WO2017133605A1 - 一种人脸跟踪方法、装置和智能终端 - Google Patents

一种人脸跟踪方法、装置和智能终端 Download PDF

Info

Publication number
WO2017133605A1
WO2017133605A1 PCT/CN2017/072545 CN2017072545W WO2017133605A1 WO 2017133605 A1 WO2017133605 A1 WO 2017133605A1 CN 2017072545 W CN2017072545 W CN 2017072545W WO 2017133605 A1 WO2017133605 A1 WO 2017133605A1
Authority
WO
WIPO (PCT)
Prior art keywords
edge feature
face
feature points
current frame
reference frame
Prior art date
Application number
PCT/CN2017/072545
Other languages
English (en)
French (fr)
Inventor
谢馥励
Original Assignee
歌尔股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 歌尔股份有限公司 filed Critical 歌尔股份有限公司
Priority to US16/068,953 priority Critical patent/US10747988B2/en
Publication of WO2017133605A1 publication Critical patent/WO2017133605A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G06V40/167—Detection; Localisation; Normalisation using comparisons between temporally consecutive images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W88/00—Devices specially adapted for wireless communication networks, e.g. terminals, base stations or access point devices
    • H04W88/02—Terminal devices

Definitions

  • the present invention relates to the field of computer vision technology, and in particular, to a face tracking method, device, and intelligent terminal.
  • Face tracking is a process of determining the trajectory of a target face based on face recognition. Face recognition can usually be divided into two parts, one is to detect the face of the image, and the other is to make a specific face determination on the detected face area, that is, to first detect the face in the input image, and then The detected face is compared with the sample in the face database to identify the target face. After identifying the target face, how to effectively track the trajectory of the face is a problem that needs to be solved by face tracking.
  • the Camshift algorithm is an improvement of the Mean Shift algorithm.
  • the basic idea of the algorithm is to video. All frames of the image are subjected to the MeanShift operation, and the calculation result of the previous frame (ie, the center and size of the search window) is taken as the initial value of the search window of the next frame MeanShift algorithm, and thus iterated.
  • the Camshift algorithm can adaptively adjust the position and size of the search window according to the tracking result of the current frame to realize real-time tracking of the face.
  • the Camshift algorithm uses skin color model tracking, which is dependent on factors such as environment and background, and has poor anti-interference ability and low tracking accuracy.
  • the invention provides a face tracking method, device and intelligent terminal, so as to solve the problem that the existing face tracking algorithm has low tracking precision.
  • a face tracking method comprising:
  • the initial position of the current frame to be tracked window is adjusted by the relative displacement, and the active trajectory of the target face in the video stream is obtained, and the face tracking is realized.
  • a face tracking device comprising:
  • a face recognition module configured to collect a frame image in the video stream as a reference frame, and perform face detection and face recognition on the reference frame to identify the target face;
  • a reference frame window position determining module configured to determine, according to the identified target face, a position of the to-be-tracked window including the target face in the reference frame, and extract an edge feature point in the reference frame to be tracked window;
  • the current frame window position determining module is configured to determine an initial position of the window to be tracked on the current frame image in the video stream according to the position of the reference frame to be tracked, and extract an edge feature point in the current frame to be tracked window;
  • a relative displacement calculation module configured to calculate a relative displacement of an edge feature point of the reference frame and an edge feature point of the current frame
  • the face tracking module is configured to adjust the initial position of the current frame to be tracked window by using relative displacement, obtain an active trajectory of the target face in the video stream, and implement face tracking.
  • an intelligent terminal comprising: a face tracking device according to another aspect of the present invention.
  • the smart terminal detects a face motion state by using a face tracking device, and the smart terminal adaptively adjusts a position of at least a part of the movable component or a function performed by at least a part of the component according to the positioned face motion state.
  • the invention has the beneficial effects that the face tracking method and device provided by the technical solution of the present invention are based on the edge feature points of the identified target face, compared with the prior art such as the skin color model based Camshift algorithm.
  • the displacement is tracked, and the motion of the target feature in the next frame image is determined by using the motion feature (ie, displacement) information of the edge feature points distributed on the target face, and the motion of the target face to be tracked and the noise tracking are performed.
  • the impact of detection improves the accuracy and tracking accuracy of face tracking.
  • the present invention further provides an intelligent terminal that uses a face tracking device to detect a face motion state, and the smart terminal adaptively adjusts a position of at least a portion of the movable component according to the positioned face motion state or at least The function performed by some components. Uninterrupted tracking can be achieved for the same target face, improving the user experience and competitiveness of the product.
  • FIG. 1 is a schematic flow chart of a face tracking method according to an embodiment of the present invention.
  • FIG. 2 is a flowchart of an implementation of a face tracking application according to another embodiment of the present invention.
  • FIG. 3 is a schematic diagram showing the principle of back tracking verification according to an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a to-be-tracked window and a target face edge feature point according to an embodiment of the present invention
  • FIG. 5 is a structural block diagram of a face tracking device according to an embodiment of the present invention.
  • FIG. 6 is a structural block diagram of an intelligent terminal according to an embodiment of the present invention.
  • the face tracking method includes the following steps 11 to 15:
  • Step 11 Collect a frame image in the video stream as a reference frame, and perform face detection and face recognition on the reference frame to identify the target face;
  • Step 12 Determine, according to the identified target face, a location of the to-be-tracked window including the target face in the reference frame, and extract an edge feature point in the reference frame to be tracked window;
  • Step 13 determining an initial position of the window to be tracked on the current frame image in the video stream according to the position of the reference frame to be tracked, and extracting edge feature points in the current frame to be tracked window;
  • the reference frame here is a frame selected from the video stream
  • the current frame is a frame that is adjacent to the reference frame and temporally adjacent.
  • the image in the video stream of this embodiment also needs to satisfy the preconditions of the calculation (for example, the image background is unchanged and the edge of the adjacent two frames of the light is stable is smaller than the image scale) to ensure the translational motion of the image, and the current frame.
  • the initial position of the to-be-tracked window is the same as the position of the to-be-tracked window of the reference frame. In this way, after determining the position of the window to be tracked in the reference frame, the initial position of the window to be tracked in the current frame can be determined according to the position.
  • Step 14 Calculate a relative displacement of an edge feature point of the reference frame and an edge feature point of the current frame; where the relative displacement is a movement amount of the feature point in the space, for example, respectively projecting each edge feature point of the target face to Constructing a corresponding plane coordinate system including four regions according to the to-be-tracked window of the current frame and the reference frame, and projecting edge feature points in the to-be-tracked window to the plane coordinate system on a plane parallel to the target human face
  • the abscissa and the ordinate value corresponding to each edge feature point are obtained, and the relative displacement is obtained by using the difference between the coordinates of the edge feature point of the reference frame and the coordinates of the edge feature point of the current frame.
  • Step 15 Adjust the initial position of the current frame to be tracked window by using relative displacement, obtain an active trajectory of the target face in the video stream, and implement face tracking.
  • the face tracking method of the embodiment determines the position of the window to be tracked in the current frame based on the position of the window to be tracked in the reference frame, and adjusts based on the displacement of the edge feature points of the target face in the two frames of images.
  • the position of the window to be tracked of the current frame is used to reduce the motion of the target face to be tracked and the influence of noise on the face tracking detection, thereby improving the accuracy and tracking accuracy of the face tracking.
  • Face tracking technology has been applied more and more to smart terminals.
  • the current application of face tracking on smart terminals still has the following problems:
  • FIG. 2 is a flowchart of an implementation of a face tracking application according to another embodiment of the present invention.
  • a face tracking method is applied to an intelligent terminal, so that the smart terminal can detect and identify the target person. Face and track the identified target face in real time.
  • the Cascade algorithm is used to scan the image, and then the Gabor wavelet transform is performed on the detected face region to extract the face feature, and then the support vector machine is used.
  • SVM performs pattern matching recognition. After matching to the determined target, the specific target is tracked by using a feature point displacement based tracking algorithm.
  • the flow of the face tracking in this embodiment is roughly as follows: first, the Cascade algorithm is used to scan and detect the face, and secondly, the face feature is extracted by using the Gabor wavelet transform according to the detected face region, and finally, the SVM is used to extract the face. The face features are identified and compared. After the comparison is successful, the target face can be tracked.
  • the Cascade algorithm is used to scan and detect the face
  • the face feature is extracted by using the Gabor wavelet transform according to the detected face region
  • the SVM is used to extract the face.
  • the face features are identified and compared. After the comparison is successful, the target face can be tracked.
  • Step 21 After the process starts, the (smart) terminal acquires a frame image in the video stream as a reference frame;
  • the video capture device is installed in the smart terminal, and the image can be collected in real time to obtain a video stream, and one frame image (generally a color image) is selected from the video stream as a reference frame, and the selected one frame color image is converted into a gray image.
  • the Haar eigenvalues of the grayscale image are then extracted.
  • a simple rectangular combination is used as the feature template, and the feature template has two rectangles of white and black, and the feature value of the template is defined as the sum of the white rectangular pixel points minus Go to the sum of the black rectangle pixels.
  • an integral image method can be employed.
  • the use of the integral image to extract the Haar-like features can be used in the prior art. For more detailed operation steps, refer to related descriptions in the prior art, and details are not described herein again.
  • face detection is to correctly find and mark the exact position of the face in the image or video, which is an important prerequisite for face recognition, tracking and other work.
  • the Cascade algorithm and the skin color discrimination method are combined to perform face detection.
  • the Cascade algorithm is based on the Haar-like features (ie, the rectangular features), and then the cascaded classifier of the Adaboost algorithm is used to select and classify the Haar-like features. That is, the face detection of the reference frame image in the embodiment includes: selecting the Haar eigenvalue of the gray image by using the trained Adaboost cascade classifier, and detecting the gray image by using the selected Haar eigenvalue. Get the initial face area.
  • the Adaboost detection algorithm based on the Haar-like feature has lower requirements on the system itself and faster calculation speed.
  • the skin color of the detected initial face region can be determined by the pre-constructed skin color model to obtain a standard face region.
  • Face recognition mainly includes feature extraction and classification.
  • Gabor wavelet transform is used to extract facial features
  • SVM feature vector machine is used for classification.
  • the face feature is extracted by using the set Gabor filter, and then the face feature extracted by the Gabor filter is sent to the trained SVM for matching recognition to identify Go to the target face.
  • the Gabor wavelet transform can take 8 directions, and a total of 40 sets of eigenvalues of 5 frequencies in each direction are used to describe the face features. Then, the extracted Gabor features are identified by SVM for matching. Next, edge feature points (such as eye corners, nose bridges, mouth corners, and other feature points of the face region) can be extracted for the face region where the SVM matches successfully, and the positions of these feature points are tracked.
  • the Gabor wavelet transform in this embodiment selects a total of 40 image texture information of 5 different frequencies in 8 different directions, which more fully describes the image texture features and can effectively eliminate the illumination. The influence of factors such as expressions on face recognition.
  • the face tracking includes step 24, determining the position of the to-be-tracked window according to the recognized target face, and performing edge feature point extraction; step 25, acquiring the current frame image in the video stream, and step 26, determining the current frame. The initial position of the window to be tracked, and the edge feature point extraction is performed; step 27 calculates the relative displacement between the edge feature points.
  • the to-be-tracked window that appears in step 24 and step 26 is determined by the target face identified in step 23 of the face recognition step.
  • the to-be-tracked window is a rectangular frame including a standard face area.
  • the face tracking is realized by SSD (Sum of Squared intensity Differences) of two frames of images to calculate the position of the to-be-tracked window appearing in the next frame, where the next frame is the first of the two frames.
  • SSD Sud of Squared intensity Differences
  • Two frames, two frames of images are not arbitrary two frames, but satisfy the preconditions of calculation (such as the image background is unchanged and the light is stable, ensuring that the edge feature points of two adjacent frames should be far smaller than the image scale), and time is as far as possible Two frames in close proximity.
  • the face tracking algorithm based on the edge feature points is a tracking algorithm in which the SSD of the video image frame is used as a metric in the window W to be tracked (including the recognized face).
  • the edge feature points refer to pixel points corresponding to the edge features of the face (such as the corners of the eyes, the sides of the mouth, etc.), and the pixel points corresponding to these features may have a sudden change in pixel values between the two frames of images.
  • the face tracking of the SSD based on the two-frame image of the present embodiment needs to ensure that the brightness is constant, the time is continuous or the motion amplitude is relatively small, and the space is consistent, and the adjacent points have similar motions and remain adjacent. Satisfying the above preconditions can ensure that the calculation process is not affected by the brightness change, and the affine transformation is performed between the windows to be tracked. Generally, in this process, the coordinate offsets of all the edge feature points in the window to be tracked are equal. It is represented by the offset variable d. Therefore, in the embodiment, the two frames of images are translational motions, and are not suitable for the case where the motion characteristics of the feature points such as image flipping, folding, and twisting are different.
  • a to-be-tracked window W containing feature texture information (ie, edge features), image frame I(x, y, t) at time t and image frame I (x, y, t at time t+ ⁇ ) The position in + ⁇ ) satisfies the following formula:
  • the displacement d tracking of a specific target of two frames of images can be achieved.
  • n(X) is the noise generated by the change of illumination conditions in time ⁇
  • n(X) is squared and integrated over the entire window to obtain the SSD of the window image
  • ⁇ (X) can generally be taken as 1. If the role of the central part feature points is emphasized, ⁇ (X) can adopt a Gaussian distribution function.
  • d is a small amount negligible compared with X
  • A(X-d) is Taylor expanded, the high-order term is removed, and only the first term of the expanded term is retained to reduce the amount of calculation, and the following formula is obtained:
  • Equation 2.4 Substituting Equation 2.4 into Equation 2.3, and taking zeros on both sides of Equation 2.3 while deriving d, you can get:
  • Equation 2.5 can be transformed into the following formula 2.6:
  • the relative displacement d of the edge feature points is obtained, and then the position of the to-be-tracked window in the next frame is adjusted according to the relative displacement, and the motion trajectory of the target face can be obtained, and the face tracking is realized.
  • the face tracking in this embodiment selects a certain frame image in the video stream as a reference frame, performs scanning detection by using the Cascade algorithm, and then performs Gabor wavelet transform on the detected face region to extract the facial feature. Then, the SVM is used for pattern matching identification. After matching the determined target face, the tracking algorithm based on the edge feature point displacement is used to determine the position of the to-be-tracked window in the current frame, so as to track the target face.
  • the tracking algorithm based on the edge feature point displacement is used to determine the position of the to-be-tracked window in the current frame, so as to track the target face.
  • FIG. 3 is a schematic diagram of the principle of back tracking verification according to an embodiment of the present invention.
  • each edge feature point in the adjusted current frame to be tracked window is verified; when the verification is successful, the current frame is confirmed.
  • the edge feature points are successfully tracked, and the edge feature points are tracked in the subsequent tracking process.
  • the verification fails, it is confirmed that the edge feature points are tracked in the current frame, and the edge feature points are discarded.
  • Edge feature points are tracked.
  • Figure 3 shows the case of tracking errors.
  • the images in the video stream are arranged clockwise, that is, from the previous frame to the current frame, and from the current frame to the next frame.
  • the images in the video stream are equivalent to being arranged counterclockwise, so that the first frame in the reverse time verification can be the last frame tracked in time.
  • Backtracking verification is an edge feature point X t+k in the image I t+k (current frame) at a t+k (k>1) time tracked in time, and backtracking back to t+1
  • By performing inverse-direction tracking calculation and verification on each edge feature point on I t+k all edge feature points of the tracking error can be determined.
  • the experimental results confirm that the method of reverse time verification can ensure the correctness of face tracking.
  • the position of the edge feature point in the current frame to be tracked window is used as the verification start position; and the relative position of the edge feature point between the current frame and the reference frame is used to determine the edge feature point in the reference.
  • the window to be tracked and is waiting Searching the edge feature points in the trace window to obtain the position of the edge feature point in the reference frame; comparing the verified edge feature point with the position of the edge feature point in the pre-acquired reference frame image, if the comparison result is the same , the verification is successful; if the comparison result is different, the verification fails.
  • the edge feature that may exist for the face motion is occluded to cause tracking error, and the edge feature points of the tracking are verified, and the edge feature points of the tracking error are eliminated in time. It is ensured that the tracking of the correct edge feature points for subsequent tracking is overcome, which overcomes the problem of tracking failure caused by misjudgment of edge feature points, and further improves the accuracy of face tracking.
  • the face tracking method based on the edge feature point displacement of the present invention can also determine the action of the target face by comparing the relative positional relationship of the same edge feature points in the adjacent two frames of the video stream. This solves the problem that the face in the video stream is always in motion, and how to determine the motion state of the user by the extracted face information.
  • Embodiment 4 The following is specifically described in conjunction with Embodiment 4:
  • FIG. 4 is a schematic diagram of a window to be tracked and a feature point of a target face according to an embodiment of the present invention.
  • the motion of the target face is mainly reflected in tilting and rotating. In the case of tilting, it can be obtained by the displacement of the edge feature points.
  • the window W to be tracked ie, the rectangular frame 40 including the face in FIG. 4
  • the corner point which is the point where the brightness of the two-dimensional image changes sharply or the point of the curvature maximum on the edge curve of the image, these points retain the important features of the image
  • the direction and size of the face tilted is the four corner points of the rectangular frame.
  • the rotation is more complicated.
  • the rotation of the face mainly includes the four directions of up, down, left and right, and the rotation of the left and right is the majority.
  • the displacement of the edge feature points is mainly reflected in the variation of the pitch. Therefore, in the present embodiment, the face motion is judged based on the relative displacement between the edge feature points. Referring to FIG.
  • the main implementation steps are: projecting each edge feature point of the target face onto a plane parallel to the target face for the current frame and the reference frame, and monitoring the spacing between the edge feature points;
  • the spacing of the edge feature points in the reference frame the spacing of the edge feature points of the left half of the current frame becomes larger, and the spacing of the edge feature points of the right half face becomes smaller, and the target face is determined to turn to the right;
  • the spacing of the edge feature points in the frame the spacing of the edge feature points of the left half of the current frame becomes smaller, and the spacing of the edge feature points of the right half face becomes larger, and the target face is determined to turn to the left;
  • the reference frame The spacing of the edge feature points, the pitch of the edge feature points of the upper half of the current frame becomes larger, and the pitch of the edge feature points of the lower half face becomes smaller, and the target face is determined to be head-up; if the edge feature points are relative to the reference frame The pitch of the edge feature points of the upper half of the current frame becomes
  • the intersection point is divided into the midpoint, including: area one 41, area two 42, area three 43 and area four 44.
  • the horizontal coordinate value of the edge feature points in the area one and the area three is less than or equal to the area two and the middle four areas.
  • the abscissa value of the sign, the ordinate value of the edge feature points in the area three and the area four is less than or equal to the ordinate value of the edge feature points in the area one and the area two;
  • the edge feature points projected into the area one and the area three are arranged in ascending order according to the abscissa X to obtain a matrix.
  • the edge feature points projected into the area 2 and the area 4 are arranged in ascending order according to the abscissa X to obtain a matrix.
  • the edge feature points projected into the area one and the area two are arranged in ascending order according to the ordinate Y value to obtain a matrix.
  • the edge feature points projected into the area three and the area four are arranged in ascending order according to the ordinate Y value to obtain a matrix.
  • the target is determined Turning to the right; and when the projections of the distances between all the feature points of the regions 2 and 4 on the x-axis are all reduced and the projections between the distances of all the feature points of the regions 1 and 3 are all increased on the x-axis, That is, the target is being turned to the left.
  • Calculate the result of the left or right turn of the target face calculate the matrix separately The spacing between each adjacent two elements is obtained by the corresponding spacing matrix with And sequentially spacing the current frame with Each element in the difference is compared with each element in the spacing matrix of the reference frame, and the heading or heading motion of the target face is judged according to the calculation result.
  • a spacing matrix representing edge feature points of the left half of the current frame The spacing matrix representing the edge feature points of the left half of the reference frame a spacing matrix representing edge feature points of the right half of the current frame, a spacing matrix representing edge feature points of the right half of the reference frame;
  • a spacing matrix representing edge feature points of the left half of the current frame The spacing matrix representing the edge feature points of the left half of the reference frame a spacing matrix representing edge feature points of the right half of the current frame, a spacing matrix representing edge feature points of the right half of the reference frame;
  • Judging whether the head face or the bowing action of the target face is based on the calculation result includes:
  • the spacing matrix representing the edge feature points of the upper half of the current frame A spacing matrix representing the edge feature points of the upper half of the reference frame
  • the spacing matrix representing the edge feature points of the lower half of the current frame The spacing matrix representing the edge feature points of the lower half of the reference frame
  • the spacing matrix representing the edge feature points of the upper half of the current frame A spacing matrix representing the edge feature points of the upper half of the reference frame
  • the spacing matrix representing the edge feature points of the lower half of the current frame The spacing matrix representing the edge feature points of the lower half of the reference frame
  • the present embodiment is directed to the problem of how to track in real time when the face in the video stream is always in motion during the face tracking process, by using the edge feature points in the reference frame and the current frame to be tracked.
  • the change of the pitch determines the motion of the target face, determines the motion state of the target, and ensures the accuracy and accuracy of the face tracking.
  • FIG. 5 is a structural block diagram of a face tracking device according to an embodiment of the present invention.
  • the face tracking device face tracking device 50 includes:
  • the face recognition module 501 is configured to collect a frame image in the video stream as a reference frame, and perform face detection and face recognition on the reference frame to identify the target face;
  • the reference frame window position determining module 502 is configured to determine, according to the identified target face, a position of the to-be-tracked window including the target face in the reference frame, and extract an edge feature point in the reference frame to be tracked window;
  • the current frame window position determining module 503 is configured to determine an initial position of the window to be tracked on the current frame image in the video stream according to the position of the reference frame to be tracked window, and extract an edge feature point in the current frame to be tracked window;
  • the relative displacement calculation module 504 is configured to calculate a relative displacement of an edge feature point of the reference frame and an edge feature point of the current frame;
  • the face tracking module 505 is configured to adjust the initial position of the current frame to be tracked window by using relative displacement, obtain an active trajectory of the target face in the video stream, and implement face tracking.
  • the apparatus 50 further includes: a tracking verification module and a face action judgment module;
  • a tracking verification module configured to verify each edge feature point in the adjusted current frame to be tracked window after adjusting the initial position of the current frame to be tracked window by using the relative displacement; when the verification is successful, confirming that the current The frame is successfully tracked to the edge In the follow-up tracking process, the edge feature points are tracked continuously; when the verification fails, it is confirmed that the edge feature points are tracked in the current frame, the edge feature points are discarded, and the edge feature points are no longer tracked. ;
  • a face motion judging module configured to determine a left-right turning motion and a head-up motion of the target human face by using a change in the spacing between the reference frame and the current frame in the edge feature to be tracked; and using the reference frame and the current frame
  • the relative displacement of the edge feature points calculates the tilt angle of the target face, and the tilt angle determines the tilting motion of the target face.
  • the tracking verification module is specifically configured to use the position of the edge feature point in the current frame to be tracked window as the verification starting position
  • the verified edge feature point is compared with the position of the edge feature point in the pre-acquired reference frame image at the position of the reference frame. If the comparison result is the same, the verification is successful; if the comparison result is different, the verification fails.
  • the face motion determining module is specifically configured to project each edge feature point of the target face onto a plane parallel to the target face for the current frame and the reference frame, and monitor each edge feature point. If the pitch of the edge feature points of the left half of the current frame becomes larger and the pitch of the edge feature points of the right half face becomes smaller than the pitch of the edge feature points in the reference frame, the target face is determined to turn right.
  • the target face is determined to turn to the left; Relative to the spacing of the edge feature points in the reference frame, the pitch of the edge feature points of the upper half face of the current frame becomes larger and the pitch of the edge feature points of the lower half face becomes smaller, and the target face is determined to be head-up; if relative to the reference The pitch of the edge feature points in the frame, the pitch of the edge feature points of the upper half face of the current frame becomes smaller, and the pitch of the edge feature points of the lower half face becomes larger, and it is determined that the target face is at the head.
  • the corresponding plane coordinate system including four regions is constructed according to the to-be-tracked window of the current frame and the reference frame, and the edge feature points in the window to be tracked are projected into the plane coordinate system to obtain corresponding points of each edge feature point.
  • the abscissa and the ordinate value specifically, the four regions are divided into the midpoints according to the intersection of the diagonal lines of the graphic of the image to be tracked in each frame, including: area one, area two, area three, and area four,
  • the abscissa value of the edge feature points in the area 1 and the area 3 is less than or equal to the abscissa value of the edge feature points in the area 2 and the area 4, and the ordinate value of the edge feature points in the area 3 and the area 4 is less than or equal to the area 1 and the area 2
  • the edge feature points projected into the area one and the area three are arranged in ascending order according to the abscissa to obtain a matrix.
  • the edge feature points projected into the area 2 and the area 4 are arranged in an ascending order according to the abscissa to obtain a matrix.
  • the edge feature points projected into the area one and the area two are arranged in ascending order according to the ordinate value to obtain a matrix.
  • the edge feature points projected into the area three and the area four are arranged in ascending order according to the ordinate value to obtain a matrix.
  • a spacing matrix representing edge feature points of the left half of the current frame The spacing matrix representing the edge feature points of the left half of the reference frame a spacing matrix representing edge feature points of the right half of the current frame, a spacing matrix representing edge feature points of the right half of the reference frame;
  • a spacing matrix representing edge feature points of the left half of the current frame The spacing matrix representing the edge feature points of the left half of the reference frame a spacing matrix representing edge feature points of the right half of the current frame, a spacing matrix representing edge feature points of the right half of the reference frame;
  • Judging whether the head face or the bowing action of the target face is based on the calculation result includes:
  • the spacing matrix representing the edge feature points of the upper half of the current frame A spacing matrix representing the edge feature points of the upper half of the reference frame
  • the spacing matrix representing the edge feature points of the lower half of the current frame The spacing matrix representing the edge feature points of the lower half of the reference frame
  • the spacing matrix representing the edge feature points of the upper half of the current frame A spacing matrix representing the edge feature points of the upper half of the reference frame
  • the spacing matrix representing the edge feature points of the lower half of the current frame The spacing matrix representing the edge feature points of the lower half of the reference frame
  • the face recognition module 501 is specifically configured to collect a frame image in the video stream and convert it into a gray image, and extract a Haar-like feature value of the gray image; use the trained Adaboost cascade classifier to gray
  • the Haar eigenvalues of the degree image are selected, and the gray face image is detected by using the selected Haar eigenvalue to obtain the initial face region;
  • the skin color model is used to determine the skin color of the initial face region to obtain the standard face region;
  • the face feature is extracted by using the set Gabor filter; the face feature extracted by the Gabor filter is sent to the trained support vector machine SVM for matching recognition to identify the target face.
  • the face tracking device in this embodiment corresponds to the foregoing face tracking method. Therefore, the working process of the face tracking device can be referred to in the embodiment of the face tracking method of the present invention. Description, no longer repeat here.
  • FIG. 6 is a structural block diagram of a smart terminal according to an embodiment of the present invention.
  • the smart terminal 60 of this embodiment includes the face tracking device 50 shown in Embodiment 5.
  • the smart terminal 60 detects the face motion state by using the face tracking device 50, and the smart terminal 60 adaptively adjusts the position of at least part of the movable component (such as rotating the position of the display screen) or at least partially according to the positioned face motion state.
  • the functions performed by the component (such as turning on or off the functionality of some applications in the terminal).
  • the smart terminal 60 includes the face tracking device, when the system of the smart terminal 60 determines that the face is rotated, the movable component of the smart terminal is adjusted when the action of positioning the face ends. Position, so that the smart terminal follows the action of the face, more humane.
  • the smart terminal 60 calculates the difference between the adjacent t time and the t-1 time, the difference of the distance matrix of the edge feature points of the left half face or the right half face of the current frame, that is, or That is, it is determined that the face rotation action is nearing the end, wherein the threshold is 0.
  • the control shifts the position of the movable part of the smart terminal by ⁇ /4 radians in the rotation direction.
  • the movable component of the smart terminal 60 moves, the background of the image in the video stream changes accordingly, so the face detection step is re-imaged. In this way, the smart terminal tracks the movement of the face through the face tracking device, so that the same person can achieve uninterrupted tracking, and the ability of the intelligent terminal to interact with humans is improved.
  • the smart terminal 60 may be a terminal device (such as an intelligent robot) that detects human activity, and then the intelligent robot performs corresponding cooperation operations according to the detected target human body motion, thereby improving human-computer interaction of the intelligent robot. Performance that optimizes the user experience.
  • a terminal device such as an intelligent robot
  • the face tracking method and the tracking device of the present invention firstly overcome the problem of background color interference and improve the face tracking. Correctness. Secondly, by using the relative positional relationship between the edge feature points, the information such as the posture and movement direction of the face is reversed, which facilitates human-computer interaction. Finally, for the course of exercise Possible features are occluded to cause tracking errors, and time-reversed verification is used to overcome the problem of tracking failure caused by misjudgment of edge feature points.
  • the smart terminal of the present invention includes a face tracking device. When the target face moves, the smart terminal can follow autonomously to achieve the effect of uninterrupted tracking, optimize the human-computer interaction experience, and improve the competitiveness of the product.

Abstract

本发明公开一种人脸跟踪方法、装置和智能终端，该跟踪方法包括：采集视频流中的一帧图像并进行人脸检测和人脸识别，识别出目标人脸；根据识别出的目标人脸，确定参考帧中包括目标人脸的待跟踪窗口的位置，并提取参考帧待跟踪窗口中的边缘特征点；根据参考帧待跟踪窗口的位置，确定视频流中当前帧待跟踪窗口的初始位置，并提取当前帧待跟踪窗口中的边缘特征点；计算参考帧的边缘特征点与当前帧的边缘特征点间的相对位移；利用相对位移对当前帧待跟踪窗口的初始位置进行调整，实现人脸跟踪。与现有技术相比，本发明实施例的这种人脸跟踪方案基于人脸边缘特征点的位移实现跟踪，对环境具有较好的适应性，人脸跟踪更精确。

Description

一种人脸跟踪方法、装置和智能终端 技术领域
本发明涉及计算机视觉技术领域，具体涉及一种人脸跟踪方法、装置和智能终端。
背景技术
人脸跟踪是在人脸识别的基础上确定目标人脸运动轨迹的过程。人脸识别通常又可分为两部分，一是对图像进行人脸检测，二是对检测到的人脸区域进行特定人脸判定，即首先要在输入图像中检测出人脸，然后再将检测到的人脸与人脸库中的样本进行比对以识别出目标人脸。在识别出目标人脸后，如何对人脸的运动轨迹进行有效的跟踪是人脸跟踪需要解决的问题。
现有技术，人脸跟踪广泛应用的是基于肤色特征的Camshift(连续自适应均值移动，Continuously Adaptive Mean Shift)算法，Camshift算法是均值移动(MeanShift)算法的改进，该算法的基本思想是将视频图像的所有帧作MeanShift运算，并将上一帧的计算结果(即搜索窗口的中心和大小)作为下一帧MeanShift算法的搜索窗口的初始值，如此迭代下去。Camshift算法能够根据当前帧的跟踪结果自适应地调整搜索窗口的位置和大小，实现人脸实时跟踪。
但是，Camshift算法采用肤色模型跟踪，对环境、背景等因素依赖大且抗干扰能力差，跟踪精度较低。
发明内容
本发明提供了一种人脸跟踪方法、装置和智能终端，以解决现有的人脸跟踪算法存在的跟踪精度低的问题。
根据本发明的一个方面，提供了一种人脸跟踪方法，该方法包括：
采集视频流中的一帧图像作为参考帧并对该参考帧进行人脸检测和人脸识别，识别出目标人脸；
根据识别出的目标人脸，确定参考帧中包括目标人脸的待跟踪窗口的位置，并提取参考帧待跟踪窗口中的边缘特征点；
根据参考帧待跟踪窗口的位置，确定视频流中当前帧图像上待跟踪窗口的初始位置，并提取当前帧待跟踪窗口中的边缘特征点；
计算参考帧的边缘特征点与当前帧的边缘特征点的相对位移；
利用相对位移对当前帧待跟踪窗口的初始位置进行调整，获得视频流中目标人脸的活动轨迹，实现人脸跟踪。
根据本发明的另一个方面，提供了一种人脸跟踪装置，该装置包括：
人脸识别模块，用于采集视频流中的一帧图像作为参考帧并对该参考帧进行人脸检测和人脸识别，识别出目标人脸；
参考帧窗口位置确定模块，用于根据识别出的目标人脸，确定参考帧中包括目标人脸的待跟踪窗口的位置，并提取参考帧待跟踪窗口中的边缘特征点；
当前帧窗口位置确定模块，用于根据参考帧待跟踪窗口的位置，确定视频流中当前帧图像上待跟踪窗口的初始位置，并提取当前帧待跟踪窗口中的边缘特征点；
相对位移计算模块，用于计算参考帧的边缘特征点与当前帧的边缘特征点的相对位移；
人脸跟踪模块，用于利用相对位移对当前帧待跟踪窗口的初始位置进行调整，获得视频流中目标人脸的活动轨迹，实现人脸跟踪。
根据本发明又一个方面，提供了一种智能终端，智能终端包括：如本发明另一个方面的人脸跟踪装置；
该智能终端利用人脸跟踪装置检测人脸运动状态，且该智能终端根据定位到的人脸运动状态自适应调整至少部分可移动部件的位置或至少部分部件所执行的功能。
本发明的有益效果是：与现有的基于肤色模型的Camshift算法等现有技术相比，本发明技术方案提供的这种人脸跟踪方法和装置，基于识别出的目标人脸的边缘特征点位移进行跟踪，通过利用分布在目标人脸的边缘特征点的运动特征(即位移)信息来确定下一帧图像中目标人脸的区域，降低待跟踪目标人脸的运动以及噪声对人脸跟踪检测的影响，从而提高了人脸跟踪的正确率和跟踪精度。另外，本发明还提供了一种智能终端，该智能终端利用人脸跟踪装置检测人脸运动状态，且该智能终端根据定位到的人脸运动状态自适应调整至少部分可移动部件的位置或至少部分部件所执行的功能。可以针对同一个目标人脸实现不间断的跟踪，提高了产品的用户体验和竞争力。
附图简要说明
图1是本发明一个实施例的一种人脸跟踪方法的流程示意图；
图2是本发明另一个实施例的一种人脸跟踪应用的实现流程图；
图3是本发明一个实施例的反向跟踪验证的原理示意图；
图4是本发明一个实施例的待跟踪窗口以及目标人脸边缘特征点的示意图；
图5是本发明一个实施例的一种人脸跟踪装置的结构框图；
图6是本发明一个实施例的一种智能终端的结构框图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显 然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其它实施例，都属于本发明保护的范围。
实施例一
图1是本发明一个实施例的一种人脸跟踪方法的流程示意图，参见图1，该人脸跟踪方法包括如下步骤11至15:
步骤11，采集视频流中的一帧图像作为参考帧并对该参考帧进行人脸检测和人脸识别，识别出目标人脸；
步骤12，根据识别出的目标人脸，确定参考帧中包括目标人脸的待跟踪窗口的位置，并提取参考帧待跟踪窗口中的边缘特征点；
步骤13，根据参考帧待跟踪窗口的位置，确定视频流中当前帧图像上待跟踪窗口的初始位置，并提取当前帧待跟踪窗口中的边缘特征点；
这里的参考帧，是从视频流中选取的一帧，当前帧是在参考帧之后并且时间上尽量相邻的一帧。此外，本实施例视频流中图像也需要满足计算的前提条件(如，图像背景不变及光线稳定相邻两帧的边缘特征点位移远小于图像尺度)以保证图像的平移运动，以及当前帧中待跟踪窗口的初始位置与参考帧的待跟踪窗口的位置相同。如此，在确定出参考帧中待跟踪窗口的位置后，即可根据该位置确定出当前帧中待跟踪窗口的初始位置。
步骤14，计算参考帧的边缘特征点与当前帧的边缘特征点的相对位移；在此，相对位移为特征点在空间中的移动量，例如，分别将目标人脸的各边缘特征点投影到与目标人脸平行的平面上，根据所述当前帧和参考帧的待跟踪窗口，构建对应的包括四个区域的平面坐标系，将待跟踪窗口中的边缘特征点投影到所述平面坐标系中，得到每个边缘特征点对应的横坐标和纵坐标值，利用参考帧的边缘特征点的坐标与当前帧的边缘特征点的坐标之间的差值得到相对位移。
步骤15，利用相对位移对当前帧待跟踪窗口的初始位置进行调整，获得视频流中目标人脸的活动轨迹，实现人脸跟踪。
由图1可知，本实施例的人脸跟踪方法，基于参考帧中待跟踪窗口的位置确定当前帧中待跟踪窗口的位置，并基于两帧图像中目标人脸的边缘特征点的位移，调整当前帧的待跟踪窗口位置，以降低待跟踪目标人脸的运动以及噪声对人脸跟踪检测的影响，从而提高了人脸跟踪的正确率和跟踪精度。
人脸跟踪技术已经越来越多的应用到智能终端中，但是，目前人脸跟踪在智能终端上的实际应用仍然存在下列问题：
(1)如何克服人脸表情、环境光线及图像集合变化产生的影响，提取有效的特征来描述人脸。
(2)在目标快速移动的时候准确的跟踪，人脸在移动的时候会产生不同程度的角度变化，左右倾斜或前后转动，也会产生一定程度的表情变化。人前后移动也会令人脸在画面中的大小产生变化。如何克服这些变化，准确的定位和跟踪人脸。
(3)如何通过跟踪的人脸来判定目标的运动状态。在视频流中的人脸始终处于运动状态，如何通过提取的人脸信息判定用户的运动状态，这也是人机交互领域不得不解决的问题之一。
以下结合本发明的其他实施例，对本发明为了解决上述技术问题采取的技术手段进行具体说明。
实施例二
图2是本发明另一个实施例的一种人脸跟踪应用的实现流程图，参见图2，本实施例中将人脸跟踪方法应用到智能终端中，使得智能终端能够检测、识别出目标人脸，并实时跟踪识别出的目标人脸。本实施例通过截取视频流中的某一帧图像，利用Cascade算法对图像进行扫描检测，然后针对检测到的人脸区域进行Gabor小波变换来提取人脸特征，随后利用支持向量机(Support Vector Machine，SVM)进行模式匹配识别，当匹配到确定目标后，采用基于特征点位移的跟踪算法对该特定目标进行跟踪。即本实施例中的人脸跟踪的流程大致为：首先，采用Cascade算法进行扫描检测人脸，其次，根据检测到的人脸区域利用Gabor小波变换提取人脸特征，最后，利用SVM来对提取的人脸特征进行识别比对。比对成功后，即可对该目标人脸进行跟踪。以下进行具体说明：
步骤21，流程开始后，由(智能)终端获取视频流中的一帧图像作为参考帧；
这里，智能终端中安装视频采集装置，能够实时采集图像得到视频流，从视频流中选取一帧图像(一般为彩色图像)作为参考帧，并将选取的一帧彩色图像转换为灰度图像，然后提取灰度图像的类Haar特征值。提取灰度图像的类Haar特征值时，本实施例中将简单矩形组合作为特征模板，特征模板内有白色和黑色两种矩形，并将此模板的特征值定义为白色矩形像素点的总和减去黑色矩形像素点的总和。
此外，为了提高计算速度，可以采用积分图像方法。而利用积分图像提取类Haar特征可以采用现有技术，更详细的操作步骤可以参见现有技术中的相关记载，这里不再赘述。
步骤22，人脸检测；
人脸检测的目的是在图像或者视频中正确地找到并标记出人脸的准确位置，是实现人脸识别、跟踪等工作的重要先决条件。
由于图像的复杂程度较高,仅利用单一特征信息往往不能得到准确的检测结果,需要融合多个特征信息进行检测。所以，本实施例中采用Cascade算法和肤色判别相结合的方法进行人脸检测。Cascade算法是基于类Haar特征(即矩形特征)的，然后采用Adaboost算法的级联分类器对类Haar特征进行选择判别。即本实施例中对参考帧图像进行人脸检测包括：利用训练好的Adaboost级联分类器对灰度图像的类Haar特征值进行选择，并利用选择的类Haar特征值对灰度图像进行检测得到初始人脸区域。基于类Haar特征的Adaboost检测算法，对系统自身的要求较低，运算速度快。
另外，进一步降低误检率，提高人脸跟踪的精度，可以利用预先构建的肤色模型对检测出的初始人脸区域进行肤色判别，得到标准人脸区域。
步骤23，人脸识别；
在对图像完成人脸检测后，需要对已检测到的人脸进行比对识别，以确认是否为目标人脸。人脸识别主要包括特征提取和分类两部分，本实施例中采用Gabor小波变换来提取人脸特征，并采用SVM特征向量机来进行分类。
具体的，对步骤22中得到的标准人脸区域，利用设定的Gabor滤波器提取人脸特征，然后将Gabor滤波器提取出的人脸特征送入训练好的SVM中进行匹配识别，以识别出目标人脸。
其中，Gabor小波变换可以取8个方向，每个方向下5个频率的一共40组特征值来描述人脸特征。然后，对提取的Gabor特征利用SVM进行匹配识别。接下来，就可以针对SVM匹配成功的人脸区域进行边缘特征点(如人脸区域的眼角、鼻梁、嘴角等特征点)提取，并对这些特征点的位置进行跟踪。相比以往人脸识别的技术，本实施例中Gabor小波变换选取了8个不同方向下的5个不同频率的总计40个图像纹理信息，更加全面的描述了图像纹理特征，可以有效的消除光照、表情等因素对人脸识别的影响。
人脸跟踪
本实施例中，人脸跟踪包括步骤24，根据识别出的目标人脸确定待跟踪窗口的位置，并进行边缘特征点提取；步骤25，获取视频流中当前帧图像，步骤26，确定当前帧的待跟踪窗口初始位置，并进行边缘特征点提取；步骤27计算边缘特征点之间的相对位移。下面结合步骤24至步骤27具体说明人脸跟踪流程：
步骤24和步骤26中出现的待跟踪窗口，待跟踪窗口是通过步骤23人脸识别步骤中识别出的目标人脸来确定的，具体的，待跟踪窗口是包括标准人脸区域一个矩形框。
人脸跟踪的实现是通过两帧图像的SSD(Sum of Squared intensity Differences，灰度差平方和)来计算待跟踪窗口出现在下一帧的位置，这里所说的下一帧是两帧中的第二帧，两帧图像不是任意的两帧，而是满足计算的前提条件(如图像背景不变及光线稳定，保证相邻两帧的边缘特征点位移应远小于图像尺度)，并且时间上尽量相近的两帧。
具体的，基于边缘特征点的人脸跟踪算法，是以待跟踪窗口W(W中包括识别出的人脸)在视频图像帧间SSD作为度量的跟踪算法。其中，边缘特征点是指人脸的边缘特征(如眼角、嘴边等)对应的像素点，这些特征对应的像素点在两帧图像间会出现像素值突变。
需要强调的是，本实施例的基于两帧图像的SSD确定位移的人脸跟踪需保证：亮度恒定，时间连续或者运动幅度相对较小，以及空间一致，临近点有相似运动，保持相邻。满足以上前提条件，可保证计算过程不受亮度变化影响，在待跟踪窗口间进行仿射变换，一般在这个过程中假设待跟踪窗口中所有边缘特征点的坐标偏移量相等。用偏移变量d来表示。因此，本实施例中两帧图像之间是平移运动，而不适用于图像翻转、折叠、扭曲等特征点运动特性不同的情况。
对于灰度图像，一个包含了特征纹理信息(即边缘特征)的待跟踪窗口W，在t时刻图像帧I(x,y,t)和在t+τ时刻图像帧I(x,y,t+τ)中的位置满足下列公式：
I(x,y,t+τ)＝I(x-Δx,y-Δy,t)           公式2.1
即，在I(x,y,t+τ)中的每个像素点，都可以由I(x,y,t)中相应窗口的像素点平移d＝(Δx,Δy)得到。求出位移d，即可实现对两帧图像的特定目标的跟踪。
假设t+τ时刻的待跟踪窗口为B(X)＝B(x,y,t+τ)，其中X＝(x,y)为视口坐标。t时刻的待跟踪窗口为A(X-d)＝A(x-Δx,y-Δy,t)。考虑一般情况，有
B(X)＝A(X-d)+n(X)       公式2.2
其中，n(X)为在时间τ内由于光照条件变化产生的噪声，将n(X)平方并在整个窗口上积分，就得到了窗口图像的SSD：
ε＝∫∫vn(X)2ω(X)dX＝∫∫v[A(X-d)-B(X)]2ω(X)dX        公式2.3
其中，X＝[x,y]T，d＝[dx,dy]T，ω(X)通常可取为1。如果强调中心部分特征点的作用，ω(X)可以采用高斯分布函数。当d为一个和X比起来可忽略的小量时,将A(X-d)进行泰勒展开，去除高次项，仅保留展开项的第一项，以降低计算量，得到下列算式：
A(X-d)＝A(X)-g·d           公式2.4
将公式2.4代入公式2.3，并对公式2.3的两边同时对d求导之后取零，可以得到：
Figure PCTCN2017072545-appb-000001
此时ε取得极小值，公式2.5可变换为下面公式2.6：
(∫∫vggTωdX)d＝∫∫v[A(X)-B(X)]gωdX          公式2.6
若令
Figure PCTCN2017072545-appb-000002
e＝∫∫v[A(X)-B(X)]g(X)ω(X)dX
则，公式2.6可以简写为：Zd＝e，
对于每两帧图像，解上式Zd＝e，即可以求出待跟踪窗口中的边缘特征点的相对位移d＝(Δx,Δy)。
至此，得到了边缘特征点的相对位移d，然后根据相对位移调整待跟踪窗口在下一帧的位置，即可得到目标人脸的运动轨迹，实现人脸跟踪。
由上可知，本实施例的人脸跟踪通过选取视频流中的某一帧图像作为参考帧，利用Cascade算法进行扫描检测，然后针对检测到的人脸区域进行Gabor小波变换来提取人脸特征，随后利用SVM进行模式匹配识别，当匹配到确定目标人脸后，采用基于边缘特征点位移的跟踪算法确定待跟踪窗口在当前帧的位置，实现对该目标人脸的跟踪。克服了人脸表情、环境光线及图像集合变化产生的影响，并提取到有效的边缘特征来描述人脸，提高了人脸跟踪的精度。
实施例三
在实际跟踪过程中，在目标快速移动的时候准确的跟踪，人脸在移动的时候会产生不同程度的角度变化，而左右倾斜或前后转动时，人脸会产生一定程度的表情变化，人前后移动也会令人脸在画面中的大小产生变化。如遇到边缘特征点被遮挡的情况，系统有可能将其他相似的边缘特征点误认为是该特征点而导致跟踪产生错误或跟踪失败的情况。为解决这一问题，本发明的一个实施例中采用逆时反向跟踪的手段来验证跟踪的准确性。以下结合本实施例对人脸跟踪的反向验证进行重点说明，其他内容参见本发明的其他实施例。
图3是本发明一个实施例的反向跟踪验证的原理示意图，本实施例中，对调整后的当前帧待跟踪窗口中的每个边缘特征点进行验证；当验证成功时，确认在当前帧成功跟踪到该边缘特征点，在后续跟踪过程中继续对该边缘特征点进行跟踪；当验证失败时，确认在当前帧中对该边缘特征点跟踪失败，舍弃该边缘特征点，不再对该边缘特征点进行跟踪。
参见图3，图3示出的是跟踪错误的情况。在人脸跟踪过程中，视频流中图像是按照顺时针排列的，即从前一帧跟踪到当前帧，并从当前帧跟踪到后一帧。但是在逆时验证时，视频流中的图像相当于按照逆时针排列，这样，逆时验证时的第一帧就可以是顺时跟踪到的最后一帧。反向跟踪验证即对于顺时跟踪到的一个t+k(k>1)时刻的图像It+k(当前帧)中的一个边缘特征点Xt+k，向前回溯跟踪至t+1时刻的图像It+1，并继续向前回溯跟踪至t时刻(参考帧)，计算出t时刻边缘特征点Xt+k对应的位置为Yt，若Yt与t时刻时记录的位置Xt没有重合(表示跟踪错误)，那么在从t+k时刻向后一帧顺时跟踪过程中，系统将抛弃该边缘特征点并利用验证通过的那些边缘特征点继续人脸跟踪。通过对It+k上的每个边缘特征点进行逆时方向跟踪计算和验证，即可判断出跟踪错误的所有边缘特征点。经实验结果证实，采用逆时验证的手段可以确保人脸跟踪的正确性。
具体实现时，是将当前帧待跟踪窗口中的边缘特征点的位置作为验证起始位置；利用验证起始位置及当前帧与参考帧之间边缘特征点的相对位移，确定边缘特征点在参考帧的待跟踪窗口，并在待跟 踪窗口内搜索边缘特征点，得到边缘特征点在参考帧的位置；将验证出的边缘特征点在参考帧的位置与预先获取的参考帧图像中边缘特征点的位置进行比较，若比较结果相同，则验证成功；若比较结果不同，则验证失败。
由上可知，在人脸跟踪过程中，针对人脸运动可能存在的边缘特征被遮挡导致跟踪错误的问题，通过对跟踪到的边缘特征点进行验证，并将跟踪错误的边缘特征点及时排除，确保利用跟踪正确的边缘特征点进行后续顺时跟踪，克服了由边缘特征点的误判导致跟踪失败的问题，进一步提高了人脸跟踪的精度。
实施例四
本发明的这种基于边缘特征点的位移的人脸跟踪方法，还可以通过比对视频流中相邻两帧画面中同一边缘特征点的相对位置关系，判断出目标人脸的动作。这解决了在视频流中的人脸始终处于运动状态的情况，如何通过提取的人脸信息判定用户的运动状态的问题。以下结合实施例四进行具体说明：
图4是本发明一个实施例的待跟踪窗口以及目标人脸边缘特征点的示意图，参见图4，目标人脸的运动主要体现在倾斜和转动。倾斜的情况下可通过边缘特征点位移得到，当人脸发生向左或向右的倾斜时，待跟踪窗口W(即图4中包括人脸的矩形框40)也会随之产生倾斜。此时，根据矩形框的四个角点(角点，是二维图像亮度变化剧烈的点或图像边缘曲线上曲率极大值的点，这些点保留了图像的重要特征)的坐标即可得出人脸倾斜的方向和大小。
而转动的情况就比较复杂，人脸的转动主要包括上、下、左、右四个方向，其中又以左和右的转动为多数情况。转动时，边缘特征点的位移主要体现在间距的变化，因此，本实施例中根据边缘特征点之间的相对位移来判断人脸动作。参见图4，主要实现步骤是：对当前帧和参考帧，分别将目标人脸的各边缘特征点投影到与目标人脸平行的平面上，并监测各边缘特征点之间的间距；若相对于参考帧中边缘特征点的间距，当前帧的左半边脸的边缘特征点的间距变大且右半边脸的边缘特征点的间距变小，则确定目标人脸向右转；若相对于参考帧中边缘特征点的间距，当前帧的左半边脸的边缘特征点的间距变小且右半边脸的边缘特征点的间距变大，则确定目标人脸向左转；若相对于参考帧中边缘特征点的间距，当前帧的上半边脸的边缘特征点的间距变大且下半边脸的边缘特征点的间距变小，则确定目标人脸在抬头；若相对于参考帧中边缘特征点的间距，当前帧的上半边脸的边缘特征点的间距变小且下半边脸的边缘特征点的间距变大，则确定目标人脸在低头。
如图4所示，在选定的人脸区域矩形框内，以对角线交点为中点划出四个区域41、42、43、44，构建对应的包括四个区域41、42、43、44的平面坐标系，将待跟踪窗口中的边缘特征点投影到平面坐标系中，得到每个边缘特征点401对应的横坐标和纵坐标值(图4中人脸中的各点投影至包括四个区域的平面坐标系，得到人脸中各点的坐标，当人脸转动时，坐标会发生变化)；具体的，四个区域是按照每帧图像待跟踪窗口所在图形的对角线的交点为中点划分而成，包括：区域一41、区域二42、区域三43和区域四44，区域一和区域三中边缘特征点的横坐标值小于等于区域二和区域四中边缘特 征点的横坐标值，区域三和区域四中边缘特征点的纵坐标值小于等于区域一和区域二中边缘特征点的纵坐标值；
将投影到区域一和区域三中的边缘特征点根据横坐标X升序排列得到矩阵
Figure PCTCN2017072545-appb-000003
将投影到区域二和区域四中的边缘特征点根据横坐标X升序排成排列得到矩阵
Figure PCTCN2017072545-appb-000004
将投影到区域一和区域二中的边缘特征点根据纵坐标Y值升序排列得到矩阵
Figure PCTCN2017072545-appb-000005
将投影到区域三和区域四中的边缘特征点根据纵坐标Y值升序排列得到矩阵
Figure PCTCN2017072545-appb-000006
当判定左右转动时，对
Figure PCTCN2017072545-appb-000007
和
Figure PCTCN2017072545-appb-000008
进行计算；当判定抬头低头动作时，则对矩阵
Figure PCTCN2017072545-appb-000009
和
Figure PCTCN2017072545-appb-000010
进行计算。判定左右转动时，分别求得矩阵
Figure PCTCN2017072545-appb-000011
相邻两个边缘特征点的间距在X轴上的映射
Figure PCTCN2017072545-appb-000012
和
Figure PCTCN2017072545-appb-000013
并将其与上一帧(即参考帧)的间距相减。当区域一、三区域的所有特征点之间的距离在x轴上的投影全部减小并且区域二、四的所有特征点之间的距离在x轴上的投影全部增加时，即判定目标正在向右转；而当区域二、四的所有特征点之间的距离在x轴上的投影全部减小并且区域一、三的所有特征点之间的距离在x轴上的投影全部增加时，即判定目标正在向左转。
Figure PCTCN2017072545-appb-000014
算结果判断目标人脸的左转或右转动作；分别计算矩阵
Figure PCTCN2017072545-appb-000015
中每相邻两个元素的间距得到相应的间距矩阵
Figure PCTCN2017072545-appb-000016
和
Figure PCTCN2017072545-appb-000017
并依次将当前帧的间距矩阵
Figure PCTCN2017072545-appb-000018
和
Figure PCTCN2017072545-appb-000019
中的每个元素与参考帧的间距矩阵中的每个元素作差，根据计算结果判断目标人脸的抬头或低头动作。
其中，当计算结果满足下列公式(1)和(2)时，确定目标人脸向右转动；
Figure PCTCN2017072545-appb-000020
Figure PCTCN2017072545-appb-000021
其中，
Figure PCTCN2017072545-appb-000022
表示当前帧左半边脸的边缘特征点的间距矩阵，
Figure PCTCN2017072545-appb-000023
表示参考帧左半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000024
表示当前帧右半边脸的边缘特征点的间距矩阵，
Figure PCTCN2017072545-appb-000025
表示参考帧右半边脸的边缘特征点的间距矩阵；
当计算结果满足下列公式(3)和(4)时，确定目标人脸向左转动：
Figure PCTCN2017072545-appb-000026
Figure PCTCN2017072545-appb-000027
其中，
Figure PCTCN2017072545-appb-000028
表示当前帧左半边脸的边缘特征点的间距矩阵，
Figure PCTCN2017072545-appb-000029
表示参考帧左半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000030
表示当前帧右半边脸的边缘特征点的间距矩阵，
Figure PCTCN2017072545-appb-000031
表示参考帧右半边脸的边缘特征点的间距矩阵；
根据计算结果判断目标人脸的抬头或低头动作包括：
当计算结果满足下列公式(5)和(6)时，确定目标人脸在抬头：
Figure PCTCN2017072545-appb-000032
Figure PCTCN2017072545-appb-000033
其中，
Figure PCTCN2017072545-appb-000034
表示当前帧上半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000035
表示参考帧上半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000036
表示当前帧下半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000037
表示参考帧下半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000038
当计算结果满足下列公式(7)和(8)时，确定目标人脸在低头：
Figure PCTCN2017072545-appb-000039
Figure PCTCN2017072545-appb-000040
其中，
Figure PCTCN2017072545-appb-000041
表示当前帧上半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000042
表示参考帧上半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000043
表示当前帧下半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000044
表示参考帧下半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000045
由上描述可知，本实施例针对人脸跟踪过程中，视频流中的人脸始终处于运动状态的情况下如何实时跟踪的问题，通过对参考帧和当前帧中待跟踪窗口中的边缘特征点的间距变化判断目标人脸的运动动作，判定目标的运动状态，同时保证了人脸跟踪的准确性和精度。
实施例五
图5是本发明一个实施例的一种人脸跟踪装置的结构框图，参见图5，该人脸跟踪装置人脸跟踪装置50包括：
人脸识别模块501，用于采集视频流中的一帧图像作为参考帧并对该参考帧进行人脸检测和人脸识别，识别出目标人脸；
参考帧窗口位置确定模块502，用于根据识别出的目标人脸，确定参考帧中包括目标人脸的待跟踪窗口的位置，并提取参考帧待跟踪窗口中的边缘特征点；
当前帧窗口位置确定模块503，用于根据参考帧待跟踪窗口的位置，确定视频流中当前帧图像上待跟踪窗口的初始位置，并提取当前帧待跟踪窗口中的边缘特征点；
相对位移计算模块504，用于计算参考帧的边缘特征点与当前帧的边缘特征点的相对位移；
人脸跟踪模块505，用于利用相对位移对当前帧待跟踪窗口的初始位置进行调整，获得视频流中目标人脸的活动轨迹，实现人脸跟踪。
本实施例中，装置50进一步包括：跟踪验证模块和人脸动作判断模块；
跟踪验证模块，用于在利用相对位移对当前帧待跟踪窗口的初始位置进行调整之后，对调整后的当前帧待跟踪窗口中的每个边缘特征点进行验证；当验证成功时，确认在当前帧成功跟踪到该边缘特 征点，在后续跟踪过程中继续对该边缘特征点进行跟踪；当验证失败时，确认在当前帧中对该边缘特征点跟踪失败，舍弃该边缘特征点，不再对该边缘特征点进行跟踪；
人脸动作判断模块，用于利用待跟踪窗口中的边缘特征点在参考帧和当前帧之间的间距变化确定目标人脸的左右转动动作和抬头低头动作；以及，利用参考帧与当前帧之间边缘特征点的相对位移计算目标人脸的倾斜角度，由倾斜角度确定目标人脸的倾斜动作。
本实施例中，跟踪验证模块，具体用于将当前帧待跟踪窗口中的边缘特征点的位置作为验证起始位置；
利用验证起始位置及当前帧与参考帧之间边缘特征点的相对位移，确定边缘特征点在参考帧的待跟踪窗口，并在待跟踪窗口内搜索边缘特征点，得到边缘特征点在参考帧的位置；
将验证出的边缘特征点在参考帧的位置与预先获取的参考帧图像中边缘特征点的位置进行比较，若比较结果相同，则验证成功；若比较结果不同，则验证失败。
本实施例中，人脸动作判断模块，具体用于对当前帧和参考帧，分别将目标人脸的各边缘特征点投影到与目标人脸平行的平面上，并监测各边缘特征点之间的间距；若相对于参考帧中边缘特征点的间距，当前帧的左半边脸的边缘特征点的间距变大且右半边脸的边缘特征点的间距变小，则确定目标人脸向右转；若相对于参考帧中边缘特征点的间距，当前帧的左半边脸的边缘特征点的间距变小且右半边脸的边缘特征点的间距变大，则确定目标人脸向左转；若相对于参考帧中边缘特征点的间距，当前帧的上半边脸的边缘特征点的间距变大且下半边脸的边缘特征点的间距变小，则确定目标人脸在抬头；若相对于参考帧中边缘特征点的间距，当前帧的上半边脸的边缘特征点的间距变小且下半边脸的边缘特征点的间距变大，则确定目标人脸在低头。
其中，分别根据当前帧和参考帧的待跟踪窗口，构建对应的包括四个区域的平面坐标系，将待跟踪窗口中的边缘特征点投影到平面坐标系中，得到每个边缘特征点对应的横坐标和纵坐标值；具体的，四个区域是按照每帧图像待跟踪窗口所在图形的对角线的交点为中点划分而成，包括：区域一、区域二、区域三和区域四，区域一和区域三中边缘特征点的横坐标值小于等于区域二和区域四中边缘特征点的横坐标值，区域三和区域四中边缘特征点的纵坐标值小于等于区域一和区域二中边缘特征点的纵坐标值；
将投影到区域一和区域三中的边缘特征点根据横坐标升序排列得到矩阵
Figure PCTCN2017072545-appb-000046
将投影到区域二和区域四中的边缘特征点根据横坐标升序排成排列得到矩阵
Figure PCTCN2017072545-appb-000047
将投影到区域一和区域二中的边缘特征点根据纵坐标值升序排列得到矩阵
Figure PCTCN2017072545-appb-000048
将投影到区域三和区域四中的边缘特征点根据纵坐标值升序排列得到矩阵
Figure PCTCN2017072545-appb-000049
分别计算矩阵
Figure PCTCN2017072545-appb-000050
中每相邻两个元素的间距得到相应的间距矩阵
Figure PCTCN2017072545-appb-000051
和
Figure PCTCN2017072545-appb-000052
并依次将当前帧的间距矩阵
Figure PCTCN2017072545-appb-000053
和
Figure PCTCN2017072545-appb-000054
中的每个元素与参考帧的间距矩阵中的每个元素作差，根据计算结果判 断目标人脸的左转或右转动作；
分别计算矩阵
Figure PCTCN2017072545-appb-000055
中每相邻两个元素的间距得到相应的间距矩阵
Figure PCTCN2017072545-appb-000056
和
Figure PCTCN2017072545-appb-000057
并依次将当前帧的间距矩阵
Figure PCTCN2017072545-appb-000058
和
Figure PCTCN2017072545-appb-000059
中的每个元素与参考帧的间距矩阵中的每个元素作差，根据计算结果判断目标人脸的抬头或低头动作。
其中，当计算结果满足下列公式(1)和(2)时，确定目标人脸向右转动；
Figure PCTCN2017072545-appb-000060
Figure PCTCN2017072545-appb-000061
其中，
Figure PCTCN2017072545-appb-000062
表示当前帧左半边脸的边缘特征点的间距矩阵，
Figure PCTCN2017072545-appb-000063
表示参考帧左半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000064
表示当前帧右半边脸的边缘特征点的间距矩阵，
Figure PCTCN2017072545-appb-000065
表示参考帧右半边脸的边缘特征点的间距矩阵；
当计算结果满足下列公式(3)和(4)时，确定目标人脸向左转动：
Figure PCTCN2017072545-appb-000066
Figure PCTCN2017072545-appb-000067
其中，
Figure PCTCN2017072545-appb-000068
表示当前帧左半边脸的边缘特征点的间距矩阵，
Figure PCTCN2017072545-appb-000069
表示参考帧左半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000070
表示当前帧右半边脸的边缘特征点的间距矩阵，
Figure PCTCN2017072545-appb-000071
表示参考帧右半边脸的边缘特征点的间距矩阵；
根据计算结果判断目标人脸的抬头或低头动作包括：
当计算结果满足下列公式(5)和(6)时，确定目标人脸在抬头：
Figure PCTCN2017072545-appb-000072
Figure PCTCN2017072545-appb-000073
其中，
Figure PCTCN2017072545-appb-000074
表示当前帧上半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000075
表示参考帧上半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000076
表示当前帧下半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000077
表示参考帧下半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000078
当计算结果满足下列公式(7)和(8)时，确定目标人脸在低头：
Figure PCTCN2017072545-appb-000079
Figure PCTCN2017072545-appb-000080
其中，
Figure PCTCN2017072545-appb-000081
表示当前帧上半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000082
表示参考帧上半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000083
表示当前帧下半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000084
表示参考帧下半边脸的边缘特征点的间距矩阵
Figure PCTCN2017072545-appb-000085
如此，实现了视频流中目标人脸的动作检测。
本实施例中，人脸识别模块501，具体用于采集视频流中的一帧图像并转换为灰度图像，提取灰度图像的类Haar特征值；利用训练好的Adaboost级联分类器对灰度图像的类Haar特征值进行选择，并利用选择的类Haar特征值对灰度图像进行检测得到初始人脸区域；利用构建的肤色模型对初始人脸区域进行肤色判别，得到标准人脸区域；对标准人脸区域，利用设定的Gabor滤波器提取人脸特征；将Gabor滤波器提取出的人脸特征送入训练好的支持向量机SVM中进行匹配识别，以识别出目标人脸。
需要说明的是，本实施例中的人脸跟踪装置是和前述的人脸跟踪方法相对应的，因此，该人脸跟踪装置的工作过程可以参见本发明的人脸跟踪方法实施例中的相关描述，在此不再赘述。
实施例六
图6是本发明一个实施例的一种智能终端的结构框图，参见图6，本实施例的这种智能终端60包括实施例五中所示的人脸跟踪装置50。该智能终端60利用人脸跟踪装置50检测人脸运动状态，且该智能终端60根据定位到的人脸运动状态自适应调整至少部分可移动部件的位置(如转动显示屏的位置)或至少部分部件所执行的功能(如开启或关闭终端中某一些应用的功能)。
一般情况下，当人脸进行抬头和低头操作时，无需移动智能终端的可移动部件位置(或朝向)，而当人脸进行左转和右转时，则可能需要对智能终端的可移动部件的位置进行移动操作。本实施例中，由于该智能终端60包括了人脸跟踪装置，所以当智能终端60的系统判定到有人脸转动动作后，定位到人脸的一次动作结束时，调整智能终端的可移动部件的位置，以使得智能终端跟随人脸的动作，更加人性化。
本实施例中，智能终端60计算相邻的t时刻和t-1时刻，当前帧的左半边脸或右半边脸的边缘特征点的间距矩阵的差值，即
Figure PCTCN2017072545-appb-000086
或
Figure PCTCN2017072545-appb-000087
即确定人脸转动动作接近结束，其中阈值为0，此时，控制将智能终端的可移动部件的位置按转动方向平移π/4弧度。智能终端60的可移动部件移动后，视频流中图像背景随之变化，因此重新对图像进行人脸检测步骤。如此，智能终端通过该人脸跟踪装置跟踪人脸的运动，使得可以针对同一个人实现不间断的跟踪，提高了智能终端的人机交互的能力。
本实施例中，智能终端60可以为检测人体活动的终端设备(如智能机器人)，然后智能机器人根据检测出的目标人脸人体动作，做出对应的配合操作，提高了智能机器人的人机交互性能，优化了用户体验。
综上所述，相比于现有的基于肤色模型算法等人脸跟踪方法，本发明的这种人脸跟踪方法和跟踪装置，首先，克服了背景颜色干扰的问题，提高了人脸跟踪的正确性。其次，利用各个边缘特征点之间的相对位置关系，反推出人脸的姿态、运动方向等信息，方便了人机交互。最后，针对运动过程中 可能存在的特征被遮挡导致跟踪出现错误的问题，采用时间反转验证，克服了由边缘特征点的误判导致跟踪失败的问题。另外，本发明的智能终端包括了人脸跟踪装置，在目标脸部运动时，智能终端可以自主跟随，实现不间断跟踪的效果，优化了人机交互体验，提升了产品的竞争力。
以上所述仅为本发明的较佳实施例而已，并非用于限定本发明的保护范围。凡在本发明的精神和原则之内所作的任何修改、等同替换、改进等，均包含在本发明的保护范围内。

Claims (20)

  1. 一种人脸跟踪方法，其特征在于，该方法包括：
    采集视频流中的一帧图像作为参考帧并对参考帧进行人脸检测和人脸识别，识别出目标人脸；
    根据识别出的目标人脸，确定参考帧中包括目标人脸的待跟踪窗口的位置，并提取参考帧待跟踪窗口中的边缘特征点；
    根据参考帧待跟踪窗口的位置，确定视频流中当前帧图像上待跟踪窗口的初始位置，并提取当前帧待跟踪窗口中的边缘特征点；
    计算参考帧的边缘特征点与当前帧的边缘特征点的相对位移；
    利用所述相对位移对当前帧待跟踪窗口的初始位置进行调整，获得视频流中目标人脸的活动轨迹，实现人脸跟踪。
  2. 根据权利要求1所述的方法，其特征在于，在利用所述相对位移对当前帧待跟踪窗口的初始位置进行调整之后，该方法进一步包括：
    对调整后的当前帧待跟踪窗口中的每个边缘特征点进行验证；
    当验证成功时，确认在当前帧成功跟踪到该边缘特征点，在后续跟踪过程中继续对该边缘特征点进行跟踪；
    当验证失败时，确认在当前帧中对该边缘特征点跟踪失败，舍弃该边缘特征点，不再对该边缘特征点进行跟踪。
  3. 根据权利要求2所述的方法，其特征在于，所述对调整后的当前帧待跟踪窗口中的每个边缘特征点进行验证包括：
    将当前帧待跟踪窗口中的边缘特征点的位置作为验证起始位置；
    利用验证起始位置及所述当前帧与所述参考帧之间边缘特征点的相对位移，确定边缘特征点在所述参考帧的待跟踪窗口，并在所述待跟踪窗口内搜索边缘特征点，得到边缘特征点在参考帧的位置；
    将验证出的边缘特征点在参考帧的位置与预先获取的参考帧图像中边缘特征点的位置进行比较，若比较结果相同，则验证成功；若比较结果不同，则验证失败。
  4. 根据权利要求1所述的方法，其特征在于，该方法进一步包括：
    利用待跟踪窗口中的边缘特征点在所述参考帧和当前帧之间的间距变化确定目标人脸的左右转动动作和抬头低头动作；
    以及，利用所述参考帧与当前帧之间边缘特征点的相对位移计算目标人脸的倾斜角度，由所述倾斜角度确定目标人脸的倾斜动作。
  5. 根据权利要求4所述的方法，其特征在于，所述利用待跟踪窗口中的边缘特征点在参考帧和当前帧之间的间距变化判断目标人脸的左右转动动作和抬头低头动作包括：
    对当前帧和参考帧，分别将目标人脸的各边缘特征点投影到与目标人脸平行的平面上，并监测各边缘特征点之间的间距；
    若相对于参考帧中边缘特征点的间距，当前帧的左半边脸的边缘特征点的间距变大且右半边脸的边缘特征点的间距变小，则确定目标人脸向右转；
    若相对于参考帧中边缘特征点的间距，当前帧的左半边脸的边缘特征点的间距变小且右半边脸的边缘特征点的间距变大，则确定目标人脸向左转；
    若相对于参考帧中边缘特征点的间距，当前帧的上半边脸的边缘特征点的间距变大且下半边脸的边缘特征点的间距变小，则确定目标人脸在抬头；
    若相对于参考帧中边缘特征点的间距，当前帧的上半边脸的边缘特征点的间距变小且下半边脸的边缘特征点的间距变大，则确定目标人脸在低头。
  6. 根据权利要求5所述的方法，其特征在于，所述对当前帧和参考帧，分别将目标人脸的各边缘特征点投影到与目标人脸平行的平面上，并监测各边缘特征点之间的间距包括：
    分别根据所述当前帧和参考帧的待跟踪窗口，构建对应的包括四个区域的平面坐标系，将待跟踪窗口中的边缘特征点投影到所述平面坐标系中，得到每个边缘特征点对应的横坐标和纵坐标值；
    其中，所述四个区域是按照每帧图像待跟踪窗口所在图形的对角线的交点为中点划分而成，包括：区域一、区域二、区域三和区域四，所述区域一和区域三中边缘特征点的横坐标值小于等于区域二和区域四中边缘特征点的横坐标值，区域三和区域四中边缘特征点的纵坐标值小于等于区域一和区域二中边缘特征点的纵坐标值；
    将投影到所述区域一和区域三中的边缘特征点根据横坐标升序排列得到矩阵
    Figure PCTCN2017072545-appb-100001
    将投影到所述区域二和区域四中的边缘特征点根据横坐标升序排成排列得到矩阵
    Figure PCTCN2017072545-appb-100002
    将投影到所述区域一和区域二中的边缘特征点根据纵坐标值升序排列得到矩阵
    Figure PCTCN2017072545-appb-100003
    将投影到所述区域三和区域四中的边缘特征点根据纵坐标值升序排列得到矩阵
    Figure PCTCN2017072545-appb-100004
    分别计算所述矩阵
    Figure PCTCN2017072545-appb-100005
    中每相邻两个元素的间距得到相应的间距矩阵
    Figure PCTCN2017072545-appb-100006
    和
    Figure PCTCN2017072545-appb-100007
    并依次将当前帧的间距矩阵
    Figure PCTCN2017072545-appb-100008
    和
    Figure PCTCN2017072545-appb-100009
    中的每个元素与参考帧的间距矩阵中的每个元素作差，根据计算结果判断目标人脸的左转或右转动作；
    分别计算所述矩阵
    Figure PCTCN2017072545-appb-100010
    中每相邻两个元素的间距得到相应的间距矩阵和，并依次将当前帧的间距矩阵
    Figure PCTCN2017072545-appb-100011
    和
    Figure PCTCN2017072545-appb-100012
    中的每个元素与参考帧的间距矩阵中的每个元素作差，根据计算结果判断目标人脸的抬头或低头动作。
  7. 根据权利要求6所述的方法，其特征在于，所述根据计算结果判断目标人脸的左转或右转动作包括：
    当计算结果满足下列公式(1)和(2)时，确定目标人脸向右转动；
    Figure PCTCN2017072545-appb-100013
    其中，
    Figure PCTCN2017072545-appb-100014
    表示当前帧左半边脸的边缘特征点的间距矩阵，
    Figure PCTCN2017072545-appb-100015
    表示参考帧左半边脸的边缘特征点的间距矩阵
    Figure PCTCN2017072545-appb-100016
    表示当前帧右半边脸的边缘特征点的间距矩阵，
    Figure PCTCN2017072545-appb-100017
    表示参考帧右半边脸的边缘特征点的间距矩阵；
    当计算结果满足下列公式(3)和(4)时，确定目标人脸向左转动：
    Figure PCTCN2017072545-appb-100018
    其中，
    Figure PCTCN2017072545-appb-100019
    表示当前帧左半边脸的边缘特征点的间距矩阵，
    Figure PCTCN2017072545-appb-100020
    表示参考帧左半边脸的边缘特征点的间距矩阵
    Figure PCTCN2017072545-appb-100021
    表示当前帧右半边脸的边缘特征点的间距矩阵，
    Figure PCTCN2017072545-appb-100022
    表示参考帧右半边脸的边缘特征点的间距矩阵；
    所述根据计算结果判断目标人脸的抬头或低头动作包括：
    当计算结果满足下列公式(5)和(6)时，确定目标人脸在抬头：
    Figure PCTCN2017072545-appb-100023
    其中，
    Figure PCTCN2017072545-appb-100024
    表示当前帧上半边脸的边缘特征点的间距矩阵
    Figure PCTCN2017072545-appb-100025
    表示参考帧上半边脸的边缘特征点的间距矩阵
    Figure PCTCN2017072545-appb-100026
    表示当前帧下半边脸的边缘特征点的间距矩阵
    Figure PCTCN2017072545-appb-100027
    表示参考帧下半边脸的边缘特征点的间距矩阵
    Figure PCTCN2017072545-appb-100028
    当计算结果满足下列公式(7)和(8)时，确定目标人脸在低头：
    Figure PCTCN2017072545-appb-100029
    其中，
    Figure PCTCN2017072545-appb-100030
    表示当前帧上半边脸的边缘特征点的间距矩阵
    Figure PCTCN2017072545-appb-100031
    表示参考帧上半边脸的边缘特征点的间距矩阵
    Figure PCTCN2017072545-appb-100032
    表示当前帧下半边脸的边缘特征点的间距矩阵
    Figure PCTCN2017072545-appb-100033
    表示参考帧下半边脸的边缘特征点的间距矩阵。
  8. 根据权利要求1所述的方法，其特征在于，所述采集视频流中的一帧图像并对参考帧进行人脸检测和人脸识别，识别出目标人脸包括：
    采集视频流中的一帧图像并转换为灰度图像，提取所述灰度图像的类Haar特征值；
    利用训练好的Adaboost级联分类器对所述灰度图像的类Haar特征值进行选择，并利用选择的类Haar特征值对所述灰度图像进行检测得到初始人脸区域；
    利用构建的肤色模型对所述初始人脸区域进行肤色判别，得到标准人脸区域；
    对所述标准人脸区域，利用设定的Gabor滤波器提取人脸特征；
    将所述Gabor滤波器提取出的人脸特征送入训练好的支持向量机SVM中进行匹配识别，以识别出目标人脸。
  9. 一种人脸跟踪装置，其特征在于，该装置包括：
    人脸识别模块，用于采集视频流中的一帧图像作为参考帧并对该参考帧进行人脸检测和人脸识别，识别出目标人脸；
    参考帧窗口位置确定模块，用于根据识别出的目标人脸，确定参考帧中包括目标人脸的待跟踪窗口的位置，并提取参考帧待跟踪窗口中的边缘特征点；
    当前帧窗口位置确定模块，用于根据参考帧待跟踪窗口的位置，确定视频流中当前帧图像上待跟踪窗口的初始位置，并提取当前帧待跟踪窗口中的边缘特征点；
    相对位移计算模块，用于计算参考帧的边缘特征点与当前帧的边缘特征点的相对位移；
    人脸跟踪模块，用于利用所述相对位移对当前帧待跟踪窗口的初始位置进行调整，获得视频流中目标人脸的活动轨迹，实现人脸跟踪。
  10. 根据权利要求9所述的人脸跟踪装置，其特征在于，该装置进一步包括跟踪验证模块；
    所述跟踪验证模块，用于在利用所述相对位移对当前帧待跟踪窗口的初始位置进行调整之后，对调整后的当前帧待跟踪窗口中的每个边缘特征点进行验证；当验证成功时，确认在当前帧成功跟踪到该边缘特征点，在后续跟踪过程中继续对该边缘特征点进行跟踪；当验证失败时，确认在当前帧中对该边缘特征点跟踪失败，舍弃该边缘特征点，不再对该边缘特征点进行跟踪。
  11. 根据权利要求10所述的人脸跟踪装置，其特征在于，所述跟踪验证模块进一步用于：
    将当前帧待跟踪窗口中的边缘特征点的位置作为验证起始位置；
    利用验证起始位置及所述当前帧与所述参考帧之间边缘特征点的相对位移，确定边缘特征点在所述参考帧的待跟踪窗口，并在所述待跟踪窗口内搜索边缘特征点，得到边缘特征点在参考帧的位置；
    将验证出的边缘特征点在参考帧的位置与预先获取的参考帧图像中边缘特征点的位置进行比较，若比较结果相同，则验证成功；若比较结果不同，则验证失败。
  12. 根据权利要求9所述的人脸跟踪装置，其特征在于，该装置进一步包括人脸动作判断模块；
    所述人脸动作判断模块，用于利用待跟踪窗口中的边缘特征点在所述参考帧和当前帧之间的间距变化确定目标人脸的左右转动动作和抬头低头动作；以及，利用所述参考帧与当前帧之间边缘特征点的相对位移计算目标人脸的倾斜角度，由所述倾斜角度确定目标人脸的倾斜动作。
  13. 根据权利要求9所述的人脸跟踪装置，其特征在于，所述人脸动作判断模块，用于利用待跟踪窗口中的边缘特征点在所述参考帧和当前帧之间的间距变化确定目标人脸的左右转动动作和抬头低头动作包括：对当前帧和参考帧，分别将目标人脸的各边缘特征点投影到与目标人脸平行的平面上，并监测各边缘特征点之间的间距；
    若相对于参考帧中边缘特征点的间距，当前帧的左半边脸的边缘特征点的间距变大且右半边脸的边缘特征点的间距变小，则确定目标人脸向右转；
    若相对于参考帧中边缘特征点的间距，当前帧的左半边脸的边缘特征点的间距变小且右半边脸的边缘特征点的间距变大，则确定目标人脸向左转；
    若相对于参考帧中边缘特征点的间距，当前帧的上半边脸的边缘特征点的间距变大且下半边脸的边缘特征点的间距变小，则确定目标人脸在抬头；
    若相对于参考帧中边缘特征点的间距，当前帧的上半边脸的边缘特征点的间距变小且下半边脸的边缘特征点的间距变大，则确定目标人脸在低头。
  14. 根据权利要求13所述的人脸跟踪装置，其特征在于，所述人脸动作判断模块，对当前帧和参考帧，分别将目标人脸的各边缘特征点投影到与目标人脸平行的平面上，并监测各边缘特征点之间的间距包括：
    分别根据所述当前帧和参考帧的待跟踪窗口，构建对应的包括四个区域的平面坐标系，将待跟踪窗口中的边缘特征点投影到所述平面坐标系中，得到每个边缘特征点对应的横坐标和纵坐标值；
    其中，所述四个区域是按照每帧图像待跟踪窗口所在图形的对角线的交点为中点划分而成，包括：区域一、区域二、区域三和区域四，所述区域一和区域三中边缘特征点的横坐标值小于等于区域二和区域四中边缘特征点的横坐标值，区域三和区域四中边缘特征点的纵坐标值小于等于区域一和区域二中边缘特征点的纵坐标值；
    将投影到所述区域一和区域三中的边缘特征点根据横坐标升序排列得到矩阵
    Figure PCTCN2017072545-appb-100034
    将投影到所述区域二和区域四中的边缘特征点根据横坐标升序排成排列得到矩阵
    Figure PCTCN2017072545-appb-100035
    将投影到所述区域一和区域二中的边缘特征点根据纵坐标值升序排列得到矩阵
    Figure PCTCN2017072545-appb-100036
    将投影到所述区域三和区域四中的边缘特征点根据纵坐标值升序排列得到矩阵
    Figure PCTCN2017072545-appb-100037
    分别计算所述矩阵
    Figure PCTCN2017072545-appb-100038
    中每相邻两个元素的间距得到相应的间距矩阵
    Figure PCTCN2017072545-appb-100039
    和
    Figure PCTCN2017072545-appb-100040
    并依 次将当前帧的间距矩阵
    Figure PCTCN2017072545-appb-100041
    和
    Figure PCTCN2017072545-appb-100042
    中的每个元素与参考帧的间距矩阵中的每个元素作差，根据计算结果判断目标人脸的左转或右转动作；
    分别计算所述矩阵
    Figure PCTCN2017072545-appb-100043
    中每相邻两个元素的间距得到相应的间距矩阵和，并依次将当前帧的间距矩阵
    Figure PCTCN2017072545-appb-100044
    和
    Figure PCTCN2017072545-appb-100045
    中的每个元素与参考帧的间距矩阵中的每个元素作差，根据计算结果判断目标人脸的抬头或低头动作。
  15. 一种智能终端，其特征在于，所述智能终端包括：如权利要求9所述的人脸跟踪装置；
    该智能终端利用所述人脸跟踪装置检测人脸运动状态，且该智能终端根据定位到的人脸运动状态自适应调整至少部分可移动部件的位置或至少部分部件所执行的功能。
  16. 根据权利要求15所述的智能终端，其特征在于，所述人脸跟踪装置进一步包括跟踪验证模块；
    所述跟踪验证模块，用于在利用所述相对位移对当前帧待跟踪窗口的初始位置进行调整之后，对调整后的当前帧待跟踪窗口中的每个边缘特征点进行验证；当验证成功时，确认在当前帧成功跟踪到该边缘特征点，在后续跟踪过程中继续对该边缘特征点进行跟踪；当验证失败时，确认在当前帧中对该边缘特征点跟踪失败，舍弃该边缘特征点，不再对该边缘特征点进行跟踪。
  17. 根据权利要求16所述的智能终端，其特征在于，所述跟踪验证模块进一步用于：
    将当前帧待跟踪窗口中的边缘特征点的位置作为验证起始位置；
    利用验证起始位置及所述当前帧与所述参考帧之间边缘特征点的相对位移，确定边缘特征点在所述参考帧的待跟踪窗口，并在所述待跟踪窗口内搜索边缘特征点，得到边缘特征点在参考帧的位置；
    将验证出的边缘特征点在参考帧的位置与预先获取的参考帧图像中边缘特征点的位置进行比较，若比较结果相同，则验证成功；若比较结果不同，则验证失败。
  18. 根据权利要求15所述的智能终端，其特征在于，所述人脸跟踪装置进一步包括人脸动作判断模块；
    所述人脸动作判断模块，用于利用待跟踪窗口中的边缘特征点在所述参考帧和当前帧之间的间距变化确定目标人脸的左右转动动作和抬头低头动作；以及，利用所述参考帧与当前帧之间边缘特征点的相对位移计算目标人脸的倾斜角度，由所述倾斜角度确定目标人脸的倾斜动作。
  19. 根据权利要求18所述的智能终端，其特征在于，所述人脸动作判断模块，用于利用待跟踪窗口中的边缘特征点在所述参考帧和当前帧之间的间距变化确定目标人脸的左右转动动作和抬头低头动作包括：对当前帧和参考帧，分别将目标人脸的各边缘特征点投影到与目标人脸平行的平面上，并监测各边缘特征点之间的间距；
    若相对于参考帧中边缘特征点的间距，当前帧的左半边脸的边缘特征点的间距变大且右半边脸的边缘特征点的间距变小，则确定目标人脸向右转；
    若相对于参考帧中边缘特征点的间距，当前帧的左半边脸的边缘特征点的间距变小且右半边脸的边缘特征点的间距变大，则确定目标人脸向左转；
    若相对于参考帧中边缘特征点的间距，当前帧的上半边脸的边缘特征点的间距变大且下半边脸的边缘特征点的间距变小，则确定目标人脸在抬头；
    若相对于参考帧中边缘特征点的间距，当前帧的上半边脸的边缘特征点的间距变小且下半边脸的边缘特征点的间距变大，则确定目标人脸在低头。
  20. 根据权利要求15所述的智能终端，其特征在于，所述该智能终端根据定位到的人脸运动状态自适应调整至少部分可移动部件的位置或至少部分部件所执行的功能包括：
    在人脸转动时，计算左半边脸或右半边脸的相邻时刻当前帧图像中边缘特征点的间距矩阵的差值，当该差值小于一阈值时，确定人脸的一次转动动作结束，则智能终端相应于该转动动作，自适应调整至少部分可移动部件的位置或至少部分部件所执行的功能。
PCT/CN2017/072545 2016-02-03 2017-01-25 一种人脸跟踪方法、装置和智能终端 WO2017133605A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/068,953 US10747988B2 (en) 2016-02-03 2017-01-25 Method and device for face tracking and smart terminal

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610076493.6 2016-02-03
CN201610076493.6A CN105760826B (zh) 2016-02-03 2016-02-03 一种人脸跟踪方法、装置和智能终端

Publications (1)

Publication Number Publication Date
WO2017133605A1 true WO2017133605A1 (zh) 2017-08-10

Family

ID=56330098

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/072545 WO2017133605A1 (zh) 2016-02-03 2017-01-25 一种人脸跟踪方法、装置和智能终端

Country Status (3)

Country Link
US (1) US10747988B2 (zh)
CN (1) CN105760826B (zh)
WO (1) WO2017133605A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111814590A (zh) * 2020-06-18 2020-10-23 浙江大华技术股份有限公司 人员安全状态监测方法、设备及计算机可读存储介质
CN112489086A (zh) * 2020-12-11 2021-03-12 北京澎思科技有限公司 目标跟踪方法、目标跟踪装置、电子设备及存储介质
CN113052119A (zh) * 2021-04-07 2021-06-29 兴体(广州)智能科技有限公司 球类运动跟踪摄像方法及系统
CN117132506A (zh) * 2023-10-23 2023-11-28 深圳市高进实业有限公司 基于视觉技术的钟表零配件质量检测方法

Families Citing this family (43)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105760826B (zh) * 2016-02-03 2020-11-13 歌尔股份有限公司 一种人脸跟踪方法、装置和智能终端
CN106214163B (zh) * 2016-07-18 2019-12-13 北京科技大学 一种下肢畸形矫正术后康复的人工心理辅导装置
CN106203370B (zh) * 2016-07-19 2019-07-09 成都通甲优博科技有限责任公司 一种基于计算机视觉技术的视力测定系统
CN106295511B (zh) * 2016-07-26 2019-05-21 北京小米移动软件有限公司 人脸跟踪方法及装置
CN106550136B (zh) * 2016-10-26 2020-12-29 努比亚技术有限公司 一种人脸提示框的追踪显示方法及移动终端
CN106650682B (zh) * 2016-12-29 2020-05-01 Tcl集团股份有限公司 一种人脸追踪的方法及装置
CN106650965B (zh) * 2016-12-30 2020-11-06 触景无限科技（北京）有限公司 一种远程视频处理方法及装置
CN106875422B (zh) * 2017-02-06 2022-02-25 腾讯科技（上海）有限公司 人脸跟踪方法和装置
CN108875460B (zh) * 2017-05-15 2023-06-20 腾讯科技（深圳）有限公司 增强现实处理方法及装置、显示终端及计算机存储介质
CN107358146B (zh) * 2017-05-22 2018-05-22 深圳云天励飞技术有限公司 视频处理方法、装置及存储介质
CN108304758B (zh) * 2017-06-21 2020-08-25 腾讯科技（深圳）有限公司 人脸特征点跟踪方法及装置
CN107506737A (zh) * 2017-08-29 2017-12-22 四川长虹电器股份有限公司 人脸性别识别方法
CN108170282A (zh) * 2018-01-19 2018-06-15 百度在线网络技术（北京）有限公司 用于控制三维场景的方法和装置
CN108596067B (zh) * 2018-04-15 2019-09-10 中少科普（北京）教育科技有限公司 一种少先队员敬礼姿势检测校正方法
TWI658815B (zh) * 2018-04-25 2019-05-11 國立交通大學 非接觸式心跳量測系統、非接觸式心跳量測方法以及非接觸式心跳量測裝置
CN110210276A (zh) * 2018-05-15 2019-09-06 腾讯科技（深圳）有限公司 一种移动轨迹获取方法及其设备、存储介质、终端
CN110795975B (zh) * 2018-08-03 2023-07-21 浙江宇视科技有限公司 人脸误检优化方法及装置
CN108960206B (zh) * 2018-08-07 2021-01-22 北京字节跳动网络技术有限公司 视频帧处理方法和装置
CN109344742B (zh) * 2018-09-14 2021-03-16 腾讯科技（深圳）有限公司 特征点定位方法、装置、存储介质和计算机设备
CN109523573A (zh) * 2018-11-23 2019-03-26 上海新世纪机器人有限公司 目标对象的跟踪方法和装置
CN109670451A (zh) * 2018-12-20 2019-04-23 天津天地伟业信息系统集成有限公司 自动人脸识别跟踪方法
JP7138199B2 (ja) * 2019-01-30 2022-09-15 富士フイルム株式会社 顔領域検出装置、撮像装置、顔領域検出方法、及び顔領域検出プログラム
US11191342B2 (en) * 2019-07-18 2021-12-07 L'oreal Techniques for identifying skin color in images having uncontrolled lighting conditions
TWI719591B (zh) * 2019-08-16 2021-02-21 緯創資通股份有限公司 物件追蹤方法及其電腦系統
KR20210024859A (ko) * 2019-08-26 2021-03-08 삼성전자주식회사 영상 처리 방법 및 장치
CN110544272B (zh) * 2019-09-06 2023-08-04 腾讯科技（深圳）有限公司 脸部跟踪方法、装置、计算机设备及存储介质
CN110991250B (zh) * 2019-11-06 2023-04-25 江苏科技大学 融合颜色干扰模型与遮挡模型的人脸跟踪方法和跟踪系统
WO2021106696A1 (ja) * 2019-11-26 2021-06-03 富士フイルム株式会社 情報処理装置及び方法並びに誘導システム及び方法
CN111105444B (zh) * 2019-12-31 2023-07-25 哈尔滨工程大学 一种适用于水下机器人目标抓取的连续跟踪方法
CN111179315A (zh) * 2019-12-31 2020-05-19 湖南快乐阳光互动娱乐传媒有限公司 一种视频目标区域跟踪方法和视频平面广告植入方法
CN111523467B (zh) * 2020-04-23 2023-08-08 北京百度网讯科技有限公司 人脸跟踪方法和装置
CN111626240B (zh) * 2020-05-29 2023-04-07 歌尔科技有限公司 一种人脸图像识别方法、装置、设备及可读存储介质
CN111783618A (zh) * 2020-06-29 2020-10-16 联通(浙江)产业互联网有限公司 一种基于视频内容分析的园区大脑感知方法和系统
CN112733719B (zh) * 2021-01-11 2022-08-02 西南交通大学 一种融合人脸及人体特征的跨境行人轨迹检测方法
CN112927268B (zh) * 2021-03-17 2022-08-05 清华大学 角点跟踪方法、装置、计算机设备及可读存储介质
CN113034458B (zh) * 2021-03-18 2023-06-23 广州市索图智能电子有限公司 室内人员轨迹分析方法、装置及存储介质
CN113242428B (zh) * 2021-04-15 2024-03-15 中南大学 视频会议场景下基于roi区域的后处理加速方法
CN113283430B (zh) * 2021-07-23 2022-12-27 深圳市赛菲姆科技有限公司 目标检测的方法、装置、电子设备及计算机可读存储介质
CN114359773A (zh) * 2021-11-10 2022-04-15 中国矿业大学 复杂地下空间轨迹融合的视频人员重识别方法
CN113793283B (zh) * 2021-11-15 2022-02-11 江苏游隼微电子有限公司 一种车载影像降噪方法
CN115114466B (zh) * 2022-08-30 2022-12-13 成都实时技术股份有限公司 一种打靶信息图像的查找方法、系统、介质及电子设备
CN115297313B (zh) * 2022-10-09 2023-04-25 南京芯视元电子有限公司 微显示动态补偿方法及系统
CN116912923B (zh) * 2023-09-12 2024-01-05 深圳须弥云图空间科技有限公司 一种图像识别模型训练方法和装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6028960A (en) * 1996-09-20 2000-02-22 Lucent Technologies Inc. Face feature analysis for automatic lipreading and character animation
CN103093212A (zh) * 2013-01-28 2013-05-08 北京信息科技大学 基于人脸检测和跟踪截取人脸图像的方法和装置
CN104866805A (zh) * 2014-02-20 2015-08-26 腾讯科技（深圳）有限公司 人脸实时跟踪的方法和装置
CN105069408A (zh) * 2015-07-24 2015-11-18 上海依图网络科技有限公司 一种复杂场景下基于人脸识别的视频人像跟踪方法
CN105760826A (zh) * 2016-02-03 2016-07-13 歌尔声学股份有限公司 一种人脸跟踪方法、装置和智能终端

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW413795B (en) * 1999-02-26 2000-12-01 Cyberlink Corp An image processing method of 3-D head motion with three face feature points
US7130446B2 (en) * 2001-12-03 2006-10-31 Microsoft Corporation Automatic detection and tracking of multiple individuals using multiple cues
EP1631937B1 (en) * 2003-06-12 2018-03-28 Honda Motor Co., Ltd. Target orientation estimation using depth sensing
KR20080073933A (ko) * 2007-02-07 2008-08-12 삼성전자주식회사 객체 트래킹 방법 및 장치, 그리고 객체 포즈 정보 산출방법 및 장치
CN101477690B (zh) * 2008-12-30 2011-04-27 清华大学 一种视频帧序列中物体轮廓跟踪的方法和装置
CN102054159B (zh) * 2009-10-28 2014-05-28 腾讯科技（深圳）有限公司 一种人脸跟踪的方法和装置
US9478033B1 (en) * 2010-08-02 2016-10-25 Red Giant Software Particle-based tracking of objects within images
US20120321134A1 (en) * 2011-06-15 2012-12-20 Samsung Electornics Co., Ltd Face tracking method and device
KR101381439B1 (ko) * 2011-09-15 2014-04-04 가부시끼가이샤 도시바 얼굴 인식 장치 및 얼굴 인식 방법
JP5795979B2 (ja) * 2012-03-15 2015-10-14 株式会社東芝 人物画像処理装置、及び人物画像処理方法
JP5895703B2 (ja) * 2012-05-22 2016-03-30 ソニー株式会社 画像処理装置及び画像処理方法、並びにコンピューター・プログラム
CN103514441B (zh) * 2013-09-21 2017-02-15 南京信息工程大学 基于移动平台的人脸特征点定位跟踪方法
US9934423B2 (en) * 2014-07-29 2018-04-03 Microsoft Technology Licensing, Llc Computerized prominent character recognition in videos
US9990780B2 (en) * 2016-10-03 2018-06-05 Ditto Technologies, Inc. Using computed facial feature points to position a product model relative to a model of a face
US10319412B2 (en) * 2016-11-16 2019-06-11 Adobe Inc. Robust tracking of objects in videos
US10068135B2 (en) * 2016-12-22 2018-09-04 TCL Research America Inc. Face detection, identification, and tracking system for robotic devices

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6028960A (en) * 1996-09-20 2000-02-22 Lucent Technologies Inc. Face feature analysis for automatic lipreading and character animation
CN103093212A (zh) * 2013-01-28 2013-05-08 北京信息科技大学 基于人脸检测和跟踪截取人脸图像的方法和装置
CN104866805A (zh) * 2014-02-20 2015-08-26 腾讯科技（深圳）有限公司 人脸实时跟踪的方法和装置
CN105069408A (zh) * 2015-07-24 2015-11-18 上海依图网络科技有限公司 一种复杂场景下基于人脸识别的视频人像跟踪方法
CN105760826A (zh) * 2016-02-03 2016-07-13 歌尔声学股份有限公司 一种人脸跟踪方法、装置和智能终端

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111814590A (zh) * 2020-06-18 2020-10-23 浙江大华技术股份有限公司 人员安全状态监测方法、设备及计算机可读存储介质
CN111814590B (zh) * 2020-06-18 2023-09-29 浙江大华技术股份有限公司 人员安全状态监测方法、设备及计算机可读存储介质
CN112489086A (zh) * 2020-12-11 2021-03-12 北京澎思科技有限公司 目标跟踪方法、目标跟踪装置、电子设备及存储介质
CN113052119A (zh) * 2021-04-07 2021-06-29 兴体(广州)智能科技有限公司 球类运动跟踪摄像方法及系统
CN113052119B (zh) * 2021-04-07 2024-03-15 兴体(广州)智能科技有限公司 球类运动跟踪摄像方法及系统
CN117132506A (zh) * 2023-10-23 2023-11-28 深圳市高进实业有限公司 基于视觉技术的钟表零配件质量检测方法
CN117132506B (zh) * 2023-10-23 2024-01-19 深圳市高进实业有限公司 基于视觉技术的钟表零配件质量检测方法

Also Published As

Publication number Publication date
CN105760826A (zh) 2016-07-13
US10747988B2 (en) 2020-08-18
CN105760826B (zh) 2020-11-13
US20190034700A1 (en) 2019-01-31

Similar Documents

Publication Publication Date Title
WO2017133605A1 (zh) 一种人脸跟踪方法、装置和智能终端
Valenti et al. Accurate eye center location and tracking using isophote curvature
CN108958473B (zh) 眼球追踪方法、电子装置及非暂态电脑可读取记录媒体
US20180211104A1 (en) Method and device for target tracking
KR101184097B1 (ko) 얼굴 정면포즈 판단 방법
Jain et al. Real-time upper-body human pose estimation using a depth camera
Kumano et al. Pose-invariant facial expression recognition using variable-intensity templates
US20120169887A1 (en) Method and system for head tracking and pose estimation
US8243993B2 (en) Method for moving object detection and hand gesture control method based on the method for moving object detection
JP6835223B2 (ja) 顔認証装置、顔認証方法およびプログラム
US9082000B2 (en) Image processing device and image processing method
JP5598751B2 (ja) 動作認識装置
CN112101208A (zh) 高龄老人特征串联融合手势识别方法及装置
WO2012117901A1 (ja) データ処理装置、データ処理システム、及びプログラム
Hata et al. Detection of distant eye-contact using spatio-temporal pedestrian skeletons
CN114402370A (zh) 基于面部-肩部特征坐标变换的自适应手部跟踪和手势识别
Hansen et al. Eye typing off the shelf
CN108694348B (zh) 一种基于自然特征的跟踪注册方法及装置
JP5829155B2 (ja) 歩行者検出装置及びプログラム
Petersen et al. Fast hand detection using posture invariant constraints
KR101844367B1 (ko) 부분 포즈 추정에 의하여 개략적인 전체 초기설정을 사용하는 머리 포즈 추정 방법 및 장치
Quesada et al. Unsupervised markerless 3-DOF motion tracking in real time using a single low-budget camera
Hannuksela et al. Face tracking for spatially aware mobile user interfaces
KR101788070B1 (ko) 코 영역 검출 방법
Nam et al. Mouth tracking for hands-free robot control systems

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17746925

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17746925

Country of ref document: EP

Kind code of ref document: A1