CN111523524A - Facial animation capturing and correcting method based on machine learning and image processing - Google Patents
Facial animation capturing and correcting method based on machine learning and image processing Download PDFInfo
- Publication number
- CN111523524A CN111523524A CN202010624435.9A CN202010624435A CN111523524A CN 111523524 A CN111523524 A CN 111523524A CN 202010624435 A CN202010624435 A CN 202010624435A CN 111523524 A CN111523524 A CN 111523524A
- Authority
- CN
- China
- Prior art keywords
- mark points
- training
- image
- machine learning
- image processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000010801 machine learning Methods 0.000 title claims abstract description 20
- 238000012545 processing Methods 0.000 title claims abstract description 18
- 230000001815 facial effect Effects 0.000 title claims abstract description 14
- 238000012549 training Methods 0.000 claims abstract description 39
- 238000001514 detection method Methods 0.000 claims abstract description 9
- 238000007781 pre-processing Methods 0.000 claims abstract description 7
- 230000003287 optical effect Effects 0.000 claims abstract description 6
- 238000001914 filtration Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 3
- 238000012937 correction Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/40—Image enhancement or restoration using histogram techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a facial animation capturing and correcting method based on machine learning and image processing, which comprises the following steps: using a head-mounted camera to carry out video shooting on the face of a person with mark points, preprocessing the collected image, selecting a picture as a training set according to the number of frames at uniform intervals, manually marking the positions of the mark points in the picture, and rotating, translating and changing the brightness of the picture in the training set; training the training set by adopting a training model; respectively inputting images shot by a camera into a network model to obtain marked mark points, and detecting and tracking the mark points by using an opencv spot detection method and an LK optical flow method; the mark points obtained by training through the training model are matched with the mark points detected by opencv, the mark points which are not matched are displayed in an emphasized mode, the positions of the mark points are corrected according to the actual situation, and the robustness is improved.
Description
Technical Field
The invention relates to the technical field of facial animation capturing and correcting, in particular to a facial animation capturing and correcting method based on machine learning and image processing.
Background
In the field of face animation production, changes such as expression and motion of a face need to be tracked, and some key points of the face need to be assisted.
Disclosure of Invention
The invention aims to provide a facial animation capturing and correcting method based on machine learning and image processing, which combines the machine learning and the image processing, and has the advantages of less required training data and high training speed;
the invention provides the following technical scheme:
a facial animation capturing and correcting method based on machine learning and image processing comprises the following steps:
s1, marking a plurality of mark points on the face of a person to be sampled, and shooting the face of the person with the mark points by using a plurality of head-mounted cameras;
s2, preprocessing each frame of image collected by each camera, and selecting the images as a training set according to the number of frames at uniform intervals;
s3, manually marking the position of the mark point in the picture on the picture in the training set;
s4, rotating, translating and changing the brightness of the pictures in the training set to amplify the data;
s5, training the amplified picture data in the training set by adopting a training model of a face key point algorithm to obtain the position information of the mark point in the picture;
s6, respectively inputting the images shot by the camera into a network model to obtain marked mark points, detecting the mark points by using an opencv spot detection method, and tracking the mark points by using an LK optical flow method;
s7, matching the mark points obtained by training by adopting the training model with the mark points detected by the opencv, displaying the mark points which are not matched in an emphasized mode, and correcting the positions of the mark points according to actual conditions;
preferably, the number of the cameras of step S1 is at least two;
preferably, the preprocessing in step S2 is to perform histogram equalization on the picture taken by the camera;
preferably, the number of the interval frames selected in step S2 is 10;
preferably, the step of detecting the mark points by using the opencv spot detection method in step S6 is as follows:
p1, setting a low threshold and a high threshold for the image to be detected, setting a threshold step, and selecting a series of thresholds from the low threshold to the high threshold according to the threshold step;
p2, binarizing the image by using each selected threshold, searching the edge of the image by using a findcontours function, detecting the outline of the image, and calculating the center of the outline of each image;
p3, suppressing the center of the outline of each image, defining a minimum distance, defining the characteristic center in the distance area as a blob, and obtaining a mark point set;
p4, filtering the mark points;
preferably, the filtering of step P4 includes color filtering and area filtering;
the invention has the beneficial effects that: the invention combines machine learning and image processing, uses opencv spot detection to detect the mark points and uses LK optical flow method to track the mark points and match with the mark points obtained by machine learning, and continuously adjusts the positions of the existing face characteristic points, so that the face characteristic points are continuously close to the real face and finally align with the real face, the required training data is less, and the training speed is fast; using histogram equalization to reduce the influence of illumination in the mark point training and detecting process; machine learning and image processing are combined, correction is carried out on the basis of tracking, and robustness is improved.
Detailed Description
A facial animation capturing and correcting method based on machine learning and image processing comprises the following steps:
s1, marking a plurality of mark points on the face of the person to be sampled, and shooting the face of the person with the mark points by using a plurality of head-mounted cameras;
s2, preprocessing each frame of image collected by each camera, and selecting the images as a training set according to the number of frames at uniform intervals;
s3, manually marking the position of a mark point in a picture on the picture in the training set;
s4, rotating, translating and changing the brightness of the pictures in the training set to amplify the data;
s5, training the picture data in the amplified training set by adopting a training model of a face key point algorithm to obtain position information of mark points in the picture;
s6, respectively inputting the images shot by the camera into a network model to obtain marked mark points, detecting the mark points by using an opencv spot detection method, and tracking the mark points by using an LK optical flow method;
s7, matching the mark points obtained by training with the training model with the mark points detected by opencv, displaying the non-matched mark points in an emphasis way, and correcting the positions of the non-matched mark points according to the actual conditions;
wherein the number of the cameras of the step S1 is at least two; the preprocessing of the step S2 is to perform histogram equalization on the picture taken by the camera; the number of the selected interval frames in the step S2 is 10;
the step of detecting mark points using opencv spot detection in step S6 is as follows:
p1, setting a low threshold and a high threshold for the image to be detected, setting a threshold step, and selecting a series of thresholds from the low threshold to the high threshold according to the threshold step;
p2, binarizing the image by each selected threshold, searching the edge of the image by using a findcontours function, detecting the outline of the image, and calculating the center of the outline of each image;
p3, suppressing the center of the outline of each image, defining a minimum distance, defining the characteristic center in the distance area as a blob, and obtaining a mark point set;
p4, filtering mark points, wherein the filtering comprises color filtering and area filtering;
the invention combines machine learning and image processing, uses opencv spot detection to detect the mark points and uses LK optical flow method to track the mark points and match with the mark points obtained by machine learning, and continuously adjusts the positions of the existing face characteristic points, so that the face characteristic points are continuously close to the real face and finally align with the real face, the required training data is less, and the training speed is fast; using histogram equalization to reduce the influence of illumination in the mark point training and detecting process; machine learning and image processing are combined, correction is carried out on the basis of tracking, and robustness is improved;
although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. A facial animation capturing and correcting method based on machine learning and image processing is characterized by comprising the following steps:
s1, marking a plurality of mark points on the face of a person to be sampled, and shooting the face of the person with the mark points by using a plurality of head-mounted cameras;
s2, preprocessing each frame of image collected by each camera, and selecting the images as a training set according to the number of frames at uniform intervals;
s3, manually marking the position of the mark point in the picture on the picture in the training set;
s4, rotating, translating and changing the brightness of the pictures in the training set to amplify the data;
s5, training the amplified picture data in the training set by adopting a training model of a face key point algorithm to obtain the position information of the mark point in the picture;
s6, respectively inputting the images shot by the camera into a network model to obtain marked mark points, detecting the mark points by using an opencv spot detection method, and tracking the mark points by using an LK optical flow method;
s7, matching the mark points obtained by training with the training model with the mark points detected by the opencv, displaying the non-matched mark points in an emphasized mode, and correcting the positions of the non-matched mark points according to actual conditions.
2. The facial animation capturing rectification method based on machine learning and image processing as claimed in claim 1, wherein the number of the cameras of step S1 is at least two.
3. The method for correcting facial animation capturing based on machine learning and image processing as claimed in claim 1, wherein the preprocessing of step S2 is to apply histogram equalization operation to the picture taken by the camera.
4. The method for correcting facial animation capturing based on machine learning and image processing as claimed in claim 1, wherein the selected interval frame number of step S2 is 10.
5. The method for correcting facial animation capturing based on machine learning and image processing as claimed in claim 1, wherein the step of detecting the mark point by opencv speckle detection method in step S6 is as follows:
p1, setting a low threshold and a high threshold for the image to be detected, setting a threshold step, and selecting a series of thresholds from the low threshold to the high threshold according to the threshold step;
p2, binarizing the image by using each selected threshold, searching the edge of the image by using a findcontours function, detecting the outline of the image, and calculating the center of the outline of each image;
p3, suppressing the center of the outline of each image, defining a minimum distance, defining the characteristic center in the distance area as a blob, and obtaining a mark point set;
p4, filtering the mark points.
6. The facial animation capturing rectification method based on machine learning and image processing as claimed in claim 5, wherein the filtering of step P4 comprises color filtering and area filtering.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010624435.9A CN111523524A (en) | 2020-07-02 | 2020-07-02 | Facial animation capturing and correcting method based on machine learning and image processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010624435.9A CN111523524A (en) | 2020-07-02 | 2020-07-02 | Facial animation capturing and correcting method based on machine learning and image processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111523524A true CN111523524A (en) | 2020-08-11 |
Family
ID=71910053
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010624435.9A Pending CN111523524A (en) | 2020-07-02 | 2020-07-02 | Facial animation capturing and correcting method based on machine learning and image processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111523524A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101271520A (en) * | 2008-04-01 | 2008-09-24 | 北京中星微电子有限公司 | Method and device for confirming characteristic point position in image |
| CN101877056A (en) * | 2009-12-21 | 2010-11-03 | 北京中星微电子有限公司 | Facial expression recognition method and system, and training method and system of expression classifier |
| US20190034706A1 (en) * | 2010-06-07 | 2019-01-31 | Affectiva, Inc. | Facial tracking with classifiers for query evaluation |
| CN110399844A (en) * | 2019-07-29 | 2019-11-01 | 南京图玩智能科技有限公司 | It is a kind of to be identified and method for tracing and system applied to cross-platform face key point |
-
2020
- 2020-07-02 CN CN202010624435.9A patent/CN111523524A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101271520A (en) * | 2008-04-01 | 2008-09-24 | 北京中星微电子有限公司 | Method and device for confirming characteristic point position in image |
| CN101877056A (en) * | 2009-12-21 | 2010-11-03 | 北京中星微电子有限公司 | Facial expression recognition method and system, and training method and system of expression classifier |
| US20190034706A1 (en) * | 2010-06-07 | 2019-01-31 | Affectiva, Inc. | Facial tracking with classifiers for query evaluation |
| CN110399844A (en) * | 2019-07-29 | 2019-11-01 | 南京图玩智能科技有限公司 | It is a kind of to be identified and method for tracing and system applied to cross-platform face key point |
Non-Patent Citations (1)
| Title |
|---|
| 邓小俊: "opencv图像特征检测之斑点检测", 《HTTPS://WWW.CNBLOGS.COM/DENGXIAOJUN/P/5307083.HTML》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107133969B (en) | A kind of mobile platform moving target detecting method based on background back projection | |
| CN105894702B (en) | A kind of intrusion detection warning system and its detection method based on multiple-camera data fusion | |
| US10327045B2 (en) | Image processing method, image processing device and monitoring system | |
| CN104754302B (en) | A kind of target detection tracking method based on rifle ball linked system | |
| CN106469444B (en) | Rapid image fusion method for eliminating splicing gap | |
| CN109685913B (en) | Augmented reality implementation method based on computer vision positioning | |
| US7982774B2 (en) | Image processing apparatus and image processing method | |
| CN106534817B (en) | Curved surface projection automatic geometric correction method based on image recognition | |
| CN111723801B (en) | Method and system for detecting and correcting target in fisheye camera picture | |
| CN105389574A (en) | Method and system for detecting human eye irises in pictures | |
| CN105704398A (en) | Video processing method | |
| US10762659B2 (en) | Real time multi-object tracking apparatus and method using global motion | |
| CN110414558A (en) | Characteristic point matching method based on event camera | |
| CN110555867A (en) | Multi-target object tracking method fusing object capturing and identifying technology | |
| CN111523524A (en) | Facial animation capturing and correcting method based on machine learning and image processing | |
| CN106971381A (en) | A kind of wide angle camera visual field line of demarcation generation method with the overlapping ken | |
| CN112906533B (en) | Safety helmet wearing detection method based on self-adaptive detection area | |
| CN112446851A (en) | Endpoint detection algorithm based on high-speed pulse type image sensor | |
| TWI479857B (en) | PTZ camera automatic tracking method | |
| CN111325073A (en) | Monitoring video abnormal behavior detection method based on motion information clustering | |
| CN110910418B (en) | Target tracking algorithm based on rotation invariance image feature descriptor | |
| CN111210384B (en) | Image splicing method suitable for airport runway scene | |
| CN106447685A (en) | Infrared tracking method | |
| US9959632B2 (en) | Object extraction from video images system and method | |
| CN110852275A (en) | Face detection re-amplification tracking method in dome camera tracking process |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200811 |
|
| RJ01 | Rejection of invention patent application after publication |