CN107369174A - The processing method and computing device of a kind of facial image - Google Patents
The processing method and computing device of a kind of facial image Download PDFInfo
- Publication number
- CN107369174A CN107369174A CN201710616812.2A CN201710616812A CN107369174A CN 107369174 A CN107369174 A CN 107369174A CN 201710616812 A CN201710616812 A CN 201710616812A CN 107369174 A CN107369174 A CN 107369174A
- Authority
- CN
- China
- Prior art keywords
- face
- dimensional
- image frame
- model
- current image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Abstract
The invention discloses the computing device of a kind of processing method of facial image and execution this method, including step：When detecting the face in video, picture frame where face is denoted as initial image frame, the projection matrix of face in initial image frame is calculated by the first calculation；Calculate three-dimensional face model corresponding to initial image frame, three-dimensional expression model, three-dimensional shape of face model；The projection matrix of current image frame is calculated by the first calculation；Calculate the three-dimensional expression model of current image frame；Compare the projection matrix and three-dimensional expression model of the projection matrix and three-dimensional expression model of current image frame with a upper picture frame；If comparative result meets predetermined condition, the three-dimensional shape of face model of current image frame is calculated, keeps three-dimensional shape of face model constant if being unsatisfactory for；Calculate the three-dimensional face model of current image frame；Calculate the texture coordinate of current image frame；Default mask is rendered into current image frame according to texture coordinate.
Description
Technical field
The present invention relates to technical field of image processing, the processing method and computing device of especially a kind of facial image.
Background technology
With mobile communication and the development of Internet technology, user is no longer only satisfied with by mobile device (such as mobile phone, phase
Machine, flat board etc.) taken pictures, shared, in daily social activity, user often wants to carry out some photos interesting processing,
To increase the interest taken pictures.For example, some processing (referred to as " facial image ") are carried out to the image comprising face, such as in people
The special dressing of glasses, cat beard, rabbit ear etc is added on the face.Usually way is, by locating human face's characteristic point phase
The dressing answered is attached to correspondence position, significantly simpler to implement, it is not necessary to considers the factors such as shape of face and the expression of different faces.
Another processing mode is that veneer has on face, for example, real for the face on one section of video or a photo
The suitable masks of Shi Shengcheng.Mask may be considered new face, and mask corresponding to different faces is also different, such as
Mask corresponding to state's word face also can be relatively square, and mask corresponding to round face is then relatively relatively round, and mask corresponding to oval face can compare
Point, etc..Even same person, in one section of video, when he/her makes different expressions, its corresponding mask also should be with
Expression shape change and change, for example it is also what is closed to correspond to the eyes of mask when closing one's eyes, and the face of mask is corresponded to when opening one's mouth
Also should open.Based on these demands, only by the scheme of locating human face's characteristic point can cause the mask of generation without
Method adapts to the shape of face of different faces, and also have can not track human faces expression the defects of.
Therefore, can, it is necessary to the face mask generation method of a kind of shape of face that can take into account different faces, posture and expression
The facial image in video is handled in real time.
The content of the invention
Therefore, the invention provides a kind of processing method of facial image and computing device, to try hard to solve or at least
Alleviate existing at least one problem above.
According to an aspect of the invention, there is provided a kind of processing method of facial image, this method be used for be in real time regarding
Face image synthesis mask in frequency, including step：When detecting the face in video, picture frame where face is denoted as just
Beginning picture frame, the projection matrix of face in initial image frame is calculated by the first calculation；According to the projection square of initial image frame
Battle array calculates three-dimensional face model corresponding to initial image frame, three-dimensional expression model, three-dimensional shape of face model；For in the video
Follow-up each picture frame, the projection matrix of current image frame is calculated by the first calculation；According to the projection square of current image frame
Battle array calculates the three-dimensional expression model of current image frame；By the projection matrix of current image frame and three-dimensional expression model respectively with upper one
The projection matrix of picture frame and three-dimensional expression model are compared；If comparative result meets predetermined condition, present image is calculated
The three-dimensional shape of face model of frame, the three-dimensional shape of face if being unsatisfactory for using the three-dimensional shape of face model of a upper picture frame as current image frame
Model；The three-dimensional face model of current image frame is calculated according to the three-dimensional shape of face model of current image frame and three-dimensional expression model；
The texture coordinate of current image frame is calculated according to the projection matrix of current image frame and three-dimensional face model；And sat according to texture
Default mask is rendered into current image frame by mark.
Alternatively, in the face image processing process according to the present invention, in addition to step：According to the three-dimensional gathered in advance
Human face data establishes face spatial base, and wherein face spatial base includes three-dimensional average face, forms the face of three-dimensional shape of face model
Type base and the expression base for forming three-dimensional expression model.
Alternatively, in the face image processing process according to the present invention, in addition to the step of presetting mask is generated：According to
The default mask of UV expanded views generation of three-dimensional average face.
Alternatively, in the face image processing process according to the present invention, people in picture frame is calculated by the first calculation
The step of projection matrix of face, includes：Extract the human face characteristic point in picture frame；And pass through extracted human face characteristic point and intend
Conjunction obtains the projection matrix of face in picture frame.
Alternatively, in the face image processing process according to the present invention, the figure is calculated according to the projection matrix of picture frame
As the step of three-dimensional face model of frame or three-dimensional expression model or three-dimensional shape of face model includes：Using least square method according to figure
As the projection matrix and human face characteristic point of frame calculate the three-dimensional face model or three-dimensional expression model or three-dimensional shape of face of the picture frame
Model.
Alternatively, in the face image processing process according to the present invention, predetermined condition includes：The projection of current image frame
Rotation parameter of the rotation parameter than rotation parameter in the projection matrix of a upper picture frame closer to three-dimensional average face in matrix；And work as
The coefficient of expression base is more in the three-dimensional expression model of a picture frame in expression base coefficient ratio in the three-dimensional expression model of preceding picture frame
Close to expression base system number in three-dimensional average face.
Alternatively, in the face image processing process according to the present invention, calculated when comparative result meets predetermined condition
The step of three-dimensional shape of face model of current image frame, includes：When comparative result meets predetermined condition, according to current image frame
Projection matrix calculates the three-dimensional shape of face model of current image frame.
Alternatively, in the face image processing process according to the present invention, according to the three-dimensional shape of face model of current image frame
The step of calculating the three-dimensional face model of current image frame with three-dimensional expression model includes：In the three-dimensional shape of face mould of current image frame
On the basis of type and three-dimensional expression model, three-dimensional average face is subtracted, obtains the three-dimensional face model of current image frame.
Alternatively, in the face image processing process according to the present invention, the three-dimensional face model M of t-th of picture framet
For：
Mt=Et+Ft-meanEF
Wherein, EtRepresent the three-dimensional expression model of t-th of picture frame, FtThe three-dimensional shape of face model of t-th of picture frame is represented,
MeanEF represents three-dimensional average face.
Alternatively, in the face image processing process according to the present invention, according to the projection matrix of current image frame and three
The step of dimension faceform calculates the texture coordinate of current image frame includes：By the projection matrix and present image of current image frame
The three-dimensional face model of frame is multiplied, and obtains the texture coordinate of current image frame.
Alternatively, in the face image processing process according to the present invention, in the projection matrix according to current image frame and
Before three-dimensional face model calculates the step of texture coordinate of current image frame, in addition to step：Utilize the throwing of a upper picture frame
Shadow matrix and three-dimensional face model are smoothed to the projection matrix and three-dimensional face model of current image frame respectively, will be flat
The projection matrix and three-dimensional face model of projection matrix and three-dimensional face model as current image frame after sliding processing.
Alternatively, in the face image processing process according to the present invention, the projection square after t-th of picture frame smoothing processing
Battle array MVPt' and three-dimensional face model Mt' be respectively：
Wherein, MVPtRepresent the projection matrix before t-th of picture frame smoothing processing, MVPt-1Represent the t-1 picture frame
Projection matrix, MtRepresent the three-dimensional face model before t-th of picture frame smoothing processing, Mt-1Represent the three-dimensional of the t-1 picture frame
Faceform.
According to another aspect of the present invention, there is provided a kind of computing device, including：One or more processors；And storage
Device；One or more programs, wherein one or more program storages in memory and be configured as by one or more handle
Device performs, and one or more programs include being used for the instruction for performing the either method in method as described above.
In accordance with a further aspect of the present invention, there is provided a kind of computer-readable storage medium for storing one or more programs
Matter, one or more programs include instruction, and the instruction is when computing device so that computing device is square as described above
Either method in method.
This programme realizes the real-time veneer tool of video based on three-dimensional facial reconstruction technology, solves only according to face characteristic
Point location causes mask can not adapt to different shapes of face, different postures, and the defects of expression can not be tracked.
Brief description of the drawings
In order to realize above-mentioned and related purpose, some illustrative sides are described herein in conjunction with following description and accompanying drawing
Face, these aspects indicate the various modes that can put into practice principles disclosed herein, and all aspects and its equivalent aspect
It is intended to fall under in the range of theme claimed.Read following detailed description in conjunction with the accompanying drawings, the disclosure it is above-mentioned
And other purposes, feature and advantage will be apparent.Throughout the disclosure, identical reference generally refers to identical
Part or element.
Fig. 1 shows the organigram of computing device 100 according to an embodiment of the invention；
Fig. 2 shows the flow chart of the processing method 200 of facial image according to an embodiment of the invention；And
Fig. 3 shows a schematic diagram of UV expanded views.
Embodiment
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although the disclosure is shown in accompanying drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
Limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
Completely it is communicated to those skilled in the art.
Fig. 1 is the block diagram of Example Computing Device 100.In basic configuration 102, computing device 100, which typically comprises, is
System memory 106 and one or more processor 104.Memory bus 108 can be used in processor 104 and system storage
Communication between device 106.
Depending on desired configuration, processor 104 can be any kind of processing, include but is not limited to：Microprocessor
(μ P), microcontroller (μ C), digital information processor (DSP) or any combination of them.Processor 104 can be included such as
The cache of one or more rank of on-chip cache 110 and second level cache 112 etc, processor core
114 and register 116.The processor core 114 of example can include arithmetic and logical unit (ALU), floating-point unit (FPU),
Digital signal processing core (DSP core) or any combination of them.The Memory Controller 118 of example can be with processor
104 are used together, or in some implementations, Memory Controller 118 can be an interior section of processor 104.
Depending on desired configuration, system storage 106 can be any type of memory, include but is not limited to：Easily
The property lost memory (RAM), nonvolatile memory (ROM, flash memory etc.) or any combination of them.System stores
Device 106 can include operating system 120, one or more apply 122 and routine data 124.In some embodiments,
It may be arranged to be operated using routine data 124 on an operating system using 122.Routine data 124 includes instruction, in root
In computing device 100 according to the present invention, routine data 124, which includes, to be used to perform the instruction based on face image processing process.
Computing device 100 can also include contributing to from various interface equipments (for example, output equipment 142, Peripheral Interface
144 and communication equipment 146) to basic configuration 102 via the communication of bus/interface controller 130 interface bus 140.Example
Output equipment 142 include graphics processing unit 148 and audio treatment unit 150.They can be configured as contributing to via
One or more A/V port 152 is communicated with the various external equipments of such as display or loudspeaker etc.Outside example
If interface 144 can include serial interface controller 154 and parallel interface controller 156, they can be configured as contributing to
Via one or more I/O port 158 and such as input equipment (for example, keyboard, mouse, pen, voice-input device, touch
Input equipment) or the external equipment of other peripheral hardwares (such as printer, scanner etc.) etc communicated.The communication of example is set
Standby 146 can include network controller 160, and it can be arranged to be easy to via one or more COM1 164 and one
The communication that other individual or multiple computing devices 162 pass through network communication link.
Network communication link can be an example of communication media.Communication media can be generally presented as in such as carrier wave
Or computer-readable instruction in the modulated data signal of other transmission mechanisms etc, data structure, program module, and can
With including any information delivery media." modulated data signal " can such signal, one in its data set or more
It is individual or it change can the mode of coding information in the signal carry out.As nonrestrictive example, communication media can be with
Include the wire medium of such as cable network or private line network etc, and it is such as sound, radio frequency (RF), microwave, infrared
(IR) the various wireless mediums or including other wireless mediums.Term computer-readable medium used herein can include depositing
Both storage media and communication media, such as store the computer-readable recording medium of one or more programs.
Computing device 100 can be implemented as a part for portable (or mobile) electronic equipment of small size, and these electronics are set
It is standby can be such as cell phone, personal digital assistant (PDA), it is personal media player device, wireless network browsing apparatus, individual
People's helmet, application specific equipment or the mixing apparatus that any of the above function can be included.Computing device 100 can be with
It is embodied as including desktop computer and the personal computer of notebook computer configuration.In certain embodiments, computing device 100
It is configured as performing face image processing process 200.
As it was noted above, in daily life, it is assumed that such scene, one section by the self-timer of computing device 100 of user
Small video, and various joyful masks are generated according to the face of user in real time in this section of small video, user wishes will so
Good friend is shared with rich in interesting small video.Below in conjunction with Fig. 2, it is described in detail and this is realized according to one embodiment of the invention
The flow chart of the method face image processing process 200 of sample effect.
Fig. 2 shows the flow chart of the processing method 200 of facial image according to an embodiment of the invention.
As shown in Fig. 2 this method 200 starts from step S210, when detecting the face in video, by image where face
Frame is denoted as initial image frame, and the projection matrix of face in the initial image frame is calculated by the first calculation.Realized according to one kind
Mode, the people in specific keys open detection video in corresponding application programs or triggering camera applications can be opened by user
Face, this programme to not being restricted in this respect.
According to an embodiment of the invention, the step of calculating the projection matrix of face in picture frame by the first calculation is wrapped
Include：
1. extract the human face characteristic point in picture frame.According to the embodiment of the present invention, facial image is divided into two
Region, it is respectively：Background area beyond human face region and human face region including face.Human face characteristic point includes but is not limited to
The key point of the features such as eyebrow, nose, canthus, face, face outline is characterized in face.It should be noted that extraction face is special
Sign point is already belonging to the algorithm of this area comparative maturity, any method for detecting human face/face spy for being currently known or developing in the future
Sign point extracting method can be used in combination with embodiments of the invention, and the scope of the present invention is not limited in this respect.
2. pass through extracted human face characteristic point to be fitted to obtain the projection matrix of face in the picture frame.According to the present invention's
Implementation, human face three-dimensional model projection matrix corresponding with its are tried to achieve by three-dimensional deformation model (3DMM).Wherein, 3DMM
Model is to be delivered by Blanz and Vetter for 1999《A Morphable Model For The Synthesis Of 3D
Faces》The method that paper is mentioned, the basic thought of this method are：Regard face space as a linear space, established with prior
The face that the projection of the linear combination of good three-dimensional face data is approached on two-dimension picture.
In view of the present invention implementation process in, it is necessary to face shape of face, human face expression carry out real-time tracking, therefore,
In embodiments of the invention, when building face spatial base according to the three-dimensional face data gathered in advance, by three-dimensional face model
Regard what is be made up of three-dimensional average face, three-dimensional shape of face model and three-dimensional expression model as, 3DMM fundamental formular is represented such as formula
(1)：
Wherein, M represents finally to be fitted obtained three-dimensional face model, and meanEF represents three-dimensional average face,Represent structure
Into the shape of face base of three-dimensional shape of face model,The expression base of the three-dimensional expression model of composition is represented, n and m represent shape of face base respectively
With the number of expression base, aiAnd bjCoefficient corresponding to shape of face base and expression base is represented respectively.
And the initial parameter of projection matrix is estimated according to the characteristic point of face spatial base, its initial parameter includes phase
The position of machine, the anglec of rotation of the plane of delineation, each component of direct light and ambient light, picture contrast etc..For given
Specific face (extracting obtained human face characteristic point), according to changing repeatedly for face spatial base and projection matrix initial parameter
In generation, the three-dimensional face model of the facial image is fitted, in other words, there is same vertices number and topological structure according to existing
Three-dimensional face model data corresponding three-dimensional feature point, after the projecting characteristic points by minimizing threedimensional model linear combination
With the distance of two dimensional character point, solve the parameter that is combined, the three-dimensional face model that is further fitted by the parameter and
Projection matrix.As described in following formula (2)：
Err=MVP*M-P (2)
In formula, MVP represents projection matrix, and P represents human face characteristic point, convolution (1), can tried to achieve by minimizing err
The projection matrix of initial image frame.Alternatively, it is contemplated that the real-time of Video processing, err is minimized using least square method.
Then in step S220, the three-dimensional people according to corresponding to the projection matrix of initial image frame calculates initial image frame
Face model, three-dimensional expression model and three-dimensional shape of face model.According to an embodiment of the invention, by three-dimensional expression model E and three-dimensional face
Pattern type F is expressed as：
Therefore, calculate three-dimensional expression model E and three-dimensional shape of face model F namely calculates corresponding coefficient aiAnd bj, such as institute above
State, when calculating projection matrix using least square method, also just obtained corresponding coefficient aiAnd bj, you can obtain initial graph
As the three-dimensional face model M of frame, equally, the three-dimensional expression model E and three of initial image frame can be calculated by formula (3), formula (4)
Tie up shape of face model F.
In subsequent step, to follow-up each picture frame for including the face in the video, handled accordingly, with
Generate corresponding mask.In step S230, the projection matrix of current image frame is calculated by the first calculation.According to the present invention
Implementation, subsequent image frames are performed with the calculation procedure same with initial image frame (as described by above-mentioned step S210
Like that), the projection matrix of calculated for subsequent picture frame.
Then in step S240, the three-dimensional expression mould of current image frame is calculated according to the projection matrix of current image frame
Type.Equally, the method that the three-dimensional expression model of the picture frame is calculated according to the projection matrix of picture frame is above specifically being explained
State, be also not repeated herein.
According to an embodiment of the invention, in actual applications, face keeps constant in one section of video, then its shape of face is general
It is fixed, but expression can constantly change frame by frame with the time, therefore, it is necessary to calculate the three-dimensional expression model E of each picture frame,
To track the change of human face expression in video, but for three-dimensional shape of face model F, then selectively calculated as needed.This
Sample, which is done, effectively to reduce amount of calculation on the basis of ensureing to calculate accurately.
Then in step s 250, by the projection matrix of current image frame and three-dimensional expression model respectively with a upper picture frame
Projection matrix and three-dimensional expression model be compared.If current image frame is t two field pictures, then compare the throwing of current image frame
Shadow matrix MVPtWith the projection matrix MVP of a upper picture framet-1Rotation parameter, and, compare the three-dimensional expression of current image frame
Model EtWith the three-dimensional expression model E of a upper picture framet-1In expression base and expression base system number.
Then in step S260, if comparative result meets predetermined condition, the three-dimensional shape of face mould of current image frame is calculated
Type, the three-dimensional shape of face model if being unsatisfactory for using the three-dimensional shape of face model of a upper picture frame as current image frame.
According to an embodiment of the invention, predetermined condition includes：1. the projection matrix MVP of current image frametMiddle rotation parameter
Than the projection matrix MVP of a upper picture framet-1Middle rotation parameter closer to three-dimensional average face rotation parameter, in other words, MVPt
Rotation parameter compare MVPt-1Rotation parameter closer to the positive face of standard, i.e., the posture of the face of current image frame is than a upper figure
As the human face posture " corrigendum " of frame；2. the three-dimensional expression model E of current image frametThe three of a picture frame in middle expression base coefficient ratio
Dimension table feelings model Et-1The coefficient of middle expression base is closer to expression base system number in three-dimensional average face, i.e. EtInPart is more
Close to 0, becauseBigger, expression is more exaggerated, so when this part is close to 0, judges expression " more neutral ".
To sum up, when judging human face posture " corrigendum " and human face expression " more neutral " of the current image frame than a upper picture frame
When, it is believed that meet predetermined condition, just recalculate the three-dimensional shape of face model F of current image framet, alternatively, according to present image
The projection matrix (being obtained by step S230) of frame calculates the three-dimensional shape of face model F of current image framet, computational methods are above
It is specifically described, is not repeated herein.Conversely, not calculating the three-dimensional shape of face model of current image frame then, a upper picture frame is kept
Three-dimensional shape of face model, i.e. Ft=Ft-1。
By the calculation so compared frame by frame, it can constantly optimize the three-dimensional shape of face model of face in the video,
It is set to be consistent as much as possible with user's shape of face.
Then in step S270, according to the three-dimensional shape of face model F of current image frametWith three-dimensional expression model EtCalculate again
The three-dimensional face model M of current image framet., can be directly the three of current image frame according to an embodiment of the invention
Tie up shape of face model FtWith three-dimensional expression model EtOn the basis of, three-dimensional average face meanEF is subtracted, obtains the three-dimensional of current image frame
Faceform Mt, as shown in formula (5)：
Mt=Et+Ft-meanEF (5)
Then in step S280, current image frame is calculated according to the projection matrix of current image frame and three-dimensional face model
Texture coordinate.According to an embodiment of the invention, by the projection matrix of current image frame and the three-dimensional face mould of current image frame
Type is multiplied, and obtains the texture coordinate of current image frame.
According to still another embodiment of the invention, after step S270, before step S280, in addition to projection matrix and
The step of three-dimensional face model is smoothed.Specifically, the projection matrix and three-dimensional face model of a upper picture frame are utilized
The projection matrix and three-dimensional face model of current image frame are smoothed respectively, by the projection matrix after smoothing processing and
Projection matrix and three-dimensional face model of the three-dimensional face model as current image frame, then calculate the texture seat of current image frame
Mark.
The specific calculation of smoothing processing can refer to formula (6), formula (7)：
Projection matrix MVP after t-th of picture frame smoothing processingt' and three-dimensional face model Mt' be：
In formula, MVPtRepresent the projection matrix before t-th of picture frame smoothing processing, MVPt-1Represent the t-1 picture frame
Projection matrix, MtRepresent the three-dimensional face model before t-th of picture frame smoothing processing, Mt-1Represent the three-dimensional of the t-1 picture frame
Faceform.
Then in step S290, default mask is rendered into current image frame according to texture coordinate.When connecting in video
After continuous multiple images frame completes above-mentioned steps S230 to step S290, just default mask is rendered on video by 3D.
According to an embodiment of the invention, default mask is generated according to the UV expanded views of three-dimensional average face.UV expanded views
A kind of normal method rendered as 3D, no longer it is described in greater detail in the present embodiment, such as Fig. 3, shows UV expanded views
An example.
Those skilled in the art should will recognize that, on the basis of this programme, default mask is further located
Reason, all it is achievable to increase the interest of veneer tool for example, putting on sunglasses for default mask, drawing the small decoration such as blush
, embodiments of the invention to not being restricted in this respect.
According to the other embodiment of the present invention, if there is Shot change in one section of video, can continue after detecting switching
Picture material in whether there is face, when detecting face again, a face may have been changed, therefore will detect again
The initial image frame new as one to this picture frame of face, repetition methods 200 carry out real-time veneer tool.The present invention is herein
Aspect is not restricted.
This programme realizes the real-time veneer tool of video based on three-dimensional facial reconstruction technology, solves only according to face characteristic
Point location causes mask can not adapt to different shapes of face, different postures, and the defects of expression can not be tracked.Meanwhile this programme leads to
Cross and solve the modes such as equation, the three-dimensional shape of face model for selectively calculating picture frame using least square method, effectively reduce and calculate
Amount, to reach the effect handled in real time, will not bring interim card to feel to user, lift Consumer's Experience.
Various technologies described herein can combine hardware or software, or combinations thereof is realized together.So as to the present invention
Method and apparatus, or some aspects of the process and apparatus of the present invention or part can take embedded tangible media, such as soft
The form of program code (instructing) in disk, CD-ROM, hard disk drive or other any machine readable storage mediums,
Wherein when program is loaded into the machine of such as computer etc, and is performed by the machine, the machine becomes to put into practice this hair
Bright equipment.
In the case where program code performs on programmable computers, computing device generally comprises processor, processor
Readable storage medium (including volatibility and nonvolatile memory and/or memory element), at least one input unit, and extremely
A few output device.Wherein, memory is arranged to store program codes；Processor is arranged to according to the memory
Instruction in the described program code of middle storage, the method for performing the present invention.
By way of example and not limitation, computer-readable medium includes computer-readable storage medium and communication media.Calculate
Machine computer-readable recording medium includes computer-readable storage medium and communication media.Computer-readable storage medium storage such as computer-readable instruction,
The information such as data structure, program module or other data.Communication media is typically modulated with carrier wave or other transmission mechanisms etc.
Data-signal processed passes to embody computer-readable instruction, data structure, program module or other data including any information
Pass medium.Any combination above is also included within the scope of computer-readable medium.
It should be appreciated that in order to simplify the disclosure and help to understand one or more of each inventive aspect, it is right above
The present invention exemplary embodiment description in, each feature of the invention be grouped together into sometimes single embodiment, figure or
In person's descriptions thereof.However, the method for the disclosure should be construed to reflect following intention：I.e. claimed hair
The bright feature more features required than being expressly recited in each claim.More precisely, as the following claims
As book reflects, inventive aspect is all features less than single embodiment disclosed above.Therefore, it then follows specific real
Thus the claims for applying mode are expressly incorporated in the embodiment, wherein each claim is used as this hair in itself
Bright separate embodiments.
Those skilled in the art should be understood the module or unit or group of the equipment in example disclosed herein
Part can be arranged in equipment as depicted in this embodiment, or alternatively can be positioned at and the equipment in the example
In different one or more equipment.Module in aforementioned exemplary can be combined as a module or be segmented into addition multiple
Submodule.
Those skilled in the art, which are appreciated that, to be carried out adaptively to the module in the equipment in embodiment
Change and they are arranged in one or more equipment different from the embodiment.Can be the module or list in embodiment
Member or component be combined into a module or unit or component, and can be divided into addition multiple submodule or subelement or
Sub-component.In addition at least some in such feature and/or process or unit exclude each other, it can use any
Combination is disclosed to all features disclosed in this specification (including adjoint claim, summary and accompanying drawing) and so to appoint
Where all processes or unit of method or equipment are combined.Unless expressly stated otherwise, this specification (including adjoint power
Profit requires, summary and accompanying drawing) disclosed in each feature can be by providing the alternative features of identical, equivalent or similar purpose come generation
Replace.
The present invention discloses in the lump：
A9, the method as described in A8, wherein, the three-dimensional face model M of t-th of picture frametFor：
Mt=Et+Ft-meanEF
Wherein, EtRepresent the three-dimensional expression model of t-th of picture frame, FtThe three-dimensional shape of face model of t-th of picture frame is represented,
MeanEF represents three-dimensional average face.
A10, the method as any one of A1-9, wherein, according to the projection matrix and three-dimensional face of current image frame
The step of texture coordinate of model calculating current image frame, includes：By the three of the projection matrix of current image frame and current image frame
Tie up faceform to be multiplied, obtain the texture coordinate of current image frame.
A11, the method as any one of A1-10, wherein, in the projection matrix according to current image frame and three-dimensional people
Before face model calculates the step of texture coordinate of current image frame, in addition to step：Utilize the projection matrix of a upper picture frame
The projection matrix and three-dimensional face model of current image frame are smoothed respectively with three-dimensional face model, by smoothing processing
Projection matrix and three-dimensional face model of the projection matrix and three-dimensional face model afterwards as current image frame.
A12, the method as described in A11, wherein, the projection matrix MVP after t-th of picture frame smoothing processingt' be：
Wherein, MVPtRepresent the projection matrix before t-th of picture frame smoothing processing, MVPt-1Represent the t-1 picture frame
Projection matrix.
A13, the method as described in A11, wherein, the three-dimensional face model M after t-th of picture frame smoothing processingt' be：
Wherein, MtRepresent the three-dimensional face model before t-th of picture frame smoothing processing, Mt-1Represent the t-1 picture frame
Three-dimensional face model.
In addition, it will be appreciated by those of skill in the art that although some embodiments described herein include other embodiments
In included some features rather than further feature, but the combination of the feature of different embodiments means in of the invention
Within the scope of and form different embodiments.For example, in the following claims, embodiment claimed is appointed
One of meaning mode can use in any combination.
In addition, be described as herein can be by the processor of computer system or by performing for some in the embodiment
The method or the combination of method element that other devices of the function are implemented.Therefore, have and be used to implement methods described or method
The processor of the necessary instruction of element forms the device for implementing this method or method element.In addition, device embodiment
Element described in this is the example of following device：The device is used to implement as in order to performed by implementing the element of the purpose of the invention
Function.
As used in this, unless specifically stated so, come using ordinal number " first ", " second ", " the 3rd " etc.
Description plain objects are merely representative of the different instances for being related to similar object, and are not intended to imply that the object being so described must
Must have the time it is upper, spatially, in terms of sequence or given order in any other manner.
Although describing the present invention according to the embodiment of limited quantity, above description, the art are benefited from
It is interior it is clear for the skilled person that in the scope of the present invention thus described, it can be envisaged that other embodiments.Additionally, it should be noted that
The language that is used in this specification primarily to readable and teaching purpose and select, rather than in order to explain or limit
Determine subject of the present invention and select.Therefore, in the case of without departing from the scope and spirit of the appended claims, for this
Many modifications and changes will be apparent from for the those of ordinary skill of technical field.For the scope of the present invention, to this
The done disclosure of invention is illustrative and not restrictive, and it is intended that the scope of the present invention be defined by the claims appended hereto.
Claims (10)
1. a kind of processing method of facial image, methods described is used to be the Face image synthesis mask in video in real time, described
Method includes step：
When detecting the face in video, picture frame where face is denoted as initial image frame, calculated by the first calculation
The projection matrix of face in the initial image frame；
According to corresponding to the projection matrix of initial image frame calculates initial image frame three-dimensional face model, three-dimensional expression model,
Three-dimensional shape of face model；
For follow-up each picture frame in the video,
The projection matrix of current image frame is calculated by the first calculation；
The three-dimensional expression model of current image frame is calculated according to the projection matrix of current image frame；
By the projection matrix of current image frame and three-dimensional expression the model projection matrix with a upper picture frame and three-dimensional expression respectively
Model is compared；
If comparative result meets predetermined condition, the three-dimensional shape of face model of current image frame is calculated, by a upper figure if being unsatisfactory for
As three-dimensional shape of face model of the three-dimensional shape of face model as current image frame of frame；
The three-dimensional face model of current image frame is calculated according to the three-dimensional shape of face model of current image frame and three-dimensional expression model；
The texture coordinate of current image frame is calculated according to the projection matrix of current image frame and three-dimensional face model；And
Default mask is rendered into current image frame according to the texture coordinate.
2. the method for claim 1, wherein also include step：
Three-dimensional face data according to gathering in advance establish face spatial base, wherein the face spatial base includes three-dimensional put down
Equal face, the shape of face base for forming three-dimensional shape of face model and the expression base for forming three-dimensional expression model.
3. method as claimed in claim 2, wherein, in addition to generate the step of presetting mask：
Default mask is generated according to the UV expanded views of the three-dimensional average face.
4. such as the method any one of claim 1-3, wherein, the throwing of face in picture frame is calculated by the first calculation
The step of shadow matrix, includes：
Extract the human face characteristic point in picture frame；And
Pass through extracted human face characteristic point to be fitted to obtain the projection matrix of face in described image frame.
5. method as claimed in claim 4, wherein, the three-dimensional face mould of the picture frame is calculated according to the projection matrix of picture frame
The step of type or three-dimensional expression model or three-dimensional shape of face model, includes：
The three-dimensional face model of the picture frame is calculated according to the projection matrix and human face characteristic point of picture frame using least square method
Or three-dimensional expression model or three-dimensional shape of face model.
6. such as the method any one of claim 1-5, wherein, the predetermined condition includes：
Rotation parameter is more closer than rotation parameter in the projection matrix of a upper picture frame three-dimensional in the projection matrix of current image frame
The rotation parameter of average face；And
Expression base in the three-dimensional expression model of a picture frame in expression base coefficient ratio in the three-dimensional expression model of current image frame
Coefficient is closer to expression base system number in three-dimensional average face.
7. the method as described in claim 5 or 6, wherein, current image frame is calculated when comparative result meets predetermined condition
The step of three-dimensional shape of face model, includes：
When comparative result meets predetermined condition, the three-dimensional shape of face of current image frame is calculated according to the projection matrix of current image frame
Model.
8. such as the method any one of claim 1-7, wherein, according to the three-dimensional shape of face model and three-dimensional of current image frame
The step of three-dimensional face model of expression model calculating current image frame, includes：
On the basis of the three-dimensional shape of face model and three-dimensional expression model of current image frame, the three-dimensional average face is subtracted, is obtained
The three-dimensional face model of current image frame.
9. a kind of computing device, including：
One or more processors；With
Memory；
One or more programs, wherein one or more of program storages are in the memory and are configured as by described one
Individual or multiple computing devices, one or more of programs include being used to perform such as appointing in claim 1-8 methods describeds
The instruction of one method.
10. a kind of computer-readable recording medium for storing one or more programs, one or more of programs include instruction,
The instruction is when computing device so that any in method of the computing device as described in claim 1-8
Method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710616812.2A CN107369174B (en) | 2017-07-26 | 2017-07-26 | Face image processing method and computing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710616812.2A CN107369174B (en) | 2017-07-26 | 2017-07-26 | Face image processing method and computing device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107369174A true CN107369174A (en) | 2017-11-21 |
| CN107369174B CN107369174B (en) | 2020-01-17 |
Family
ID=60307047
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710616812.2A Active CN107369174B (en) | 2017-07-26 | 2017-07-26 | Face image processing method and computing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107369174B (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108564659A (en) * | 2018-02-12 | 2018-09-21 | 北京奇虎科技有限公司 | The expression control method and device of face-image, computing device |
| CN108898068A (en) * | 2018-06-06 | 2018-11-27 | 腾讯科技（深圳）有限公司 | A kind for the treatment of method and apparatus and computer readable storage medium of facial image |
| CN109308725A (en) * | 2018-08-29 | 2019-02-05 | 华南理工大学 | A kind of system that expression interest figure in mobile terminal generates |
| CN109886244A (en) * | 2019-03-01 | 2019-06-14 | 北京视甄智能科技有限公司 | A kind of recognition of face biopsy method and device |
| CN110032959A (en) * | 2019-03-29 | 2019-07-19 | 北京迈格威科技有限公司 | A kind of face shape of face judgment method and device |
| CN110246224A (en) * | 2018-03-08 | 2019-09-17 | 北京京东尚科信息技术有限公司 | The surface denoising method and system of grid model |
| CN110580733A (en) * | 2018-06-08 | 2019-12-17 | 北京搜狗科技发展有限公司 | Data processing method and device and data processing device |
| CN111161395A (en) * | 2019-11-19 | 2020-05-15 | 深圳市三维人工智能科技有限公司 | Method and device for tracking facial expression and electronic equipment |
| CN111368593A (en) * | 2018-12-25 | 2020-07-03 | 北京右划网络科技有限公司 | Mosaic processing method and device, electronic equipment and storage medium |
| CN111768477A (en) * | 2020-07-06 | 2020-10-13 | 网易（杭州）网络有限公司 | Three-dimensional facial expression base establishment method and device, storage medium and electronic equipment |
| CN112347870A (en) * | 2020-10-23 | 2021-02-09 | 歌尔光学科技有限公司 | Image processing method, device and equipment of head-mounted equipment and storage medium |
| CN113887408A (en) * | 2021-09-30 | 2022-01-04 | 平安银行股份有限公司 | Method, device and equipment for detecting activated face video and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011162352A1 (en) * | 2010-06-23 | 2011-12-29 | 株式会社 山武 | Three-dimensional data generating apparatus, three-dimensional data generating method, and three-dimensional data generating program |
| CN103035022A (en) * | 2012-12-07 | 2013-04-10 | 大连大学 | Facial expression synthetic method based on feature points |
| CN106127196A (en) * | 2016-09-14 | 2016-11-16 | 河北工业大学 | The classification of human face expression based on dynamic texture feature and recognition methods |
| CN106204430A (en) * | 2016-07-25 | 2016-12-07 | 浙江工业大学 | Characteristic point interpolation based on face naive model and image distortion method |
| CN106874825A (en) * | 2015-12-10 | 2017-06-20 | 展讯通信(天津)有限公司 | The training method of Face datection, detection method and device |
| CN106910247A (en) * | 2017-03-20 | 2017-06-30 | 厦门幻世网络科技有限公司 | Method and apparatus for generating three-dimensional head portrait model |
-
2017
- 2017-07-26 CN CN201710616812.2A patent/CN107369174B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011162352A1 (en) * | 2010-06-23 | 2011-12-29 | 株式会社 山武 | Three-dimensional data generating apparatus, three-dimensional data generating method, and three-dimensional data generating program |
| CN103035022A (en) * | 2012-12-07 | 2013-04-10 | 大连大学 | Facial expression synthetic method based on feature points |
| CN106874825A (en) * | 2015-12-10 | 2017-06-20 | 展讯通信(天津)有限公司 | The training method of Face datection, detection method and device |
| CN106204430A (en) * | 2016-07-25 | 2016-12-07 | 浙江工业大学 | Characteristic point interpolation based on face naive model and image distortion method |
| CN106127196A (en) * | 2016-09-14 | 2016-11-16 | 河北工业大学 | The classification of human face expression based on dynamic texture feature and recognition methods |
| CN106910247A (en) * | 2017-03-20 | 2017-06-30 | 厦门幻世网络科技有限公司 | Method and apparatus for generating three-dimensional head portrait model |
Non-Patent Citations (2)
| Title |
|---|
| BLANZ V ET AL.: "A Morphable Model For The Synthesis Of 3D Faces", 《SIGGRAPH》 * |
| CHEN CAO ET AL.: "FaceWarehouse: A 3D Facial Expression Database for Visual Computing", 《IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS》 * |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108564659A (en) * | 2018-02-12 | 2018-09-21 | 北京奇虎科技有限公司 | The expression control method and device of face-image, computing device |
| CN110246224A (en) * | 2018-03-08 | 2019-09-17 | 北京京东尚科信息技术有限公司 | The surface denoising method and system of grid model |
| CN108898068A (en) * | 2018-06-06 | 2018-11-27 | 腾讯科技（深圳）有限公司 | A kind for the treatment of method and apparatus and computer readable storage medium of facial image |
| CN110580733A (en) * | 2018-06-08 | 2019-12-17 | 北京搜狗科技发展有限公司 | Data processing method and device and data processing device |
| CN109308725B (en) * | 2018-08-29 | 2020-09-22 | 华南理工大学 | System for generating mobile terminal table sentiment picture |
| CN109308725A (en) * | 2018-08-29 | 2019-02-05 | 华南理工大学 | A kind of system that expression interest figure in mobile terminal generates |
| CN111368593B (en) * | 2018-12-25 | 2023-11-28 | 北京右划网络科技有限公司 | Mosaic processing method and device, electronic equipment and storage medium |
| CN111368593A (en) * | 2018-12-25 | 2020-07-03 | 北京右划网络科技有限公司 | Mosaic processing method and device, electronic equipment and storage medium |
| CN109886244A (en) * | 2019-03-01 | 2019-06-14 | 北京视甄智能科技有限公司 | A kind of recognition of face biopsy method and device |
| CN110032959A (en) * | 2019-03-29 | 2019-07-19 | 北京迈格威科技有限公司 | A kind of face shape of face judgment method and device |
| CN111161395A (en) * | 2019-11-19 | 2020-05-15 | 深圳市三维人工智能科技有限公司 | Method and device for tracking facial expression and electronic equipment |
| CN111161395B (en) * | 2019-11-19 | 2023-12-08 | 深圳市三维人工智能科技有限公司 | Facial expression tracking method and device and electronic equipment |
| CN111768477A (en) * | 2020-07-06 | 2020-10-13 | 网易（杭州）网络有限公司 | Three-dimensional facial expression base establishment method and device, storage medium and electronic equipment |
| CN112347870A (en) * | 2020-10-23 | 2021-02-09 | 歌尔光学科技有限公司 | Image processing method, device and equipment of head-mounted equipment and storage medium |
| CN112347870B (en) * | 2020-10-23 | 2023-03-24 | 歌尔科技有限公司 | Image processing method, device and equipment of head-mounted equipment and storage medium |
| CN113887408A (en) * | 2021-09-30 | 2022-01-04 | 平安银行股份有限公司 | Method, device and equipment for detecting activated face video and storage medium |
| CN113887408B (en) * | 2021-09-30 | 2024-04-23 | 平安银行股份有限公司 | Method, device, equipment and storage medium for detecting activated face video |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107369174B (en) | 2020-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107369174A (en) | The processing method and computing device of a kind of facial image | |
| US11055906B2 (en) | Method, device and computing device of face image fusion | |
| CN107392984A (en) | A kind of method and computing device based on Face image synthesis animation | |
| KR102616010B1 (en) | System and method for photorealistic real-time human animation | |
| CN108062526A (en) | A kind of estimation method of human posture and mobile terminal | |
| CN110111418A (en) | Create the method, apparatus and electronic equipment of facial model | |
| WO2019201042A1 (en) | Image object recognition method and device, storage medium, and electronic device | |
| CN108197602A (en) | A kind of convolutional neural networks generation method and expression recognition method | |
| US11727617B2 (en) | Single image-based real-time body animation | |
| KR101944112B1 (en) | Method and apparatus for creating user-created sticker, system for sharing user-created sticker | |
| JP7268071B2 (en) | Virtual avatar generation method and generation device | |
| CN110969682B (en) | Virtual image switching method and device, electronic equipment and storage medium | |
| CN107808147A (en) | A kind of face Confidence method based on the tracking of real-time face point | |
| CN110020600A (en) | Generate the method for training the data set of face alignment model | |
| CN107886516A (en) | The method and computing device that hair moves towards in a kind of calculating portrait | |
| CN110620884B (en) | Expression-driven-based virtual video synthesis method and device and storage medium | |
| CN110580733A (en) | Data processing method and device and data processing device | |
| CN112712578A (en) | Virtual character model creating method and device, electronic equipment and storage medium | |
| CN115601484B (en) | Virtual character face driving method and device, terminal equipment and readable storage medium | |
| CN107341841A (en) | The generation method and computing device of a kind of gradual-change animation | |
| CN110580677A (en) | Data processing method and device and data processing device | |
| CN107644455B (en) | Face image synthesis method and device | |
| CN107657657A (en) | A kind of three-dimensional human modeling method, device, system and storage medium | |
| CN109117736B (en) | Method and computing device for judging visibility of face points | |
| CN117173734A (en) | Palm contour extraction and control instruction generation method and device and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20211208 Address after: 361100 568, No. 942, tonglong Second Road, torch high tech Zone (Xiang'an) Industrial Zone, Xiang'an District, Xiamen City, Fujian Province Patentee after: Xiamen Meitu Yifu Technology Co.,Ltd. Address before: Room 11, Chuangye building, Chuangye Park, torch hi tech Zone, Huli District, Xiamen City, Fujian Province Patentee before: XIAMEN HOME MEITU TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |