CN110298329B - Expression degree prediction model obtaining method and device, storage medium and terminal - Google Patents
Expression degree prediction model obtaining method and device, storage medium and terminal Download PDFInfo
- Publication number
- CN110298329B CN110298329B CN201910594664.8A CN201910594664A CN110298329B CN 110298329 B CN110298329 B CN 110298329B CN 201910594664 A CN201910594664 A CN 201910594664A CN 110298329 B CN110298329 B CN 110298329B
- Authority
- CN
- China
- Prior art keywords
- sample
- expression degree
- expression
- coefficient
- prediction model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
Abstract
The disclosure provides an expression degree prediction model obtaining method and device, a storage medium and a terminal. The method comprises the following steps: obtaining a sample set, wherein the face sample set comprises: the method comprises the steps of obtaining a first sample and a second sample, wherein the first sample is a facial image sample with a known expression degree, the second sample is a facial image sample with an unknown expression degree, then obtaining a mapping relation between the expression degree and a facial motion coding unit coefficient according to the first sample, processing the second sample by utilizing the mapping relation, obtaining the expression degree of each facial image sample in a sample set, and further obtaining an expression degree prediction model according to the expression degree of each facial image sample in the sample set. Therefore, the method disclosed by the invention can realize the recognition of the fine expression degree and improve the recognition accuracy to a certain extent.
Description
Technical Field
The present disclosure relates to computer technologies, and in particular, to an expression degree prediction model obtaining method and apparatus, a storage medium, and a terminal.
Background
With the development of internet technology, the expression degree of a user is used as an index for evaluating the satisfaction degree of the user, and the expression degree is widely developed.
At present, the analysis technology for the expression degree is mainly realized by a neural network technology. Specifically, after the expression degree of the initial sample data is divided manually according to a preset degree dividing rule, the initial neural network model is trained and learned to obtain an expression degree prediction model.
However, because the expression degree of the sample data is obtained according to the preset partition rule, the expression degree of the sample data and the expression degree prediction model obtained by training are all discrete numerical values, so that the existing expression degree prediction model cannot realize degree analysis of fine expressions, and the application scene of the expression degree obtained by training is limited.
Disclosure of Invention
The disclosure provides an expression degree prediction model obtaining method and device, a storage medium and a terminal, which are used for solving the problem that fine expression degree identification cannot be realized in the prior art.
In a first aspect, the present disclosure provides an expression degree prediction model obtaining method, including:
obtaining a sample set, wherein the face sample set comprises: the method comprises the steps of obtaining a first sample and a second sample, wherein the first sample is a facial image sample with known expression degree, and the second sample is a facial image sample with unknown expression degree;
acquiring a mapping relation between the expression degree and the face motion coding unit coefficient according to the first sample;
processing the second sample by using the mapping relation to obtain the expression degree of each face image sample in the sample set;
and acquiring an expression degree prediction model according to the expression degree of each face image sample in the sample set.
In a second aspect, the present disclosure provides an expression degree prediction model obtaining apparatus, including:
a first obtaining module, configured to obtain a sample set, where the face sample set includes: the method comprises the steps of obtaining a first sample and a second sample, wherein the first sample is a facial image sample with known expression degree, and the second sample is a facial image sample with unknown expression degree;
the second acquisition module is used for acquiring the mapping relation between the expression degree and the face motion coding unit coefficient according to the first sample;
the processing module is used for processing the second sample by utilizing the mapping relation to obtain the expression degree of each face image sample in the sample set;
and the third acquisition module is used for acquiring an expression degree prediction model according to the expression degree of each face image sample in the sample set.
In a third aspect, the present disclosure provides an expression degree prediction model obtaining apparatus, including:
a memory;
a processor; and
a computer program;
wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of the first aspect.
In a fourth aspect, the present disclosure provides a terminal comprising:
expression degree prediction model obtaining means for implementing the method according to the first aspect;
a terminal body.
In a fifth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon,
the computer program is executed by a processor to implement the method as described in the first aspect.
The expression degree prediction model obtaining method and device, the storage medium and the terminal provided by the present disclosure can obtain the mapping relationship between the face motion coding unit coefficient and the expression degree by labeling a small number of first samples in the sample set, so that the expression degrees of other unknown expression degree second samples can be obtained according to the mapping relationship, and thus, the expression degree of each second sample is a continuous numerical value obtained through the mapping relationship, and further, the expression degree prediction model is trained by using each sample, so that the expression degree output by the expression degree prediction model is also a continuous numerical value, therefore, the method provided by the embodiment of the present disclosure can realize the recognition of the slight emotional change of the face, solve the technical problem that the existing expression degree recognition method cannot recognize the micro-expression, and widen the application scene of the expression degree, and, this disclosure can also effectual reduction mark cost and improve mark efficiency.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
Fig. 1 is a schematic flow chart of an expression degree prediction model obtaining method according to an embodiment of the present disclosure;
fig. 2 is a schematic flow chart of another expression degree prediction model obtaining method according to an embodiment of the present disclosure;
fig. 3 is a schematic flow chart of another expression degree prediction model obtaining method according to an embodiment of the present disclosure;
fig. 4 is a functional block diagram of an expression degree prediction model obtaining apparatus according to an embodiment of the present disclosure;
fig. 5 is a schematic physical structure diagram of an expression degree prediction model obtaining apparatus according to an embodiment of the present disclosure;
fig. 6 is a schematic diagram of an architecture of a terminal according to an embodiment of the present disclosure.
With the foregoing drawings in mind, certain embodiments of the disclosure have been shown and described in more detail below. These drawings and written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the concepts of the disclosure to those skilled in the art by reference to specific embodiments.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The specific application scenarios of the present disclosure are: and aiming at the predicted scene of the expression degree of the user. Further, the method may further specifically include: and evaluating the scene of the satisfaction degree of the user. For example, whether the user is a scene of a satisfactory object to the current recommended object is evaluated based on a face image of the user. Or, the following can be further included: and identifying the user preference as a scene portrait of the user based on the expression degree of the user.
As described above, the existing expression level prediction mainly focuses on prediction of a smiling expression of a user. Before the prediction of the smile expression is realized, expression degree labeling is carried out on an initial sample manually according to a preset degree division rule, and then the prediction is realized by training a neural network model by using sample data of a label number. However, due to the expression degree dividing, training and predicting mode, the prediction result output by the trained expression degree prediction model is a discrete numerical value, so that the recognition of the tiny expression cannot be realized, and the further application and development of the expression degree are limited.
The technical scheme provided by the disclosure aims to solve the technical problems in the prior art.
The following describes the technical solutions of the present disclosure and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present disclosure will be described below with reference to the accompanying drawings.
Example one
The embodiment of the disclosure provides an expression degree prediction model obtaining method. Referring to fig. 1, the method includes the following steps:
s102, obtaining a sample set, wherein the face sample set comprises: the facial image analysis method comprises a first sample and a second sample, wherein the first sample is a facial image sample with known expression degree, and the second sample is a facial image sample with unknown expression degree.
In other words, the face image is used as a sample to form a sample set. In the case that the sample set includes a first sample with a part of known expression degrees, and the expression degree of the second sample is unknown, the embodiment of the present disclosure uses the first sample to achieve the obtaining of the expression degree of the second sample. The expression level of the first sample may be manually annotated.
And S104, acquiring a mapping relation between the expression degree and the face motion coding unit coefficient according to the first sample.
Specifically, the face motion Coding unit coefficient (hereinafter, referred to as FACS coefficient) may be obtained by processing a Facial Action Coding System (FACS) technique. FACS divides human face into several independent and connected motion coding units, and based on the motion characteristics of these motion coding units and their controlled main area, and their relative expression, obtains great amount of base vectors.
When the step is implemented, a first FACS coefficient of the first sample may be acquired, and then based on the known expression degree, mapping from the FACS coefficient to the expression degree may be implemented to obtain a mapping relationship thereof. The mapping relation is used for representing the mapping relation between the continuous expression degrees and the continuous FACS, and thus, the mapping processing of the step realizes the transformation of the expression degrees from discrete data to continuous data.
And S106, processing the second sample by using the mapping relation to obtain the expression degree of each face image sample in the sample set.
For any second sample, only the second FACS coefficient of the second sample needs to be obtained, and then the expression degree corresponding to the second FACS coefficient is obtained according to the mapping relationship to serve as the expression degree of the second sample.
Therefore, the expression degree of each face image sample in the sample set is obtained. And, the expression level is continuous type data.
And S108, acquiring an expression degree prediction model according to the expression degree of each face image sample in the sample set.
In other words, the expression degree of each face image sample in the sample set is utilized to train the initial neural network model, and the expression degree prediction model is obtained. The input data of the expression degree prediction model is a human face image, and the output data is an expression degree prediction result.
As shown, by the method shown in fig. 1, only a small number of first samples in the sample set need to be labeled, the mapping relation between the face motion coding unit coefficient and the expression degree can be obtained, therefore, the expression degrees of the second samples with other unknown expression degrees can be obtained according to the mapping relation, so that the expression degree of each second sample is a continuous numerical value obtained through the mapping relation, furthermore, each sample is used for training the expression degree prediction model, so that the expression degree output by the expression degree prediction model is also a continuous numerical value, therefore, the method provided by the embodiment of the disclosure can realize the recognition of the slight emotional changes of the human face, solve the technical problem that the existing expression degree recognition method can not recognize the micro-expression, this has also widened the application scene of expression degree to, this disclosure can also effectual reduction mark cost and improve mark efficiency.
The embodiments of the present disclosure do not specifically limit the number of the first samples and the second samples included in the sample set. It can be known that, when the number of the first samples is greater than the number of the second samples, the more accurate the mapping relationship is, and at this time, the prediction accuracy of the expression degree prediction model is favorably improved.
In another implementation scenario, the number of first samples may be smaller, and in some implementations may even be much smaller than the number of second samples. At this moment, only need the manual work in advance to label a small amount of first samples in the sample set can, be favorable to reducing the human cost, and realize the acquirement to the second sample of unknown expression degree through the mapping relation, also be favorable to improving the treatment effeciency.
The following further describes the implementation of the above steps.
For the aforementioned step of acquiring the mapping relationship in S104, reference may be made to the flow illustrated in fig. 2. As shown in fig. 2, S104 may include the steps of:
s1042, obtaining a first face motion coding unit coefficient of the first sample.
That is, for each first sample in the sample set, its corresponding first FACS coefficient is obtained.
Specifically, the FACS includes a plurality of (e.g., 51) basis vectors, and the basis vectors are used to represent the variation of an object in a human face, such as eyes and nose. For example, a base vector that can be used to characterize mouth corners tilting, a base vector that can be used to characterize eyes squinting.
And obtaining the sum of the products of each base vector and the FACS coefficient thereof to obtain an expression image for representing any expression in the human face. In other words, only the FACS coefficients of the basis vectors need to be adjusted to obtain an expression image for representing any expression.
Based on this, in performing this step, the first sample needs to be regressed using the basis vector in FACS to obtain the first FACS coefficient.
In a possible implementation manner, initial coefficients may be configured for a plurality of preset basis vectors to obtain an initial expression image, and then, the initial expression and the first sample are subjected to key point matching to obtain a key point matching result, so that the initial coefficients are adjusted by using the key point matching result, so that the matching difference degree between the expression image obtained according to the adjusted coefficients and the first sample is minimum, and thus, the adjusted coefficients are used as the coefficients of the first face motion coding unit. The key point matching refers to matching key points of the human face.
And S1044, acquiring a mapping relation between the expression degree and the coefficient of the face motion coding unit by using the expression degree of the first sample and the coefficient of the first face motion coding unit.
As described above, for any first sample, the expression degree and the first FACS coefficient are known, and at this time, the mapping relationship may be obtained based on the high-quality data. The embodiment of the disclosure at least provides the following implementation modes:
in a possible implementation manner, the expression degree of the first sample and the coefficient of the first facial motion coding unit may be subjected to fitting processing to obtain the mapping relationship.
Specifically, a curve fitting process may be performed on the initial mathematical model between the FACS coefficients and the expression level using a mathematical fitting algorithm to obtain a curve with the minimum fitting error rate, and the curve or an expression of the curve may be used as the mapping relationship.
In this implementation scenario, when the step described in S106 is executed, for any second sample, only the second FACS coefficient of the second sample needs to be obtained, and the second FACS coefficient is substituted into the curve expression, so as to obtain the expression degree of the second sample, which is used as the expression degree of the second sample.
Or, when step S106 is executed, for any second sample, only the second FACS coefficient of the second sample needs to be acquired, the second FACS coefficient is found in the FACS axis of the coordinate system of the curve, and the expression degree corresponding to the second FACS coefficient is directly determined from the curve to serve as the expression degree of the second sample.
It should be noted that, when the mapping relationship is obtained by a fitting process, the curve may be represented as a straight line, and at this time, the FACS coefficient and the expression degree are in a linear relationship. In the implementation scene, at least 2 first samples are needed, the expression degrees of all face image samples in the sample set can be acquired, and the labor cost and the time cost for manually marking the expression degrees are greatly reduced.
Of course, the curve may also be represented as a curve, which may also be a piecewise curve, or a continuous curve; the relationship between the FACS coefficient and the expression degree represented by the curve can be unary or multivariate. In a specific implementation scene, the user-defined presetting can be carried out according to actual needs.
In another possible implementation, the mapping relationship may be obtained by using a neural network technology. At this time, an initial mapping model is trained by using the expression degree of the first sample and the coefficient of the first face motion coding unit, so as to obtain a trained mapping model as the mapping relation.
In other words, the first sample is used as a training sample of the initial mapping model, the FACS coefficients are input to the initial mapping model, and the expression degrees corresponding to the FACS coefficients are output.
The specific training process for the initial mapping model may be: constructing each processing subunit of an initial training model, presetting an initial training coefficient for each processing subunit, then inputting the FACS coefficient of a first sample into the initial mapping model to obtain a predicted expression degree output by the initial mapping model, then adjusting each initial training coefficient in the initial mapping model based on the difference condition of the predicted expression degree and the expression degree of the first sample, repeating the previous steps until the difference condition reaches a preset difference allowable value, and substituting the training coefficient at the moment into the initial mapping model to obtain the trained mapping model.
The embodiment of the present disclosure has no particular limitation on the model type of the mapping model, which may include but is not limited to: convolutional Neural Network (CNN) or Recurrent Neural Network (RNN).
When the step of S106 is executed, for any second sample, it is only necessary to obtain a second FACS coefficient of the second sample, input the second FACS coefficient to the mapping model, and obtain an expression degree output by the mapping model, so as to use the expression degree as the expression degree of the second sample.
Furthermore, in the embodiments of the present disclosure, the mapping model may be used to map the expression degree of at least one second sample at the same time. At this time, based on different design ideas, one FACS coefficient can be used as an input when the mapping model is trained, and the mapping model is used for mapping the expression degree corresponding to the FACS coefficient; alternatively, a plurality of FACS coefficients may be used as inputs, and a mapping model is used to map the degree of expression for each of the plurality of FACS coefficients.
Through any one of the implementation manners, the mapping relationship between the continuous FACS coefficients and the continuous expression degrees can be obtained, and the expression degrees of the second samples are obtained based on different processing manners.
In this way, before the step S108 is executed, the expression degree labeling may be further performed on each second sample based on the acquisition result of S106, and stored. Therefore, the expression labeling of each face image sample in the sample set can be realized right, the problem of manpower cost and time cost waste caused by artificial labeling of the expression degree is solved, the problem of discrete expression degree caused by the artificial labeling of the expression degree is solved, and the problem that fine expressions cannot be identified is solved.
Based on the foregoing processing, when step S108 is executed, the initial neural network model is trained by using the sample set with the obtained expression degree as training data, so as to obtain a trained expression degree prediction model. The model training process of the expression degree prediction model is similar to the training process of the mapping model, and is not repeated here.
In the embodiment of the disclosure, the expression degree prediction model can be used for simultaneously predicting the expression degree of at least one face image. At this time, based on different design ideas, when the expression degree prediction model is trained, one face image sample is used as input, and the expression degree prediction model is used for predicting the expression degree of the face image sample; alternatively, a plurality of facial image samples may be used as input, and the expression degree prediction model is used to predict the respective expression degree of each facial image sample in the plurality of facial image samples.
In addition to the foregoing design, the expression degree prediction model obtained by the method provided in the embodiment of the present disclosure may be used to predict the expression degree of at least one facial expression. The facial expressions related to the embodiments of the present disclosure may include, but are not limited to: smile expression, crying expression, anger expression and the like, and in an actual scene, the preset can be defined according to actual needs.
Therefore, compared with the prior art that only smile expression recognition can be realized, the technical scheme provided by the embodiment of the disclosure can further realize recognition of other facial expressions, and the application range is wider and comprehensive.
Based on any one of the above designs, an expression degree prediction model can be obtained, and at this time, referring to fig. 3, for any one or more face images to be recognized, the method may further include the following steps:
and S110, acquiring a face image to be recognized.
The face image to be recognized may be a face image acquired by an image acquisition device in real time, or may be a historical face image.
And S112, inputting the facial image to be recognized into the expression degree prediction model to obtain an expression degree prediction result output by the expression degree prediction model.
At this time, the expression degree prediction model may be used to identify the expression degree of any specified expression, such as a smiling expression or a crying expression, and details are not repeated.
And based on the expression degree recognition result aiming at the face image to be recognized, the satisfaction degree of the user to the current output or current recommended object can be further judged according to the expression degree recognition result, and the expression degree recognition result can be further used for generating the interesting portrait of the user and the like.
It is to be understood that some or all of the steps or operations in the above-described embodiments are merely examples, and other operations or variations of various operations may be performed by the embodiments of the present application. Further, the various steps may be performed in a different order presented in the above-described embodiments, and it is possible that not all of the operations in the above-described embodiments are performed.
The words used in this application are words of description only and not of limitation of the claims. As used in the description of the embodiments and the claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the term "and/or" as used in this application is meant to encompass any and all possible combinations of one or more of the associated listed. Furthermore, the terms "comprises" and/or "comprising," when used in this application, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Example two
Based on the expression degree prediction model obtaining method provided in the first embodiment, the embodiment of the present disclosure further provides an embodiment of a device for implementing each step and method in the embodiment of the method.
An embodiment of the present disclosure provides an expression degree prediction model obtaining apparatus, please refer to fig. 4, where the expression degree prediction model obtaining apparatus 400 includes:
a first obtaining module 41, configured to obtain a sample set, where the face sample set includes: the method comprises the steps of obtaining a first sample and a second sample, wherein the first sample is a facial image sample with known expression degree, and the second sample is a facial image sample with unknown expression degree;
a second obtaining module 42, configured to obtain, according to the first sample, a mapping relationship between an expression degree and a face motion coding unit coefficient;
a processing module 43, configured to process the second sample by using the mapping relationship, so as to obtain an expression degree of each face image sample in the sample set;
and a third obtaining module 44, configured to obtain an expression degree prediction model according to the expression degree of each face image sample in the sample set.
In one possible design, the second obtaining module 42 is specifically configured to:
acquiring a first face motion coding unit coefficient of the first sample;
and acquiring the mapping relation by using the expression degree of the first sample and the coefficient of the first face motion coding unit.
In a possible implementation manner, the second obtaining module 42 is specifically configured to:
and fitting the expression degree of the first sample with the coefficient of the first face motion coding unit to obtain the mapping relation.
In a possible implementation scenario, the mapping relationship is a linear relationship.
In another possible implementation manner, the second obtaining module 42 is specifically configured to:
and training an initial mapping model by using the expression degree of the first sample and the coefficient of the first face motion coding unit to obtain a trained mapping model serving as the mapping relation.
In another possible design, the second obtaining module 42 is further specifically configured to:
configuring initial coefficients for a plurality of preset base vectors to obtain an initial expression image;
carrying out key point matching on the initial expression and the first sample to obtain a key point matching result;
adjusting the initial coefficient by using the key point matching result, so that the matching difference degree between the expression image obtained according to the adjusted coefficient and the first sample is minimum;
and taking the adjusted coefficient as the coefficient of the first face motion coding unit.
In another possible design, the processing module 43 is specifically configured to:
acquiring a second face motion coding unit coefficient of the second sample;
and acquiring the expression degree corresponding to the second face motion coding unit coefficient according to the mapping relation to be used as the expression degree of the second sample.
In another possible design, the third obtaining module 44 is further specifically configured to:
training an initial neural network model by using the expression degree of each face image sample to obtain an expression degree prediction model;
and inputting the expression degree into a human face image, and outputting the expression degree prediction result.
In another possible implementation scenario, the expression degree prediction model is used for predicting the expression degree of at least one facial expression.
In another possible implementation scenario, the expression degree prediction model obtaining apparatus 400 further includes:
a fourth obtaining module (not shown in fig. 4) for obtaining a face image to be recognized;
a fifth obtaining module (not shown in fig. 4), configured to input the facial image to be recognized into the expression degree prediction model, so as to obtain an expression degree prediction result output by the expression degree prediction model.
The expression degree prediction model obtaining apparatus 400 in the embodiment shown in fig. 4 may be used to implement the technical solution of the above method embodiment, and the implementation principle and technical effect of the apparatus may further refer to the relevant description in the method embodiment, and optionally, the expression degree prediction model obtaining apparatus 400 may be a terminal.
It should be understood that the above division of the modules of the expression degree prediction model obtaining apparatus 400 shown in fig. 4 is only a division of logical functions, and all or part of the actual implementation may be integrated into one physical entity, or may be physically separated. And these modules can be realized in the form of software called by processing element; or may be implemented entirely in hardware; and part of the modules can be realized in the form of calling by the processing element in software, and part of the modules can be realized in the form of hardware. For example, the second obtaining module 42 may be a processing element separately installed, or may be integrated into the expression degree prediction model obtaining apparatus 400, for example, a chip of a terminal, or may be stored in a memory of the expression degree prediction model obtaining apparatus 400 in the form of a program, and a processing element of the expression degree prediction model obtaining apparatus 400 calls and executes the functions of the above modules. Other modules are implemented similarly. In addition, all or part of the modules can be integrated together or can be independently realized. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.
For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), among others. As another example, when one of the above modules is implemented in the form of a Processing element scheduler, the Processing element may be a general purpose processor, such as a Central Processing Unit (CPU) or other processor capable of invoking programs. As another example, these modules may be integrated together, implemented in the form of a system-on-a-chip (SOC).
Also, an embodiment of the present disclosure provides an expression degree prediction model obtaining apparatus, please refer to fig. 5, where the expression degree prediction model obtaining apparatus 400 includes:
a memory 410;
a processor 420; and
a computer program;
wherein the computer program is stored in the memory 410 and configured to be executed by the processor 420 to implement the methods as described in the above embodiments.
The number of the processors 420 in the expression degree prediction model obtaining apparatus 400 may be one or more, and the processors 420 may also be referred to as processing units, which may implement a certain control function. The processor 420 may be a general purpose processor, a special purpose processor, or the like. In an alternative design, the processor 420 may also store instructions, which can be executed by the processor 420, so that the expression degree prediction model obtaining apparatus 400 performs the method described in the above method embodiment.
In yet another possible design, the expression degree prediction model obtaining device 400 may include a circuit, and the circuit may implement the functions of sending or receiving or communication in the foregoing method embodiments.
Optionally, the number of the memories 410 in the expression degree prediction model obtaining apparatus 400 may be one or more, and the memories 410 have instructions or intermediate data stored thereon, and the instructions may be executed on the processor 420, so that the expression degree prediction model obtaining apparatus 400 performs the method described in the above method embodiments. Optionally, other related data may also be stored in the memory 410. Optionally, instructions and/or data may also be stored in processor 420. The processor 420 and the memory 410 may be provided separately or integrated together.
In addition, as shown in fig. 5, a transceiver 430 is further disposed in the expression degree prediction model obtaining apparatus 400, where the transceiver 430 may be referred to as a transceiver unit, a transceiver circuit, or a transceiver, and is used for data transmission or communication with a testing device or other terminal devices, and will not be described herein again.
As shown in fig. 5, the memory 410, the processor 420, and the transceiver 430 are connected by a bus and communicate.
If the expression degree prediction model obtaining apparatus 400 is used to implement the method corresponding to fig. 1, the processor 420 is used to perform corresponding determination or control operations, and optionally, corresponding instructions may also be stored in the memory 410. The specific processing manner of each component can be referred to the related description of the previous embodiment.
Furthermore, the disclosed embodiments provide a readable storage medium, on which a computer program is stored, the computer program being executed by a processor to implement the method according to the first embodiment.
Also, an embodiment of the present disclosure provides a terminal, please refer to fig. 6, where the terminal 600 includes: an expression degree prediction model acquisition device 400 and a terminal body 610. The expression degree prediction model obtaining apparatus 400 is configured to execute an expression degree prediction model obtaining method according to any implementation manner of the embodiment.
The terminal body 610 generally further includes an image capturing device (e.g., a camera), a display device (e.g., a display screen), and the like. At this time, the expression degree prediction model acquisition means 400 as shown in fig. 4 or 5 may acquire the face image to be processed by an image acquisition means that is already present in the terminal.
The disclosed embodiment is not particularly limited with respect to the components included in the terminal body 610. In a practical implementation scenario, one or more of the following components may be included: a processing component, a memory, a power component, a multimedia component, an audio component, an input/output (I/O) interface, a sensor component, and a communication component.
The terminal 600 according to the embodiments of the present disclosure may be a wireless terminal or a wired terminal. A wireless terminal may refer to a device that provides voice and/or other traffic data connectivity to a target user, a handheld device having wireless connection capability, or other processing device connected to a wireless modem. A wireless terminal, which may be a mobile terminal such as a mobile telephone (or "cellular" telephone) and a computer having a mobile terminal, for example, a portable, pocket, hand-held, computer-included, or vehicle-mounted mobile device, may communicate with one or more core Network devices via a Radio Access Network (RAN), and may exchange language and/or data with the RAN. For another example, the Wireless terminal may also be a Personal Communication Service (PCS) phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA), and other devices. The wireless Terminal may also be referred to as a system, a Subscriber Unit (Subscriber Unit), a Subscriber Station (Subscriber Station), a Mobile Station (Mobile), a Remote Station (Remote Station), a Remote Terminal (Remote Terminal), an Access Terminal (Access Terminal), a target User Terminal (User Terminal), a target User Agent (User Agent), and a target User Equipment (User Device or User Equipment), which are not limited herein. Optionally, the terminal device may also be a smart watch, a tablet computer, or the like.
Since each module in this embodiment can execute the method shown in the first embodiment, reference may be made to the related description of the first embodiment for a part of this embodiment that is not described in detail.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (11)
1. An expression degree prediction model acquisition method is characterized by comprising the following steps:
obtaining a set of samples, the set of samples comprising: the method comprises the steps of obtaining a first sample and a second sample, wherein the first sample is a facial image sample with known expression degree, and the second sample is a facial image sample with unknown expression degree;
acquiring a mapping relation between the expression degree and the face motion coding unit coefficient according to the first sample, wherein the mapping relation is used for representing the mapping relation between the continuous expression degree and the continuous face motion coding unit coefficient;
processing the second sample by using the mapping relation to obtain the expression degree of each face image sample in the sample set;
acquiring an expression degree prediction model according to the expression degree of each face image sample in the sample set;
the obtaining of the mapping relationship between the expression degree and the face motion coding unit coefficient according to the first sample includes:
configuring an initial coefficient through a plurality of preset base vectors to obtain an initial expression image;
carrying out key point matching on the initial expression and the first sample to obtain a key point matching result;
adjusting the initial coefficient by using the key point matching result, so that the matching difference degree between the expression image obtained according to the adjusted coefficient and the first sample is minimum;
taking the adjusted coefficient as a first face motion coding unit coefficient;
and acquiring the mapping relation by using the expression degree of the first sample and the coefficient of the first face motion coding unit.
2. The method according to claim 1, wherein said obtaining the mapping relationship by using the expression degree of the first sample and the first facial motion coding unit coefficient comprises:
and fitting the expression degree of the first sample with the coefficient of the first face motion coding unit to obtain the mapping relation.
3. The method of claim 2, wherein the mapping is a linear relationship.
4. The method according to claim 1, wherein said obtaining the mapping relationship by using the expression degree of the first sample and the first facial motion coding unit coefficient comprises:
and training an initial mapping model by using the expression degree of the first sample and the coefficient of the first face motion coding unit to obtain a trained mapping model serving as the mapping relation.
5. The method according to any one of claims 1 to 4, wherein the processing the second sample by using the mapping relationship to obtain the expression degree of each face image sample in the sample set comprises:
acquiring a second face motion coding unit coefficient of the second sample;
and acquiring the expression degree corresponding to the second face motion coding unit coefficient according to the mapping relation to be used as the expression degree of the second sample.
6. The method according to claim 1, wherein the obtaining an expression degree prediction model according to the expression degree of each face image sample comprises:
training an initial neural network model by using the expression degree of each face image sample to obtain an expression degree prediction model;
and inputting the expression degree into a human face image, and outputting the expression degree prediction result.
7. The method of claim 1 or 6, wherein the expression level prediction model is used for predicting the expression level of at least one facial expression.
8. The method of claim 1, further comprising:
acquiring a face image to be recognized;
and inputting the facial image to be recognized into the expression degree prediction model to obtain an expression degree prediction result output by the expression degree prediction model.
9. An expression degree prediction model acquisition apparatus, comprising:
a first obtaining module configured to obtain a sample set, the sample set comprising: the method comprises the steps of obtaining a first sample and a second sample, wherein the first sample is a facial image sample with known expression degree, and the second sample is a facial image sample with unknown expression degree;
the second obtaining module is used for obtaining the mapping relation between the expression degree and the face motion coding unit coefficient according to the first sample, and the mapping relation is used for representing the mapping relation between the continuous expression degree and the continuous face motion coding unit coefficient;
the processing module is used for processing the second sample by utilizing the mapping relation to obtain the expression degree of each face image sample in the sample set;
the third acquisition module is used for acquiring an expression degree prediction model according to the expression degree of each face image sample in the sample set;
the second obtaining module is specifically configured to: configuring an initial coefficient through a plurality of preset base vectors to obtain an initial expression image; carrying out key point matching on the initial expression and the first sample to obtain a key point matching result; adjusting the initial coefficient by using the key point matching result, so that the matching difference degree between the expression image obtained according to the adjusted coefficient and the first sample is minimum; taking the adjusted coefficient as a first face motion coding unit coefficient; and acquiring the mapping relation by using the expression degree of the first sample and the coefficient of the first face motion coding unit.
10. An expression degree prediction model acquisition apparatus, comprising:
a memory;
a processor; and
a computer program;
wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-8.
11. A computer-readable storage medium, having stored thereon a computer program,
the computer program is executed by a processor to implement the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910594664.8A CN110298329B (en) | 2019-07-03 | 2019-07-03 | Expression degree prediction model obtaining method and device, storage medium and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910594664.8A CN110298329B (en) | 2019-07-03 | 2019-07-03 | Expression degree prediction model obtaining method and device, storage medium and terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110298329A CN110298329A (en) | 2019-10-01 |
| CN110298329B true CN110298329B (en) | 2021-11-16 |
Family
ID=68030121
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910594664.8A Active CN110298329B (en) | 2019-07-03 | 2019-07-03 | Expression degree prediction model obtaining method and device, storage medium and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110298329B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111445417B (en) * | 2020-03-31 | 2023-12-19 | 维沃移动通信有限公司 | Image processing method, device, electronic equipment and medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109583431A (en) * | 2019-01-02 | 2019-04-05 | 上海极链网络科技有限公司 | A kind of face Emotion identification model, method and its electronic device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109255830B (en) * | 2018-08-31 | 2020-06-05 | 百度在线网络技术（北京）有限公司 | Three-dimensional face reconstruction method and device |
| CN109903363A (en) * | 2019-01-31 | 2019-06-18 | 天津大学 | Condition generates confrontation Network Three-dimensional human face expression moving cell synthetic method |
-
2019
- 2019-07-03 CN CN201910594664.8A patent/CN110298329B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109583431A (en) * | 2019-01-02 | 2019-04-05 | 上海极链网络科技有限公司 | A kind of face Emotion identification model, method and its electronic device |
Non-Patent Citations (2)
| Title |
|---|
| "基于DWT-KPCA的区域化人脸运动单元识别的研究";宋晓娜;《中国优秀硕士学位论文全文数据库 信息科技辑》;20140715(第07期);第15页 * |
| "基于人脸运动单元及表情关系模型的自动表情识别";解仑等;《北京理工大学学报》;20160225;第36卷(第2期);第163-169页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110298329A (en) | 2019-10-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111369427B (en) | Image processing method, image processing device, readable medium and electronic equipment | |
| US20190205616A1 (en) | Method and apparatus for detecting face occlusion | |
| WO2020248841A1 (en) | Au detection method and apparatus for image, and electronic device and storage medium | |
| CN110298327B (en) | Visual special effect processing method and device, storage medium and terminal | |
| CN110930296B (en) | Image processing method, device, equipment and storage medium | |
| CN110287836B (en) | Image classification method and device, computer equipment and storage medium | |
| WO2021004322A1 (en) | Head special effect processing method and apparatus, and storage medium | |
| CN110225400B (en) | Motion capture method and device, mobile terminal and storage medium | |
| WO2016165614A1 (en) | Method for expression recognition in instant video and electronic equipment | |
| CN108347578A (en) | The processing method and processing device of video image in video calling | |
| CN109784304A (en) | Method and apparatus for marking dental imaging | |
| CN111107278B (en) | Image processing method and device, electronic equipment and readable storage medium | |
| CN104123741A (en) | Method and device for generating human face sketch | |
| CN110298405A (en) | Classification recognition methods and device, storage medium and terminal | |
| CN111178298A (en) | Human body key point detection method and device, electronic equipment and storage medium | |
| CN110298329B (en) | Expression degree prediction model obtaining method and device, storage medium and terminal | |
| CN111833242A (en) | Face transformation method and device, electronic equipment and computer readable medium | |
| CN110570383B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN111881740A (en) | Face recognition method, face recognition device, electronic equipment and medium | |
| CN110097004B (en) | Facial expression recognition method and device | |
| CN111126159A (en) | Method, apparatus, electronic device, and medium for tracking pedestrian in real time | |
| CN109003272B (en) | Image processing method, device and system | |
| CN114359963A (en) | Gesture recognition method and communication system | |
| CN107729886B (en) | Method and device for processing face image | |
| CN111104827A (en) | Image processing method and device, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |