CN110895685A - Smile service quality evaluation system and evaluation method based on deep learning - Google Patents
Smile service quality evaluation system and evaluation method based on deep learning Download PDFInfo
- Publication number
- CN110895685A CN110895685A CN201911166917.8A CN201911166917A CN110895685A CN 110895685 A CN110895685 A CN 110895685A CN 201911166917 A CN201911166917 A CN 201911166917A CN 110895685 A CN110895685 A CN 110895685A
- Authority
- CN
- China
- Prior art keywords
- smile
- service quality
- quality evaluation
- mouth corner
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
- G06V40/175—Static expression
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
Abstract
The invention discloses a smile service quality evaluation system based on deep learning, which comprises a human body image acquisition module, a human body image acquisition module and a smile service quality evaluation module, wherein the human body image acquisition module is used for acquiring human body images of service personnel; the human face recognition module is used for carrying out human face recognition on the human body image; the face key point identification module is used for further identifying face key points of the identified face area; the mouth angle identification module is used for further identifying the mouth angle key points from the identified face key points; the expression recognition module is used for recognizing the mouth angle state of the key point of the mouth angle and recognizing the mouth angle state of the service staff; the smile quality judging module is used for carrying out image matching on the identified mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judging result, and outputting the mouth corner state compliance judging result as a smile service quality evaluating result.
Description
Technical Field
The invention relates to the technical field of service quality evaluation, in particular to a smile service quality evaluation system and method based on deep learning.
Background
In certain service scenarios, such as toll booths, airports, train stations, restaurants, etc., the smile service quality of the service personnel needs to be evaluated. At present, the evaluation mode of smile service quality of service personnel is generally carried out by adopting a manual on-site evaluation and supervision mode, the manual evaluation mode has strong subjectivity, and objective evaluation on smile service quality of the service personnel cannot be carried out.
Disclosure of Invention
The invention provides a smiling service quality evaluation system based on deep learning, which aims to solve the technical problem.
The technical scheme for solving the technical problem is to provide a smile service quality evaluation system based on deep learning, which is used for evaluating whether the smile service quality of service personnel is qualified or not, and comprises the following steps of
The human body image acquisition module is used for acquiring human body images of the service personnel;
the human face recognition module is connected with the human body image acquisition module and used for carrying out human face recognition detection on the human body image according to a preset human face recognition model so as to obtain a human face area through recognition;
the face key point recognition module is connected with the face recognition module and used for further recognizing face key points in the face area according to a preset face key point recognition model so as to obtain a plurality of face key points through recognition;
the mouth corner recognition module is connected with the face key point recognition module and used for recognizing mouth corner key points in the face key points according to a preset mouth corner recognition model;
the expression recognition module is respectively connected with the face key point recognition module and the mouth corner recognition module and is used for carrying out mouth corner state recognition on the mouth corner key points according to a preset expression recognition model and recognizing to obtain the current mouth corner state related to the service personnel;
and the smile quality judging module is connected with the expression recognition module and is used for carrying out image matching on the recognized mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judging result and outputting the mouth corner state compliance judging result as a smile service quality evaluation result.
Further, the smile service quality evaluation system further includes:
a head gesture recognition module respectively connected with the face recognition module and the smile quality judgment module and used for recognizing the head gesture of the service personnel in the face area according to a preset head gesture recognition model to obtain a head gesture recognition result,
and the smile quality judging module integrates the mouth angle state compliance judging result and the head posture identifying result, judges whether the smile service quality of the service personnel reaches the standard or not, and outputs the smile service quality evaluating result.
Further, the head pose is an euler angle of the head of the service person with respect to a setting position of the camera shooting and collecting device.
Further, the smile service quality evaluation system further includes:
the voice acquisition module is used for acquiring voice information of the service personnel;
the voice recognition module is connected with the voice acquisition module and used for recognizing the voice content in the voice information;
a language library, in which a plurality of expressions are pre-stored;
a speech content compliance determination module, connected to the speech recognition module and the smile quality determination module respectively, and connected to the phrase library, for performing phrase matching between the recognized speech content and each phrase in the phrase library to determine whether the speech content contains an unqualified phrase, and generating and storing a speech content compliance determination result,
and the smile quality judging module integrates the mouth corner state compliance judging result and the voice content compliance judging result, judges whether the smile service quality of the service personnel is compliant or not, and finally forms and outputs the smile service quality evaluation result.
Further, the smile service quality evaluation system further includes:
the limb key point identification module is connected with the human body image acquisition module and used for continuously identifying limb key points of a plurality of continuously acquired human body images according to a preset limb key point identification model so as to identify and obtain a limb key point identification result aiming at each human body image;
the limb action identification module is connected with the limb key point identification module and used for identifying and detecting the corresponding limb action of each limb key point in the human body image shooting period according to the identification result of each limb key point;
a limb action compliance judging module which is respectively connected with the limb action recognition module and the smile quality judging module and is used for judging whether each limb action is compliant or not, generating a limb action compliance judging result and storing the result,
and the smile quality judging module integrates the mouth corner state compliance judging result and the limb action compliance judging result, judges whether the smile service quality of the service personnel is compliant or not, and finally forms and outputs the smile quality judging result.
The invention also provides a smile service quality evaluation method which is realized by applying the smile service quality evaluation system and specifically comprises the following steps:
step S1, the smile service quality evaluation system collects the human body image of the service personnel;
step S2, the smile service quality evaluation system carries out face recognition detection on the human body image, and a face area is obtained through recognition;
step S3, the smile service quality evaluation system carries out further face key point identification on the face area to obtain a plurality of face key points;
step S4, the smile service quality evaluation system identifies a mouth corner key point in each face key point;
step S5, the smile service quality evaluation system identifies the mouth corner state of the key point of the mouth corner, and identifies the current mouth corner state associated with the service personnel;
and step S6, the smile service quality evaluation system performs image matching on the identified mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judgment result, and outputs the mouth corner state compliance judgment result as a smile service quality evaluation result.
The invention also provides another smile service quality evaluation method which is realized by applying the smile service quality evaluation system and specifically comprises the following steps:
step L1, the smile service quality evaluation system collects the human body image of the service personnel and the voice information of the service personnel;
l2, the smile service quality evaluation system carries out face recognition detection on the human body image to obtain a face area,
performing voice content recognition on the voice information, and outputting a voice content recognition result;
step L3, the smile service quality evaluation system further identifies the face key points to obtain a plurality of face key points,
performing content compliance evaluation on the voice content to generate and store a voice content compliance judgment result;
step L4, the smile service quality evaluation system identifies a mouth corner key point in each face key point;
l5, the smile service quality evaluation system identifies the mouth corner state of the key point of the mouth corner, and identifies the current mouth corner state associated with the service personnel;
step L6, the smile service quality evaluation system performs image matching on the identified mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judgment result;
and L7, the smile service quality evaluation system integrates the speech content compliance judgment result obtained in the step L3 and the mouth corner state compliance judgment result obtained in the step L6, judges whether the smile service quality of the service staff is compliant, and finally forms and outputs the smile service quality evaluation result.
The invention also provides a smile service quality evaluation method, which is realized by applying the smile service quality evaluation system and specifically comprises the following steps:
step M1, the smile service quality evaluation system continuously collects a plurality of human body images related to the service personnel;
step M2, the smile service quality evaluation system carries out face recognition detection on each human body image to obtain a face area on each human body image,
continuously identifying limb key points of each human body image to obtain a limb key point identification result for each human body image;
step M3, the smile service quality evaluation system further identifies the key points of the human face to obtain a plurality of key points of the human face,
according to the identification result of each limb key point, identifying and detecting the corresponding limb action of each limb key point in the human body image shooting period;
step M4, the smile service quality evaluation system identifies corresponding mouth corner key points in each face key point, judges whether each limb action is in compliance, and then generates and stores a limb action compliance judgment result;
step M5, the smile service quality evaluation system identifies the mouth corner state of each key point of the mouth corner, and identifies the mouth corner state associated with the service staff during the process of shooting the human body image;
step M6, the smile service quality evaluation system performs image matching on each identified mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judgment result for the service staff during the process of shooting the human body image;
and step M7, the smile service quality evaluation system integrates the limb action compliance judgment result obtained in the step M4 and the mouth angle state compliance judgment result obtained in the step M6, judges whether the smile service quality of the service staff during the human body image shooting is compliant, and finally forms and outputs the smile service quality evaluation result.
According to the invention, the service site of the service personnel is automatically acquired through the image acquisition equipment, then the smile state, the site voice content or the limb state of the service personnel is identified and detected through a machine identification technology, and finally whether the smile state, the voice content or the limb state of the service personnel is qualified or not is evaluated through a preset evaluation rule, so that the whole evaluation process is completely automatic, and the technical problems of strong subjectivity of manual evaluation and insufficient scientific and objective evaluation results are solved.
Drawings
FIG. 1 is a schematic structural diagram of a smile service quality evaluation system based on deep learning according to an embodiment;
fig. 2 is a schematic structural diagram of a smile service quality evaluation system based on deep learning provided in the second embodiment;
fig. 3 is a schematic structural diagram of a smile service quality evaluation system based on deep learning according to a third embodiment;
FIG. 4 is a diagram of the method steps for evaluating smile service quality of a service person using the deep learning based smile service quality evaluation system according to an embodiment;
FIG. 5 is a diagram of the method steps for evaluating smile service quality of a service person using the deep learning based smile service quality evaluation system provided in example two;
FIG. 6 is a diagram of the method steps for evaluating smile service quality of a service person by applying the smile service quality evaluation system based on deep learning provided in the third embodiment;
fig. 7 is a schematic block diagram of a smile service quality evaluation system based on deep learning according to the present invention for evaluating smile service quality of a service person.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
The invention is further described with reference to the following drawings and specific examples, which are not intended to be limiting.
Example one
Referring to fig. 1, the smile service quality evaluation system according to the first embodiment includes:
the human body image acquisition module 1 is used for acquiring human body images of service personnel;
the face recognition module 2 is connected with the human body image acquisition module 1 and used for carrying out face recognition detection on a human body image according to a preset face recognition model and recognizing to obtain a face region;
the face key point recognition module 3 is connected with the face recognition module 2 and is used for further recognizing face key points in the face area according to a preset face key point recognition model so as to obtain a plurality of face key points through recognition;
the mouth corner recognition module 4 is connected with the face key point recognition module 3 and used for recognizing mouth corner key points in each face key point according to a preset mouth corner recognition model;
the expression recognition module 5 is respectively connected with the face key point recognition module 3 and the mouth corner recognition module 4 and is used for carrying out mouth corner state recognition on the mouth corner key points according to a preset expression recognition model and recognizing to obtain the current mouth corner state related to service personnel;
and the smile quality judging module 6 is connected with the expression recognition module 5 and is used for carrying out image matching on the recognized mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judging result and outputting the mouth corner state compliance judging result as a smile service quality evaluation result.
In the technical scheme, the face recognition model, the face key point recognition model, the mouth corner recognition model and the expression recognition model are pre-trained recognition models. The training process of each recognition model is known, and is not claimed, so it is not described here.
The above-mentioned face key points include 64 common face key points, such as the corner of the mouth, eyes, nose, eyebrows, forehead, etc., in this embodiment, the smile service quality evaluation system determines whether the smile service quality of the service staff meets the standard by identifying the corner of the mouth state of the corner of the mouth key point. Specifically, when a person smiles, the mouth corner is in a sipping state and usually a few teeth are exposed, and the system can judge whether the smile service quality reaches the standard by identifying the sipping mouth amplitude and the number of exposed teeth.
Therefore, the system carries out picture matching on the recognized mouth corner state and a standard mouth corner state picture meeting smile service quality, and if the puckering mouth amplitude and the number of exposed teeth of the recognized mouth corner state are basically consistent with the standard mouth corner state, the smile service quality of the service personnel is judged to meet the standard.
In order to improve the accuracy of whether the smile service quality of the service staff meets the standard, preferably, with continuing to refer to fig. 1, the smile service quality evaluation system further includes:
a head gesture recognition module 7 respectively connected with the face recognition module 2 and the smile quality judgment module 6 and used for recognizing the head gesture of the service personnel in the face area according to a preset head gesture recognition model to obtain a head gesture recognition result,
the smile quality determination module 6 integrates the mouth angle state compliance determination result and the head posture recognition result, determines whether the smile service quality of the service staff reaches the standard, and finally outputs a smile service quality evaluation result.
In the above technical solution, the process of the system integrating the mouth angle state compliance determination result and the head posture recognition result is briefly described as follows:
in some specific service scenes, such as a toll gate or an airport, when the service staff smiles, the head of the service staff is inclined, so that the system takes the head posture as one of the factors of the smiling service quality evaluation, which is beneficial to improving the accuracy of the smiling service quality evaluation.
The head pose is preferably calculated by calculating the euler angle of the head of the attendant with respect to the assumed setup position of the acquisition device.
The head posture recognition model is also trained in advance, the training method of the head posture recognition model is the prior art, and the training process is not explained here.
The invention provides a method for evaluating smile service quality of service personnel by applying a smile service quality evaluation system provided by an embodiment, and please refer to fig. 4 and 7, and the method specifically comprises the following steps:
step S1, the smile service quality evaluation system collects the human body image of the service personnel;
step S2, the smile service quality evaluation system carries out face recognition detection on the human body image, and a face area is obtained through recognition;
step S3, the smile service quality evaluation system carries out further face key point identification on the face area to obtain a plurality of face key points;
step S4, the smile service quality evaluation system identifies key points of mouth corners in key points of each face;
step S5, the smile service quality evaluation system identifies the mouth corner state of the key point of the mouth corner to obtain the current mouth corner state associated with the service personnel;
and step S6, the smile service quality evaluation system performs picture matching on the recognized mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judgment result, and the mouth corner state compliance judgment result is used as a smile service quality evaluation result and is output.
Example two
The difference between the second embodiment and the first embodiment is that, referring to fig. 2, the smile service quality evaluation system provided by the second embodiment further includes:
the voice acquisition module 8 is used for acquiring voice information of service personnel;
the voice recognition module 9 is connected with the voice acquisition module 8 and used for recognizing the voice content in the voice information;
a phrase library 10, in which a plurality of phrases are pre-stored;
the speech content compliance judgment module 11 is respectively connected with the speech recognition module 9 and the smile quality judgment module 6, is connected with the phrase library 10, and is used for performing phrase matching on the recognized speech content and each phrase in the phrase library so as to judge whether the speech content contains an unqualified phrase and generate and store a speech content compliance judgment result;
the smile quality determination module 6 integrates the mouth corner state compliance determination result and the voice content compliance determination result, determines whether the smile service quality of the service staff is compliant, and finally forms and outputs a smile service quality evaluation result.
Compared with the first embodiment, the second embodiment provides a smiling service quality evaluation system which is more scientific, reasonable and accurate in determining the smiling service quality of the service staff because the second embodiment determines whether the smiling service quality of the service staff is in compliance and additionally considers the speaking content of the service staff during smiling service. When the service personnel is in the smile state, but the speaking content does not meet the requirement, the system judges that the smile service of the service personnel is unqualified.
Referring to fig. 5 and 7, the present invention further provides a method for implementing smile service quality evaluation on service personnel by using the smile service quality evaluation system provided in the second embodiment, where the evaluation method specifically includes the following steps:
l1, the smile service quality evaluation system collects the human body image of the service personnel and the voice information of the service personnel;
l2, the smile service quality evaluation system carries out face recognition detection on the human body image, a face area is obtained by recognition, voice content recognition is carried out on the voice information, and a voice content recognition result is output;
step L3, the smile service quality evaluation system further detects the key points of the face to identify a plurality of key points of the face,
performing content compliance evaluation on the voice content to generate and store a voice content compliance judgment result;
l4, the smile service quality evaluation system identifies key points of mouth corners in key points of each face;
l5, the smile service quality evaluation system identifies the mouth corner state of the key point of the mouth corner to obtain the current mouth corner state associated with the service personnel;
l6, the smile service quality evaluation system performs image matching on the recognized mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judgment result;
and L7, the smile service quality evaluation system integrates the speech content compliance judgment result and the mouth corner state compliance judgment result obtained in the step L3, judges whether the smile service quality of the service staff is compliant or not, and finally forms and outputs a smile service quality evaluation result.
In the steps of the method, when the speech content compliance judgment result is qualified, and the mouth corner state compliance judgment result is also judged to be qualified, the system finally confirms that the smiling service quality of the service personnel is qualified, otherwise, the smiling service quality is unqualified.
EXAMPLE III
The difference between the third embodiment and the second embodiment is that, referring to fig. 3, the smile service quality evaluation system provided in the third embodiment further includes:
the limb key point identification module 12 is connected with the human body image acquisition module 1 and used for continuously identifying limb key points of a plurality of continuously acquired human body images according to a preset limb key point identification model and identifying to obtain a limb key point identification result aiming at each human body image;
the limb action recognition module 13 is connected with the limb key point recognition module 12 and used for recognizing and detecting the corresponding limb action of each limb key point during the period of shooting the human body image according to the recognition result of each limb key point;
a limb action compliance judging module 14 which is respectively connected with the limb action recognition module 13 and the smile quality judging module 6 and is used for judging whether each limb action is compliant or not, generating and storing a limb action compliance judging result,
the smile quality determination module 6 integrates the mouth corner state compliance determination result and the limb movement compliance determination result, determines whether the smile service quality of the service staff is compliant, and finally forms and outputs a smile quality determination result.
The body actions such as bending actions of service personnel at a toll station, handing off a toll card and the like can effectively evaluate the service quality of the service personnel, so that the system brings the body actions into smile service quality evaluation of the service personnel, and is favorable for improving the scientificity and rationality of the smile service quality evaluation.
In the above technical solution, the limb key point recognition model is trained in advance, and the training of the limb key point recognition model is an existing training method, and a detailed training process thereof is not described herein. In addition, the recognition and detection of the limb movement by the system are also prior art and are not described herein.
In addition, the system may determine whether the limb movement is compliant by image matching the recognized limb movement with a standard limb movement.
The invention also provides a method for evaluating smile service quality of service personnel by applying the smile service quality evaluation system provided by the third embodiment, please refer to fig. 6 and 7, and the evaluation method specifically comprises the following steps:
step M1, the smile service quality evaluation system continuously collects a plurality of human body images related to service personnel;
step M2, the smile service quality evaluation system carries out face recognition detection on each human body image to obtain the face area on each human body image,
continuously identifying limb key points of each human body image to obtain a limb key point identification result for each human body image;
step M3, the smile service quality evaluation system carries out further face key point identification to each face area to obtain a plurality of face key points,
according to the identification result of each limb key point, identifying and detecting the corresponding limb action of each limb key point during the period of shooting each human body image;
step M4, the smile service quality evaluation system identifies corresponding key points of the mouth corner in each key point of the face,
judging whether each limb action is in compliance, and generating and storing a limb action compliance judgment result;
step M5, the smile service quality evaluation system identifies the mouth corner state of each mouth corner key point to obtain the mouth corner state related to service personnel during the process of shooting the human body image;
step M6, the smile service quality evaluation system carries out image matching on each recognized mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judgment result aiming at service personnel during human body image shooting;
and step M7, integrating the limb action compliance judgment result and the mouth corner state compliance judgment result obtained in the step M4 by the smile service quality evaluation system, judging whether the smile service quality of the service staff during the process of shooting the human body image is compliant or not, and finally forming and outputting a smile service quality evaluation result.
In the above steps, when the system judges that the limb action of the service personnel is qualified and the mouth corner state is also qualified in the period of shooting the human body image, the smiling service quality of the service personnel is judged to be qualified, otherwise, the smiling service quality is not qualified.
In addition, the invention also integrates the mouth corner state compliance judgment result, the limb action compliance judgment result and the voice content compliance judgment result of the service personnel in the period of shooting the human body image, and judges whether the smile service quality of the service personnel is qualified. And when the mouth corner state of the service personnel is qualified, the limb action is in compliance, and the voice content is in compliance, judging that the smiling service quality of the service personnel is qualified, otherwise, judging that the smiling service quality of the service personnel is unqualified.
The smile service quality evaluation system provided by the invention integrates the mouth corner state compliance judgment result, the limb action compliance judgment result and the voice content compliance judgment result of the service personnel, and the specific method steps for forming the smile service quality evaluation result of the service personnel are not repeated herein.
While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.
Claims (8)
1. A smile service quality evaluation system based on deep learning is used for evaluating whether smile service quality of service personnel is qualified or not, and is characterized by comprising the following steps:
the human body image acquisition module is used for acquiring human body images of the service personnel;
the human face recognition module is connected with the human body image acquisition module and used for carrying out human face recognition detection on the human body image according to a preset human face recognition model so as to obtain a human face area through recognition;
the face key point recognition module is connected with the face recognition module and used for further recognizing face key points in the face area according to a preset face key point recognition model so as to obtain a plurality of face key points through recognition;
the mouth corner recognition module is connected with the face key point recognition module and used for recognizing mouth corner key points in the face key points according to a preset mouth corner recognition model;
the expression recognition module is respectively connected with the face key point recognition module and the mouth corner recognition module and is used for carrying out mouth corner state recognition on the mouth corner key points according to a preset expression recognition model and recognizing to obtain the current mouth corner state related to the service personnel;
and the smile quality judging module is connected with the expression recognition module and is used for carrying out image matching on the recognized mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judging result and outputting the mouth corner state compliance judging result as a smile service quality evaluation result.
2. The smile service quality evaluation system of claim 1, further comprising:
the head posture recognition module is respectively connected with the face recognition module and the smile quality judgment module and is used for recognizing the head posture of the service personnel in the face area according to a preset head posture recognition model to obtain a head posture recognition result;
and the smile quality judging module integrates the mouth angle state compliance judging result and the head posture identifying result, judges whether the smile service quality of the service personnel reaches the standard or not, and outputs the smile service quality evaluating result.
3. The smiling service quality evaluation system of claim 2, wherein the head pose is an euler angle of the head of the service person with respect to a setting position of a camera capturing apparatus.
4. The smile service quality evaluation system of claim 1, further comprising:
the voice acquisition module is used for acquiring voice information of the service personnel;
the voice recognition module is connected with the voice acquisition module and used for recognizing the voice content in the voice information;
a language library, in which a plurality of expressions are pre-stored;
a speech content compliance determination module, which is respectively connected with the speech recognition module and the smile quality determination module, and connected with the phrase library, and is used for performing phrase matching on the recognized speech content and each phrase in the phrase library to determine whether the speech content contains an unqualified phrase, and generating and storing a speech content compliance determination result;
and the smile quality judging module integrates the mouth corner state compliance judging result and the voice content compliance judging result, judges whether the smile service quality of the service personnel is compliant or not, and finally forms and outputs the smile service quality evaluation result.
5. The smile service quality evaluation system of claim 1, further comprising:
the limb key point identification module is connected with the human body image acquisition module and used for continuously identifying limb key points of a plurality of continuously acquired human body images according to a preset limb key point identification model so as to identify and obtain a limb key point identification result aiming at each human body image;
the limb action identification module is connected with the limb key point identification module and used for identifying and detecting the corresponding limb action of each limb key point in the human body image shooting period according to the identification result of each limb key point;
the limb action compliance judging module is respectively connected with the limb action identification module and the smile quality judging module and is used for judging whether each limb action is compliant or not, generating a limb action compliance judging result and storing the result;
and the smile quality judging module integrates the mouth corner state compliance judging result and the limb action compliance judging result, judges whether the smile service quality of the service personnel is compliant or not, and finally forms and outputs the smile quality judging result.
6. A smile service quality evaluation method is realized by applying the smile service quality evaluation system as any one of claims 1 to 5, and is characterized by specifically comprising the following steps of:
step S1, the smile service quality evaluation system collects the human body image of the service personnel;
step S2, the smile service quality evaluation system carries out face recognition detection on the human body image, and a face area is obtained through recognition;
step S3, the smile service quality evaluation system carries out further face key point identification on the face area to obtain a plurality of face key points;
step S4, the smile service quality evaluation system identifies a mouth corner key point in each face key point;
step S5, the smile service quality evaluation system identifies the mouth corner state of the key point of the mouth corner, and identifies the current mouth corner state associated with the service personnel;
and step S6, the smile service quality evaluation system performs image matching on the identified mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judgment result, and outputs the mouth corner state compliance judgment result as a smile service quality evaluation result.
7. A smile service quality evaluation method is realized by applying the smile service quality evaluation system as any one of claims 1 to 5, and is characterized by specifically comprising the following steps of:
step L1, the smile service quality evaluation system collects the human body image of the service personnel and the voice information of the service personnel;
l2, the smile service quality evaluation system performs face recognition detection on the human body image, recognizes to obtain a face area, performs voice content recognition on the voice information, and outputs a voice content recognition result;
step L3, the smile service quality evaluation system carries out further face key point identification on the face area to obtain a plurality of face key points, and carries out content compliance evaluation on the voice content to generate and store a voice content compliance judgment result;
step L4, the smile service quality evaluation system identifies a mouth corner key point in each face key point;
l5, the smile service quality evaluation system identifies the mouth corner state of the key point of the mouth corner, and identifies the current mouth corner state associated with the service personnel;
step L6, the smile service quality evaluation system performs image matching on the identified mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judgment result;
and L7, the smile service quality evaluation system integrates the speech content compliance judgment result obtained in the step L3 and the mouth corner state compliance judgment result obtained in the step L6, judges whether the smile service quality of the service staff is compliant, and finally forms and outputs the smile service quality evaluation result.
8. A smile service quality evaluation method is realized by applying the smile service quality evaluation system as any one of claims 1 to 5, and is characterized by specifically comprising the following steps of:
step M1, the smile service quality evaluation system continuously collects a plurality of human body images related to the service personnel;
step M2, the smile service quality evaluation system performs face recognition detection on each human body image to obtain a face area on each human body image, and performs limb key point recognition on each human body image continuously to obtain a limb key point recognition result for each human body image;
step M3, the smile service quality evaluation system further identifies the face key points of each face area to obtain a plurality of face key points, and identifies and detects the corresponding limb actions of each limb key point during the human body image shooting according to the identification result of each limb key point;
step M4, the smile service quality evaluation system identifies corresponding mouth corner key points in each face key point,
judging whether each limb action is in compliance, and generating and storing a limb action compliance judgment result;
step M5, the smile service quality evaluation system identifies the mouth corner state of each key point of the mouth corner, and identifies the mouth corner state associated with the service staff during the process of shooting the human body image;
step M6, the smile service quality evaluation system performs image matching on each identified mouth corner state and a preset standard mouth corner state to obtain a mouth corner state compliance judgment result for the service staff during the process of shooting the human body image;
and step M7, the smile service quality evaluation system integrates the limb action compliance judgment result obtained in the step M4 and the mouth angle state compliance judgment result obtained in the step M6, judges whether the smile service quality of the service staff during the human body image shooting is compliant, and finally forms and outputs the smile service quality evaluation result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911166917.8A CN110895685A (en) | 2019-11-25 | 2019-11-25 | Smile service quality evaluation system and evaluation method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911166917.8A CN110895685A (en) | 2019-11-25 | 2019-11-25 | Smile service quality evaluation system and evaluation method based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110895685A true CN110895685A (en) | 2020-03-20 |
Family
ID=69787959
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911166917.8A Withdrawn CN110895685A (en) | 2019-11-25 | 2019-11-25 | Smile service quality evaluation system and evaluation method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110895685A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112348390A (en) * | 2020-11-17 | 2021-02-09 | 广州顶新时代电子系统科技有限公司 | Intelligent assessment system based on service quality |
| CN113642503A (en) * | 2021-08-23 | 2021-11-12 | 国网山东省电力公司金乡县供电公司 | Window service scoring method and system based on image and voice recognition |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102007499A (en) * | 2008-01-29 | 2011-04-06 | 泰塞拉技术爱尔兰公司 | Detecting facial expressions in digital images |
| CN103098078A (en) * | 2010-09-13 | 2013-05-08 | 惠普发展公司，有限责任合伙企业 | Smile detection systems and methods |
| CN105608447A (en) * | 2016-02-17 | 2016-05-25 | 陕西师范大学 | Method for detecting human face smile expression depth convolution nerve network |
| CN106096831A (en) * | 2016-06-08 | 2016-11-09 | 山西万立科技有限公司 | Expressway Civilization services overall evaluation system |
| US20180039821A1 (en) * | 2016-08-03 | 2018-02-08 | Microsoft Technology Licensing, Llc | Combining images aligned to reference frame |
| CN107820018A (en) * | 2017-11-30 | 2018-03-20 | 广东欧珀移动通信有限公司 | User's photographic method, device and equipment |
| CN108229855A (en) * | 2018-02-06 | 2018-06-29 | 上海小蚁科技有限公司 | Method for monitoring service quality and device, computer readable storage medium, terminal |
| CN108694372A (en) * | 2018-03-23 | 2018-10-23 | 广东亿迅科技有限公司 | A kind of net cast customer service attitude evaluation method and device |
| CN108830633A (en) * | 2018-04-26 | 2018-11-16 | 华慧视科技（天津）有限公司 | A kind of friendly service evaluation method based on smiling face's detection |
| CN109033377A (en) * | 2018-07-27 | 2018-12-18 | 重庆柚瓣家科技有限公司 | A kind of chat system and method that auxiliary is social |
| CN109472198A (en) * | 2018-09-28 | 2019-03-15 | 武汉工程大学 | A kind of video smiling face's recognition methods of attitude robust |
| CN109858379A (en) * | 2019-01-03 | 2019-06-07 | 深圳壹账通智能科技有限公司 | Smile's sincerity degree detection method, device, storage medium and electronic equipment |
| CN109871751A (en) * | 2019-01-04 | 2019-06-11 | 平安科技（深圳）有限公司 | Attitude appraisal procedure, device and storage medium based on facial expression recognition |
-
2019
- 2019-11-25 CN CN201911166917.8A patent/CN110895685A/en not_active Withdrawn
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102007499A (en) * | 2008-01-29 | 2011-04-06 | 泰塞拉技术爱尔兰公司 | Detecting facial expressions in digital images |
| CN103098078A (en) * | 2010-09-13 | 2013-05-08 | 惠普发展公司，有限责任合伙企业 | Smile detection systems and methods |
| CN105608447A (en) * | 2016-02-17 | 2016-05-25 | 陕西师范大学 | Method for detecting human face smile expression depth convolution nerve network |
| CN106096831A (en) * | 2016-06-08 | 2016-11-09 | 山西万立科技有限公司 | Expressway Civilization services overall evaluation system |
| US20180039821A1 (en) * | 2016-08-03 | 2018-02-08 | Microsoft Technology Licensing, Llc | Combining images aligned to reference frame |
| CN107820018A (en) * | 2017-11-30 | 2018-03-20 | 广东欧珀移动通信有限公司 | User's photographic method, device and equipment |
| CN108229855A (en) * | 2018-02-06 | 2018-06-29 | 上海小蚁科技有限公司 | Method for monitoring service quality and device, computer readable storage medium, terminal |
| CN108694372A (en) * | 2018-03-23 | 2018-10-23 | 广东亿迅科技有限公司 | A kind of net cast customer service attitude evaluation method and device |
| CN108830633A (en) * | 2018-04-26 | 2018-11-16 | 华慧视科技（天津）有限公司 | A kind of friendly service evaluation method based on smiling face's detection |
| CN109033377A (en) * | 2018-07-27 | 2018-12-18 | 重庆柚瓣家科技有限公司 | A kind of chat system and method that auxiliary is social |
| CN109472198A (en) * | 2018-09-28 | 2019-03-15 | 武汉工程大学 | A kind of video smiling face's recognition methods of attitude robust |
| CN109858379A (en) * | 2019-01-03 | 2019-06-07 | 深圳壹账通智能科技有限公司 | Smile's sincerity degree detection method, device, storage medium and electronic equipment |
| CN109871751A (en) * | 2019-01-04 | 2019-06-11 | 平安科技（深圳）有限公司 | Attitude appraisal procedure, device and storage medium based on facial expression recognition |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112348390A (en) * | 2020-11-17 | 2021-02-09 | 广州顶新时代电子系统科技有限公司 | Intelligent assessment system based on service quality |
| CN113642503A (en) * | 2021-08-23 | 2021-11-12 | 国网山东省电力公司金乡县供电公司 | Window service scoring method and system based on image and voice recognition |
| CN113642503B (en) * | 2021-08-23 | 2024-03-15 | 国网山东省电力公司金乡县供电公司 | Window service scoring method and system based on image and voice recognition |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111563487B (en) | Dance scoring method based on gesture recognition model and related equipment | |
| US10068128B2 (en) | Face key point positioning method and terminal | |
| CN104361276B (en) | A kind of multi-modal biological characteristic identity identifying method and system | |
| CN105809144B (en) | A kind of gesture recognition system and method using movement cutting | |
| WO2019127262A1 (en) | Cloud end-based human face in vivo detection method, electronic device and program product | |
| Aran et al. | Signtutor: An interactive system for sign language tutoring | |
| CN105518708A (en) | Method and equipment for verifying living human face, and computer program product | |
| EP3716159A1 (en) | Multi-modal emotion recognition device, method and storage medium using artificial intelligence | |
| TWI621999B (en) | Method for face detection | |
| CN110895685A (en) | Smile service quality evaluation system and evaluation method based on deep learning | |
| WO2020073403A1 (en) | Silent voice input identification method, computing apparatus, and computer-readable medium | |
| CN110348378A (en) | A kind of authentication method, device and storage medium | |
| CN111291613A (en) | Classroom performance evaluation method and system | |
| CN112651714A (en) | Interview evaluation method and system based on multi-mode information | |
| CN112286364A (en) | Man-machine interaction method and device | |
| CN112149553A (en) | Examination cheating behavior identification method | |
| KR101747712B1 (en) | interview auto recognizetion real-time management method by smart phone | |
| CN109345427B (en) | Classroom video frequency point arrival method combining face recognition technology and pedestrian recognition technology | |
| CN112149517A (en) | Face attendance checking method and system, computer equipment and storage medium | |
| CN109087694B (en) | Method for helping students to exercise bodies and family education equipment | |
| CN108960191B (en) | Multi-mode fusion emotion calculation method and system for robot | |
| CN113837112A (en) | Video data processing method and electronic equipment | |
| CN113255535A (en) | Depression identification method based on micro-expression analysis | |
| CN113343948A (en) | Satisfaction degree analysis method and system and computer storage medium thereof | |
| CN114466179A (en) | Method and device for measuring synchronism of voice and image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200320 |