WO2020119630A1 - 一种多模态客户满意度综合评价系统、方法 - Google Patents

一种多模态客户满意度综合评价系统、方法 Download PDF

Info

Publication number
WO2020119630A1
WO2020119630A1 PCT/CN2019/124007 CN2019124007W WO2020119630A1 WO 2020119630 A1 WO2020119630 A1 WO 2020119630A1 CN 2019124007 W CN2019124007 W CN 2019124007W WO 2020119630 A1 WO2020119630 A1 WO 2020119630A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
evaluation
satisfaction
facial expression
customer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/124007
Other languages
English (en)
French (fr)
Inventor
沈志娟
程俊
胡希平
王磊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Institute of Advanced Technology of CAS
Publication of WO2020119630A1 publication Critical patent/WO2020119630A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G06Q40/02—Banking, e.g. interest calculation or account maintenance

Definitions

  • the invention relates to the field of intelligent analysis technology, in particular to a multi-modal customer satisfaction comprehensive evaluation system and method.
  • the main purpose of the present invention is to propose a multi-modal customer satisfaction comprehensive evaluation system and method, aiming to solve the technical problem that the traditional customer satisfaction evaluation method is too singular and has low objectivity.
  • a multimodal customer satisfaction comprehensive evaluation system provided by the present invention includes:
  • the data acquisition module acquires at least one or more of the posture information, facial expression information, text information, and voice information of the customer's body;
  • the user sentiment analysis and evaluation module performs user sentiment analysis based on one or more items of information obtained by the data acquisition module and obtains a corresponding satisfaction score
  • the information fusion module is based on one or more satisfaction scores analyzed and obtained based on user emotions and evaluation analysis modules to obtain final evaluation information.
  • the user emotion analysis and evaluation module includes one or more of a human posture evaluation module, a facial expression evaluation module, an information evaluation module, and a voice information evaluation module;
  • the human posture evaluation module is used to detect and identify the client's body movements corresponding to different emotions, and correspondingly obtain corresponding posture satisfaction scores;
  • the facial expression evaluation module is used for detecting and recognizing facial expression actions of customers corresponding to different emotions, and correspondingly obtaining corresponding facial expression satisfaction scores;
  • the information evaluation module is used to obtain a text information satisfaction score corresponding to the score option manually clicked by the customer;
  • the voice information evaluation module is used for detecting and recognizing voices of customers corresponding to different emotions, and correspondingly obtaining corresponding voice information satisfaction scores.
  • the information fusion module gives one or more pieces of information of human posture information, facial expression information, text information, and voice information, assigns different weight levels, and obtains a comprehensive score by weighting.
  • the weighted levels from high to low are: text information, facial expression information, voice information, and human posture information.
  • the satisfaction score of the human posture evaluation module and the facial expression evaluation module includes at least 3 levels;
  • the satisfaction score of the text information evaluation module includes at least 6 levels
  • the satisfaction score of the voice information evaluation module includes at least 4 scores.
  • the present invention also proposes a multimodal customer satisfaction comprehensive evaluation method, the method includes:
  • the final evaluation information is obtained.
  • the user emotion analysis is performed based on one or more pieces of information acquired by the data acquisition module, and acquiring corresponding scores includes a body posture evaluation acquisition step, a facial expression evaluation acquisition step, an information evaluation acquisition step, and voice information Evaluate one or more of the acquisition steps;
  • the human body posture evaluation obtaining step is used to detect and identify the client's body movements corresponding to different emotions, and correspondingly obtain corresponding posture satisfaction scores;
  • the facial expression evaluation and acquisition step is used to detect and identify facial expression actions of customers corresponding to different emotions, and correspondingly obtain corresponding facial expression satisfaction scores;
  • the information evaluation obtaining step is used to obtain the text information satisfaction score corresponding to the scoring option manually clicked by the customer;
  • the voice information evaluation obtaining step is used to detect and recognize the voices of the customers corresponding to different emotions, and correspondingly obtain corresponding voice information satisfaction scores.
  • the one or more pieces of evaluation information obtained through analysis and analysis based on the user emotion and evaluation analysis module to obtain the final evaluation information include:
  • the weighted levels from high to low are: text information, facial expression information, voice information, and human posture information.
  • the satisfaction score of the human body posture evaluation obtaining step and the facial expression evaluation obtaining step includes at least 3 levels;
  • the satisfaction score of the text information evaluation acquisition step includes at least 6 levels
  • the satisfaction score of the voice information evaluation obtaining step includes at least 4 scores.
  • the multi-modal customer satisfaction comprehensive evaluation system and method proposed by the present invention acquire at least one or more of the human body posture information, facial expression information, text information and voice information of the customer through collection; through detection and recognition corresponding to the human body posture Satisfaction score of information, facial expression information, text information and voice information, and further based on the weighted fusion algorithm to achieve the fusion of one or more satisfaction score information of human posture information, facial expression information, text information and voice information, thereby Obtain the final comprehensive evaluation information, avoiding the influence of a single index on the final evaluation information, and assign different weights to different indexes according to user habits and actual information collection to achieve a more objective and accurate comprehensive evaluation of final satisfaction .
  • FIG. 2 is a diagram of a method for acquiring data by the data acquisition module in the multi-modal customer satisfaction comprehensive evaluation system of the present invention
  • FIG. 3 is a flow chart of human body posture evaluation in the multi-modal customer satisfaction comprehensive evaluation system of the present invention.
  • FIG. 6 is a flowchart of voice content recognition in the voice information evaluation module in the multi-modal customer satisfaction comprehensive evaluation system of the present invention.
  • a multi-modal customer satisfaction comprehensive evaluation system includes:
  • the data acquisition module acquires at least one or more of the posture information, facial expression information, text information, and voice information of the customer's body;
  • the data acquisition method is shown in Figure 2.
  • the data acquisition follows the principle of easy access, real and effective.
  • the existing surveillance cameras in the bank lobby, as well as the camera at the smart counter, touch screen and microphone are selected. There are four existing or The easy-to-implement acquisition method.
  • the video of the customer's body posture is obtained from the surveillance camera in the bank lobby, and the facial expression video of the customer is obtained from the camera at the smart counter.
  • the text message of the satisfaction score selected by the customer is obtained from the touch screen of the smart counter , And the microphone at the smart counter to obtain voice messages of customers' suggestions and opinions on the bank.
  • the user sentiment analysis and evaluation module performs user sentiment analysis based on one or more items of information obtained by the data acquisition module and obtains a corresponding satisfaction score
  • the information fusion module is based on one or more satisfaction scores analyzed and obtained based on user emotions and evaluation analysis modules to obtain final evaluation information.
  • the user emotion analysis and evaluation module includes one or more of a human posture evaluation module, a facial expression evaluation module, an information evaluation module, and a voice information evaluation module;
  • the human body posture evaluation module is used to detect and identify the customer's limb movements corresponding to different emotions, and correspondingly obtain corresponding posture satisfaction scores;
  • the satisfaction score of the human posture evaluation module includes at least 3 levels;
  • the human body posture evaluation module in the multimodal customer satisfaction comprehensive evaluation system of the present invention executes human body posture information for satisfaction evaluation.
  • human body posture evaluation module For the unmanned bank customer satisfaction evaluation system, it is the main task of the human body posture evaluation module to detect whether the customer has suspicious, angry, or very angry body postures. When the customer is in a neutral emotional action, 100 points will be scored; if there is a doubtful action, 80 points will be recorded; if the customer is angry, 40 points will be recorded; if he is very angry, 20 points will be recorded.
  • the human body posture evaluation module first performs human body detection from the customer's human body posture video to find the customer. Then, extract the relative position of the hand and the center of gravity, the relative position of the hand and the head, etc., and match the features extracted from the action database mainly based on doubt and anger to obtain the results of the action recognition and analyze whether the customer has doubts and anger Actions.
  • the human posture evaluation module also extracts features such as hand movement speed and angular velocity from the customer's human posture video, recognizes the degree of customer anger, and builds an emotion database. Combined with the action recognition results, scientifically and objectively obtain the human body evaluation results.
  • the facial expression evaluation module is configured to detect and recognize facial expression actions of customers corresponding to different emotions, and correspondingly obtain corresponding facial expression satisfaction scores;
  • the satisfaction score of the facial expression evaluation module includes at least 3 levels
  • Facial expression refers to various emotions expressed through the changes of various facial organs such as eyes, forehead, nose, cheeks, lips and other muscles, and it can express emotions of different natures in detail. Facial expressions are used in an unmanned bank customer satisfaction evaluation system, which can not only reflect the customer's overall satisfaction with the bank's service, but also quickly locate the part of the service that needs improvement according to the changes in customer emotions.
  • the facial expression evaluation module of the present invention scores 100 points when the customer shows positive and neutral emotions; when the customer shows negative emotions, such as doubt, anxiety, and anger, according to the proportion and intensity of the duration of the negative emotions, the gradient is 20 points in sequence Points are deducted, the scores are: 80 points, 60 points, 40 points, 20 points, the lowest point is 0 points.
  • the facial expression evaluation module of the present invention adopts the currently effective deep learning method.
  • FIG. 4 is a flowchart of the facial expression evaluation in the multi-modal customer satisfaction comprehensive evaluation system of the present invention.
  • the preprocessing includes face detection, face alignment, data enhancement, and normalization. There are four steps. Because the distance between the camera of the smart counter and the customer is basically fixed, and the face size of the collected video is basically the same, the present invention uses the sliding window method to detect the face. After that, the SDM algorithm (supervised descent method) is used to detect the key points of the face and perform the face alignment operation.
  • the data enhancement part uses scale change, rotation, color change, noise interference and other methods to obtain enhanced data.
  • the normalization is divided into two parts: brightness normalization and attitude normalization.
  • the brightness normalization adopts the histogram normalization method
  • the attitude normalization adopts the TP-GAN network (dual path generation adversarial network).
  • Facial expression recognition is currently a hotspot in the field of artificial intelligence research.
  • the present invention adopts the CK+ data set (extended Cohn-Kanade data set) to obtain a model pre-trained, which is used to recognize the facial expressions of customers, and finally analyze the facial expressions of customers.
  • the information evaluation module is used to obtain a text information satisfaction score corresponding to the score option manually clicked by the customer;
  • the satisfaction score of the text information evaluation module includes at least 6 levels
  • the text information evaluation is based on the questions in the traditional customer satisfaction questionnaire, which can be selected by the customer by clicking the corresponding option at the touch screen of the smart counter. Customers are requested to evaluate this service, which is divided into “very satisfied”, “satisfied”, “basic satisfaction”, “general”, “unsatisfied”, and “very dissatisfied”, a total of six types of evaluation.
  • the text information evaluation module records the above six evaluations as 100 points, 90 points, 80 points, 60 points, 40 points, and 20 points, respectively.
  • the voice information evaluation module is configured to detect and recognize voices of customers corresponding to different emotions, and obtain corresponding voice information satisfaction scores accordingly.
  • the satisfaction score of the voice information evaluation module includes at least 4 scores.
  • the voice information evaluation module of the present invention scores 100 points when the customer shows positive and neutral emotions; when the customer shows negative emotions, such as doubt, anxiety, and anger, according to the proportion and intensity of the duration of the negative emotions, the gradient is 20 points in sequence Points are deducted, the scores are: 80 points, 60 points, 40 points, 20 points, the lowest point is 0 points.
  • the voice information is collected by the smart counter microphone, mainly after the customer clicks on the service evaluation of the touch screen to collect the user's opinions and suggestions. The tedious problem of manual input by the customer is avoided, and the evaluation time is reduced.
  • the voice information evaluation module mainly includes two units of voice content recognition and voice sentiment analysis.
  • Speech content recognition technology and speech emotion recognition technology are relatively mature technologies at this stage.
  • Speech content recognition technology is a technology that allows machines to convert speech signals into corresponding text or commands through recognition and understanding.
  • 6 is a flowchart of the voice information evaluation in the multi-modal customer satisfaction comprehensive evaluation system of the present invention; the voice content recognition unit is essentially a pattern recognition system. It mainly includes four parts: signal preprocessing, feature extraction, pattern matching, and reference pattern library.
  • the invention directly uses the pre-trained Mandarin speech frame data (Mandarin Speech Frame Data) as the reference pattern library, which saves the model training part.
  • the preprocessing part filters out the secondary information and background noise in the original voice signal, including anti-aliasing filtering, pre-emphasis, analog-to-digital conversion, automatic gain control, etc.
  • the feature extraction part analyzes the acoustic parameters of speech to extract speech feature parameters, including short-term average amplitude, short-term average energy, linear prediction coding coefficients, short-term frequency spectrum, etc.
  • the pattern matching part the current representative hidden Markov method is selected, and the analyzed characteristic parameters are pattern-matched with the template in Mandarin speech frame data (Mandarin Speech Frame Data) to obtain the recognition result.
  • the recognition result judges whether the customer has suggestions on the one hand, and is used to record the opinions and suggestions of the customer on the other hand, which is convenient for the improvement of bank services.
  • the basic structure of the speech emotion analysis unit is consistent with the speech content recognition unit, and the reference template library can be transposed to the pre-trained speech emotion template library.
  • the information fusion module gives one or more pieces of information of human posture information, facial expression information, text information, and voice information, assigns different weight levels, and obtains a comprehensive score by weighting.
  • the invention adopts decision layer fusion.
  • Decision layer fusion is to preprocess, feature extract and classify the information of each sub-source first, and establish a preliminary evaluation score for the observed target, and then the fusion center integrates the processing results of each sub-source to obtain the final score result.
  • Decision-level fusion has high flexibility in information processing, can effectively reflect different types of information on the environment and target sides, and can handle asynchronous information.
  • the invention adopts the simplest weighted fusion method.
  • the text information evaluation module is the module that directly reflects customer satisfaction, with the largest weight; the voice information evaluation module and facial expression evaluation module reflect the customer satisfaction to a certain extent, and the weight is second. ;
  • the weight of the human pose evaluation module is the smallest. However, when the customer is in a hurry, it is inevitable that the satisfaction evaluation will not be performed.
  • the present invention also lists the weight of the fusion module in this case.
  • the weighted levels from high to low are: text information, facial expression information, voice information, and human posture information.
  • the weights are assigned as follows: human body posture 10%, facial expression 20%, text information 50%, voice information 20%;
  • the weights are assigned as follows: human posture 10%, facial expression 20%, text information 70%, voice information 0%;
  • the weights are assigned as follows: human body posture 30%, facial expression 70%, text information 0%, voice information 0%.
  • the data obtaining step at least one or more items of the client's body posture information, facial expression information, text information, and voice information are obtained;
  • the data acquisition method is shown in Figure 2.
  • the data acquisition follows the principle of easy access, real and effective.
  • the existing surveillance cameras in the bank lobby, as well as the camera at the smart counter, touch screen and microphone are selected. There are four existing or The easy-to-implement acquisition method.
  • the video of the customer's human posture is obtained from the surveillance camera in the bank lobby, and the facial expression video of the customer is obtained from the camera at the smart counter.
  • the text message of the satisfaction score selected by the customer is obtained from the touch screen of the smart counter , And the microphone at the smart counter to obtain voice messages of customers' suggestions and opinions on the bank.
  • user emotion analysis and evaluation step user emotion analysis is performed based on one or more pieces of information obtained in the data acquisition step, and a corresponding satisfaction score is obtained;
  • one or more satisfaction scores are analyzed and obtained based on user emotions and evaluation analysis steps to obtain final evaluation information.
  • the user emotion analysis and evaluation step includes one or more of a human posture evaluation step, a facial expression evaluation step, an information evaluation step, and a voice information evaluation step;
  • the human body posture evaluation step is used to detect and recognize the customer's limb movements corresponding to different emotions, and correspondingly obtain a corresponding posture satisfaction score;
  • the satisfaction score of the human body posture evaluation step includes at least 3 levels;
  • the human body posture evaluation step in the multimodal customer satisfaction comprehensive evaluation system of the present invention executes human body posture information for satisfaction evaluation.
  • the unmanned bank customer satisfaction evaluation system it is the main task of the human body posture evaluation step to detect whether the customer has a suspicious, angry, or very angry posture.
  • 100 points will be scored; if there is a doubtful action, 80 points will be recorded; if the customer is angry, 40 points will be recorded; if he is very angry, 20 points will be recorded.
  • the human body posture evaluation step first performs human body detection from the customer's human body posture video to find the customer. Then, extract the relative position of the hand and the center of gravity, the relative position of the hand and the head, etc., and match the features extracted from the action database mainly based on doubt and anger to obtain the results of the action recognition and analyze whether the customer has doubts and anger Actions.
  • the human body posture evaluation step also extracts hand movement speed, angular velocity and other features from the customer's human body posture video, recognizes the degree of customer anger, and builds an emotion database. Combined with the action recognition results, scientifically and objectively obtain the human body evaluation results.
  • the facial expression evaluation step is used to detect and identify facial expression actions of the customer corresponding to different emotions, and correspondingly obtain a corresponding facial expression satisfaction score;
  • the satisfaction score of the facial expression evaluation step includes at least 3 levels;
  • Facial expression refers to various emotions expressed through the changes of various facial organs such as eyes, forehead, nose, cheeks, lips and other muscles, and it can express emotions of different natures in detail. Facial expressions are used in an unmanned bank customer satisfaction evaluation system, which can not only reflect the customer's overall satisfaction with the bank's service, but also quickly locate the part of the service that needs improvement according to the changes in customer emotions.
  • the facial expression evaluation step of the present invention when the client shows positive and neutral emotions, 100 points are recorded; when the client has negative emotions, such as doubt, anxiety, and anger, according to the proportion and intensity of the duration of the negative emotions, the gradient is 20 points in sequence Points are deducted, the scores are: 80 points, 60 points, 40 points, 20 points, the lowest point is 0 points.
  • the facial expression evaluation step of the present invention adopts the currently effective deep learning method.
  • FIG. 4 is a flowchart of the facial expression evaluation in the multi-modal customer satisfaction comprehensive evaluation system of the present invention.
  • the preprocessing includes face detection, face alignment, data enhancement, and normalization. There are four steps. Because the distance between the camera of the smart counter and the customer is basically fixed, and the face size of the collected video is basically the same, the present invention uses the sliding window method to detect the face. After that, the SDM algorithm (supervised descent method) is used to detect the key points of the face and perform the face alignment operation.
  • the data enhancement part uses scale change, rotation, color change, noise interference and other methods to obtain enhanced data.
  • the normalization is divided into two parts: brightness normalization and attitude normalization.
  • the brightness normalization adopts the histogram normalization method
  • the attitude normalization adopts the TP-GAN network (dual path generation adversarial network).
  • Facial expression recognition is currently a hotspot in the field of artificial intelligence research.
  • the present invention adopts the CK+ data set (extended Cohn-Kanade data set) to obtain a model pre-trained, which is used to recognize the facial expressions of customers, and finally analyze the facial expressions of customers.
  • the information evaluation step is used to obtain a text information satisfaction score corresponding to the scoring option manually clicked by the customer;
  • the satisfaction score of the text information evaluation step includes at least 6 levels
  • the text information evaluation is based on the questions in the traditional customer satisfaction questionnaire, which can be selected by the customer by clicking the corresponding option at the touch screen of the smart counter. Customers are requested to evaluate this service, which is divided into “very satisfied”, “satisfied”, “basic satisfaction”, “general”, “unsatisfied”, and “very dissatisfied”, a total of six types of evaluation.
  • the text information evaluation step records the above six evaluations as 100 points, 90 points, 80 points, 60 points, 40 points, and 20 points, respectively.
  • the voice information evaluation step is used to detect and recognize voices of customers corresponding to different emotions, and correspondingly obtain corresponding voice information satisfaction scores.
  • the satisfaction score of the voice information evaluation step includes at least 4 scores.
  • the voice information evaluation step of the present invention when the customer shows positive and neutral emotions, 100 points are recorded; when the customer shows negative emotions, such as doubt, anxiety, and anger, according to the proportion and intensity of the duration of the negative emotions, the gradient is 20 points in sequence Points are deducted, the scores are: 80 points, 60 points, 40 points, 20 points, the lowest point is 0 points.
  • the voice information is collected by the smart counter microphone, mainly after the customer clicks on the service evaluation of the touch screen to collect the user's opinions and suggestions. The tedious problem of manual input by the customer is avoided, and the evaluation time is reduced.
  • the speech information evaluation step mainly includes two steps of speech content recognition and speech sentiment analysis.
  • Speech content recognition technology and speech emotion recognition technology are relatively mature technologies at this stage.
  • Speech content recognition technology is a technology that allows machines to convert speech signals into corresponding text or commands through recognition and understanding.
  • 6 is a flowchart of the voice information evaluation in the multi-modal customer satisfaction comprehensive evaluation system of the present invention; the voice content recognition unit is essentially a pattern recognition system. It mainly includes four parts: signal preprocessing, feature extraction, pattern matching, and reference pattern library.
  • the invention directly uses the pre-trained Mandarin speech frame data (Mandarin Speech Frame Data) as the reference pattern library, which saves the model training part.
  • the preprocessing part filters out the secondary information and background noise in the original voice signal, including anti-aliasing filtering, pre-emphasis, analog-to-digital conversion, automatic gain control, etc.
  • the feature extraction part analyzes the acoustic parameters of speech to extract speech feature parameters, including short-term average amplitude, short-term average energy, linear prediction coding coefficients, short-term frequency spectrum, etc.
  • the pattern matching part the current representative hidden Markov method is selected, and the analyzed characteristic parameters are pattern-matched with the template in Mandarin speech frame data (Mandarin Speech Frame Data) to obtain the recognition result.
  • the recognition result on the one hand judges whether the customer has suggestions, on the other hand is used to record the customer's opinions and suggestions, which is convenient for the improvement of bank services.
  • the basic structure of the speech emotion analysis step is consistent with the speech content recognition unit, and the reference template library can be transposed to pre-train the speech emotion template library.
  • one or more pieces of information of human body posture information, facial expression information, text information, and voice information are assigned with different weight levels, and weighted to obtain a comprehensive score.
  • the invention adopts decision layer fusion.
  • Decision layer fusion is to preprocess, feature extract and classify the information of each sub-source first, and establish a preliminary evaluation score for the observed target, and then the fusion center integrates the processing results of each sub-source to obtain the final score result.
  • Decision-level fusion has high flexibility in information processing, can effectively reflect different types of information on the environment and target sides, and can handle asynchronous information.
  • the present invention adopts the simplest weighted fusion method.
  • the text information evaluation step is the most direct step of customer satisfaction, with the largest weight; the voice information evaluation step and facial expression evaluation step, because they reflect customer satisfaction to a certain extent, the weight is second ;
  • the weight of the human body posture evaluation step is the smallest. However, when the customer is in a hurry, it is inevitable that the satisfaction evaluation will not be performed.
  • the present invention also lists the weight of fusion in this case.
  • the weighted levels from high to low are: text information, facial expression information, voice information, and human posture information.
  • the weights are assigned as follows: human body posture 10%, facial expression 20%, text information 50%, voice information 20%;
  • the weights are assigned as follows: human posture 10%, facial expression 20%, text information 70%, voice information 0%;
  • the weights are assigned as follows: human body posture 30%, facial expression 70%, text information 0%, voice information 0%.
  • the multi-modal customer satisfaction comprehensive evaluation system and method proposed by the present invention acquire at least one or more of the human body posture information, facial expression information, text information and voice information of the customer through collection; through detection and recognition corresponding to the human body posture Satisfaction score of information, facial expression information, text information and voice information, and further based on the weighted fusion algorithm to achieve the fusion of one or more satisfaction score information of human posture information, facial expression information, text information and voice information, thereby Obtain the final comprehensive evaluation information, avoiding the influence of a single index on the final evaluation information, and assign different weights to different indexes according to user habits and actual information collection to achieve a more objective and accurate comprehensive evaluation of final satisfaction .
  • each functional unit in each embodiment of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units may be integrated into one module.
  • the above integrated modules may be implemented in the form of hardware or software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
  • the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program code.
  • a computer usable storage media including but not limited to disk storage and optical storage, etc.

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Strategic Management (AREA)
  • Development Economics (AREA)
  • General Physics & Mathematics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • Game Theory and Decision Science (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Technology Law (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

一种多模态客户满意度综合评价系统、方法，属于智能分析技术领域，数据获取模块，至少获取客户人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项；用户情绪分析与评价模块，基于所述数据获取模块获取的一项或多项信息执行用户情绪分析，并获取相应的满意度评分；信息融合模块，基于用户情绪与评价分析模块进行分析获取的一项或多项满意度评分，获取最终评价信息；从而避免了单一化的指标对于最终评价信息的影响，且根据用户习惯以及实际信息采集情况给予不同的指标分配不同的权重从而更加客观、精准地实现最终的满意度综合评价。

Description

一种多模态客户满意度综合评价系统、方法 技术领域
本发明涉及智能分析技术领域，尤其涉及一种多模态客户满意度综合评价系统、方法。
背景技术
目前，随着人工智能技术的飞速发展，智能柜台正逐步取代银行柜员，成为未来银行的主要“服务者”。银行作为一种服务与盈利相结合的企业。一方面，客户满意度体现银行服务的价值与质量，其调查结果可以对银行服务的改进提供依据，作为市场驱动质量方法的一种手段。另一方面，无人银行因为其“无人”的特性，客户满意度的调研手段与传统方式有很大不同。无扰、便捷的数据获取方式，智能、合理的评价体系，是无人银行客户满意度评价系统的关键。传统银行的客户满意度调查，是通过客户口头反馈和和问卷调查的形式实现的。调查花费人力物力财力巨大，效率和准确率不高，还不一定适用于无人银行。
传统银行的客户满意度调查，一般通过客户口头反馈和和问卷调查的形式实现的。此类调查一般会花费巨大的人力物力财力，但是效率和准确率不高，数据人为主观因素很大；而现有技术存在单模态的面部表情信号并不能准确评价客户的满意度，一方面，人的面部表情数据可能因为被遮挡等无法获取，或者数据质量很差，导致无法分析出客户的满意度；另一方面，仅从面部表情分析，容易忽略情绪强度对满意度的影响，客观性、准确性低。
发明内容
本发明的主要目的在于提出一种多模态客户满意度综合评价系统、方法，旨在解决传统的客户满意度评价方式过于单一、客观性较低的技术问题。
为实现上述目的，本发明提供的一种多模态客户满意度综合评价系统，包括：
数据获取模块，至少获取客户人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项；
用户情绪分析与评价模块，基于所述数据获取模块获取的一项或多项信息执行用户情绪分析，并获取相应的满意度评分；
信息融合模块，基于用户情绪与评价分析模块进行分析获取的一项或多项满意度评分，获取最终评价信息。
可选地，所述用户情绪分析与评价模块包括人体姿态评价模块、面部表情评价模块、信息评价模块、语音信息评价模块的一项或多项；
所述人体姿态评价模块，用于检测并识别客户对应于不同情绪的肢体动作，并对应获取相应的姿态满意度评分；
所述面部表情评价模块，用于检测并识别客户对应于不同情绪的面部表情动作，并对应获取相应的面部表情满意度评分；
所述信息评价模块，用于获取客户手动点击的评分选项对应的文本信息满意度评分；
所述语音信息评价模块，用于检测并识别客户对应于不同情绪的语音，并对应获取相应的语音信息满意度评分。
可选地，所述信息融合模块，给予人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项信息，分配不同的权重等级，加权获取综合评分。
可选地，所权重等级由高到低依次为：文本信息、面部表情信息、语音信息、人体姿态信息。
可选地，所述人体姿态评价模块、面部表情评价模块的满意度评分至少包括3个等级；
所述文本信息评价模块的满意度评分至少包括6个等级；
所述语音信息评价模块的满意度评分至少包括4个评分格次。
此外，为实现上述目的，本发明还提出一种多模态客户满意度综合评价方法，所述方法包括：
至少获取客户人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项；
基于所述数据获取模块获取的一项或多项信息执行用户情绪分析，并获取相应的满意度评分；
基于用户情绪与评价分析模块进行分析获取的一项或多项满意度评分，获取最终评价信息。
可选地，所述基于所述数据获取模块获取的一项或多项信息执行用户情绪分析，并获取相应的评分包括人体姿态评价获取步骤、面部表情评价获取步骤、信息评价获取步骤、语音信息评价获取步骤的一项或多项；
所述人体姿态评价获取步骤，用于检测并识别客户对应于不同情绪的肢体动 作，并对应获取相应的姿态满意度评分；
所述面部表情评价获取步骤，用于检测并识别客户对应于不同情绪的面部表情动作，并对应获取相应的面部表情满意度评分；
所述信息评价获取步骤，用于获取客户手动点击的评分选项对应的文本信息满意度评分；
所述语音信息评价获取步骤，用于检测并识别客户对应于不同情绪的语音，并对应获取相应的语音信息满意度评分。
可选地，所述基于用户情绪与评价分析模块进行分析获取的一项或多项评价信息，获取最终评价信息，包括：
给予人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项信息，分配不同的权重等级，加权获取综合评分。
可选地，所权重等级由高到低依次为：文本信息、面部表情信息、语音信息、人体姿态信息。
可选地，所述人体姿态评价获取步骤、面部表情评价获取步骤的满意度评分至少包括3个等级；
所述文本信息评价获取步骤的满意度评分至少包括6个等级；
所述语音信息评价获取步骤的满意度评分至少包括4个评分格次。
本发明提出的多模态客户满意度综合评价系统、方法，通过采集获取至少获取客户人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项；通过检测、识别对应于人体姿态信息、面部表情信息、文本信息及语音信息的满意度评分，并进一步基于加权融合算法，实现融合人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项满意度评分信息，从而获取最终的综合评价信息，避免了单一化的指标对于最终评价信息的影响，且根据用户习惯以及实际信息采集情况给予不同的指标分配不同的权重从而更加客观、精准地实现最终的满意度综合评价。
附图说明
图1为本发明所述多模态客户满意度综合评价系统结构以及流程图；
图2为本发明所述多模态客户满意度综合评价系统中所述数据获取模块进行获取数据的途径图；
图3为本发明所述多模态客户满意度综合评价系统中所述人体姿态评价流程图；
图4为本发明所述多模态客户满意度综合评价系统中所述面部表情评价流程图；
图5为本发明所述多模态客户满意度综合评价系统中所述语音信息评价流程图；
图6为本发明所述多模态客户满意度综合评价系统中所述语音信息评价模块中语音内容识别的流程图；
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
实施例一
如图1所示，本发明实施例一种多模态客户满意度综合评价系统，包括：
数据获取模块，至少获取客户人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项；
相应的，数据获取途径如图2所示，数据获取遵循容易获得，真实有效的原则，选择了银行大厅已有的监控摄像头，以及智能柜台处的摄像头，触摸屏和麦克风，共四种现有或者改装容易实现的获取方法。
在业务办理过程中，从银行大厅监控摄像头获得客户人体姿态视频，从智能柜台处的摄像头获得客户面部表情视频，在业务办理后，从智能柜台触摸屏处获得由客户选择的满意度评分的文本信息，以及智能柜台处的麦克风获得客户对银行建议及意见的语音信息。
用户情绪分析与评价模块，基于所述数据获取模块获取的一项或多项信息执行用户情绪分析，并获取相应的满意度评分；
信息融合模块，基于用户情绪与评价分析模块进行分析获取的一项或多项满意度评分，获取最终评价信息。
可选地，所述用户情绪分析与评价模块包括人体姿态评价模块、面部表情评价模块、信息评价模块、语音信息评价模块的一项或多项；
可选地，如图3所示，所述人体姿态评价模块，用于检测并识别客户对应于不同情绪的肢体动作，并对应获取相应的姿态满意度评分；
可选地，所述人体姿态评价模块的满意度评分至少包括3个等级；
相应的，本发明所述多模态客户满意度综合评价系统中所述人体姿态评价模块执行人体姿态信息进行满意度评价。对于无人银行客户满意度评价系统，检测客户是否出现疑惑、生气、非常生气的身体姿态，是人体姿态评价模块的主要任务。当客户为中性情绪的动作时，记100分；出现疑惑的动作，记80分；客户出现生气的动作，记40分；非常生气，记20分。
人在疑惑时，通常会做出手托腮，摸下巴，摸头等动作。人在生气时，通常会做出握拳，快速捶打桌面等动作。因此，人体姿态评价模块从客户人体姿态视频中先进行人体检测，找到客户。之后提取手部与重心的相对位置，手部与头部的相对位置等特征，与以疑惑、生气为主的动作数据库提取的特征进行匹配，获得动作识别的结果，分析客户是否有疑惑、生气的动作。人体姿态评价模块还从客户人体姿态视频中提取手部运动的速度，角速度等特征，识别客户生气的程度，构建情绪数据库。与动作识别结果结合，科学客观地得出人体姿态评价结果。
可选地，如图4所示，所述面部表情评价模块，用于检测并识别客户对应于不同情绪的面部表情动作，并对应获取相应的面部表情满意度评分；
可选地，所述面部表情评价模块的满意度评分至少包括3个等级；
面部表情是指通过面部的各个器官如眼睛、额眉、鼻颊、口唇等肌肉变化所表现出来的各种情绪,它能精细的表达出不同性质的情绪。面部表情用于无人银行客户满意度评价系统，既能反映出客户对银行此次服务的总体满意度，又能根据客户情绪的变化，快速定位到服务中需要改进的部分。本发明面部表情评价模块，当客户表现积极和中性情绪，记100分；当客户出现消极情绪，如疑惑，焦虑，生气时，根据消极情绪的时长占比和强度，以20分为梯度依次扣分，格次分别为：80分，60分，40分，20分，最低分0分。本发明面部表情评价模块采用目前效果很好的深度学习方法，图4为本发明所述多模态客户满意度综合评价系统中所述面部表情评价流程图。
获得客户面部表情视频后，先进行关键帧检测，检测出带有客户的帧。之后，进行预处理。预处理包括人脸检测，人脸对齐，数据增强，归一化，共四步。因为智能柜台摄像头与客户距离基本固定，采集到的视频人脸大小基本一致，本发明采用滑动窗口法检测人脸。之后，采用SDM算法(监督下降法)检测出人脸 关键点，进行人脸对齐操作。数据增强部分采用了尺度变化，旋转，颜色变化，噪声干扰等方式，获得增强后的数据。归一化分为亮度归一化和姿态归一化两部分，亮度归一化采用直方图归一化法，姿态归一化采用TP-GAN网络(双路径生成对抗网络)。表情识别是目前人工智能领域研究的热点，有很多公开的大型数据集可以用来预训练CNN(卷积神经网络)模型。本发明采用CK+数据集(扩展的Cohn-Kanade数据集)预训练得到模型，用于客户面部表情的识别，最终分析出客户面部表情。
可选地，如图5所示，所述信息评价模块，用于获取客户手动点击的评分选项对应的文本信息满意度评分；
所述文本信息评价模块的满意度评分至少包括6个等级；
文本信息评价基于传统客户满意度问卷表中的问题得到，由客户在智能柜台触摸屏处点击对应选项选择即可。请客户对本次服务进行评价，分为“非常满意”、“满意”、“基本满意”、“一般”、“不满意”、“非常不满意”，共计六种评价。文本信息评价模块分别将上述六种评价记为100分，90分，80分，60分，40分，20分。
可选地，如图6所示，所述语音信息评价模块，用于检测并识别客户对应于不同情绪的语音，并对应获取相应的语音信息满意度评分。
所述语音信息评价模块的满意度评分至少包括4个评分格次。
本发明语音信息评价模块，当客户表现积极和中性情绪，记100分；当客户出现消极情绪，如疑惑，焦虑，生气时，根据消极情绪的时长占比和强度，以20分为梯度依次扣分，格次分别为：80分，60分，40分，20分，最低分0分。
语音信息由智能柜台麦克风收录，主要是在客户点击触摸屏的服务评价之后，收录用户的意见建议。避免了客户手工输入的繁琐问题，减少了评价耗时。所述语音信息评价模块主要包括语音内容识别和语音情绪分析两个单元。
语音内容识别技术和语音情绪识别技术，是现阶段发展较为成熟的技术。
语音内容识别技术，就是让机器通过识别和理解，把语音信号转变为相应的文本或命令的技术。图6为本发明所述多模态客户满意度综合评价系统中所述语音信息评价流程图；语音内容识别单元本质上是一种模式识别系统。主要包括信号预处理、特征提取、模式匹配、参考模式库四个部分。
本发明直接采用预训练好的普通话语音帧数据(Mandarin Speech Frame  Data)作为参考模式库，省去了模型训练部分。其中，预处理部分，滤除原始语音信号中的次要信息及背景噪音等,包括抗混叠滤波、预加重、模/数转换、自动增益控制等。特征提取部分，对语音的声学参数进行分析后提取出语音特征参数,包括短时平均幅度、短时平均能量、线性预测编码系数、短时频谱等。模式匹配部分,选择目前有代表性的隐马尔可夫法，将分析出来的特征参数与普通话语音帧数据(Mandarin Speech Frame Data)中的模板进行模式匹配，得到识别结果。识别结果一方面判断客户是否有建议，另一方面用于记录客户的意见建议，便于银行服务的改进。
语音情绪分析单元的基本结构与语音内容识别单元一致，将参考模板库换位预先训练好的语音情绪模板库即可。
可选地，，所述信息融合模块，给予人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项信息，分配不同的权重等级，加权获取综合评分。
本发明采用决策层融合。决策层融合就是将各子源信息先分别进行预处理、特征提取、分类识别，建立对所观测目标的初步评价得分,然后融合中心对各子源处理的结果进行整合得到最后的得分结果。决策层融合在信息处理方面具有很高的灵活性,能有效地反映环境和目标各个侧面的不同类型信息,而且可以处理异步信息。
本发明采用最简单的加权融合方式，文本信息评价模块是客户满意度最直接体现的模块，权重最大；语音信息评价模块和面部表情评价模块，因其一定程度反应了客户满意度，权重次之；人体姿态评价模块权重最小。但是，客户赶时间之时，难免出现不进行满意度评价的情况，本发明还列出了此情况下融合模块的权重。
可选地，所权重等级由高到低依次为：文本信息、面部表情信息、语音信息、人体姿态信息。
，当具有语音信息满意度评分以及文本信息满意度评分时，权重分配如下：人体姿态10％、面部表情20％、文本信息50％、语音信息20％；
当没有语音信息满意度评分，但具有文本信息满意度评分时，权重分配如下：人体姿态10％、面部表情20％、文本信息70％、语音信息0％；
当没有语音信息满意度评分，且没有文本信息满意度评分时，权重分配如下： 人体姿态30％、面部表情70％、文本信息0％、语音信息0％。
实施例二
本发明实施例一种多模态客户满意度综合评价方法，包括：
数据获取步骤，至少获取客户人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项；
相应的，数据获取途径如图2所示，数据获取遵循容易获得，真实有效的原则，选择了银行大厅已有的监控摄像头，以及智能柜台处的摄像头，触摸屏和麦克风，共四种现有或者改装容易实现的获取方法。
在业务办理过程中，从银行大厅监控摄像头获得客户人体姿态视频，从智能柜台处的摄像头获得客户面部表情视频，在业务办理后，从智能柜台触摸屏处获得由客户选择的满意度评分的文本信息，以及智能柜台处的麦克风获得客户对银行建议及意见的语音信息。
用户情绪分析与评价步骤，基于所述数据获取步骤获取的一项或多项信息执行用户情绪分析，并获取相应的满意度评分；
信息融合步骤，基于用户情绪与评价分析步骤进行分析获取的一项或多项满意度评分，获取最终评价信息。
可选地，所述用户情绪分析与评价步骤包括人体姿态评价步骤、面部表情评价步骤、信息评价步骤、语音信息评价步骤的一项或多项；
可选地，如图3所示，所述人体姿态评价步骤，用于检测并识别客户对应于不同情绪的肢体动作，并对应获取相应的姿态满意度评分；
可选地，所述人体姿态评价步骤的满意度评分至少包括3个等级；
相应的，本发明所述多模态客户满意度综合评价系统中所述人体姿态评价步骤执行人体姿态信息进行满意度评价。对于无人银行客户满意度评价系统，检测客户是否出现疑惑、生气、非常生气的身体姿态，是人体姿态评价步骤的主要任务。当客户为中性情绪的动作时，记100分；出现疑惑的动作，记80分；客户出现生气的动作，记40分；非常生气，记20分。
人在疑惑时，通常会做出手托腮，摸下巴，摸头等动作。人在生气时，通常会做出握拳，快速捶打桌面等动作。因此，人体姿态评价步骤从客户人体姿态视频中先进行人体检测，找到客户。之后提取手部与重心的相对位置，手部与头部 的相对位置等特征，与以疑惑、生气为主的动作数据库提取的特征进行匹配，获得动作识别的结果，分析客户是否有疑惑、生气的动作。人体姿态评价步骤还从客户人体姿态视频中提取手部运动的速度，角速度等特征，识别客户生气的程度，构建情绪数据库。与动作识别结果结合，科学客观地得出人体姿态评价结果。
可选地，如图4所示，所述面部表情评价步骤，用于检测并识别客户对应于不同情绪的面部表情动作，并对应获取相应的面部表情满意度评分；
可选地，所述面部表情评价步骤的满意度评分至少包括3个等级；
面部表情是指通过面部的各个器官如眼睛、额眉、鼻颊、口唇等肌肉变化所表现出来的各种情绪,它能精细的表达出不同性质的情绪。面部表情用于无人银行客户满意度评价系统，既能反映出客户对银行此次服务的总体满意度，又能根据客户情绪的变化，快速定位到服务中需要改进的部分。本发明面部表情评价步骤，当客户表现积极和中性情绪，记100分；当客户出现消极情绪，如疑惑，焦虑，生气时，根据消极情绪的时长占比和强度，以20分为梯度依次扣分，格次分别为：80分，60分，40分，20分，最低分0分。本发明面部表情评价步骤采用目前效果很好的深度学习方法，图4为本发明所述多模态客户满意度综合评价系统中所述面部表情评价流程图。
获得客户面部表情视频后，先进行关键帧检测，检测出带有客户的帧。之后，进行预处理。预处理包括人脸检测，人脸对齐，数据增强，归一化，共四步。因为智能柜台摄像头与客户距离基本固定，采集到的视频人脸大小基本一致，本发明采用滑动窗口法检测人脸。之后，采用SDM算法(监督下降法)检测出人脸关键点，进行人脸对齐操作。数据增强部分采用了尺度变化，旋转，颜色变化，噪声干扰等方式，获得增强后的数据。归一化分为亮度归一化和姿态归一化两部分，亮度归一化采用直方图归一化法，姿态归一化采用TP-GAN网络(双路径生成对抗网络)。表情识别是目前人工智能领域研究的热点，有很多公开的大型数据集可以用来预训练CNN(卷积神经网络)模型。本发明采用CK+数据集(扩展的Cohn-Kanade数据集)预训练得到模型，用于客户面部表情的识别，最终分析出客户面部表情。
可选地，如图5所示，所述信息评价步骤，用于获取客户手动点击的评分选项对应的文本信息满意度评分；
所述文本信息评价步骤的满意度评分至少包括6个等级；
文本信息评价基于传统客户满意度问卷表中的问题得到，由客户在智能柜台触摸屏处点击对应选项选择即可。请客户对本次服务进行评价，分为“非常满意”、“满意”、“基本满意”、“一般”、“不满意”、“非常不满意”，共计六种评价。文本信息评价步骤分别将上述六种评价记为100分，90分，80分，60分，40分，20分。
可选地，如图6所示，所述语音信息评价步骤，用于检测并识别客户对应于不同情绪的语音，并对应获取相应的语音信息满意度评分。
所述语音信息评价步骤的满意度评分至少包括4个评分格次。
本发明语音信息评价步骤，当客户表现积极和中性情绪，记100分；当客户出现消极情绪，如疑惑，焦虑，生气时，根据消极情绪的时长占比和强度，以20分为梯度依次扣分，格次分别为：80分，60分，40分，20分，最低分0分。
语音信息由智能柜台麦克风收录，主要是在客户点击触摸屏的服务评价之后，收录用户的意见建议。避免了客户手工输入的繁琐问题，减少了评价耗时。所述语音信息评价步骤主要包括语音内容识别和语音情绪分析两个步骤。
语音内容识别技术和语音情绪识别技术，是现阶段发展较为成熟的技术。
语音内容识别技术，就是让机器通过识别和理解，把语音信号转变为相应的文本或命令的技术。图6为本发明所述多模态客户满意度综合评价系统中所述语音信息评价流程图；语音内容识别单元本质上是一种模式识别系统。主要包括信号预处理、特征提取、模式匹配、参考模式库四个部分。
本发明直接采用预训练好的普通话语音帧数据(Mandarin Speech Frame Data)作为参考模式库，省去了模型训练部分。其中，预处理部分，滤除原始语音信号中的次要信息及背景噪音等,包括抗混叠滤波、预加重、模/数转换、自动增益控制等。特征提取部分，对语音的声学参数进行分析后提取出语音特征参数,包括短时平均幅度、短时平均能量、线性预测编码系数、短时频谱等。模式匹配部分,选择目前有代表性的隐马尔可夫法，将分析出来的特征参数与普通话语音帧数据(Mandarin Speech Frame Data)中的模板进行模式匹配，得到识别结果。识别结果一方面判断客户是否有建议，另一方面用于记录客户的意见建议，便于银行服务的改进。
语音情绪分析步骤的基本结构与语音内容识别单元一致，将参考模板库换位 预先训练好的语音情绪模板库即可。
可选地，，所述信息融合步骤，给予人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项信息，分配不同的权重等级，加权获取综合评分。
本发明采用决策层融合。决策层融合就是将各子源信息先分别进行预处理、特征提取、分类识别，建立对所观测目标的初步评价得分,然后融合中心对各子源处理的结果进行整合得到最后的得分结果。决策层融合在信息处理方面具有很高的灵活性,能有效地反映环境和目标各个侧面的不同类型信息,而且可以处理异步信息。
本发明采用最简单的加权融合方式，文本信息评价步骤是客户满意度最直接体现的步骤，权重最大；语音信息评价步骤和面部表情评价步骤，因其一定程度反应了客户满意度，权重次之；人体姿态评价步骤权重最小。但是，客户赶时间之时，难免出现不进行满意度评价的情况，本发明还列出了此情况下融合的权重。
可选地，所权重等级由高到低依次为：文本信息、面部表情信息、语音信息、人体姿态信息。
，当具有语音信息满意度评分以及文本信息满意度评分时，权重分配如下：人体姿态10％、面部表情20％、文本信息50％、语音信息20％；
当没有语音信息满意度评分，但具有文本信息满意度评分时，权重分配如下：人体姿态10％、面部表情20％、文本信息70％、语音信息0％；
当没有语音信息满意度评分，且没有文本信息满意度评分时，权重分配如下：人体姿态30％、面部表情70％、文本信息0％、语音信息0％。
本发明提出的多模态客户满意度综合评价系统、方法，通过采集获取至少获取客户人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项；通过检测、识别对应于人体姿态信息、面部表情信息、文本信息及语音信息的满意度评分，并进一步基于加权融合算法，实现融合人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项满意度评分信息，从而获取最终的综合评价信息，避免了单一化的指标对于最终评价信息的影响，且根据用户习惯以及实际信息采集情况给予不同的指标分配不同的权重从而更加客观、精准地实现最终的满意度综合评价。
本领域普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可 读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器和光学存储器等)上实施的计算机程序产品的形式。
显然，本领域的技术人员可以对本发明进行各种改动和变型而不脱离本发明的精神和范围。这样，倘若本发明的这些修改和变型属于本发明权利要求及其等同技术的范围之内，则本发明也意图包含这些改动和变型在内。

Claims (10)

  1. 一种多模态客户满意度综合评价系统，其特征在于，包括：
    数据获取模块，至少获取客户人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项；
    用户情绪分析与评价模块，基于所述数据获取模块获取的一项或多项信息执行用户情绪分析，并获取相应的满意度评分；
    信息融合模块，基于用户情绪与评价分析模块进行分析获取的一项或多项满意度评分，获取最终评价信息。
  2. 根据权利要求1所述的多模态客户满意度综合评价系统，其特征在于，所述用户情绪分析与评价模块包括人体姿态评价模块、面部表情评价模块、信息评价模块、语音信息评价模块的一项或多项；
    所述人体姿态评价模块，用于检测并识别客户对应于不同情绪的肢体动作，并对应获取相应的姿态满意度评分；
    所述面部表情评价模块，用于检测并识别客户对应于不同情绪的面部表情动作，并对应获取相应的面部表情满意度评分；
    所述文本信息评价模块，用于获取客户手动点击的评分选项对应的文本信息满意度评分；
    所述语音信息评价模块，用于检测并识别客户对应于不同情绪的语音，并对应获取相应的语音信息满意度评分。
  3. 根据权利要求1或2所述的多模态客户满意度综合评价系统，其特征在于，所述信息融合模块，给予人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项信息，分配不同的权重等级，加权获取综合评分。
  4. 根据权利要求3所述的多模态客户满意度综合评价系统，其特征在于，所权重等级由高到低依次为：文本信息、面部表情信息、语音信息、人体姿态信息。
  5. 根据权利要求2所述的多模态客户满意度综合评价系统，其特征在于，
    所述人体姿态评价模块、面部表情评价模块的满意度评分至少包括3个等级；
    所述文本信息评价模块的满意度评分至少包括6个等级；
    所述语音信息评价模块的满意度评分至少包括4个评分格次。
  6. 一种多模态客户满意度综合评价方法，其特征在于，包括：
    至少获取客户人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项；
    基于所述数据获取模块获取的一项或多项信息执行用户情绪分析，并获取相应的满意度评分；
    基于用户情绪与评价分析模块进行分析获取的一项或多项满意度评分，获取最终评价信息。
  7. 根据权利要求6所述的多模态客户满意度综合评价系统，其特征在于，所述基于所述数据获取模块获取的一项或多项信息执行用户情绪分析，并获取相应的评分包括人体姿态评价获取步骤、面部表情评价获取步骤、信息评价获取步骤、语音信息评价获取步骤的一项或多项；
    所述人体姿态评价获取步骤，用于检测并识别客户对应于不同情绪的肢体动作，并对应获取相应的姿态满意度评分；
    所述面部表情评价获取步骤，用于检测并识别客户对应于不同情绪的面部表情动作，并对应获取相应的面部表情满意度评分；
    所述文本信息评价获取步骤，用于获取客户手动点击的评分选项对应的文本信息满意度评分；
    所述语音信息评价获取步骤，用于检测并识别客户对应于不同情绪的语音，并对应获取相应的语音信息满意度评分。
  8. 根据权利要求6或7所述的多模态客户满意度综合评价系统，其特征在于，所述基于用户情绪与评价分析模块进行分析获取的一项或多项评价信息，获取最终评价信息，包括：
    给予人体姿态信息、面部表情信息、文本信息及语音信息的一项或多项信息，分配不同的权重等级，加权获取综合评分。
  9. 根据权利要求8所述的多模态客户满意度综合评价系统，其特征在于，所权重等级由高到低依次为：文本信息、面部表情信息、语音信息、人体姿态信息。
  10. 根据权利要求7所述的多模态客户满意度综合评价系统，其特征在于，
    所述人体姿态评价获取步骤、面部表情评价获取步骤的满意度评分至少包括3个等级；
    所述文本信息评价获取步骤的满意度评分至少包括6个等级；
    所述语音信息评价获取步骤的满意度评分至少包括4个评分格次。
PCT/CN2019/124007 2018-12-14 2019-12-09 一种多模态客户满意度综合评价系统、方法 Ceased WO2020119630A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811535581.3A CN109801096A (zh) 2018-12-14 2018-12-14 一种多模态客户满意度综合评价系统、方法
CN201811535581.3 2018-12-14

Publications (1)

Publication Number Publication Date
WO2020119630A1 true WO2020119630A1 (zh) 2020-06-18

Family

ID=66556780

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/124007 Ceased WO2020119630A1 (zh) 2018-12-14 2019-12-09 一种多模态客户满意度综合评价系统、方法

Country Status (2)

Country Link
CN (1) CN109801096A (zh)
WO (1) WO2020119630A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115239355A (zh) * 2022-08-15 2022-10-25 中国工商银行股份有限公司 客户分类方法、装置、计算机设备和存储介质
CN118096226A (zh) * 2024-04-26 2024-05-28 北京五洲天宇认证中心 一种基于监控数据的顾客满意度评估方法和评估系统

Families Citing this family (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109801096A (zh) * 2018-12-14 2019-05-24 中国科学院深圳先进技术研究院 一种多模态客户满意度综合评价系统、方法
CN112232101A (zh) * 2019-07-15 2021-01-15 北京正和思齐数据科技有限公司 一种用户沟通状态的评价方法、装置及系统
CN112699716A (zh) * 2019-10-23 2021-04-23 深圳云天励飞技术有限公司 服务评分方法及相关装置
CN110718293A (zh) * 2019-10-23 2020-01-21 合肥盛东信息科技有限公司 一种护理员服务质量监测评价系统
KR102318661B1 (ko) * 2020-02-03 2021-11-03 주식회사 지앤 현장 공간에서의 동작 인식을 통한 만족도 조사 시스템
CN111324865A (zh) * 2020-02-24 2020-06-23 浪潮天元通信信息系统有限公司 一种基于物联网的店面满意度智能分析方法和系统
CN111523575B (zh) * 2020-04-13 2023-12-12 中南大学 基于短视频多模态特征的短视频推荐方法
CN111582653A (zh) * 2020-04-14 2020-08-25 五邑大学 一种政务服务评价处理方法、系统、装置和存储介质
CN111563160B (zh) * 2020-04-15 2023-03-31 华南理工大学 基于全局语义的文本自动摘要方法、装置、介质及设备
CN111523460A (zh) * 2020-04-23 2020-08-11 上海铠盾科技有限公司 一种标准作业行为检测系统
CN111967311B (zh) * 2020-07-06 2021-09-10 广东技术师范大学 情绪识别方法、装置、计算机设备及存储介质
CN112052740A (zh) * 2020-08-09 2020-12-08 中信银行股份有限公司 一种客服评价方法、装置及存储介质
CN112331316A (zh) * 2020-10-23 2021-02-05 江苏亚寰软件股份有限公司 一种移动式信息安全医疗评价满意度反馈评测方法
CN114519596B (zh) * 2020-11-18 2025-09-16 中国移动通信有限公司研究院 一种数据处理方法、装置及设备
CN112200697B (zh) * 2020-12-04 2021-03-02 深圳市房多多网络科技有限公司 远程视频看房方法、装置、设备以及计算机存储介质
CN112581321A (zh) * 2020-12-23 2021-03-30 嘉兴学院 一种吴越文化旅游产品用户体验测试系统及方法
CN113011341B (zh) * 2021-03-22 2024-06-28 平安科技（深圳）有限公司 佛经抄写辅助方法、装置、设备及存储介质
CN113345177B (zh) * 2021-05-11 2022-01-18 深圳市海恒智能科技有限公司 一种基于人脸识别的图书自助借还系统
CN113240345A (zh) * 2021-06-16 2021-08-10 中国银行股份有限公司 客户服务满意度的管理方法及装置、存储介质及电子设备
CN113256154A (zh) * 2021-06-16 2021-08-13 中国银行股份有限公司 客户服务满意度的评估方法及装置、存储介质及电子设备
CN113921098A (zh) * 2021-10-14 2022-01-11 江苏木盟智能科技有限公司 一种医疗服务评价方法和系统
CN114663134A (zh) * 2022-03-01 2022-06-24 招商银行股份有限公司 满意度评估方法、装置、设备及存储介质
CN114783023B (zh) * 2022-04-11 2025-04-25 台州学院 一种融合ai面部表情识别和主观打分的满意度评价系统
CN117152850A (zh) * 2022-05-20 2023-12-01 四川大学 基于多模态评分及深度学习的客服情绪视频数据库构建
CN115063016B (zh) * 2022-07-04 2023-07-14 华贸云(北京)科技有限公司 一种基于智慧办公的数据分析方法、系统及存储介质
CN115907528A (zh) * 2022-11-15 2023-04-04 中国银行股份有限公司 一种服务评价确定方法及装置
CN116258513A (zh) * 2023-03-01 2023-06-13 东华云都技术有限公司 一种基于大数据的分析预警系统
CN116682168B (zh) * 2023-08-04 2023-10-17 阳光学院 一种多模态表情识别方法、介质及系统
CN117876047B (zh) * 2023-12-28 2025-01-28 深圳市亿莱顿科技有限公司 评价终端的控制方法、系统、计算机设备及可读存储介质
CN118037140B (zh) * 2024-04-12 2024-06-25 泉州市金诺保洁服务有限公司 基于数字化的家政服务质量分析系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103811009A (zh) * 2014-03-13 2014-05-21 华东理工大学 一种基于语音分析的智能电话客服系统
US20160125419A1 (en) * 2014-10-30 2016-05-05 Sestek Ses Ve Iletisim Bilgisayar Teknolojileri Sanayii Ve Ticaret Anonim Sirketi Speech analysis and evaluation system and method
CN108227932A (zh) * 2018-01-26 2018-06-29 上海智臻智能网络科技股份有限公司 交互意图确定方法及装置、计算机设备及存储介质
CN108334583A (zh) * 2018-01-26 2018-07-27 上海智臻智能网络科技股份有限公司 情感交互方法及装置、计算机可读存储介质、计算机设备
CN109801096A (zh) * 2018-12-14 2019-05-24 中国科学院深圳先进技术研究院 一种多模态客户满意度综合评价系统、方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107194316A (zh) * 2017-04-20 2017-09-22 广东数相智能科技有限公司 一种情绪满意度的评价方法、装置及系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103811009A (zh) * 2014-03-13 2014-05-21 华东理工大学 一种基于语音分析的智能电话客服系统
US20160125419A1 (en) * 2014-10-30 2016-05-05 Sestek Ses Ve Iletisim Bilgisayar Teknolojileri Sanayii Ve Ticaret Anonim Sirketi Speech analysis and evaluation system and method
CN108227932A (zh) * 2018-01-26 2018-06-29 上海智臻智能网络科技股份有限公司 交互意图确定方法及装置、计算机设备及存储介质
CN108334583A (zh) * 2018-01-26 2018-07-27 上海智臻智能网络科技股份有限公司 情感交互方法及装置、计算机可读存储介质、计算机设备
CN109801096A (zh) * 2018-12-14 2019-05-24 中国科学院深圳先进技术研究院 一种多模态客户满意度综合评价系统、方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115239355A (zh) * 2022-08-15 2022-10-25 中国工商银行股份有限公司 客户分类方法、装置、计算机设备和存储介质
CN118096226A (zh) * 2024-04-26 2024-05-28 北京五洲天宇认证中心 一种基于监控数据的顾客满意度评估方法和评估系统

Also Published As

Publication number Publication date
CN109801096A (zh) 2019-05-24

Similar Documents

Publication Publication Date Title
WO2020119630A1 (zh) 一种多模态客户满意度综合评价系统、方法
Caridakis et al. Multimodal emotion recognition from expressive faces, body gestures and speech
CN1157710C (zh) 会话数据开采
CN101101752B (zh) 基于视觉特征的单音节语言唇读识别系统
CN106127156A (zh) 基于声纹和人脸识别的机器人交互方法
CN103996155A (zh) 智能交互及心理慰藉机器人服务系统
CN120849791A (zh) 基于大模型的多模态访谈自动化质性分析评估方法及系统
CN112818742A (zh) 一种用于智能面试的表达能力维度评价方法及装置
CN119167308B (zh) 基于多模态数据和垂类大模型的心理咨询机器人
CN120935432B (zh) 一种基于多模态分析的短视频智能剪辑方法及系统
CN113392918A (zh) 一种基于多源信息融合的抑郁障碍相关因素识别方法
CN113255442A (zh) 金融从业人员行为识别与风险内控方法、装置、电子设备及存储介质
Jazouli et al. Automatic detection of stereotyped movements in autistic children using the Kinect sensor
CN116883888A (zh) 基于多模态特征融合的银行柜面服务问题溯源系统及方法
CN111144359B (zh) 一种展品评价装置和方法及展品推送方法
CN114492579A (zh) 情绪识别方法、摄像装置、情绪识别装置及存储装置
Ansari et al. Multimodal depression detection system using machine learning
CN120599155B (zh) 一种面向教学互动的数字人三维重构系统
CN110705523A (zh) 一种基于神经网络的创业者路演能力测评方法及其测评系统
CN121400830A (zh) 一种基于多头注意力机制的多模态情绪识别融合方法
CN116108396A (zh) 一种非接触式的共同参与人工智能分析系统
CN120636702A (zh) 基于大语言模型与多模态数据的心理健康评估方法及系统
Sánchez-Ancajima et al. Gesture Phase Segmentation Dataset: An Extension for Development of Gesture Analysis Models.
CN118865465A (zh) 一种人机交互图像情绪识别方法及其系统
CN117219088A (zh) 一种复杂环境下生猪连续咳嗽声音识别方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19896263

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 03.11.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19896263

Country of ref document: EP

Kind code of ref document: A1