RU2703969C1 - Method and system for evaluating quality of customer service based on analysis of video and audio streams using machine learning tools - Google Patents

Abstract

FIELD: monitoring systems.

SUBSTANCE: invention relates to a method and system for assessing the quality of customer service. Method, in which in background mode in real time receive data from data source, containing video and audio recording of space with employee workplace; transmitting obtained data to a server; using a computing device, determining the beginning and closing moment of the client dialogue with the employee, storing the video file of the client dialogue with the employee; retrieving client behavior parameters from video recording; based on parameters extracted above, client behavior at the beginning of dialogue is monitored and behavior dynamics of the client is monitored during the whole dialogue; calculating an integral level of client satisfaction, using a trained neural network, based on analysis of the above parameters; analysis results are recorded in a database.

EFFECT: high accuracy of determining quality of customer service based on analysis of video and audio streams using machine learning tools in real time.

14 cl, 2 dwg

Description

FIELD OF TECHNOLOGY

[0001] This technical solution relates to the field of digital data processing based on machine learning algorithms, in particular, to a method and system for assessing the quality of customer service based on the analysis of video and audio streams using machine learning tools.

BACKGROUND

[0002] As solutions at the current level of technology that locally solve the problems of determining human emotions, one can consider publications that describe methods for determining human emotions. Such solutions, for example, are disclosed on the following websites of companies: Affectiva (https://www.affectiva.com/), Kairos (https://www.kairos.com/), Nviso (https: //www.nviso-insights .com / en), VisionLabs (https://visionlabs.ai/ru/), NTechLab (https://ntechlab.ru/).

[0003] Also known are solutions disclosing speech analytics. Such solutions, for example, are disclosed on the following websites of companies: Nice (https://www.nice.com/), Verint (https://ru.verint.com/), Nuance (https://www.nuance.com /index.html).

[0004] A key disadvantage of such solutions is that they provide only one analytical tool, namely, either an assessment of emotions or speech analytics. Also, they are not focused on assessing the quality of customer service and should be customized.

SUMMARY OF THE INVENTION

[0005] The technical problem to which the claimed technical solution is directed is the creation of a method and system for assessing the quality of customer service based on the analysis of the video stream using machine learning tools.

[0006] The technical result consists in increasing the accuracy of determining customer service quality assessment based on the analysis of video and audio streams using real-time machine learning tools.

[0007] In a preferred embodiment, a computer-implemented method for assessing the quality of customer service based on the analysis of video and audio streams using machine learning tools is claimed, comprising the steps of:

- in the background, in real time, they receive data from a data source containing video and audio recording of space with the workplace of the employee;

- transmit the received data to a server containing a computing device and a database;

using a computing device:

- process the obtained data using neural networks in real time, namely, determine the start and end time of the client’s dialogue with the employee, save this video file of the client’s dialogue with the employee;

- analyze the saved video file using neural networks, namely:

- extract the following parameters from the video: socio-demographic characteristics, facial emotions of the client and their changes during the dialogue, the number of emotional states of the client in the framework of this dialogue, head turns and the direction of the gaze, the client’s posture and its change during this dialogue;

- extract the following parameters from the audio recording: the tone of the client’s voice and its changes during the dialogue, the speech speed of the client and its changes during the dialogue, the length of pauses between the client’s remarks during the dialogue, the proportion of positive and negative emotions of the client, the positive content of the client’s speech, voice tone the employee and his changes during the dialogue, the employee’s speech speed and its changes during the dialogue, the length of pauses between employee replicas during the dialogue, the share of positive and negative emotions of the employee, positivity the content of the employee’s speech, the number of interruptions by the employee;

- evaluate, based on the parameters extracted above, the client’s behavior at the beginning of the dialogue and monitor the dynamics of changes in client behavior throughout the dialogue;

- calculate the integral level of customer satisfaction using a trained neural network based on an analysis of the above parameters;

- write the analysis results to the database.

[0008] In a particular embodiment, the time a system is turned on and exited by a specific employee is equal to the total time of the employee.

[0009] In another particular embodiment, the total time of all conversations with various clients that took place at one particular employee during one business day is additionally calculated and the total percentage of dialogs of this employee is determined.

[0010] In another particular embodiment, when analyzing a client’s face, at least eight different emotions are distinguished.

[0011] In another particular embodiment, when analyzing socio-demographic parameters, the age category, gender and race of the client are determined.

[0012] In another particular embodiment, when analyzing the integral level of customer satisfaction, the company type, country of location, and field of activity are additionally taken into account.

[0013] In another particular embodiment, the integral level of customer satisfaction can be adjusted based on the input of additional data: self-assessment by an employee, customer assessment, evaluation by an external expert.

[0014] In another particular embodiment, when analyzing the client’s speech, key phrases are additionally identified, highlighted intonationally, and checked for compliance with the script library, while a standard library built into the system or a library created independently by the client company can be used with given synonym.

[0015] In another particular embodiment, when analyzing the employee’s speech, key phrases are additionally identified, highlighted intonationally, and checked for compliance with the script library, and a standard library built into the system or a library created independently by the client company can be used with given synonym.

[0016] In another particular embodiment of the system, neural networks are additionally used to determine individual recommendations for each employee to improve the quality of customer service.

[0017] In another particular embodiment, the average employee load with client work is additionally calculated based on the ratio of the total percentage of employee conversations to the total system uptime.

[0018] In another particular embodiment, the system additionally displays the total workload of the entire office, which is determined based on an analysis of the percentage of the duration of all dialogs of company employees from the total time of all employees during a selected period of time.

[0019] In another particular embodiment, the data source may be:

- stationary webcam;

- mobile phone camera;

- employee screen using video communication;

- A tablet aimed at the client, with a built-in webcam on the front side of the screen.

[0020] The claimed solution is also implemented through a system for assessing the quality of customer service based on the analysis of video and audio streams using machine learning tools, containing:

- a data source configured to fix video and audio data;

- a server connected by a data transmission channel to a data source, and containing a database and a computing device, the computing device being configured to perform the method according to any one of claims. 1-13.

DESCRIPTION OF DRAWINGS

[0021] The implementation of the invention will be described hereinafter in accordance with the accompanying drawings, which are presented to illustrate the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

FIG. 1 illustrates a general view of the claimed system;

Fig 2. illustrates a general diagram of a computing device.

DETAILED DESCRIPTION OF THE INVENTION

[0022] The following detailed description of an embodiment of the invention provides numerous implementation details in order to provide a clear understanding of the present invention. However, to a person skilled in the art, it will be apparent how the present invention can be used, both with and without implementation details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the understanding of the features of the present invention.

[0023] Furthermore, it will be clear from the foregoing that the invention is not limited to the foregoing implementation. Numerous possible modifications, changes, variations and replacements preserving the essence and form of the present invention will be apparent to those skilled in the subject field.

[0024] The present invention is directed to providing a computer-implemented method and system for assessing the quality of customer service based on the analysis of video and audio streams using machine learning tools.

[0025] The claimed solution can be implemented on a cloud infrastructure platform. Cloud technologies provide the user with a convenient virtual environment for storing and processing information, combining hardware, software, communication channels, as well as technical support. Storing information in the “cloud” with access to the Internet makes it possible to access it from anywhere in the world from virtually any computing device.

[0026] As shown in FIG. 1, the claimed system (100) includes a data source (101) from which data containing video and audio recording space with the workplace of the employee. Moreover, the data source is connected to the server (102) containing the computing device (103) and the database (104).

[0027] The components of the system are connected by means of data transmission, which are selected from means designed to implement the communication process between the various units via wired and / or wireless communication, in particular, these types of communication can be: GSM, GPRS, LTE, 5G, WLAN , Wi-Fi, Bluetooth, BLE, satellite communications, NFC, Ethernet, USB, IrDa, Lightning, UART, CAN, etc.

[0028] The architecturally software part of the computing device (103) is a collection of software modules that provide a functional part of its operation.

[0029] The data source (101) is made in the form of a camera, which can be a stationary or mobile webcam, located, for example, in a bank or pharmacy, or at the MFC, or in stores, or the like, customer service enterprises.

[0030] The following is an example of the operation of the claimed method.

[0031] Video and audio recording of the space with the employee’s workplace is received from the camera (101) and transmitted to the server (102).

[0032] On the server (102) using a computing device (103), the data received from the camera (101) is processed in real time. They process the data using neural networks in real time, namely, they determine the start and end times of the client’s dialogue with the employee, and save this video file of the client’s dialogue with the employee.

[0033] Neural networks are learning algorithms that are based on a model of the human brain and are aimed at solving multiclass, two-class, and regression problems.

[0034] Next, analyze this saved video file using neural networks, namely: extract the following parameters from the video: socio-demographic characteristics, facial emotions of the client and their changes during the dialogue, the number of emotional states of the client within this dialogue, head turns and direction view, customer posture and its change during this dialogue.

[0035] The following is an example of a head rotation:

The head of each person has a characteristic symmetry, therefore, it is possible to determine the longitudinal axis, vertical axis and transverse axis of rotation of the human head, with respect to which three angles of rotation of the head are determined. The neural network was trained on labeled data, in which three rotation angles were determined, and at the output it finds angles of deviation from the corresponding axes for each image.

[0036] According to facial emotions:

The system extracts frames from the dialog every three seconds. As a result, for each frame, a trained neural network receives facial emotions, provided that there is a human face on the frame. Thus, during the entire dialogue, the system has the ability to evaluate the emotions of the client and their change.

[0037] By the number of emotional states:

In each frame, the system finds the following emotional states

- anger / rage (Anger);

- contempt (Contempt);

- disgust (Disgust);

- fear (Fear);

- happiness (Happiness);

- disinterest (Neutral);

- sadness / sadness (Sadness);

- surprise (Surprise)

and builds the probability distribution of all these emotions for each frame.

[0038] A trained neural network of emotional content within the face includes detecting at least eight different emotions from, for example, sadness, stress, happiness, anger, frustration, confusion, hesitation, cognitive overload, focusing, participation, attention, boredom, research, confidence, trust, enthusiasm, disgust, skepticism, doubt, satisfaction, excitement, laughter, calmness, curiosity, humor, sadness, pungency or fun.

[0039] A trained neural network can be a convolutional neural network, a decision tree, or any other classifying algorithm, that is, an algorithm capable of classifying input data into a finite set of classes.

[0040] The neural network is trained on a large data set. A neural network can work with both color images and grayscale images.

[0041] Moreover, when analyzing socio-demographic parameters using a trained neural network, the age category, gender and race of the client are determined.

[0042] A neural network is used to determine a person’s age, gender and race from his image, which is built on the basis of a large sample of labeled data on faces of people of all races, sexes and ages. For each new image of a person’s face, the neural network finds a number of characteristic principles that are characteristic of a particular race, gender or age and builds the probability distribution of compliance with the final result. For example, a typical result of the operation of a neural network is a triple of indicators: race - European (90%), gender - male (99%), age - 30 (80%).

[0043] Next, the following parameters are extracted from the audio recording: the tone of the client’s voice and its changes during the dialogue, the speech speed of the client and its changes during the dialogue, the length of the pauses between the client’s replicas during the dialogue, the proportion of positive and negative emotions of the client, the positive content of the client’s speech , the tone of the employee’s voice and its changes during the dialogue, the employee’s speech speed and its changes during the dialogue, the length of pauses between employee’s remarks during the dialogue, the share of positive and negative emotions of the employee, position ivnost content of employee speech, the amount of heckling from the employee.

[0044] In voice analysis, two neural networks are used.

1. Analysis of the tone of speech. It is based on labeled data - small fragments of audio recordings that express certain emotions, for example, fear or joy. According to the marked data, there are some signs that correspond to one or another category of emotions. As a result, for each audio recording, you can get average indicators of various categories of emotions.

2. Speech recognition.

[0045] Based on the data of neural networks, it is possible to calculate the following indicators:

1. The speed of speech. The system indicates the start and end time of each spoken word, so you can determine the number of spoken sounds per second, as well as control the change in speech speed.

2. The positivity / negativity of emotions. The results of the neural network for the analysis of the tone of speech just find positive / negative emotions in speech.

3. Positive content. For each word, its vector form is constructed. The meaning of the vector form of words is that the distance between the vectors of similar words is small (for example, beautiful and beautiful). As a result, all words are clustered by similar types and within each cluster a level of positivity is determined (for example, all words synonymous with a beautiful word have a large positive level of positivity, and words synonymous with a bad word have a negative level of positivity). As a result, all the word vectors in the text are added up and the content positivity indicator is built based on the final indicator.

[0046] Based on all the parameters extracted above, the client’s behavior at the beginning of the dialogue is evaluated and the dynamics of changes in client behavior throughout the dialogue are monitored.

[0047] The integral level of customer satisfaction is calculated using a trained neural network and the analysis results are recorded in a database (104) located on the server.

[0048] A special technique was developed to calculate the integral indicator, which allowed viewing the dialogue to rate the dialogue on a 10-point scale, depending on how the dialogue went. Several independent people put down dialogue ratings in accordance with the developed methodology. In addition, the dialogs themselves were evaluated by the dialogue participants themselves (client and employee). As a result, according to the obtained indicators, a neural network was constructed that estimates the integral indicator of the dialogue depending on many of the input parameters described above.

[0049] In addition to the above, the system additionally makes it possible for each employee to determine the time the system is turned on and exit by a specific employee, which is equivalent to the total working time of this employee.

[0050] Additionally, the total time of all dialogs with various clients that took place with one particular employee during one business day is calculated and the total percentage of dialogs of this employee is determined.

[0051] The total percentage of workload for this employee is the ratio of the time of the dialogs in which the employee participated, to the total working time during the day.

[0052] Additionally, when analyzing the integral level of customer satisfaction, parameters such as the type of company, country of location, and scope of activity may be taken into account. Also, the integral level of customer satisfaction can be adjusted based on the input of additional data: self-assessment by an employee, customer assessment, evaluation by an external expert.

[0053] When analyzing the client’s speech, key phrases are additionally identified, highlighted intonationally, and checked for compliance with the script library, while a standard library built into the system or a library created independently by the client company can be used, taking into account a synonym. To do this, all words are given in the infinitive and a search is made for all phrases from the company’s script library or the standard script library with the ability to replace some words with synonyms. Also, when searching for phrases, in addition to synonyms, the number of additional words that can be included in the phrase is taken into account (for example, the phrase “how are you?” Can be found in the sentence “how are you, dear Ivan Ivanovich, deeds?”, Despite the fact that the words "how" and "things" are not near).

[0054] When analyzing the employee’s speech, key phrases are additionally identified, highlighted intonationally, and checked for compliance with the script library, while a standard library built into the system or a library created independently by the client company, taking into account a synonym, can be used.

[0055] Additionally, the system uses a neural network to form and provide personal advice (individual recommendations) to each employee to improve the quality of customer service. Recommendations are made as a result of the analysis of the 3 main components of the dialogue:

- facial expressions;

- tones;

- content (text) of speech.

For each of these indicators, when training the neural network, a standard is set, which is considered an ideal dialogue. Subsequently, if in facial expressions, for example, there are 10% more bad emotions than in the standard, then a hint is given: “Smile more”, but if the tone is 20% more frightened than in the standard, then a hint is given: “Speak more calmly ".

[0056] The system also additionally for each employee of the company, in addition to the average satisfaction indicator, the ability to build the average workload for each employee and the indicator of cross-sales, in part, the percentage of dialogs in which the cross-sale was heard, can be implemented. After that, on the basis of three indicators, a linear model is built in which efficiency is the sum of the above indicators with given weights.

[0057] The average employee load for client work is defined as follows:

1. The time the employee turned on the system and exited it (total operating time) is fixed;

2. The total time of all dialogs that took place at this employee within one day is taken (total percentage of dialogs);

3. The time of dialogs is divided by the total time of work, it turns out the percentage of the employee loading client work.

[0058] In the system, you can find information on the workload of the entire office, which is determined based on an analysis of the percentage of the duration of all company dialogs from the total time of all employees during the selected time period (how many people worked and how many dialogs there were in a certain period of time). If the workload is more than 80%, the system gives a recommendation on the withdrawal of additional employees during this time period.

[0059] The data source may also be a stationary webcam; mobile phone / tablet camera; employee screen using video calling.

[0060] The claimed solution allows you to accurately determine the quality of customer service based on the analysis of the video stream using real-time machine learning tools.

[0061] In addition, the use of the claimed solution additionally allows you to: identify customer satisfaction in various services and identify the reasons for high and low customer satisfaction; allows you to rate employees on the quality of service and the use of sales scripts, determine the load when working with clients.

[0062] FIG. 2, a general diagram of a computer device (200) that provides the data processing necessary to implement the claimed solution will be presented.

[0063] In general, device (200) comprises components such as: one or more processors (201), at least one memory (202), data storage means (203), input / output interfaces (204), means B / In (205), means of network interaction (206).

[0064] The processor (201) of the device performs the basic computing operations necessary for the operation of the device (200) or the functionality of one or more of its components. The processor (201) executes the necessary computer-readable instructions contained in the random access memory (202).

[0065] The memory (202), as a rule, is made in the form of RAM and contains the necessary program logic that provides the required functionality.

[0066] The storage medium (203) may be in the form of HDDs, SSDs, an array raid, network attached storage, flash memory, optical information storage devices (CD, DVD, MD, Blue-Ray drives) and the like. The tool (203) allows for long-term storage of various types of information, for example, the history of processing transactional requests (logs), user identifiers, etc.

[0067] Interfaces (204) are standard means for connecting and working with the server part (102), for example, USB, RS232, RJ45, LPT, COM, HDMI, PS / 2, Lightning, FireWire, etc.

[0068] The choice of interfaces (204) depends on the specific implementation of the device (200), which may be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, etc.

[0069] As the I / O data means (205), the following can be used: keyboard, joystick, display (touch screen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

[0070] Network communication tools (206) are selected from a device providing network reception and data transmission, for example, an Ethernet card, WLAN / Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. .P. Using means (205), the organization of data exchange via a wired or wireless data channel is provided, for example, WAN, PAN, LAN (LAN), Intranet, Internet, WLAN, WMAN or GSM.

[0071] The components of the device (200) are interfaced via a common data bus (210).

[0072] In the present application materials, a preferred disclosure was provided for the implementation of the claimed technical solution, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims (30)

1. A method for assessing the quality of customer service based on the analysis of video and audio streams using machine learning tools, which consists in performing the steps in which: - in the background, in real time, they receive data from a data source containing video and audio recordings of the space with the employee’s workplace; - transmit the received data to a server containing a computing device and a database; using a computing device: - process the obtained data using neural networks in real time, namely, determine the start and end time of the client’s dialogue with the employee, save this video file of the client’s dialogue with the employee; - analyze the saved video file using neural networks, namely: - extract the following parameters from the video: socio-demographic characteristics, facial emotions of the client and their changes during the dialogue, the number of emotional states of the client in the framework of this dialogue, head turns and the direction of the gaze, the client’s posture and its change during this dialogue; - extract the following parameters from the audio recording: the tone of the client’s voice and its changes during the dialogue, the speech speed of the client and its changes during the dialogue, the length of pauses between the client’s remarks during the dialogue, the proportion of positive and negative emotions of the client, the positive content of the client’s speech, voice tone the employee and his changes during the dialogue, the employee’s speech speed and its changes during the dialogue, the length of pauses between the employee’s remarks during the dialogue, the share of positive and negative emotions of the employee, positivity the content of the employee’s speech, the number of interruptions by the employee; - evaluate, based on the parameters extracted above, the client’s behavior at the beginning of the dialogue and monitor the dynamics of changes in client behavior throughout the dialogue; - calculate the integral level of customer satisfaction using a trained neural network based on an analysis of the above parameters; - write the analysis results to the database. 2. The method according to p. 1, characterized in that the time the system is turned on and out of a particular employee is equal to the total working time of this employee. 3. The method according to p. 1, characterized in that it further counts the total time of all dialogs with various clients that took place at one particular employee within one working day and determines the total percentage of dialogs of this employee. 4. The method according to p. 1, characterized in that when analyzing the face of the client, at least eight different emotions are distinguished. 5. The method according to p. 1, characterized in that in the analysis of socio-demographic parameters determine the age category, gender and race of the client. 6. The method according to p. 1, characterized in that when analyzing the integral level of customer satisfaction, they additionally take into account the type of company, the country of its location, and the scope of activity. 7. The method according to claim 1, characterized in that the integral level of customer satisfaction can be adjusted based on the input of additional data: self-assessment by an employee, customer assessment, evaluation by an external expert. 8. The method according to claim 1, characterized in that when analyzing the client’s speech, key phrases are additionally identified, highlighted intonationally, and checked for compliance with the script library, while the standard library built into the system or a library created independently can be used client company, taking into account synonyms. 9. The method according to p. 1, characterized in that when analyzing the employee’s speech, key phrases are additionally identified, highlighted intonationally, and checked for compliance with the script library, while the standard library built into the system or a library created independently can be used client company, taking into account synonyms. 10. The method according to p. 1, characterized in that the system additionally uses neural networks to determine individual recommendations to each employee to improve the quality of customer service. 11. The method according to p. 1, characterized in that it further calculates the average workload of the employee with client work based on the ratio of the total percentage of employee dialogs to the total system time. 12. The method according to p. 1, characterized in that the system additionally displays the total workload of the entire office, which is determined based on the analysis of the percentage of the duration of all dialogs of company employees from the total time of all employees during the selected time period. 13. The method according to p. 1, characterized in that the data source may be: - stationary webcam; - mobile phone camera; - employee screen using video communication; - A tablet aimed at the client, with a built-in webcam on the front side of the screen. 14. A system for assessing the quality of customer service based on the analysis of video and audio streams using machine learning tools, containing: - a data source configured to capture video and audio data; - a server connected by a data transmission channel to a data source and containing a database and a computing device, the computing device being configured to perform the method according to any one of claims. 1-13.

Images