JP2023005289A - Dialogue support server, dialogue support system, dialogue support method, and program - Google Patents

Abstract

To enable a person in charge to get close to client's feelings in a dialogue between a client and the person in charge.SOLUTION: An acquisition unit is provided to acquire first image data including the face of a client imaged during a dialogue between the client and a person in charge and second image data including the face of the person in charge imaged during the dialogue. An image processing unit is provided to estimate a first emotion expressed by the face indicated by the first image data, estimate a second emotion expressed by the face indicated by the second image data, and calculate a degree of similarity between the first emotion and the second emotion as a degree of sympathy during the dialogue. A communication unit is provided to transmit the degree of sympathy calculated by the image processing unit to a terminal device of the person in charge.SELECTED DRAWING: Figure 2

Description

The present invention relates to a dialogue support server, a dialogue support system, a dialogue support method, and a program.

Conventionally, technology has been used to detect the emotional state of customers using voice and images. For example, Patent Literature 1 discloses a technique capable of more accurately determining whether or not a customer is interested in a product by including not only the result of sentiment analysis at the time of proposal but also the results of daily emotion analysis. .

JP 2020-184216 A

However, even if the customer's emotional state can be detected, it is not always possible to get close to the customer's emotions. In particular, even if an inexperienced person in charge notices that a customer is worried, he or she may not know how to respond. As a result, the customer's trust could not be obtained, and there was a possibility of opportunity loss.

The present invention has been made in view of such circumstances, and its object is to provide a dialogue support server, a dialogue support system, and a dialogue support method that enable the person in charge to be close to the customer's feelings in the dialogue between the customer and the person in charge. , and to provide a program.

In order to solve the above-described problems, a dialogue support server according to the present invention provides first image data in which the customer's face is imaged during the dialogue between the customer and the person in charge, and the face of the person in charge during the dialogue. an acquisition unit for acquiring imaged second image data; and a first emotion indicated by the face indicated by the first image data, and estimating the second emotion indicated by the face indicated by the second image data. and calculating the degree of similarity between the first emotion and the second emotion as a degree of empathy during the dialogue; and a communication unit that transmits to the terminal device.

Further, according to the present invention, in the above-described dialogue support server, the image processing unit estimates, as the first emotion, a first degree to which the face of the customer indicates a specific target emotion, and the face of the person in charge A second degree indicating the target emotion is estimated as the second emotion, and the degree of empathy is calculated based on an absolute value of a difference between the first degree and the second degree.

Further, in the above-described dialogue support server, the image processing unit calculates the degree of concern of the customer based on the feature points extracted from the first image data and the concern level estimation model. and the communication unit transmits the level of empathy and the level of worry calculated by the image processing unit to the terminal device of the person in charge, and the level of worry estimation model uses the facial image for learning as the learning It is a model that machine-learned the correspondence relationship between the face image and the worried face using a training data set that has a label indicating whether the face image for the target is a worried face. It is a model that estimates the degree to which a given face image is a distressed face.

Further, in the present invention, the above-described dialogue support server further comprises a speech processing unit, the acquisition unit acquires audio data collected by a microphone that collects the speech during the dialogue, and the speech processing unit converts the speech contained in the speech data into text, extracts a keyword having a frequency of occurrence equal to or higher than a threshold value from the text-converted speech, and the communication unit transfers the keyword extracted by the speech processing unit to the person in charge to the terminal device of the other party.

Further, according to the present invention, in the above-described dialogue support server, the speech processing unit refers to a dialogue database in which dialogues are stored based on the keyword, and the appearance frequency of the keyword from the dialogue database is equal to or higher than a threshold. A dialogue is extracted as a similar case, and the communication unit transmits the similar case extracted by the speech processing unit to the terminal device of the person in charge.

Further, the dialogue support system according to the present invention includes a first camera that captures the face of the customer during the dialogue between the customer and the person in charge, a second camera that captures the face of the person in charge during the dialogue, and a terminal device of a person, and the dialogue support server described above, which obtains first image data captured by the first camera and second image data captured by the second camera, and obtains the empathy level to the terminal device.

Further, a dialogue support method according to the present invention is a dialogue support method performed by a computer device, wherein an acquisition unit obtains first image data in which the customer's face is imaged during a dialogue between the customer and the person in charge, and the dialogue acquires second image data in which the face of the person in charge is imaged at the time, an image processing unit estimates a first emotion indicated by the face indicated in the first image data, and calculates the first emotion in the second image data estimating a second emotion indicated by the indicated face, calculating a degree of similarity between the first emotion and the second emotion as a degree of empathy during the dialogue, and calculating by the communication unit by the image processing unit The degree of empathy obtained is transmitted to the terminal device of the person in charge.

Further, in order to solve the above-described problems, the present invention provides a program for causing a computer to operate as the dialogue support server described above, comprising: It's a program.

According to the present invention, the empathy level can be transmitted to the terminal device of the person in charge. Therefore, the person in charge can know the degree of sympathy in the dialogue with the customer, and by conducting dialogue according to the degree of sympathy, it becomes possible to get close to the customer's feelings.

It is a figure showing an example of application of dialogue support system 1 concerning an embodiment. 1 is a block diagram showing a configuration example of a dialogue support system 1 according to an embodiment; FIG. It is a figure showing an example of assessment data 120 concerning an embodiment. It is a figure which shows the example of the customer's U emotion data 121A which concerns on embodiment. It is a figure which shows the example of the person in charge T's emotion data 121B which concerns on embodiment. It is a figure which shows the example of the degree-of-worry data 122 which concerns on embodiment. It is a figure which shows the example of empathy degree data 123 which concerns on embodiment. It is a figure which shows the example of the keyword data 124 which concern on embodiment. FIG. 4 is a diagram showing an example of case text data 125 according to the embodiment; 4A and 4B are diagrams showing examples of images displayed on the terminal device 20 according to the embodiment; FIG. 4A and 4B are diagrams showing examples of images displayed on the terminal device 20 according to the embodiment; FIG. 4 is a flowchart showing the flow of processing performed by the server device 10 according to the embodiment; 4 is a flowchart showing the flow of processing performed by the server device 10 according to the embodiment;

An embodiment of the present invention will be described below with reference to the drawings.

(Regarding Dialogue Support System 1)
FIG. 1 is a diagram showing an application example of a dialogue support system 1 according to an embodiment. As shown in FIG. 1, the dialogue support system 1 is applied when a customer U and a person in charge T have a dialogue.

In the dialogue support system 1, during dialogue, the camera C1 captures the face of the customer U and transmits the captured image to the server device 10 as the image data D1. Camera C2 captures the face of person in charge T and transmits the captured image to server device 10 as image data D2. In addition, the microphone M collects the voice related to the conversation between the customer U and the person in charge T, and transmits the collected voice as the voice data V to the server device 10 . The server device 10 analyzes the dialogue using the image data D1, the image data D2, and the voice data V, and transmits the processing result K of the analysis to the terminal device 20. FIG.

Here, the method by which the server device 10 analyzes the dialogue will be described. As an example of dialogue analysis, the server device 10 estimates the emotion of the customer U during the dialogue. The server device 10 estimates the degree of a plurality of emotions, such as joy, sadness, surprise, anger, fear, and neutrality, from the features of the face image of the customer U shown in the image data D1, and outputs the estimation result to the terminal. Send to device 20 . Thereby, the person in charge T can understand the emotion of the customer U while having a conversation.

The server device 10 also calculates the degree of empathy. The degree of empathy is the degree of empathy between the customer U and the person in charge T in the dialogue. In this embodiment, when the feelings of the customer U and the feelings of the person in charge T are similar, it is assumed that the customer U and the person in charge T empathize. On the other hand, when the feelings of the customer U and the feelings of the person in charge T are not similar, it is considered that the customer U and the person in charge T do not sympathize. For example, the server device 10 estimates the emotion of the customer U from the features of the face image of the customer U shown in the image data D1. Further, the server device 10 estimates the emotion of the person in charge T from the features of the face image of the person in charge T shown in the image data D2. Then, the server device 10 calculates the degree of similarity between the estimated emotion of the customer U and the emotion of the person in charge T as the degree of empathy. Thereby, the person in charge T can recognize whether he/she can share the feelings of the customer U, that is, whether he or she is close to the customer U while having a conversation.

In addition, the server device 10 estimates the degree of concern (degree of concern) of the customer U during the dialogue. In this embodiment, a machine learning technique is used to estimate the degree of concern of the customer U. FIG. Specifically, in this embodiment, a worry level estimation model is generated in advance. The distress level estimation model is, for example, a model that estimates the distress level of the face shown in the face image based on the feature points extracted from the face image. In this case, the worry level estimation model learns a learning data set in which a learning face image, its feature points, and a label indicating whether or not the image is a worried face are associated with each other. It is a model that has learned the correspondence between the feature points of the image and the degree of concern. By the server device 10 estimating the degree of concern of the customer U, the person in charge T can grasp whether the customer U is worried while talking.

In addition, the server device 10 associates the degree of empathy and the degree of anxieties with the elapsed time of the dialogue and transmits them to the terminal device 20 . As a result, the person in charge T can grasp how the customer U's degree of anxieties is changing and how the degree of sympathy is changing according to the progress of the dialogue.

The server device 10 also extracts keywords that frequently appear in the dialogue (frequent keywords). The server device 10 converts the voice collected as the voice data V into text using a voice recognition technology, and out of the nouns included in the textual information, for example, the nouns present in a number equal to or greater than a threshold value are identified as frequent occurrences. Keywords are extracted, and the extracted frequent keywords are transmitted to the terminal device 20 . As a result, the person in charge T can grasp the keywords that frequently appear in the dialogue.

The server device 10 also extracts similar cases. Similar cases are interactions that are similar to the current interaction. Similar cases are extracted, for example, from a database of past interactions. The server device 10 searches a database of past dialogues using the frequent keywords extracted from the dialogues, and extracts dialogues associated with keywords equivalent to the frequent keywords as similar cases. The server device 10 transmits the extracted similar cases to the terminal device 20 . Thereby, the person in charge T can refer to the content of the dialogue when a similar dialogue has been held in the past.

The server device 10 also transmits the assessment data of the customer U to the terminal device 20 . The assessment data are the results of preliminary research on the customer U, such as the results of a questionnaire to the customer U conducted before the dialogue. As a result, the person in charge T can confirm the assessment data of the customer U before or during the dialogue.

FIG. 2 is a block diagram showing a configuration example of the dialogue support system 1 according to the embodiment. The dialogue support system 1 includes, for example, two cameras C (a camera C1 and a camera C2), a microphone M, a server device 10, and a terminal device 20. These components of the dialogue support system 1 (camera C, microphone M, server device 10, and terminal device 20) are communicably connected via a communication network NW.

The server device 10 includes a communication unit 11, a storage unit 12, and a control unit 13, for example. The communication unit 11 communicates with the camera C, the microphone M, and the terminal device 20 . For example, the communication unit 11 acquires the image data D from the camera C. The communication unit 11 acquires the voice data V from the microphone M. The communication unit 11 transmits the analysis result of the dialogue as the processing result K to the terminal device 20 .

The storage unit 12 is a storage medium such as a HDD (Hard Disk Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access read/write Memory), ROM (Read Only Memory), or any of these any combination of storage media. The storage unit 12 stores programs for executing various processes of the server device 10 and temporary data used when performing various processes.

The storage unit 12 stores, for example, assessment data 120, emotion data 121, empathy data 123, keyword data 124, case text data 125, and learned models 126. FIG. The assessment data 120 is information indicating the result of a questionnaire given to the customer U in advance. The emotion data 121 is information indicating the emotion of the customer U in the dialogue. The trouble level data 122 is information indicating the trouble level of the customer U in the dialogue. The degree of empathy data 123 is information indicating the degree of empathy between the customer U and the person in charge T in the dialogue. The keyword data 124 is information indicating frequently appearing keywords in dialogue. The case text data 125 is information indicating similar cases. The learned model 126 is information indicating the configuration of the worry level estimation model.

The control unit 13 is implemented by causing a CPU (Central Processing Unit) provided as hardware in the server device 10 to execute a program. The control unit 13 comprehensively controls the server device 10 . The control unit 13 includes an acquisition unit 130, a questionnaire processing unit 131, an image processing unit 132, an audio processing unit 133, and a device control unit 134, for example.

Acquisition unit 130 acquires a questionnaire to customer U conducted in advance. For the questionnaire, for example, the customer U inputs an answer to a questionnaire displayed on a terminal device (not shown) such as his/her own smartphone, and presses a button such as "send answer". is transmitted from the terminal device to the server device 10. Acquisition unit 130 outputs the acquired questionnaire to questionnaire processing unit 131 .

The questionnaire processing unit 131 generates assessment data 120 based on the answers obtained from the questionnaire. The questionnaire processing unit 131 acquires, for example, the gender, age, and family relationship of the customer U obtained from the questionnaire, and performs an error check. For example, the questionnaire processing unit 131 extracts, as an error, the case where the age is extremely young or the age is over 100 years old. The questionnaire processing unit 131 causes the storage unit 12 to store the questionnaire result after the error check as the assessment data 120 .

Acquisition unit 130 acquires image data D<b>1 and D<b>2 and outputs the acquired image data D<b>1 and D<b>2 to image processing unit 132 .

Image processing section 132 generates emotion data 121 based on image data D1. The image processing unit 132 extracts feature points from the facial image represented by the image data D1. A feature point is a point that can be a feature when estimating an emotion or a degree of worry. A feature point is, for example, a point group indicating the shape of eyebrows, eyes, and lips, and the positions of the corners of the eyes and the corners of the mouth. The image processing unit 132 estimates the emotion of the customer U based on the extracted feature points. For example, if the shape of the eyebrows tends to fall toward the outer ends of the eyebrows and the position of the corners of the mouth is below the center of the lips, the image processing unit 132 indicates that the emotion of sadness is high and the emotion of joy is low. We estimate that On the other hand, the image processing unit 132 does not tend to lower the shape of the eyebrows in the direction toward the outer ends of the eyebrows, and when the corners of the mouth are above the center of the lips, the emotion of sadness is low and the emotion of joy is low. Estimate high. The image processor 132 estimates other emotions such as surprise, anger, fear, and neutrality in a similar manner. For example, the image processing unit 132 estimates the degree of joy, sadness, surprise, anger, fear, neutrality, etc., as a percentage. The image processing unit 132 stores the estimated emotion in the storage unit 12 as the emotion data 121 .

The image processing unit 132 generates the worry level data 122 based on the image data D1. The image processing unit 132 extracts feature points from the face image represented by the image data D1, and inputs information indicating the extracted feature points to the degree-of-worry estimation model. Anxiety level estimation model outputs anxiety level estimated from input feature points. The image processing unit 132 estimates the distress level output from the distress level estimation model as the distress level of the face image. The image processing unit 132 stores the estimated degree of concern in the storage unit 12 as the degree of concern data 122 .

The image processing unit 132 generates empathy data 123 based on the image data D1 and D2. The image processing unit 132 estimates the emotions of the customer U and the person in charge T from the facial images indicated by the image data D1 and D2. The method by which the image processing unit 132 estimates the emotion of each of the customer U and the person in charge T is the same as the method used by the image processing unit 132 to generate the emotion data 121, so description thereof will be omitted.

The image processing unit 132 generates the empathy level data 123 based on the emotional closeness between the customer U and the person in charge T. FIG. The image processing unit 132, for example, calculates the position of the customer U and the position of the person in charge T in a multidimensional emotion vector space in which each emotion is regarded as a vector. The image processing unit 132 calculates the distance from the position of the customer U to the person in charge T, and sets a value corresponding to the magnitude of the calculated distance as the degree of empathy. For example, the image processing unit 132 calculates the degree of empathy using the following formula (1). In the expression (1), a _i indicates the estimated value of the customer U's emotion (i) of the i-th type. b _i denotes the estimated value of customer U's i-th type of emotion (i). i is a natural number from i=1 to n, and n is the number of emotions estimated by the image processing unit 132 . The type of emotion (i) may be set arbitrarily. For example, emotion (1) is set to joy and emotion (2) is set to be sad.

As shown in the formula (1), when the estimated values of the feelings (i) of the customer U and the person in charge T are close to each other and the difference is close to 0 (zero), the degree of empathy approaches 1. On the other hand, when the difference between the estimated values of feelings (i) of the customer U and the person in charge T is large, the degree of empathy approaches zero. In this case, a higher degree of empathy indicates that the customer U and the person in charge T are in empathy, and a smaller degree of empathy indicates that the customer U and the person in charge T are less empathetic. The image processing unit 132 causes the storage unit 12 to store the calculated degree of empathy as the degree of empathy data 123 .

Acquisition unit 130 acquires audio data V and outputs acquired audio data V to audio processing unit 133 .

The voice processing unit 133 generates keyword data 124 based on the voice data V. FIG. The speech processing unit 133 converts the speech in the speech data V into text using speech recognition technology. The speech recognition technology used here may be any technology. For example, the speech processing unit 133 performs speech recognition by converting speech into phonemes using an acoustic model for each phoneme, converting a group of phonemes connecting the converted phonemes into words, and outputting the speech recognition result. to text. The speech processing unit 133 searches for words contained in the text data, and extracts words that appear more than a predetermined number of times from the start of the dialogue to the present time. The speech processing unit 133 stores the extracted words in the storage unit 12 as the keyword data 124 .

The voice processing unit 133 generates case text data 125 based on the voice data V. FIG. The speech processing unit 133, for example, expresses a sentence obtained by converting the speech data V into text (hereinafter referred to as a first sentence) as a vector based on the content of the sentence. For example, the speech processing unit 133 uses Doc2Vec to express a sentence as a vector. When texts are vector-expressed using Doc2Vec, conversations with similar content, such as nursing care and family topics, are digitized into vectors having similar vector values.

The speech processing unit 133 refers to a database in which sentences (hereinafter, referred to as second sentences) in which past dialogues have been converted into text are expressed in vectors. The speech processing unit 133 generates positional coordinates in the vector space when the first sentence is expressed in vectors (hereinafter referred to as the first position) and positional coordinates in the vector space when the second sentence is expressed in vectors (hereinafter referred to as the first position). 2 positions) are within a predetermined distance are extracted as similar cases. The image processing unit 132 stores the extracted similar cases in the storage unit 12 as case text data 125 .

The device control unit 134 controls the server device 10 as a whole. For example, when the communication unit 11 receives a questionnaire answer, the device control unit 134 outputs the questionnaire answer to the acquisition unit 130 . When the image processing unit 132 generates data such as the emotion data 121 , the device control unit 134 outputs the data to the communication unit 11 and notifies the terminal device 20 of the data. When the voice processing unit 133 generates data such as the keyword data 124, the device control unit 134 outputs the data to the communication unit 11 and notifies the terminal device 20 of it.

FIG. 3 is a diagram showing an example of assessment data 120 according to the embodiment. The assessment data 120 includes, for example, items such as assessment number, customer U, and tenant applicant. Here, a case is exemplified in which the content of the dialogue is the content of consultation about the nursing care facility where the family of the customer U resides. For this reason, the questionnaire includes items about family members to be moved into nursing care facilities, that is, applicants.

The assessment number is identification information such as a number that uniquely identifies the assessment data 120 . In the item of customer U, information about customers obtained from responses to questionnaires, such as information indicating names and addresses, is stored. For the prospective tenant, information about the prospective tenant obtained from answers to questionnaires, for example, information indicating the name and address of the prospective tenant is stored.

FIG. 4 is a diagram showing an example of customer U emotion data 121A according to the embodiment. The emotion data 121A includes, for example, items such as customer U face image number, time, joy, sadness, surprise, and anger. The customer U face image No. is identification information such as a number that uniquely identifies the face image of the customer U obtained from the image data D1. Time is information indicating the time when the face image was captured. The time is set based on, for example, the dialogue start time. Items such as joy, sadness, surprise, anger, etc. store the ratios of emotions estimated by the speech processing unit 133 for each emotion.

FIG. 5 is a diagram showing an example of the emotion data 121B of the person in charge T according to the embodiment. Since the configuration of the emotion data 121B is the same as that of the emotion data 121A, its explanation is omitted. The emotion estimated from the face image of the person in charge T may also be stored in the storage unit 12 in the same manner as the emotion data 121Ato.

FIG. 6 is a diagram showing an example of the worry level data 122 according to the embodiment. The worry level data 122 includes, for example, items such as the customer U face image number, time, and worry level. The customer U face image No. and time are the same as the emotion data 121A, so description thereof will be omitted. The degree of concern estimated by the voice processing unit 133 is stored in the degree of concern item.

FIG. 7 is a diagram showing an example of empathy data 123 according to the embodiment. The empathy level data 123 includes items such as time and empathy level, for example. Since the time is the same as the time in the emotion data 121A, its explanation is omitted. The degree of empathy calculated by the voice processing unit 133 is stored in the item of degree of empathy.

FIG. 8 is a diagram showing an example of keyword data 124 according to the embodiment. The keyword data 124 includes, for example, items such as text data and frequently appearing keywords. The text data is information obtained by converting the voice data V into text. The text data is updated, for example, along with the passage of time of dialogue. A frequent keyword is a word that appears more than a predetermined number of times in a dialogue, and a plurality of words such as keyword 1 and keyword 2 may be stored.

FIG. 9 is a diagram showing an example of case text data 125 according to the embodiment. The case text data 125 includes items such as text data and similar cases, for example. Since the text data is the same as the text data in the keyword data 124, the description thereof is omitted. A similar case is a similar case extracted by the speech processing unit 133, and a plurality of similar cases such as case 1 and case 2 may be stored. Also, the case text data 125 may store a “similarity score” that indicates the degree of similarity of similar cases. A similarity score is calculated, for example, according to the closeness of the distance between sentences in the vector space.

Note that the learned model 126 stores information necessary for constructing a worry level estimation model. The information necessary for constructing the model is, specifically, the configuration of the model, the set values of the parameters to be used, and the like. For example, if the model is a CNN (Convolutional Neural Network), the configuration of the model is information indicating the number of units in each layer of the CNN input layer, intermediate layer, and output layer, the number of intermediate layers, the activation function, and so on. be. The parameters to be used are information indicating coupling coefficients and weights for coupling nodes in each hierarchy.

Here, examples of images displayed on the terminal device 20 will be described. 10 and 11 are diagrams showing examples of images displayed on the terminal device 20 according to the embodiment.

As shown in FIG. 10, images 200 to 207 are displayed on the display of the terminal device 20, for example. Images 200 to 202 show examples of button images that the person in charge T operates. For example, the image 200 is a button image for displaying the progress of dialogue in real time. An image 201 is a button image for displaying customer U's assessment data. An image 202 is a button image that is selected when ending the display of the image. Note that the example of FIG. 10 shows an example in which the image 200 is selected and the situation in which the dialogue progresses in real time is displayed.

An image 203 shows the facial images of the customer U and the person in charge T. FIG. The image 203 changes in real time, for example, according to the progress of the dialogue. A feature point may be indicated in each of the face images of the customer U and the person in charge T in the image 203 .

The image 204 shows frequent keywords. The image 204 is displayed based on the keyword data 124 notified from the server device 10, and changes in real time according to the progress of the dialogue, for example.

Image 205 shows the result of customer U's sentiment analysis. The image 205 is displayed based on the emotion data 121 notified from the server device 10, and changes in real time according to the progress of the dialogue, for example.

Image 206 shows a similar case. The image 206 is displayed based on the case text data 125 notified from the server device 10, and changes in real time according to the progress of the dialogue, for example.

An image 207 shows the snuggling state. FIG. 11 shows an example in which the image 207 is enlarged. As shown in FIG. 11, in the image 207, the degree of worry is shown on the upper side, and the degree of sympathy is shown on the lower side in chronological order. In the example of this figure, when the degree of worry is low, it is in a relaxed state, and when the degree of worry is high, it is in a serious state. In addition, when the degree of empathy is low, it indicates dissonance, and when the degree of empathy is high, it indicates that the person in charge T is accompanying the customer U, indicating that the person in charge T is close to the customer U's feelings.

Here, the flow of processing performed by the server device 10 will be described. 12 and 13 are flowcharts showing the flow of processing performed by the server device 10 according to the embodiment.

FIG. 12 shows the flow of processing performed using the image data D1 and D2. FIG. 13 shows the flow of processing performed using the voice data V. As shown in FIG.

As shown in FIG. 12, the server device 10 acquires image data D1 at time t (step S10). The server device 10 extracts facial feature points from the acquired image data D1 (step S11). The server device 10 estimates the emotion of the customer U based on the extracted feature points, and stores the estimated emotion as the emotion data 121A (step S12). The server device 10 also estimates the degree of concern of the customer U based on the feature points and the degree-of-concern estimation model extracted in step S11, and stores the estimated degree of concern as the degree-of-concern data 122 (step S13).

On the other hand, the server device 10 acquires the image data D2 at time t (step S14). The server device 10 extracts facial feature points from the acquired image data D2 (step S15). Server device 10 estimates the emotion of person in charge T based on the extracted feature points, and stores the estimated emotion as emotion data 121B (step S16).

The server device 10 estimates the degree of empathy of the customer U and the person in charge T based on the emotion of the customer U estimated in step S12 and the emotion of the person in charge T estimated in step S16, and calculates the estimated degree of empathy as This is stored as empathy data 123 (step S17).

The server device 10 advances the time from t to t+Δt (step S18), and repeats the processing shown in steps S10 to S17 until the acquisition of the image data D1 and D2 ends, such as when the dialogue ends. Regarding the process of estimating emotion, either the customer U or the person in charge T may be estimated first. Specifically, the processes shown in steps S14 to S16 may be performed before performing the processes shown in steps S10 to S12.

As shown in FIG. 13, the server device 10 acquires voice data V at time t (step S20). The server device 10 converts the obtained voice data V into text using voice recognition technology and stores it as text data of the keyword data 124 (step S21). The server device 10 extracts frequently appearing keywords from the text data, and stores the extracted frequently appearing keywords in the keyword data 124 (step S22).

The server device 10 also extracts similar cases and stores the extracted similar cases as case text data 125 (step S23). The server device 10 vector-expresses the sentence converted to text in step S21 using Doc2Vec or the like. The server device 10 refers to a database of past conversations, etc., and extracts, as similar cases, conversations having vector values close to the current conversation in the vector space. The server device 10 advances the time from t to t+Δt (step S24), and repeats the processing shown in steps S20 to S23 until acquisition of the voice data V ends, such as when the dialogue ends.

The server device 10 periodically transmits the emotion data 121 to case text data 125 (hereinafter referred to as emotion data 121 and the like) stored in the flowchart above to the terminal device 20 . The server device 10 may transmit the emotion data 121 or the like to the terminal device 20 each time it generates the emotion data 121 or the like, or may collectively transmit the emotion data 121 or the like to the terminal device 20 to some extent. .

As described above, the server device 10 of the embodiment includes the acquisition unit 130 , the image processing unit 132 and the communication unit 11 . Acquisition unit 130 acquires image data D1 and image data D2. The image processing unit 132 estimates the facial emotion of the customer U indicated by the image data D1. The image processing unit 132 estimates the emotion of the person in charge T's face indicated by the image data D2. The image processing unit 132 calculates the degree of similarity between the estimated emotion of the customer U and the emotion of the person in charge T as the degree of sympathy in the dialogue. The communication unit 11 transmits the degree of empathy calculated by the image processing unit 132 to the terminal device 20 .

Here, the server device 10 is an example of a "dialogue support server." The camera C1 is an example of a "first camera". Camera C2 is an example of a “second camera”. The image data D1 is an example of "first image data". The image data D2 is an example of "second image data". The emotion of the customer U's face indicated by the image data D1 is an example of the "first emotion". The emotion of the person in charge T's face indicated by the image data D2 is an example of the "second emotion".

As a result, the server device 10 of the embodiment can estimate the degree of empathy in the dialogue and present the estimated degree of empathy to the person in charge T. FIG. The degree of sympathy is the degree of similarity between the feelings of the customer U and the feelings of the person in charge T. Therefore, the person in charge T can confirm whether or not he/she sympathizes with the customer U in the dialogue with the customer. If the degree of empathy is low, the person in charge T can change his/her facial expression to make it closer to the emotion of the customer U, thereby increasing the degree of empathy. Therefore, the person in charge can get close to the customer's feelings.

In order to build a relationship of trust with the customer U, the usefulness of being emotionally accepted and sympathetic has been pointed out Academic Journal Vol. 39 No. 2 2016", "Imai, Yusai, Bando, ``Convincing Factors for Treatment of Elderly Cancer Patients with Metastasis'', Journal of Japanese Society of Cancer Nursing, Vol. 30 (2016), No. 3 pp.19-28”). It is also suggested that mirroring is useful as a technique for building trusting relationships. In this embodiment, the degree of similarity (synchronization rate) between the emotions of the customer U and the person in charge T is calculated from the face image as the degree of empathy, and is presented to the person in charge. As a result, even an inexperienced person in charge can accept the customer's feelings and proceed with the dialogue so as to gain sympathy, making it possible to build a relationship of trust with the customer U.

Further, in the server device 10 of the embodiment, the image processing unit 132 estimates the degree (first degree) to which the face of the customer U expresses a specific target emotion (for example, an emotion such as joy or sadness). The image processing unit 132 also estimates the degree (second degree) of the person in charge T's face indicating the specific target emotion. For example, the image processing unit 132 calculates the degree of empathy based on the absolute value of the difference between the first degree and the second degree, as shown in Equation (1). As a result, the server device 10 of the embodiment can quantitatively calculate the degree of empathy.

Further, in the server device 10 of the embodiment, the image processing unit 132 estimates the customer U's degree of concern. The image processing unit 132 estimates the output obtained by inputting the feature points of the face of the customer U indicated by the image data D1 into the worry level estimation model as the worry level. The worry level estimation model is a model obtained by machine-learning the correspondence relationship between a face image and whether or not the face is worried using a learning data set. The learning data set is a data set in which a learning face image is paired with a label indicating whether or not the learning face image is a face of concern. The degree-of-worry estimation model is a model for estimating the degree to which an input face image is the face that worries. As a result, in the server device 10 of the embodiment, the degree of worry can be calculated using a machine learning method.

Further, the server device 10 of the embodiment further includes an audio processing unit 133 . Acquisition unit 130 acquires voice data V. FIG. The voice processing unit 133 extracts frequent keywords from the voice data V. FIG. The speech processing unit 133 converts the speech contained in the speech data V into text, and extracts keywords whose frequency of appearance is equal to or higher than a threshold from the converted speech as frequent keywords. The communication unit 11 transmits the frequent keywords extracted by the speech processing unit 133 to the terminal device 20 . As a result, the server device 10 of the embodiment can present the person in charge T with words that frequently appear in the dialogue. Therefore, the person in charge T can recognize the words repeatedly spoken by the customer U in the dialogue, and can proceed with the dialogue along the matters in which the customer U is interested.

In addition, in the server device 10 of the embodiment, the speech processing unit 133 extracts similar cases based on frequent keywords extracted from the speech data V. FIG. The speech processing unit 133 uses Doc2Vec or the like to calculate the position in the vector space according to the content of the sentence obtained by converting the speech contained in the speech data V into text, and expresses it as a vector. The speech processing unit 133 extracts dialogues at close positions in the vector space as similar cases. The communication unit 11 transmits the similar cases extracted by the speech processing unit 133 to the terminal device 20 . As a result, in the server device 10 of the embodiment, it is possible to present the person in charge T with a dialogue similar to the current dialogue. Therefore, the person in charge T can recognize cases of dialogue similar to the current dialogue, and can proceed with the dialogue while referring to similar cases.

In the above-described embodiment, the case where the customer U and the person in charge T have a face-to-face conversation has been exemplified and explained. However, it is not limited to this. The dialogue support system 1 can also be applied when the customer U and the person in charge T have a dialogue remotely. In this case, the dialogue support system 1 includes two microphones M (hereinafter referred to as microphones M1 and M2). The microphone M<b>1 collects the voice of the customer U and transmits first voice data, which is the collected voice, to the server device 10 . The microphone M<b>2 collects the voice of the person in charge T and transmits second voice data, which is the collected voice, to the server device 10 . The server device 10 generates the keyword data 124 based on the first voice data notified from the microphone M1 and the second voice data notified from the microphone M2. The server device 10 also generates the case text data 125 based on the first voice data and the second voice data.

Further, in the above-described embodiment, the case in which steps S11 to S13 are sequentially executed in the flow of FIG. 12 has been exemplified and explained. However, it is not limited to this. Both steps S12 and S13 are processes using the facial feature points extracted in step S11. Therefore, either of steps S12 and S13 may be executed first. That is, step S13 may be executed after step S11.

Moreover, in the above-described embodiment, the case where the degree of empathy is calculated using the formula (1) has been exemplified and explained. However, it is not limited to this. The formula for calculating the degree of empathy may be any formula as long as it can at least calculate the degree of similarity between the emotions of the customer U and the person in charge T. Also, a table for calculating empathy, a calculation model, a learning model, or the like may be used.

In addition to formula (1), for example, formula (2) below may be used as a formula for calculating the degree of empathy. a _i and b _i in formula (2) are the same as a _i and b _i in formula (1).

When using formula (2), the degree of empathy can be calculated so as to fall within a predetermined range (here, the range of 0 to 1). As a result, for example, as shown in FIG. 11, when the degree of empathy is displayed as a time-series graph on the display of the terminal device 20, extremely low values and extremely large values are not displayed. Therefore, when the person in charge T looks at the screen, it is possible to avoid a situation in which the display resolution of the empathy level graph is coarse and it becomes difficult to grasp the change in the empathy level.

All or part of the dialogue support system 1 and the server device 10 in the above-described embodiments may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" means a medium that dynamically retains a program for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include something that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in that case. Further, the program may be for realizing a part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design and the like are included within the scope of the gist of the present invention.

DESCRIPTION OF SYMBOLS 1... Dialogue support system 10... Server apparatus (dialogue support server) 12... Storage part 120... Assessment data 121... Emotion data 122... Worry level data 123... Sympathy level data 124... Keyword data 125... Case text data 13... Control unit 130... Acquisition unit 131... Questionnaire processing unit 132... Image processing unit 133... Audio processing unit 20... Terminal device

Claims (8)

an acquisition unit that acquires first image data in which the customer's face is imaged during the dialogue between the customer and the person in charge, and second image data in which the face of the person in charge is imaged during the dialogue;
estimating a first emotion indicated by the face indicated by the first image data, estimating a second emotion indicated by the face indicated by the second image data, and calculating the first emotion and the second emotion; an image processing unit that calculates the degree of similarity as the degree of empathy during the dialogue;
a communication unit that transmits the empathy level calculated by the image processing unit to the terminal device of the person in charge;
A dialogue support server comprising: The image processing unit estimates a first degree that the face of the customer indicates the specific target emotion as the first emotion, and estimates a second degree that the face of the person in charge indicates the target emotion as the first emotion. 2 estimating as emotion, and calculating the empathy level based on the absolute value of the difference between the first degree and the second degree;
A dialogue support server according to claim 1. The image processing unit estimates, as a degree of concern, the degree of concern of the customer based on the feature points extracted from the first image data and the concern degree estimation model,
The communication unit transmits the empathy level and the distress level calculated by the image processing unit to the terminal device of the person in charge,
The worry level estimation model uses a learning data set in which a label indicating whether or not the learning face image is a face that the user is worried about is associated with the learning face image. It is a model that performs machine learning of the correspondence relationship with the face that is in question, and is a model that estimates the degree to which the input face image is the face that worries.
3. A dialogue support server according to claim 1 or 2. further comprising an audio processing unit,
The acquisition unit acquires audio data collected by a microphone that collects audio during the dialogue,
The speech processing unit converts the speech contained in the speech data into text, extracts a keyword having a frequency of appearance equal to or higher than a threshold from the textualized speech,
The communication unit transmits the keyword extracted by the voice processing unit to the terminal device of the person in charge.
A dialogue support server according to any one of claims 1 to 3. The speech processing unit refers to a dialogue database in which dialogues are stored based on the keywords, extracts dialogues in which the appearance frequency of the keywords is equal to or higher than a threshold value from the dialogue database as similar cases,
The communication unit transmits the similar case extracted by the speech processing unit to the terminal device of the person in charge.
5. A dialogue support server according to claim 4. a first camera that captures the face of the customer during conversation between the customer and the person in charge;
a second camera that captures the face of the person in charge during the dialogue;
a terminal device of the person in charge;
6. The dialogue support server according to any one of claims 1 to 5, wherein first image data captured by said first camera and second image data captured by said second camera are acquired. and a dialogue support server that transmits the empathy level to the terminal device;
A dialogue support system. A dialogue support method performed by a computer device,
An acquisition unit acquires first image data in which the customer's face is imaged during the dialogue between the customer and the person in charge, and second image data in which the face of the person in charge is imaged during the dialogue,
An image processing unit estimates a first emotion indicated by the face indicated by the first image data, estimates a second emotion indicated by the face indicated by the second image data, and calculates the first emotion. calculating the degree of similarity with the second emotion as the degree of empathy during the dialogue;
The communication unit transmits the empathy level calculated by the image processing unit to the terminal device of the person in charge;
Dialogue support method. 6. A program for causing a computer to operate as the dialogue support server according to any one of claims 1 to 5, the program for causing the computer to function as each unit included in the dialogue support server.

Images