JP7108184B2 - Keyword extraction program, keyword extraction method and keyword extraction device - Google Patents

Description

The present invention relates to a keyword extraction program, a keyword extraction method and a keyword extraction device.

There are times when it is desired to extract keywords that are important to the conversation from among the conversations between users. For example, it is conceivable to extract positive keywords that have greatly contributed to the customer's empathy from the conversation between the customer and the person in charge of customer service, and use them for subsequent customer service. Further, for example, it is conceivable to extract keywords requiring attention that were used in unfavorable customer service from conversations between customers and customer service personnel, and use them in subsequent customer service.

A digital photo frame has been proposed in which a keyword is extracted by recognizing a user's speech while an image is being displayed, and the extracted keyword is added to the image as a tag. In addition, an empathic interpretation estimating device has been proposed that presents a video of another user to a certain user, photographs the head of the user while the video is being viewed, analyzes the actions of the two users, and estimates the empathic interpretation. there is In addition, understanding that collects signals indicating the activity status of each conference participant, determines the communication difficulty level from each conference participant's utterance, and estimates the level of understanding of each conference participant based on the activity status and communication difficulty level A state estimator has been proposed.

Also, the speech rate is calculated from the user's voice, the time change of the user area is detected from the moving image of the user speaking, and the user's response evaluation value is calculated based on the speech rate and the time conversion of the user area. A response quality evaluation device has been proposed. In addition, an information processing apparatus is proposed that detects the state of each of a plurality of users using a sensor device, calculates the degree of synchronization between users based on the detected state, and changes the information presented to the user according to the degree of synchronization. It is

JP 2010-224715 A JP 2015-64827 A JP 2016-213631 A Japanese Patent Application Laid-Open No. 2017-162100 JP 2018-45676 A

Chiori Hori, Sadahiro Furui, "Speech Summary Sentence Generation Method and Its Evaluation by Extracting Words", Transactions of the Institute of Electronics, Information and Communication Engineers, J85-D-II, pp.200-209, February 2002

However, in the conventional technology for extracting keywords from conversation, there is room for improvement in the accuracy of extracting keywords that are highly important from the viewpoint of the psychological state of the user. For example, if the degree of importance is simply determined based on the frequency of appearance of keywords, there is a risk that keywords that are highly relevant to psychological states, such as customer empathy and customer service levels, will not be extracted.

An object of the present invention is to provide a keyword extraction program, a keyword extraction method, and a keyword extraction device that improve the accuracy of extracting keywords that reflect the psychological state of the user.

In one aspect, a computer-executable keyword extraction program is provided. Keywords from voice data indicating utterances made by at least one of the first user and the second user in a conversation between a first user who is a service provider and a second user who is a service receiver to detect Detecting the timing of the first action by the first user and the timing of the second action by the second user from the action data indicating the action performed by the first user and the action performed by the second user in the conversation do. An evaluation value indicating the importance of the keyword is calculated based on the relationship between the timing of the first action and the timing of the second action.

Also, in one aspect, a computer-implemented method for keyword extraction is provided. Also, in one aspect, a keyword extracting device having a storage unit and a processing unit is provided.

In one aspect, the accuracy of extracting keywords reflecting the user's mental state is improved.

It is a figure explaining the example of the keyword extraction apparatus of 1st Embodiment. It is a figure which shows the example of the information processing system of 2nd Embodiment. It is a block diagram which shows the hardware example of a conversation analysis apparatus. It is a figure which shows the keyword extraction example of 2nd Embodiment. It is a figure which shows the example of calculation of a keyword evaluation value. FIG. 11 is a block diagram showing an example of functions of a conversation analysis device according to a second embodiment; FIG. FIG. 4 is a first diagram showing an example of a table held by the conversation analysis device; FIG. 4 is a second diagram showing an example of a table held by the conversation analysis device; FIG. 11 is a flow chart showing an example of a conversation analysis procedure according to the second embodiment; FIG. It is a figure which shows the example of the information processing system of 3rd Embodiment. It is a figure which shows the example of the information processing system of 4th Embodiment. It is a figure which shows the keyword extraction example of 5th Embodiment. FIG. 14 is a flow chart showing an example of a conversation analysis procedure according to the fifth embodiment; FIG. FIG. 12 is a block diagram showing an example of functions of a conversation analysis device according to a sixth embodiment; FIG. FIG. 21 is a flow chart showing an example of a conversation analysis procedure according to the sixth embodiment; FIG.

Hereinafter, this embodiment will be described with reference to the drawings.
[First embodiment]
A first embodiment will be described.

FIG. 1 is a diagram for explaining an example of a keyword extracting device according to the first embodiment.
The keyword extraction device 10 of the first embodiment extracts important keywords reflecting the psychological state of at least one user from conversations between users. For example, the keyword extraction device 10 extracts important keywords related to the customer's degree of empathy from the conversation between the customer and the person in charge of customer service. Further, for example, the keyword extraction device 10 extracts important keywords related to the customer service level of the customer service representative from the conversation between the customer and the customer service representative.

The keyword extraction device 10 can also be called a computer or an information processing device. The keyword extraction device 10 may be a client device or a server device. Moreover, the keyword extraction device 10 may extract important keywords in real time during a conversation between users, or may extract important keywords as a batch process after the end of the conversation.

The keyword extraction device 10 has a storage section 11 and a processing section 12 . The storage unit 11 may be a volatile semiconductor memory such as a RAM (Random Access Memory), or may be a non-volatile storage such as an HDD (Hard Disk Drive) or flash memory. The processing unit 12 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). However, the processing unit 12 may include electronic circuits for specific purposes such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays). A collection of multiple processors is sometimes called a "multiprocessor" or simply a "processor."

The storage unit 11 stores voice data 13 and action data 14 . Audio data 13 and action data 14 are records of conversations between the first user and the second user. The first user is a service provider, and the second user is a service receiver. The service field includes, for example, operations performed through communication such as retail, education, and medical care. The first user is, for example, a person in charge of customer service, such as a product explainer at a store, a lecturer at an educational institution, or a doctor or counselor at a medical institution. A second user is, for example, a customer who receives service, such as a consumer visiting a store, a student attending an educational institution, or a patient visiting a medical institution.

The voice data 13 indicates an utterance made by at least one of the first user and the second user. For example, the audio data 13 is an audio signal obtained by recording speech of at least one of the first user and the second user using a microphone. In that case, the audio data 13 may be a recording of only the first user's utterances, a recording of only the second user's utterances, or a recording of both the first user's and second user's utterances. may be recorded. Also, the first user's speech and the second user's speech may be recorded using the same microphone, or may be recorded using different microphones.

Action data 14 indicates actions taken by the first user and actions taken by the second user in the conversation. Actions are physical actions that can be visually recognized by the other user, such as changes in facial expressions, gestures of the head, arms, and legs, changes in line of sight, and changes in posture. Facial expressions include smiling. Head gestures include nodding. Changing the line of sight includes looking at the other user's head. Posture changes include leaning forward.

For example, the motion data 14 is image data such as moving images captured by an image sensor of the first user and the second user. However, the motion data 14 may be sensor data generated using a sensor device other than an image sensor. For example, the motion data 14 may be head gestures detected using an accelerometer built into the headset. Alternatively, the motion data 14 may be arm gestures detected using an acceleration sensor incorporated in the wristwatch. Alternatively, the motion data 14 may be obtained by detecting a change in posture using a pressure sensor incorporated in the chair.

The data relating to the first user and the data relating to the second user in the motion data 14 may be generated using the same device or different devices. For example, when the motion data 14 is image data, both the first user and the second user may appear in an image captured using one image sensor, or may be captured using different image sensors. Different users may appear in the different images produced.

The processing unit 12 detects keywords 15 from the voice data 13 . For example, the processing unit 12 converts the voice data 13 into an uttered character string (text) by voice recognition, and searches the character string for a predetermined search target keyword. Keywords to be searched are defined in advance as a keyword list, for example. Further, for example, the processing unit 12 continuously compares the feature quantity of the speech signal of the speech and the feature quantity of the speech signal of the search target keyword by word spotting without converting the entire speech into a character string, Only recognize keywords directly.

The keyword 15 may be an utterance of the first user or an utterance of the second user. The processing unit 12 may search for a search target keyword by distinguishing between the first user's utterance and the second user's utterance, or may distinguish between the first user's utterance and the second user's utterance. You may search for the search target keyword without In addition, the processing unit 12 may search for a search target keyword by limiting to one of the first user's utterance and the second user's utterance.

The processing unit 12 also detects the timing of the action 16 (first action) by the first user and the timing of the action 17 (second action) by the second user from the action data 14 . Either of the processing of the voice data 13 and the processing of the motion data 14 may be executed first, or may be executed in parallel. The motion 16 of the first user and the motion 17 of the second user are detected separately. For example, the processing unit 12 uses image recognition to convert physical actions such as changes in facial expression, gestures of the head, arms, and legs, changes in line of sight, and changes in posture of the first user and the second user into image data. is detected from the motion data 14 . When a sensor device other than an image sensor generates motion data 14, there may be no need to perform special recognition processing.

Then, the processing unit 12 calculates an evaluation value 18 indicating the importance of the detected keyword 15 based on the relationship between the timing of the action 16 by the first user and the timing of the action 17 by the second user. . The processing unit 12 determines whether or not to extract the keyword 15 as an important keyword based on the evaluation value 18, for example. The processing unit 12 may extract the keyword 15 as a preferable keyword when the evaluation value 18 exceeds a predetermined first threshold. Moreover, the processing unit 12 may extract the keyword 15 as a keyword requiring caution when the evaluation value 18 is less than a predetermined second threshold that is smaller than the first threshold. The keyword 15 is, for example, a keyword issued within a predetermined range from the actions 16 and 17 on the time axis.

As for the relationship between the timings of the actions 16 and 17, the processing unit 12 detects that the actions 16 and 17 are the same type of actions, that the action 16 is performed first, and that the action 17 is performed within a predetermined time from the action 16. You may At this time, the condition may be that the second user has not performed an action within a predetermined period of time immediately before the action 16, that is, the first user has started the action. It can be said that this relationship reflects the psychological state of the second user.

For example, if the first user is a waiter and the second user is a customer, this relationship may be linked to the waiter's laughter, nod, or other gesture, and the customer's laughter, nod, or other gesture of the same type. indicates a synchronization that the action of Therefore, it can be presumed that this indicates that the customer is closely observing the behavior of the customer service representative, and that the customer is empathizing with the customer service representative's story, which indicates the excitement of the conversation. Further, for example, when the action 17 is performed first and the action 16 is performed within a predetermined time after the action 17, the behavior of the customer, such as laughter or nod, is interlocked with the action of the customer, such as laughter or nod. indicates a synchronization that the action of Therefore, it can be inferred that the person in charge of customer service observes the behavior of the customer well and shows good customer service.

If the actions 16 and 17 are the same kind of actions, and the action 16 is performed first and the action 17 is performed within a predetermined time after the action 16, the processing unit 12 selects the keyword 15 near the actions 16 and 17. It can be highly evaluated. A high evaluation of the keyword 15 corresponds to a good mental state of the second user. If the second user is a customer, keyword 15 may be a preferred keyword that resonates with the customer.

On the other hand, if the action 17 has not been performed within a predetermined time from the action 16, the processing unit 12 may evaluate the keyword 15 near the actions 16 and 17 low. A low rating for keyword 15 corresponds to a poor mental state of the second user. If the second user is a customer, keyword 15 may be a cautionary keyword that did not resonate with the customer.

The processing unit 12 may output the extracted important keywords. For example, the processing unit 12 may store the important keywords in a storage device included in the keyword extraction device 10 . Further, for example, the processing unit 12 may output the important keywords to an output device provided in the keyword extraction device 10, such as displaying them on a display. Also, for example, the processing unit 12 may transmit the important keyword to another information processing device via a network.

According to the keyword extraction device 10 of the first embodiment, the keyword 15 is detected from the voice data 13, and the timing of the first user's action 16 and the timing of the second user's action 17 are detected from the action data 14. be done. Then, based on the relationship between the timing of the action 16 and the timing of the action 17, the evaluation value 18 of the keyword 15 is calculated. This makes it possible to accurately extract important keywords from the viewpoint of the psychological state of at least one of the first user and the second user. Therefore, it is possible to utilize the important keywords extracted by the keyword extracting device 10 for a predetermined purpose such as improving customer service.

[Second embodiment]
Next, a second embodiment will be described.
FIG. 2 is a diagram illustrating an example of an information processing system according to the second embodiment.

The information processing system according to the second embodiment analyzes conversations and supports improvement of customer service in an industry in which customers and customer service personnel converse with each other. This information processing system can be applied to various industries such as product explanations and health guidance.

The information processing system of the second embodiment includes a management device 41 and a conversation analysis device 100 connected to a network 40. FIG. A camera device 50 is connected to the conversation analysis device 100 . The management device 41 is a terminal device used by an administrator, such as the boss of the person in charge of customer service, who instructs the person in charge of customer service. A conversation analysis device 100 is a terminal device installed at a place where a customer and a customer service representative have a conversation. For example, the conversation analysis device 100 is installed on or around the counter where the customer and the customer service representative face each other. The camera device 50 is a device having a moving image shooting function and an audio recording function. The camera device 50 is installed so that the conversation between the customer and the customer service representative can be photographed and recorded.

The camera device 50 captures images so that both the customer and the customer service representative are included in the image while the customer and the customer service representative are having a conversation. In addition, the camera device 50 records both the voice of the customer and the voice of the person in charge of customer service during the conversation between the customer and the person in charge of customer service. Conversation analysis device 100 collects image data representing captured moving images and voice data representing recorded voices, and analyzes the conversation between the customer and the customer service representative. Specifically, the conversation analysis device 100 detects a keyword uttered by a customer or a customer service representative from voice data, and detects actions of a customer and a customer service representative during a conversation from image data. The conversation analysis device 100 extracts important keywords by evaluating keywords from the actions of the customer and the person in charge of customer service. The conversation analysis device 100 reports the extracted important keywords to the management device 41 .

Conversation analysis by the conversation analysis device 100 and reporting of important keywords from the conversation analysis device 100 to the management device 41 may be performed in real time during the customer service staff's work, or as a batch process after the customer service staff's work is completed. may For example, the conversation analysis device 100 analyzes the audio data and image data output by the camera device 50 in real time, determines important keywords for each segment of the conversation, and transmits them to the management device 41 . The end of the conversation may be, for example, the end of customer service for one customer, the continuation of no speech for a predetermined period of time or more, or the lapse of a certain period of time from the start of the conversation. Also, for example, the conversation analysis device 100 saves the voice data and image data output by the camera device 50 , analyzes the voice data and the image data collectively after the end of work, and transmits important keywords to the management device 41 .

At least part of the important keywords transmitted from conversation analysis device 100 to management device 41 is displayed on the display of management device 41 . The manager may confirm the important keywords during the customer service staff's work, or may confirm the important keywords after the work is finished.

The conversation analysis device 100 corresponds to the keyword extraction device 10 of the first embodiment. Image data representing a moving image captured using the camera device 50 corresponds to the motion data 14 of the first embodiment. Audio data representing audio recorded using the camera device 50 corresponds to the audio data 13 of the first embodiment.

FIG. 3 is a block diagram showing a hardware example of the conversation analysis device.
Conversation analysis device 100 has CPU 101, RAM 102, HDD 103, image signal processing section 104, input signal processing sections 105 and 106, medium reader 107, and communication interface 108, which are connected to a bus. A CPU 101 corresponds to the processing unit 12 of the first embodiment. A RAM 102 or HDD 103 corresponds to the storage unit 11 of the first embodiment. The management device 41 can also be realized using similar hardware.

The CPU 101 is a processor that executes program instructions. The CPU 101 loads at least part of the programs and data stored in the HDD 103 into the RAM 102 and executes the programs. Note that CPU 101 may include a plurality of processor cores, and conversation analysis device 100 may include a plurality of processors. A collection of multiple processors is sometimes called a "multiprocessor" or simply a "processor."

The RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by the CPU 101 and data used by the CPU 101 for calculation. Conversation analysis device 100 may be provided with a type of memory other than RAM, and may be provided with a plurality of memories.

The HDD 103 is a nonvolatile storage that stores an OS (Operating System), software programs such as middleware and application software, and data. Conversation analysis device 100 may include other types of storage such as flash memory and SSD (Solid State Drive), or may include multiple storages.

Image signal processing unit 104 outputs an image to display 111 connected to conversation analysis device 100 according to a command from CPU 101 . As the display 111, any type of display such as a CRT (Cathode Ray Tube) display, a liquid crystal display (LCD: Liquid Crystal Display), an organic EL (OEL: Organic Electro-Luminescence) display, or the like can be used.

Input signal processing unit 105 receives an input signal from input device 112 connected to conversation analysis apparatus 100 . Input device 112 can be any type of input device, such as a mouse, touch panel, touchpad, keyboard, or the like. Also, a plurality of types of input devices may be connected to conversation analysis apparatus 100 .

Input signal processing unit 106 receives image signals and audio signals from camera device 50 connected to conversation analysis device 100 . Camera device 50 has image sensor 51 and microphone 52 . The image sensor 51 is an imaging device that converts light into an electrical signal (image signal). Any type of image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor can be used as the image sensor 51 . The microphone 52 converts sound into an electrical signal (audio signal). Any type of microphone can be used as the microphone 52, such as a dynamic microphone or a condenser microphone.

The medium reader 107 is a reading device that reads programs and data recorded on the recording medium 113 . Examples of the recording medium 113 include magnetic disks such as flexible disks (FDs) and HDDs, optical disks such as CDs (Compact Discs) and DVDs (Digital Versatile Discs), magneto-optical disks (MOs), A semiconductor memory or the like can be used. The medium reader 107 stores programs and data read from the recording medium 113 in the RAM 102 or the HDD 103, for example.

The communication interface 108 is connected to the network 40 and communicates with the management device 41 via the network 40 . The communication interface 108 may be a wired communication interface connected to a wired communication device such as a switch or router, or a wireless communication interface connected to a wireless communication device such as a base station or access point.

Next, a keyword extraction method according to the second embodiment will be described. Conversation analysis device 100 of the second embodiment estimates and extracts as important keywords preferred keywords that empathize with customers from among keywords that appear in conversations between customers and service personnel. In addition, the conversation analysis device 100 of the second embodiment estimates, from among the keywords appearing in the conversation between the customer and the customer service representative, the keyword requiring attention that did not gain the customer's sympathy and is important. Extract as keywords. As a result, it is possible to improve the customer service skills of the person in charge of customer service so that the customer's sympathy can be obtained in the next customer service.

FIG. 4 is a diagram showing an example of keyword extraction according to the second embodiment.
In order to evaluate whether the conversation between the customer and the customer service representative is lively and whether the customer sympathizes with the customer service representative's story, in the second embodiment, the behavior of the customer and the behavior of the customer service representative Detect synchronization between Synchronization in the second embodiment is a chain of actions in which the person in charge of customer service starts an action, and immediately after that, the customer performs the same type of action. In this case, it is presumed that the customer is looking carefully at the customer service representative and listening positively to what the customer service representative has to say. Thus, a keyword uttered by a customer or waiter while such synchronization is occurring may be a preferred keyword. On the other hand, keywords uttered by customers or attendants when such synchronization is not occurring may be sensitive keywords.

For example, consider the scenes 71-73 described below.
In scene 71, the person in charge of customer service laughs, and immediately after that, the customer laughs. The customer does not act within the time S1 immediately before the customer service representative's action, and the customer performs an action within the time S2 immediately after the customer service representative's action. In scene 71, the behavior of the customer and the behavior of the person in charge of customer service are synchronized, starting from the person in charge of customer service. Then, the evaluation value of the keyword "response" issued by the customer or the person in charge of customer service around the scene 71 becomes high.

In scene 72, although the customer service clerk made a laughing motion, the customer did not make any movement within the time S1 immediately before the customer service clerk's movement, and the customer did not make any movement within the time S2 immediately after the customer service clerk's movement. Customer is not taking action. In scene 72, it is determined that the customer's sympathy level is low because the customer's behavior is not synchronized with the customer service representative's behavior. Then, the evaluation value of the keyword "new function" issued by the customer or the person in charge of customer service around the scene 72 is low.

In scene 73, the customer nods his head, and immediately after that, the receptionist nods his head. The customer performs an action within the time S1 immediately before the action of the customer service representative, and the customer does not perform an action within the time S2 immediately after the action of the customer service representative. In scene 73, it is determined that the customer's sympathy level is low, since the behavior of the customer is not synchronized with the behavior of the customer service representative, starting with the customer service representative. Then, the evaluation value of the keyword "image quality" issued by the customer or the person in charge of customer service around the scene 73 becomes low.

Conversation analysis device 100 presumes that a keyword whose evaluation value is larger than threshold value T1 is a preferable keyword, and extracts it as an important keyword. In addition, conversation analysis device 100 presumes keywords with evaluation values smaller than threshold value T2 to be caution-required keywords, and similarly extracts them as important keywords. Thresholds T1 and T2 are predetermined, and T1>T2. In the above example, "response" may be extracted as an important keyword.

Here, the conversation analysis device 100 detects the actions of the customer and the person in charge of customer service from the image data. Actions to be detected are visible physical actions such as changes in facial expressions, gestures of the head, arms, and legs, changes in line of sight, and changes in posture. Facial expressions include smiling. Head gestures include nodding. A change of line of sight involves looking at the opponent's head. Posture changes include leaning forward.

The conversation analysis device 100 recognizes a customer and a customer service representative from image data by image recognition. For example, feature information of the appearance of the person in charge of customer service is registered in advance in the conversation analysis device 100, and the person in charge of customer service is recognized based on the feature information. In that case, a person other than the person in charge of serving customers is recognized as a customer. Further, the conversation analysis device 100 recognizes the types of actions of each of the customer and the customer service representative from the image data through image recognition. At this time, the size of the motion, such as the size of the nod or the size of the gesture of the arm, may be recognized together.

A technique described in Patent Document 2 (Japanese Patent Application Laid-Open No. 2015-64827) may be used for motion detection. For example, with respect to facial expressions, speech analysis device 100 extracts the contours of the eyes and mouth from each frame of the image data, and determines changes in facial expressions from changes in the contours between frames. Further, for example, regarding a nod, the conversation analysis device 100 extracts the positions of the eyes, nose, and mouth from each frame of the image data, and determines the nod from changes in the positions of the eyes, nose, and mouth between frames. The magnitude of motion can be determined from the amount of change.

In addition, the conversation analysis device 100 recognizes keywords from voice data by voice recognition. At this time, the conversation analysis device 100 may recognize the customer's utterance and the customer service clerk's utterance by distinguishing between them, or may recognize them without distinguishing between them. As a method of distinguishing between the customer's utterance and the customer service staff's utterance, for example, characteristic information of the customer service staff's voice quality is registered in advance in the conversation analysis device 100, and out of the two different voice quality utterances, the customer service staff's A possible method is to determine one utterance first and regard the other utterance as the customer's utterance. Also, as a method of distinguishing between the customer's utterance and the customer service clerk's utterance, for example, a method of judging from the direction of arrival of voice during recording is conceivable.

The technique described in Patent Document 1 (Japanese Unexamined Patent Application Publication No. 2010-224715) may be used for keyword detection. For example, in the conversation analysis device 100, search target keywords that may affect the customer's empathy are registered in advance. The conversation analysis device 100 converts the speech waveform indicated by the speech data into speech feature information by Fourier transform or the like, inputs the speech feature information into a speech recognition model prepared in advance, converts it into a word string, and searches from the word string. Search for the target keyword. However, conversation analysis device 100 may directly detect only the search target keyword by word spotting without converting the entire utterance into a word string.

Next, a method of calculating the keyword evaluation value will be described.
FIG. 5 is a diagram showing an example of calculating a keyword evaluation value.
First, the conversation analysis device 100 calculates a scene evaluation value at predetermined time intervals using image data in which a customer and a customer service representative are recorded. In the second embodiment, the scene evaluation value indicates the liveliness of the conversation between the customer and the customer service representative at that time, and corresponds to the customer's degree of empathy. It is estimated that the greater the scene evaluation value, the greater the liveliness of the conversation, and the greater the degree of empathy. It is estimated that the smaller the scene evaluation value, the smaller the liveliness of the conversation and the smaller the degree of empathy.

Conversation analysis device 100 then calculates a keyword evaluation value for the keyword extracted from the voice data using the scene evaluation values around the time when the keyword was uttered. A keyword with a higher keyword evaluation value is more likely to be a keyword that has gained customer sympathy. A keyword with a lower keyword evaluation value is more likely to be a keyword that did not gain customer sympathy. Conversation analysis device 100 extracts keywords whose keyword evaluation value exceeds threshold value T1 as important keywords. Conversation analysis device 100 also extracts keywords whose keyword evaluation value is less than threshold value T2 as important keywords.

In calculating the scene evaluation value, conversation analysis device 100 sets a section from before time S0 to after time S0 around a certain time (section of time S0 before and after) as sliding window 81 . The position of the sliding window 81 is shifted by time Δt. The time Δt is, for example, about 1 frame time to 1 second. The time S0 is, for example, approximately 1 to 2 minutes. Since one scene evaluation value is calculated for the center time of the sliding window 81, the scene evaluation value is calculated at intervals of time Δt.

In the sliding window 81, let F(x) be the time at which the person in charge of customer service performed the action, and let w(x) be the weight of the action at time F(x). Weight w(x) is determined based on the customer's actions during time S1 immediately preceding time F(x). The time S1 is, for example, about 1 to 2 seconds. Weight w(x)=w1 if the customer did not make a move during the previous time S1, and weight w(x)=w2 if the customer made a move during the previous time S1. . However, the magnitude relationship between the weight w1 and the weight w2 is w1>w2.

Further, each action belonging to the sliding window 81 is classified into sets r1 and r2 according to the presence or absence of synchronization of the customer. If the customer performs the same kind of action during the time S2 immediately after the time F(x), that is, if there is synchronization, the action of the customer service representative is classified into the set r1. If the customer does not perform the same type of action during the immediately following time S2, ie, no synchronization, the waiter's actions are classified into set r2. The time S2 is, for example, about 1 to 2 seconds and may be the same as the time S1. Also, the value of the coefficient a used below is determined in advance. The value of the coefficient a is a real number less than 1 (a<1) and may be a negative value.

From such a sliding window 81, for example, the scene evaluation value Vt at the center time is calculated as shown in Equation (1). That is, the scene evaluation value is the average of the weights of the motions belonging to the set r1 and the weights of the motions belonging to the set r2 multiplied by the coefficient a. Since the value of the coefficient a is less than 1, the scene evaluation value increases as the percentage of the customer service clerk's actions belonging to the sliding window 81 that are synchronized with the customer's actions increases.

However, when the behavior of the person in charge of customer service and the behavior of the customer are synchronized, it is also possible to calculate the scene evaluation value by further considering the delay time from the behavior of the person in charge of customer service to the behavior of the customer. Let E(x) be the delay time from time F(x), and determine the value of the coefficient b in advance. The value of coefficient b is a positive real number. In this case, for example, the scene evaluation value Vt at the center time is calculated as shown in Equation (2). In equation (2), the weights of actions belonging to set r1 are corrected by delay time E(x) and coefficient b. In the sliding window 81, the shorter the delay time from the customer's action to the customer's action, the larger the scene evaluation value.

In addition, when the behavior of the person in charge of customer service and the behavior of the customer are synchronized, it is also possible to calculate the scene evaluation value by further considering the magnitude of the behavior of the customer. Let D(x) be the magnitude of the customer's movement synchronized with the customer's movement, and determine the value of the coefficient c in advance. The magnitude D(x) indicates the amount of positional change per unit time recognized from the image data, such as the amount of change in the position of the head or the amount of movement of the arm. take a value. The value of coefficient c is a positive real number. In this case, for example, the scene evaluation value Vt at the central time is calculated as shown in Equation (3). In equation (3), the weights of actions belonging to set r1 are corrected by magnitude D(x) and coefficient c. In the sliding window 81, the scene evaluation value increases as the customer's motion synchronized with the customer's motion increases. Although the delay time E(x) is also taken into account in Equation (3), the delay time E(x) may not be taken into account.

In calculating the keyword evaluation value, conversation analysis device 100 sets a window 82 as a window 82 from time S3 before time S3 to time S3 after time G(y) when the keyword was issued (time S3 before and after). do. The time S3 is, for example, about two minutes to several minutes. The same keyword may be uttered multiple times in a single segment of conversation (for example, a series of conversations from the beginning to the end of serving a customer). Here, it is assumed that attention is paid to one keyword, and that the keyword is uttered Y+1 times in one segment of conversation. A window 82 is set for each of y=0, 1, .

Let H(y) be the average value of the scene evaluation values within the range of the window 82 for one appearance of the keyword of interest. Then, for example, the keyword evaluation value Vk of the keyword in one segment of conversation is calculated as shown in Equation (4). That is, the keyword evaluation value of the keyword is obtained by averaging the average values of the keyword evaluation values around each time when the keyword was issued over the multiple occurrences of the keyword.

If the keyword evaluation value exceeds the threshold T1, the keyword is an important keyword. Also, if the keyword evaluation value is less than the threshold T2, the keyword is an important keyword. Conversation analysis device 100 transmits the extracted important keywords to management device 41 . Management device 41 displays all or part of the important keywords received from conversation analysis device 100 on the display. For example, the management device 41 displays the N (top N) important keywords having the highest keyword evaluation value among the received important keywords. In addition, the management device 41 displays the N (lowest N) important keywords among the received important keywords having the lowest keyword evaluation values. N is a predetermined integer of 1 or more. However, the conversation analysis device 100 may narrow down the important keywords to the top N items and the bottom N items.

Next, the functions and processing procedures of conversation analysis device 100 will be described.
FIG. 6 is a block diagram showing a functional example of the conversation analysis device of the second embodiment.
Conversation analysis device 100 has voice storage unit 121 , image storage unit 122 , keyword storage unit 123 and evaluation result storage unit 124 . These storage units are implemented using storage areas of the RAM 102 or the HDD 103, for example. Conversation analysis device 100 also has keyword detection section 125 , action detection section 126 , scene evaluation section 127 and keyword evaluation section 128 . These processing units are implemented using, for example, programs.

The audio storage unit 121 stores audio data including audio signals received from the camera device 50 . The image storage unit 122 stores image data including image signals received from the camera device 50 . The keyword storage unit 123 stores search target keywords. A search target keyword is specified in advance. The administrator may be allowed to add or delete search target keywords. The evaluation result storage unit 124 stores keyword evaluation results. The evaluation results include the important keywords extracted from the voice data and their ranks.

The keyword detection unit 125 converts the voice data stored in the voice storage unit 121 into a word string by voice recognition. The keyword detection unit 125 detects the search target keyword stored in the keyword storage unit 123 from the word string. The keyword detection result includes the detected keyword and the time when the keyword appears.

The motion detection unit 126 recognizes the customer and the customer service representative by image recognition from each frame included in the image data stored in the image storage unit 122 . The motion detection unit 126 extracts the characteristic information of the area in which the customer service representative is captured from each frame, and detects the customer service representative's movement based on the change in the characteristic information between the frames. In addition, the motion detection unit 126 extracts the feature information of the area in which the customer is captured from each frame, and detects the motion of the customer based on changes in the feature information between frames. The motion detection result includes the time, the subject of the motion, the type of motion, and the magnitude of the motion.

The scene evaluation unit 127 calculates a scene evaluation value at intervals of time Δt using the motion detection result obtained by the motion detection unit 126 . As described above, the scene evaluation unit 127 determines whether or not the customer's action occurs during the time S1 immediately before the customer service staff's action time, and whether the customer's action occurs during the time S2 immediately after that. Determine if the same type of customer activity is occurring. The scene evaluation unit 127 calculates a scene evaluation value based on the state of synchronization between the behavior of the person in charge of customer service and the behavior of the customer. The evaluation of the synchronization status may further take into account the delay time from the customer's action to the customer's action, and the magnitude of the customer's action. The scene evaluation result includes multiple times and multiple scene evaluation values corresponding to the multiple times.

The keyword evaluation unit 128 calculates a keyword evaluation value for each detected keyword using the keyword detection result by the keyword detection unit 125 and the scene evaluation result by the scene evaluation unit 127 . As described above, the keyword evaluation unit 128 extracts one or more appearance times of the keyword for each keyword, averages the scene evaluation values of surrounding times for each appearance time, Further, the values are averaged to obtain a keyword evaluation value.

The keyword evaluation unit 128 extracts keywords whose keyword evaluation values exceed the threshold T1 and keywords whose keyword evaluation values are less than the threshold T2 as important keywords. Then, the keyword evaluation unit 128 stores rankings (best 1, best 2, worst 1, worst 2, etc.) determined by the extracted important keywords and keyword evaluation values in the evaluation result storage unit 124 as evaluation results. The keyword evaluation unit 128 transmits evaluation results to the management device 41 .

FIG. 7 is a first diagram showing an example of a table held by the conversation analysis device.
The keyword table 131 is stored in the keyword storage unit 123. FIG. In the keyword table 131, character strings of keywords specified as search target keywords are registered. Only the keywords registered in the keyword table 131 are extracted from the speech indicated by the voice data, and no other words are extracted.

The keyword detection table 132 is generated by the keyword detection unit 125. FIG. Keyword detection table 132 may be stored in RAM 102 or HDD 103 . The keyword detection table 132 includes time and keyword items. In the item of time, the time at which any keyword registered in the keyword table 131 was issued is registered. The issued keyword is registered in the keyword item. In addition, when distinguishing and recognizing an utterance by a customer and an utterance by a person in charge of customer service, the keyword detection table 132 may further include an item indicating a speaker. A speaker is a customer or a waiter.

The motion detection table 133 is generated by the motion detection unit 126 . The motion detection table 133 may be stored in the RAM 102 or HDD 103. FIG. The motion detection table 133 includes items of time, subject of motion, type and size. The time at which the action was performed is registered in the item of time. In the action subject item, a "customer" or a "person in charge of customer service" is registered as the subject who performed the action. In the type item, types of actions such as "laughing" and "nodding" are registered. A numerical value indicating the magnitude of the motion is registered in the magnitude item.

FIG. 8 is a second diagram showing an example of a table held by the conversation analysis device.
A scene evaluation table 134 is generated by the scene evaluation unit 127 . Scene evaluation table 134 may be stored in RAM 102 or HDD 103 . The scene evaluation table 134 includes items of time and evaluation value. The time item registers the time at which the degree of liveliness of the conversation was evaluated, that is, the time at which the customer's degree of empathy was evaluated. The calculated scene evaluation value is registered in the evaluation value item.

Keyword evaluation table 135 is generated by keyword evaluation unit 128 . Keyword evaluation table 135 may be stored in RAM 102 or HDD 103 . The keyword evaluation table 135 includes items of keywords and evaluation values. Keywords appearing in the keyword detection table 132 are registered in the keyword field. The calculated keyword evaluation value is registered in the evaluation value item.

The important keyword table 136 is stored in the evaluation result storage unit 124. FIG. The important keyword table 136 includes ranking and keyword items. In the rank item, ranks of important keywords such as best 1, best 2, worst 1, worst 2, etc. determined by keyword evaluation values are registered. An important keyword selected based on the keyword evaluation value among the keywords registered in the keyword evaluation table 135 is registered in the keyword item. The contents of the important keyword table 136 are sent to the management device 41 .

FIG. 9 is a flow chart showing an example of a conversation analysis procedure according to the second embodiment.
( S<b>10 ) The motion detection unit 126 reads image data from the image storage unit 122 . The image data to be read is the image data for a certain period of time following the processed image data. Also, the keyword detection unit 125 reads the voice data from the voice storage unit 121 . The audio data to be read is the audio data for a certain period of time following the processed audio data.

(S11) The keyword detection unit 125 converts the voice data read in step S10 into a word string by voice recognition. The keyword detection unit 125 searches for search target keywords registered in the keyword table 131 from the converted word string, and generates a keyword detection table 132 indicating the searched keywords and the appearance time of the keywords.

(S12) The motion detection unit 126 recognizes the area in which the customer and the customer service representative appear by image recognition from each frame included in the image data read out in step S10. The motion detection unit 126 detects the type of motion of the customer, the time of the motion, and the magnitude of the motion from the position change between frames. Further, the motion detection unit 126 detects the type of motion, the time of motion, and the magnitude of motion of the person in charge of customer service from the position change between frames. The motion detector 126 generates a motion detection table 133 including the detected information.

(S13) The scene evaluation unit 127 extracts the action time of the person in charge of customer service from the action detection table 133 generated in step S12.
(S14) The scene evaluation unit 127 searches the action detection table 133 for the previous customer action for each action time of the person in charge of customer service extracted in step S13, determines the presence or absence of the customer action, and determines the presence or absence of the customer action. determine the weights. Specifically, the scene evaluation unit 127 selects the weight w1 when there is no customer movement during the previous time S1, and selects a weight smaller than the weight w1 when there is a customer movement during the previous time S1. Select w2.

(S15) The scene evaluation unit 127 searches the action detection table 133 for the immediately following customer action for each action time of the customer service representative extracted in step S13, determines whether there is synchronization by the same kind of customer action, Decide the coefficient according to the presence or absence of Specifically, when there is synchronization, that is, when there is the same type of customer action during the immediately following time S2, the scene evaluation unit 127 selects coefficient=1. On the other hand, if there is no synchronization, that is, if there is no customer action of the same type during the immediately following time S2, the scene evaluator 127 selects coefficient=a. These coefficients are weights multiplied by a<1.

(S16) The scene evaluation unit 127 sets a sliding window having a time width of time S0×2. The scene evaluation unit 127 calculates the scene evaluation value at the center time of the sliding window using the weights and coefficients calculated in steps S14 and S15 for the behavior of the customer service representative belonging to the sliding window. This scene evaluation value represents the customer's degree of empathy. The scene evaluation unit 127 calculates a scene evaluation value at intervals of time Δt by sliding the sliding window by time Δt. The scene evaluation unit 127 generates a scene evaluation table 134 indicating scene evaluation values for each of a plurality of times.

(S17) The motion detector 126 determines whether the image data has ended. The keyword detection unit 125 determines whether the voice data has ended. For example, image data and voice data end when service to one customer ends. If the image data and the audio data are completed, the process proceeds to step S18, and if not completed, the process proceeds to step S10.

(S18) The keyword evaluation unit 128 extracts the appearance times of keywords from the keyword detection table 132 generated in step S11. For each appearance time of the keyword, the keyword evaluation unit 128 searches the scene evaluation table 134 generated in step S16 for surrounding scene evaluation values belonging to the time S3 immediately before and the time S3 immediately after the appearance time. The keyword evaluation unit 128 calculates an average value of surrounding scene evaluation values.

(S19) The keyword evaluation unit 128 classifies the average values of the scene evaluation values calculated in step S18 according to the identity of the keywords. The keyword evaluation unit 128 calculates a keyword evaluation value by further averaging the average value of the scene evaluation values for each keyword. The keyword evaluation unit 128 generates a keyword evaluation table 135 indicating keyword evaluation values.

(S20) The keyword evaluation unit 128 extracts keywords whose keyword evaluation value exceeds the threshold value T1 and keywords whose keyword evaluation value is less than the threshold value T2 as important keywords from the keyword evaluation table 135 generated in step S19. The keyword evaluation unit 128 generates an important keyword table 136 showing the extracted important keywords and their ranks, and stores it in the evaluation result storage unit 124 . The keyword evaluation unit 128 transmits the contents of the important keyword table 136 to the management device 41 . The management device 41 displays the top N and bottom N important keywords based on the contents of the important keyword table 136 .

According to the information processing system of the second embodiment, keywords are detected from voice data, and actions of a customer and actions of a customer service representative are detected from image data. Synchronization is detected in which the customer performs an action first and the customer performs the same type of action immediately after, and a scene evaluation value indicating the customer's degree of empathy is calculated based on the synchronization of actions, and the surroundings of the keyword are calculated. A keyword evaluation value is calculated from the scene evaluation value of the time. Then, a desirable keyword with a high keyword evaluation value and a keyword requiring caution with a low keyword evaluation value are extracted and reported to the administrator.

This makes it possible to estimate important keywords that have a high possibility of having a positive impact on the customer's psychological state and important keywords that have a high possibility of having a negative impact on the customer's psychological condition. can help improve customer service skills. In addition, since keywords are evaluated based on the synchronization between the behavior of the customer service staff and the behavior of the customer, it is possible to accurately extract important keywords that reflect the customer's psychological state compared to methods such as evaluation based on the number of occurrences of keywords. can. In addition, in order to determine the condition that the customer did not perform an action immediately before the customer service representative's action, and that the customer performed the same type of action immediately after the customer service representative's action, the psychological state of the customer receiving the service was determined. can be estimated with high accuracy.

[Third embodiment]
Next, a third embodiment will be described. Differences from the second embodiment will be mainly described, and descriptions of the same contents as those of the second embodiment may be omitted. The information processing system of the third embodiment differs from that of the second embodiment in the arrangement of conversation analysis device 100 .

FIG. 10 is a diagram illustrating an example of an information processing system according to the third embodiment.
The information processing system of the third embodiment includes management device 41, camera device 50 and conversation analysis device 100, as in the second embodiment. However, in the third embodiment, camera device 50 is connected to network 40 and conversation analysis device 100 operates as a server device that communicates with camera device 50 via network 40 . Camera device 50 transmits an audio signal and an image signal to conversation analysis device 100 via network 40 . According to the information processing system of the third embodiment, effects similar to those of the second embodiment are obtained.

[Fourth embodiment]
Next, a fourth embodiment will be described. Differences from the second embodiment will be mainly described, and descriptions of the same contents as those of the second embodiment may be omitted. The information processing system according to the fourth embodiment is applied to a business in which a person in charge of customer service attends a customer remotely.

FIG. 11 is a diagram illustrating an example of an information processing system according to the fourth embodiment.
The information processing system of the fourth embodiment includes user device 42 , camera devices 50 and 60 and conversation analysis device 100 . A camera device 60 is connected to the user device 42 . A camera device 50 is connected to the conversation analysis device 100 . User device 42 and speech analysis device 100 are connected to network 40 .

The user device 42 is a terminal device installed at a location different from the conversation analysis device 100, such as the customer's home, and used by the customer. A conversation analysis device 100 is a terminal device installed in an office or the like and used by a person in charge of customer service. The camera devices 50 and 60 are devices having a moving image shooting function using an image sensor and an audio recording function using a microphone. The camera device 50 is set to photograph the person in charge of customer service and record the speech of the person in charge of customer service. A camera device 60 is installed to photograph the customer and record the customer's speech.

User device 42 collects image data of the customer and voice data of recorded voice of the customer, and transmits the data to conversation analysis device 100 via network 40 . Further, the user device 42 receives image data of the reception staff and audio data of the reception staff recorded from the conversation analysis device 100 via the network 40 . The user device 42 displays the image of the customer service representative on the display based on the received image data, and reproduces the customer service representative's speech from the speaker based on the received audio data.

Conversation analysis device 100 collects the image data of the customer service representative and the voice data of the customer service representative, and transmits them to user device 42 via network 40 . Conversation analysis device 100 also receives customer image data and customer voice data from user device 42 via network 40 . Conversation analysis device 100 displays the customer's video on the display based on the received image data, and reproduces the customer's utterance from the speaker based on the received audio data. As a result, the customer and the customer service representative can have a teleconference conversation.

In addition, the conversation analysis device 100, based on the image data of the customer, the image data of the reception staff, the voice data of the customer, and the voice data of the reception staff, analyzes the relationship between the customer and the reception staff in the same manner as in the second embodiment. analyze conversations between That is, the conversation analysis device 100 detects a keyword uttered by a customer or a customer service representative from voice data, and detects the customer's and customer service representative's actions during a conversation from image data. The conversation analysis device 100 extracts important keywords by evaluating keywords from the actions of the customer and the person in charge of customer service. Conversation analysis device 100 displays the top N and bottom N important keywords of the extracted important keywords on the display.

However, conversation analysis device 100 may transmit important keywords to management device 41 via network 40 . Also, the terminal device used by the person in charge of customer service may be separated from the conversation analysis device 100 . According to the information processing system of the fourth embodiment, the same effects as those of the second embodiment can be obtained for remote customer service.

[Fifth embodiment]
Next, a fifth embodiment will be described. Differences from the second embodiment will be mainly described, and descriptions of the same contents as those of the second embodiment may be omitted. In the second embodiment, the degree of empathy of the customer is estimated from the state of synchronization between the behavior of the customer and the behavior of the person in charge of customer service, and the degree of importance of the keyword is evaluated according to the degree of empathy of the customer. On the other hand, in the fifth embodiment, the degree of customer service of the person in charge of customer service is estimated from the state of synchronization between the behavior of the customer and that of the person in charge of customer service, and the degree of importance of the keyword is evaluated according to the degree of customer service of the person in charge of customer service. do. The degree of customer service represents the customer service attitude including positiveness, enthusiasm, and politeness in customer service. The scene evaluation value calculated in the fifth embodiment corresponds to the degree of customer service of the person in charge of customer service, and the keyword evaluation value calculated in the fifth embodiment corresponds to the degree of customer service when the customer or the person in charge of customer service issues a keyword. It reflects the degree of customer service. Therefore, the important keywords extracted in the fifth embodiment are keywords presumed to have a strong relationship with good customer service and keywords presumed to have a strong relationship with bad customer service. It can be said that the important keywords extracted in the fifth embodiment reflect the mental state of the person in charge of customer service.

The information processing system of the fifth embodiment can be realized by a configuration similar to that of the information processing system of the second embodiment shown in FIGS. Therefore, hereinafter, the fifth embodiment may be described using the same reference numerals as in FIGS. Note that the information processing system of the fifth embodiment can have the same system configuration as the information processing system of the third embodiment shown in FIG. 10, and the fourth embodiment shown in FIG. A system configuration similar to that of the information processing system of the embodiment is also possible.

FIG. 12 is a diagram showing an example of keyword extraction according to the fifth embodiment.
In a fifth embodiment, synchronization between the customer's actions and the waiter's actions is detected. Synchronization in the fifth embodiment is a chain of actions in which the customer initiates an action and immediately after that, the customer service representative performs the same kind of action. The synchronization detected in the fifth embodiment differs from the synchronization in the second embodiment in that the order of operations is different. In this case, it is presumed that the customer service representative is watching the customer carefully and listening carefully to what the customer has to say. Thus, keywords uttered by customers or waitstaff when such synchronization occurs are likely to be keywords associated with good customer service. On the other hand, keywords uttered by customers or attendants when such synchronization has not occurred may be keywords associated with bad customer service.

For example, consider the scenes 74-76 described below.
In scene 74, the waiter makes a laughing action, and immediately after that, the customer makes a laughing action. Scene 74 corresponds to scene 71 in FIG. The customer service clerk performs an action within the time S1 immediately before the customer's action, and the customer service clerk does not perform an action within the time S2 immediately after the customer's action. In scene 74, starting from the customer, the behavior of the customer and the behavior of the person in charge of customer service are not synchronized, and it is determined that the degree of customer service is low. Then, the evaluation value of the keyword "fast" uttered by the customer or the customer service representative around the scene 74 is low.

In scene 75, although the customer makes an action of laughing, the customer service representative does not make any action within the time S1 immediately before the customer's action, and the customer service representative does not make any action within the time S2 immediately after the customer's action. not operating. In scene 75, since the behavior of the customer and the behavior of the person in charge of customer service are not synchronized, it is determined that the degree of customer service is low. Then, the evaluation value of the keyword "interesting" uttered by the customer or the person in charge of customer service around the scene 75 is low.

In scene 76, the customer nods, and immediately after that, the customer service representative nods. Scene 76 corresponds to scene 73 in FIG. Within time S1 immediately before the customer's action, the waiter did not perform any action, and within time S2 immediately after the customer's action, the waiter performed the same kind of action. Starting from the customer, the behavior of the customer and the behavior of the person in charge of customer service are synchronized, so it is determined that the degree of customer service is high. Then, the evaluation value of the keyword "beautiful" uttered by the customer or the person in charge of customer service around the scene 76 becomes high.

When the keyword evaluation value is calculated, similarly to the second embodiment, conversation analysis device 100 presumes that the keyword whose keyword evaluation value is larger than threshold value T1 is a preferable keyword, and extracts it as an important keyword. In addition, conversation analysis device 100 presumes keywords whose keyword evaluation value is smaller than threshold value T2 to be caution-required keywords, and extracts them as important keywords. In the above example, "beautiful" may be extracted as an important keyword.

FIG. 13 is a flow chart showing an example of a conversation analysis procedure according to the fifth embodiment.
(S30) The motion detection unit 126 reads image data from the image storage unit 122. FIG. Also, the keyword detection unit 125 reads the voice data from the voice storage unit 121 .

(S31) The keyword detection unit 125 converts the voice data read in step S30 into a word string by voice recognition. The keyword detection unit 125 searches for search target keywords registered in the keyword table 131 from the converted word string, and generates a keyword detection table 132 indicating the searched keywords and the appearance time of the keywords.

(S32) The motion detection unit 126 recognizes the area in which the customer and the customer service representative appear by image recognition from each frame included in the image data read out in step S30. The motion detection unit 126 detects the type of motion of the customer, the time of the motion, and the magnitude of the motion from the position change between frames. Further, the motion detection unit 126 detects the type of motion, the time of motion, and the magnitude of motion of the person in charge of customer service from the position change between frames. The motion detector 126 generates a motion detection table 133 including the detected information.

(S33) The scene evaluation unit 127 extracts the customer's action time from the action detection table 133 generated in step S32.
(S34) The scene evaluation unit 127 searches the motion detection table 133 for the immediately preceding service staff motion for each of the customer action times extracted in step S33, determines whether or not there is a service staff motion, and determines whether or not there is a service staff motion. Determine the weight according to the presence or absence of Specifically, the scene evaluation unit 127 selects the weight w1 if there is no action of the customer service representative during the immediately preceding time S1, and selects the weight w1 if there is any action of the customer service representative during the immediately preceding time S1. Select w2.

(S35) The scene evaluation unit 127 searches the motion detection table 133 for the immediately following service staff motion for each of the customer action times extracted in step S33, and determines whether there is synchronization by the same kind of service staff motion. , determines the coefficient according to the presence or absence of synchronization. Specifically, when there is synchronization, that is, when there is the same type of customer service representative action during the immediately following time S2, the scene evaluation unit 127 selects coefficient=1. On the other hand, if there is no synchronization, that is, if there is no same type of customer service representative action during the immediately following time S2, the scene evaluation unit 127 selects coefficient=a. These coefficients are weights multiplied by a<1.

(S36) The scene evaluation unit 127 sets a sliding window having a time width of time S0×2. The scene evaluation unit 127 calculates the scene evaluation value at the center time of the sliding window using the weights and coefficients calculated in steps S34 and S35 for the customer's motion belonging to the sliding window. This scene evaluation value represents the customer service level of the person in charge of customer service. The scene evaluation unit 127 calculates a scene evaluation value at intervals of time Δt by sliding the sliding window by time Δt. The scene evaluation unit 127 generates a scene evaluation table 134 indicating scene evaluation values for each of a plurality of times.

(S37) The motion detector 126 determines whether the image data has ended. The keyword detection unit 125 determines whether the voice data has ended. If the image data and the audio data have ended, the process proceeds to step S38, and if not, the process proceeds to step S30.

(S38) The keyword evaluation unit 128 extracts the appearance times of keywords from the keyword detection table 132 generated in step S31. For each appearance time of the keyword, the keyword evaluation unit 128 searches the scene evaluation table 134 generated in step S36 for surrounding scene evaluation values belonging to the time S3 immediately before and the time S3 immediately after the appearance time. The keyword evaluation unit 128 calculates an average value of surrounding scene evaluation values.

(S39) The keyword evaluation unit 128 classifies the average values of the scene evaluation values calculated in step S38 according to the identity of the keywords. The keyword evaluation unit 128 calculates a keyword evaluation value by further averaging the average value of the scene evaluation values for each keyword. The keyword evaluation unit 128 generates a keyword evaluation table 135 indicating keyword evaluation values.

(S40) The keyword evaluation unit 128 extracts keywords whose keyword evaluation value exceeds the threshold value T1 and keywords whose keyword evaluation value is less than the threshold value T2 as important keywords from the keyword evaluation table 135 generated in step S39. The keyword evaluation unit 128 generates an important keyword table 136 showing the extracted important keywords and their ranks, and stores it in the evaluation result storage unit 124 . The keyword evaluation unit 128 transmits the contents of the important keyword table 136 to the management device 41 . The management device 41 displays the top N and bottom N important keywords based on the contents of the important keyword table 136 .

According to the information processing system of the fifth embodiment, a keyword is detected from voice data, and a customer's action and a customer service representative's action are detected from image data. Synchronization is detected in which the customer performs an action first and the receptionist performs the same kind of action immediately after that, and a scene evaluation value indicating the degree of customer service is calculated based on the synchronization of the action, and the time around the keyword is calculated. A keyword evaluation value is calculated from the scene evaluation value. Then, a desirable keyword with a high keyword evaluation value and a keyword requiring caution with a low keyword evaluation value are extracted and reported to the administrator.

As a result, it is possible to estimate the important keywords that are highly likely to be related to the customer service staff's good customer service attitude and the important keywords that are highly likely to be related to the customer customer service attitude of the customer service staff. can help improve In addition, since the keywords are evaluated based on the synchronization between the customer's actions and the customer service staff's actions, it is possible to accurately extract important keywords that reflect the customer service staff's psychological state, compared to methods such as evaluation based on the number of occurrences of keywords. be able to. In addition, in order to determine the condition that the customer service clerk did not make a movement immediately before the customer's movement and that the customer service clerk performed the same kind of movement immediately after the customer's movement, It is possible to accurately estimate the psychological state of

[Sixth embodiment]
Next, a sixth embodiment will be described. Differences from the second embodiment will be mainly described, and descriptions of the same contents as those of the second embodiment may be omitted. In the second embodiment, search target keywords to be detected from voice data are specified in advance. On the other hand, in the sixth embodiment, search target keywords are automatically added through conversation analysis, thereby reducing the burden of manually designating search target keywords.

An information processing system according to the sixth embodiment can be realized by a system configuration similar to that of the information processing system according to the second embodiment shown in FIG. The information processing system of the sixth embodiment can have the same system configuration as the information processing system of the third embodiment shown in FIG. It is also possible to have a system configuration similar to the information processing system of the form. However, instead of the conversation analysis device 100, a conversation analysis device 200, which will be described later, is used. A conversation analysis device 200 according to the sixth embodiment can be realized by a hardware configuration similar to that of the information processing system according to the second embodiment shown in FIG. It should be noted that it is also possible to extract important keywords that reflect the degree of customer service, as in the fifth embodiment.

FIG. 14 is a block diagram showing a functional example of the conversation analysis device of the sixth embodiment.
Conversation analysis device 200 has voice storage unit 221 , image storage unit 222 , keyword storage unit 223 and evaluation result storage unit 224 . These storage units are implemented using, for example, RAM or HDD storage areas. Conversation analysis device 200 also has keyword detection section 225 , motion detection section 226 , scene evaluation section 227 , word extraction section 228 and keyword evaluation section 229 . These processing units are implemented using, for example, programs.

The audio storage unit 221 stores audio data including audio signals received from the camera device 50 . The image storage unit 222 stores image data including image signals received from the camera device 50 . The keyword storage unit 223 stores the keyword table 131 shown in FIG. The keyword table 131 registers search target keywords that are specified in advance by an administrator or the like. Further, search target keywords automatically added by the keyword evaluation unit 229 are registered in the keyword table 131 . The evaluation result storage unit 224 stores the important keyword table 136 shown in FIG.

The keyword detection unit 225 converts the voice data stored in the voice storage unit 221 into a word string by voice recognition. The keyword detection unit 225 detects the search target keyword indicated by the keyword table 131 stored in the keyword storage unit 223 from the word string, and generates the keyword detection table 132 shown in FIG.

The motion detection unit 226 recognizes the customer and the customer service representative by image recognition from each frame included in the image data stored in the image storage unit 222 . The motion detection unit 226 extracts the characteristic information of the area in which the customer service representative is captured from each frame, and detects the customer service representative's movement based on the change in the characteristic information between the frames. Further, the motion detection unit 226 extracts the feature information of the area in which the customer is captured from each frame, and detects the motion of the customer based on the change in the feature information between the frames. The motion detection unit 226 generates the motion detection table 133 shown in FIG.

The scene evaluation unit 227 calculates scene evaluation values at intervals of time Δt based on the motion detection table 133 . As described above, the scene evaluation unit 227 determines whether or not the customer's action occurs during the time S1 immediately before the customer service staff's action time, and during the time S2 immediately after that. Determine if the same type of customer activity is occurring. The scene evaluation unit 227 calculates a scene evaluation value based on the synchronization between the behavior of the person in charge of customer service and the behavior of the customer, and generates the scene evaluation table 134 shown in FIG.

The word extraction unit 228 converts the voice data stored in the voice storage unit 221 into a word string by voice recognition. The word extraction unit 228 extracts unregistered words that are not registered in the keyword table 131 from the word string. However, general-purpose words (stop words) such as Japanese particles and auxiliary verbs, which may appear many times during speech, are excluded. For extraction of unregistered words, the technique described in Non-Patent Document 1 (“Speech Summary Sentence Generating Method by Word Extraction and Its Evaluation”) may be used. The word extraction unit 228 notifies the keyword evaluation unit 229 of the unregistered word extraction result. The extraction result includes the extracted unregistered word and the time when the unregistered word appears.

The keyword evaluation unit 229 calculates word evaluation values for each of the keyword and the unregistered word based on the keyword detection table 132, the scene evaluation table 134, and the extraction result of the unregistered word. A method of calculating the word evaluation value is the same as that of the keyword evaluation value in the second embodiment. That is, the word evaluation value for the keyword is the same as the keyword evaluation value, and the word evaluation value for the unregistered word is calculated by the same method as the keyword evaluation value.

Therefore, the keyword evaluation unit 229 extracts one or more appearance times of the keyword for each keyword, averages the scene evaluation values of surrounding times for each appearance time, and further averages the one or more appearance times. is the word evaluation value. In addition, the keyword evaluation unit 229 extracts, for each unregistered word, one or more appearance times of the unregistered word, averages the scene evaluation values of surrounding times for each appearance time, to obtain a word evaluation value. The keyword evaluation unit 229 then generates a word evaluation table having the same data structure as the keyword evaluation table 135 shown in FIG. The word evaluation table contains both word evaluation values for keywords and word evaluation values for unregistered words.

The keyword evaluation unit 229 extracts keywords whose word evaluation values exceed the threshold T1 and keywords whose word evaluation values are less than the threshold T2 as important keywords. The keyword evaluation section 229 then generates the important keyword table 136 shown in FIG. The keyword evaluation unit 229 also extracts unregistered words whose word evaluation values exceed the threshold value T1 and those whose word evaluation values are less than the threshold value T2 from among the unregistered words. The keyword evaluation unit 229 adds the extracted unregistered word to the keyword table 131 as a search target keyword.

FIG. 15 is a flow chart showing an example of a conversation analysis procedure according to the sixth embodiment.
(S50) The motion detection unit 226 reads image data from the image storage unit 222. FIG. Also, the keyword detection unit 225 reads the voice data from the voice storage unit 221 . Also, the word extraction unit 228 reads the same voice data from the voice storage unit 221 .

(S51) The keyword detection unit 225 converts the speech data read out in step S50 into a word string by speech recognition, searches the keyword table 131 for the search target keyword registered in the word string, and generates the keyword detection table 132. . The word extraction unit 228 also converts the voice data read in step S50 into a word string by voice recognition, and extracts unregistered words that are not registered in the keyword table 131 from the word string.

(S52) The motion detection unit 226 recognizes the area in which the customer and the customer service representative appear by image recognition from each frame included in the image data read out in step S50. The motion detection unit 226 detects the type of motion of the customer, the time of the motion, and the magnitude of the motion from the position change between frames. Further, the motion detection unit 226 detects the type of motion, the time of motion, and the magnitude of motion of the person in charge of customer service from the position change between frames. The motion detection unit 226 generates the motion detection table 133 including the detected information.

(S53) The scene evaluation unit 227 extracts the action time of the person in charge of customer service from the action detection table 133 generated in step S52.
(S54) The scene evaluation unit 227 searches the action detection table 133 for the previous customer action for each action time of the customer service representative extracted in step S53, determines whether or not there is a customer action, and determines whether or not there is a customer action. determine the weights.

(S55) The scene evaluation unit 227 searches the action detection table 133 for the immediately following customer action for each action time of the person in charge of customer service extracted in step S53, determines whether there is synchronization by the same kind of customer action, Decide the coefficient according to the presence or absence of

(S56) The scene evaluation unit 227 sets a sliding window having a predetermined time width. The scene evaluation unit 227 calculates the scene evaluation value at the center time of the sliding window using the weights and coefficients calculated in steps S54 and S55 for the behavior of the customer service representative belonging to the sliding window. The scene evaluation unit 227 slides the sliding window by time Δt to generate the scene evaluation table 134 .

(S57) The motion detector 226 determines whether the image data has ended. The keyword detection unit 225 determines whether the voice data has ended. The word extractor 228 determines whether the voice data has ended. If the image data and the audio data have ended, the process proceeds to step S58; otherwise, the process proceeds to step S50.

(S58) The keyword evaluation unit 229 extracts the appearance times of keywords from the keyword detection table 132 generated in step S51. For each keyword appearance time, the keyword evaluation unit 229 searches the scene evaluation values around the appearance time from the scene evaluation table 134 generated in step S56, and calculates the average value of the surrounding scene evaluation values. Similarly, the keyword evaluation unit 229 extracts the appearance times of unregistered words from the unregistered word extraction results of step S51. For each appearance time of an unregistered word, the keyword evaluation unit 229 searches the scene evaluation values around the appearance time from the scene evaluation table 134 and calculates the average value of the surrounding scene evaluation values.

(S59) The keyword evaluation unit 229 classifies the average values of the scene evaluation values calculated in step S58 for the keywords according to the identity of the keywords, and further averages the average values of the scene evaluation values for each keyword to obtain word values. Calculate the evaluation value. Similarly, the keyword evaluation unit 229 classifies the average value of the scene evaluation values calculated in step S58 for the unregistered words according to the identity of the unregistered words, and classifies the average value of the scene evaluation values for each unregistered word. are further averaged to calculate the word evaluation value. The keyword evaluation unit 229 generates a word evaluation table indicating word evaluation values of keywords and unregistered words.

(S60) The keyword evaluation unit 229 extracts keywords whose word evaluation value exceeds the threshold value T1 and keywords whose word evaluation value is less than the threshold value T2 as important keywords from the word evaluation table generated in step S59. The keyword evaluation unit 229 generates an important keyword table 136 showing the extracted important keywords and their ranks, and stores it in the evaluation result storage unit 224 . The keyword evaluation section 229 transmits the contents of the important keyword table 136 to the management device 41 . The management device 41 displays the top N and bottom N important keywords based on the contents of the important keyword table 136 .

(S61) The keyword evaluation unit 229 extracts unregistered words whose word evaluation value exceeds the threshold value T1 and unregistered words whose word evaluation value is less than the threshold value T2 from the word evaluation table generated in step S59. The keyword evaluation unit 229 adds the extracted unregistered word to the keyword table 131 as a new search target keyword.

According to the information processing system of the sixth embodiment, effects similar to those of the second embodiment are obtained. Further, in the sixth embodiment, search target keywords that are likely to be closely related to customer service are automatically added through conversation analysis. Therefore, it is not necessary to specify search target keywords exhaustively in advance, and the burden of specifying search target keywords can be reduced. In addition, since search target keywords are automatically learned, the accuracy of important keywords extracted from conversations is improved, and the usefulness of conversation analysis results is improved.

REFERENCE SIGNS LIST 10 keyword extraction device 11 storage unit 12 processing unit 13 voice data 14 motion data 15 keywords 16, 17 motion 18 evaluation value

Claims (8)

to the computer,
Audio data representing an utterance made by at least one of the first user and the second user in a conversation between a first user who is a service provider and a second user who is a receiver of the service to detect keywords,
The timing of the first action by the first user and the second action by the second user are obtained from the action data indicating the action performed by the first user and the action performed by the second user in the conversation. detect the timing of motion,
calculating an evaluation value indicating the importance of the keyword based on the relationship between the timing of the first action and the timing of the second action;
A keyword extraction program that runs the process. The motion data is image data obtained by imaging the first user and the second user while the conversation is taking place.
The keyword extraction program according to claim 1. the second action is the same type of action performed after the first action;
In calculating the evaluation value, when the timing of the second action exists within a predetermined time from the timing of the first action, the timing of the second action is higher than when the timing of the second action does not exist within the predetermined time. appreciate the importance of
The keyword extraction program according to claim 1. the second action is the same type of action performed after the first action;
In calculating the evaluation value, the shorter the elapsed time from the timing of the first action to the timing of the second action, the higher the importance is evaluated.
The keyword extraction program according to claim 1. In the calculation of the evaluation value, as a predetermined condition that the timing of the second action is later than the timing of the first action, the second user within a predetermined time immediately before the first action Determining the state of mind of the second user according to whether or not the second user is not performing an action and the timing of the second action exists within the predetermined time period immediately after the first action. and calculating the evaluation value based on the determination result of the mental state;
The keyword extraction program according to claim 1. In the keyword detection, a predetermined search target keyword is searched from the voice data,
The computer further comprises:
extracting words other than the search target keyword from the audio data;
calculating another evaluation value indicating the degree of importance of the extracted word based on the relationship;
If the other evaluation value satisfies a predetermined condition, executing a process of adding the extracted word to the search target keyword;
The keyword extraction program according to claim 1. the computer
Audio data representing an utterance made by at least one of the first user and the second user in a conversation between a first user who is a service provider and a second user who is a receiver of the service to detect keywords,
The timing of the first action by the first user and the second action by the second user are obtained from the action data indicating the action performed by the first user and the action performed by the second user in the conversation. detect the timing of motion,
calculating an evaluation value indicating the importance of the keyword based on the relationship between the timing of the first action and the timing of the second action;
Keyword extraction method. Audio data representing an utterance made by at least one of the first user and the second user in a conversation between a first user who is a service provider and a second user who is a receiver of the service and a storage unit for storing action data indicating actions performed by the first user and actions performed by the second user in the conversation;
A keyword is detected from the voice data, a timing of a first action by the first user and a timing of a second action by the second user are detected from the action data, and the timing of the first action and the timing of the second action are detected. a processing unit that calculates an evaluation value indicating the importance of the keyword based on the relationship with the timing of the second action;
A keyword extraction device having

Images