JP2020067790A - Keyword extracting program, keyword extracting method and keyword extracting apparatus - Google Patents

Abstract

To improve the extraction precision on a keyword that reflects the psychological state of a user.SOLUTION: In a conversation between a first user who becomes a service provider and a second user who becomes a service receiver, a keyword is extracted from speech data that indicates uttered speech by at least either the first user and the second user. A first action timing by the first user and a second action timing by a second user are detected from action data indicating the action taken by the first user in the conversation and the action taken by the second user therein. An evaluation value that indicates the importance level of the keyword is calculated on the basis of the relationship between the first action timing and the second action timing.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a keyword extraction program, a keyword extraction method, and a keyword extraction device.

  From the conversation between users, it may be desired to extract a keyword that is important for the conversation. For example, it is conceivable to extract a positive keyword that greatly contributes to the empathy of the customer from the conversation between the customer and the person in charge of the customer, and utilize it for later customer service. Further, for example, it is conceivable to extract a keyword of caution used in an unfavorable customer service from a conversation between the customer and the person in charge of the customer service and utilize it for later customer service.

  A digital photo frame has been proposed in which voice recognition of a user's utterance is performed while an image is displayed, a keyword is extracted, and the extracted keyword is added as a tag to the image. Further, there has been proposed an empathic interpretation estimation device that presents a video of another user to a user, photographs the heads of the users while watching the video, analyzes the actions of the two users, and estimates the empathic interpretation. There is. Also, a signal indicating the activity state of each conference participant is collected, the communication difficulty level is determined from each utterance of each conference participant, and the understanding level of each conference participant is estimated based on the activity state and communication difficulty level. A state estimation device has been proposed.

  Also, the speech rate is calculated from the user's voice, the time change of the user area is detected from the moving image of the user who is speaking, and the user response evaluation value is calculated based on the speech rate and the time conversion of the user area. A response quality evaluation device has been proposed. In addition, an information processing device is proposed which detects a state of each of a plurality of users using a sensor device, calculates the degree of synchronization between users based on the detected states, and changes information presented to the user according to the degree of synchronization. Has been done.

JP, 2010-224715, A JP, 2005-64827, A JP, 2016-213631, A JP, 2017-162100, A JP, 2018-45676, A

Tomoori Hori, Sadahiro Furui, "A Method for Generating Speech Summarization by Word Extraction and Its Evaluation", IEICE Transactions, J85-D-II, pp. 200-209, February 2002.

  However, in the conventional technique of extracting a keyword from a conversation, there is room for improvement in the accuracy of extracting a keyword of high importance from the viewpoint of the psychological state of the user. For example, if the importance degree is simply determined from the appearance frequency of the keyword, there is a possibility that the keyword having a high degree of relation with the psychological state such as the customer's sympathy or the degree of customer service of the customer is not extracted.

  In one aspect, an object of the present invention is to provide a keyword extraction program, a keyword extraction method, and a keyword extraction device that improve the extraction accuracy of keywords that reflect the psychological state of the user.

  In one aspect, a keyword extraction program to be executed by a computer is provided. From the voice data indicating the utterance made by at least one of the first user and the second user in the conversation between the first user who is the service providing side and the second user who is the service receiving side, a keyword To detect. Detecting the timing of the first operation by the first user and the timing of the second operation by the second user from the operation data indicating the operation performed by the first user and the operation performed by the second user in the conversation. To do. An evaluation value indicating the importance of the keyword is calculated based on the relationship between the timing of the first operation and the timing of the second operation.

  Further, in one aspect, a computer-executed keyword extraction method is provided. Further, in one aspect, a keyword extraction device having a storage unit and a processing unit is provided.

  In one aspect, the keyword extraction accuracy that reflects the psychological state of the user is improved.

It is a figure explaining the example of the keyword extraction device of 1st Embodiment. It is a figure which shows the example of the information processing system of 2nd Embodiment. It is a block diagram which shows the hardware example of a conversation analysis apparatus. It is a figure which shows the keyword extraction example of 2nd Embodiment. It is a figure which shows the example of calculation of a keyword evaluation value. It is a block diagram which shows the example of a function of the conversation analysis apparatus of 2nd Embodiment. It is a 1st figure which shows the example of the table which a conversation analysis apparatus hold | maintains. It is a 2nd figure which shows the example of the table which a conversation analysis apparatus hold | maintains. It is a flow chart which shows the example of a procedure of conversation analysis of a 2nd embodiment. It is a figure which shows the example of the information processing system of 3rd Embodiment. It is a figure which shows the example of the information processing system of 4th Embodiment. It is a figure which shows the keyword extraction example of 5th Embodiment. It is a flowchart which shows the example of a procedure of the conversation analysis of 5th Embodiment. It is a block diagram which shows the example of a function of the conversation analysis apparatus of 6th Embodiment. It is a flow chart which shows the example of a procedure of conversation analysis of a 6th embodiment.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
The first embodiment will be described.

FIG. 1 is a diagram illustrating an example of the keyword extracting device according to the first embodiment.
The keyword extracting device 10 according to the first embodiment extracts an important keyword that reflects the psychological state of at least one of the conversations between users. For example, the keyword extraction device 10 extracts an important keyword related to the customer's sympathy from the conversation between the customer and the customer service representative. Further, for example, the keyword extracting device 10 extracts an important keyword related to the degree of customer service of the customer service representative from the conversation between the customer and the customer service representative.

  The keyword extraction device 10 can also be called a computer or an information processing device. The keyword extracting device 10 may be a client device or a server device. Further, the keyword extracting device 10 may extract the important keywords in real time during the conversation between the users, or may extract the important keywords as a batch process after the conversation ends.

  The keyword extracting device 10 has a storage unit 11 and a processing unit 12. The storage unit 11 may be a volatile semiconductor memory such as a RAM (Random Access Memory) or a non-volatile storage such as a HDD (Hard Disk Drive) or a flash memory. The processing unit 12 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a DSP (Digital Signal Processor). However, the processing unit 12 may include a specific-purpose electronic circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). A set of multiple processors may be referred to as a "multiprocessor" or simply "processor."

  The storage unit 11 stores voice data 13 and operation data 14. The voice data 13 and the motion data 14 are records of a conversation between the first user and the second user. The first user is the person who provides the service, and the second user is the person who receives the service. Examples of the service field include business performed through communication such as retailing, education, and medical care. The first user is, for example, a person in charge of customer service, such as a product explainer in a store, a lecturer engaged in an educational institution, a doctor or a counselor engaged in a medical institution. The second user is, for example, a customer who receives a customer, such as a consumer who visits a store, a student who attends an educational institution, or a patient who visits a medical institution.

  The voice data 13 indicates an utterance made by at least one of the first user and the second user. For example, the voice data 13 is a voice signal obtained by recording the utterance of at least one of the first user and the second user using a microphone. In that case, the voice data 13 may be the one in which only the utterance of the first user is recorded, the one in which only the utterance of the second user is recorded, or the utterance of both the first user and the second user. It may be a recording of. The utterance of the first user and the utterance of the second user may be recorded using the same microphone or different microphones.

  The motion data 14 indicates a motion performed by the first user and a motion performed by the second user in the conversation. The action is a physical action that can be visually recognized by the other user, such as a change in facial expression, a gesture of the head, arms, or legs, a change of the line of sight, or a change of the posture. The change in facial expression includes laughing. Head gestures include nodding. Changing the line of sight includes looking at the head of the other user. Changing the posture includes getting leaned forward.

  For example, the operation data 14 is image data such as a moving image obtained by shooting the first user and the second user using the image sensor. However, the operation data 14 may be sensor data generated using a sensor device other than the image sensor. For example, the motion data 14 may be data obtained by detecting a gesture of the head using an acceleration sensor incorporated in the headset. The motion data 14 may be data obtained by detecting an arm gesture using an acceleration sensor incorporated in a wristwatch. Further, the motion data 14 may be data obtained by detecting a change in posture using a pressure-sensitive sensor incorporated in a chair.

  The data regarding the first user and the data regarding the second user of the operation data 14 may be generated using the same device or different devices. For example, when the motion data 14 is image data, both the first user and the second user may be shown in an image taken by using one image sensor, or images taken by using different image sensors. Different users may be shown in the different images displayed.

  The processing unit 12 detects the keyword 15 from the voice data 13. For example, the processing unit 12 converts the voice data 13 into a character string (text) of utterance by voice recognition, and searches a predetermined search target keyword from the character string. The search target keyword is defined in advance as a keyword list, for example. Further, for example, the processing unit 12 continuously compares the feature amount of the voice signal of the utterance with the feature amount of the voice signal of the search target keyword by word spotting, without converting the entire utterance into a character string, and searches for the search target. Directly recognize only keywords.

  The keyword 15 may be the utterance of the first user or the utterance of the second user. The processing unit 12 may search the search target keyword by distinguishing the utterance of the first user and the utterance of the second user, or distinguish the utterance of the first user from the utterance of the second user. You may search for a search target keyword, without being. Further, the processing unit 12 may search for the search target keyword by limiting to either one of the utterance of the first user and the utterance of the second user.

  Further, the processing unit 12 detects, from the operation data 14, the timing of the operation 16 (first operation) by the first user and the timing of operation 17 (second operation) by the second user. Either of the processing of the audio data 13 and the processing of the motion data 14 may be executed first or in parallel. The first user action 16 and the second user action 17 are detected separately. For example, the processing unit 12 uses image data to recognize physical changes such as facial expression changes of the first user and the second user, gestures of the head, arms, and legs, changes of line of sight, changes of posture, and the like by image recognition. Is detected from the motion data 14. When the sensor device other than the image sensor generates the operation data 14, it may not be necessary to perform special recognition processing.

  Then, the processing unit 12 calculates an evaluation value 18 indicating the degree of importance of the detected keyword 15 based on the relationship between the timing of the action 16 by the first user and the timing of the action 17 by the second user. . The processing unit 12 determines whether to extract the keyword 15 as an important keyword based on the evaluation value 18, for example. When the evaluation value 18 exceeds the predetermined first threshold value, the processing unit 12 may extract the keyword 15 as a preferable keyword. In addition, the processing unit 12 may extract the keyword 15 as a keyword requiring attention when the evaluation value 18 is less than a predetermined second threshold value that is smaller than the first threshold value. The keyword 15 is, for example, a keyword issued within a predetermined range from the actions 16 and 17 on the time axis.

  As a timing relationship between the operations 16 and 17, the processing unit 12 detects that the operations 16 and 17 are the same type of operation, the operation 16 is performed first, and the operation 17 is performed within a predetermined time after the operation 16. You may. At this time, it may be added to the condition that the second user has not performed an operation within a predetermined time immediately before the operation 16, that is, that the operation has been started by the first user. It can be said that this relationship reflects the psychological state of the second user.

  For example, when the first user is a customer service representative and the second user is a customer, this relationship is linked to the behavior of the customer service representative, such as laughing or nodding, and the same type of customer laughing or nodding. Indicates that synchronization has occurred. Therefore, it can be inferred that this indicates the excitement of the conversation in which the customer often observes the behavior of the customer service representative and has an empathy for the story of the customer service representative. Further, for example, when the action 17 is performed first and the action 16 is performed within a predetermined time from the action 17, the same type of laughing or nodding of the customer service representative is linked with the action of the customer laughing or nodding. Indicates that synchronization has occurred. Therefore, this can be presumed to be that the customer service representative observes the customer's behavior well and shows a good customer service.

  When the actions 16 and 17 are the same type of action and the action 16 is performed first and the action 17 is performed within a predetermined time from the action 16, the processing unit 12 sets the keyword 15 in the vicinity of the actions 16 and 17. It is considered highly appreciated. Highly evaluating the keyword 15 corresponds to the good psychological state of the second user. When the second user is a customer, the keyword 15 may be a preferable keyword that has the customer's empathy.

  On the other hand, when the operation 17 is not performed within the predetermined time from the operation 16, the processing unit 12 may consider the keyword 15 near the operations 16 and 17 to be low. Evaluating the keyword 15 to be low corresponds to that the psychological state of the second user is not good. When the second user is a customer, there is a possibility that the keyword 15 is a keyword that needs attention because the customer's empathy was not obtained.

  The processing unit 12 may output the extracted important keyword. For example, the processing unit 12 may store the important keyword in a storage device included in the keyword extracting device 10. Further, for example, the processing unit 12 may output the important keyword to an output device included in the keyword extracting device 10 such as displaying on the display. Further, for example, the processing unit 12 may transmit the important keyword to another information processing device via the network.

  According to the keyword extracting device 10 of the first embodiment, the keyword 15 is detected from the voice data 13, and the timing of the action 16 of the first user and the timing of the action 17 of the second user are detected from the action data 14. To be done. Then, the evaluation value 18 of the keyword 15 is calculated based on the relationship between the timing of the action 16 and the timing of the action 17. This makes it possible to accurately extract the important keyword from the viewpoint of the psychological state of at least one of the first user and the second user. Therefore, it is possible to utilize the important keywords extracted by the keyword extracting device 10 for a predetermined purpose such as improving customer service.

[Second Embodiment]
Next, a second embodiment will be described.
FIG. 2 is a diagram illustrating an example of the information processing system according to the second embodiment.

  The information processing system according to the second embodiment is for supporting the improvement of customer service by analyzing the conversation in an industry in which the customer and the person in charge of customer service have a conversation. This information processing system can be applied to various industries such as product explanations and health guidance.

  The information processing system according to the second embodiment includes a management device 41 and a conversation analysis device 100 connected to the network 40. A camera device 50 is connected to the conversation analysis device 100. The management device 41 is a terminal device used by a manager who guides a customer service representative, such as a boss of the customer service representative. The conversation analysis device 100 is a terminal device installed in a place where a customer and a person in charge of customer service have a conversation. For example, the conversation analysis device 100 is installed on or near a counter where a customer and a person in charge of customer service face each other. The camera device 50 is a device device having a moving image shooting function and a voice recording function. The camera device 50 is installed so as to capture and record a conversation between a customer and a customer service representative.

  The camera device 50 captures images so that both the customer and the customer service representative fit within the image while the customer and the customer service agent have a conversation. Further, the camera device 50 performs a recording so that both the voice of the customer and the voice of the customer service representative are included while the customer and the customer service representative have a conversation. The conversation analysis device 100 collects image data indicating a captured moving image and voice data indicating a recorded voice, and analyzes a conversation between a customer and a customer service representative. Specifically, the conversation analysis device 100 detects a keyword uttered by a customer or a contact person from the voice data, and detects a customer's action during conversation and an action of the contact person from the image data. The conversation analysis device 100 evaluates the keywords from the actions of the customer and the person in charge of serving the customer, and extracts the important keywords. The conversation analysis device 100 reports the extracted important keyword to the management device 41.

  The conversation analysis by the conversation analysis device 100 and the report of the important keyword from the conversation analysis device 100 to the management device 41 may be performed in real time during the work of the customer service representative, or as a batch process after the work of the customer service representative is completed. May be. For example, the conversation analysis device 100 analyzes the voice data and the image data output from the camera device 50 in real time, determines the important keyword for each conversation break, and transmits the important keyword to the management device 41. Possible conversation breaks include when the customer service for one customer is completed, when no speech continues for a predetermined period of time, or when a certain period of time elapses from the start of the conversation. Further, for example, the conversation analysis device 100 saves the voice data and the image data output by the camera device 50, analyzes the voice data and the image data collectively after the work ends, and transmits the important keyword to the management device 41.

  At least a part of the important keywords transmitted from the conversation analysis device 100 to the management device 41 is displayed on the display of the management device 41. The administrator may check the important keywords during the work of the customer service staff, or may check the important keywords after the work is completed.

  The conversation analysis device 100 corresponds to the keyword extraction device 10 of the first embodiment. Image data showing a moving image captured by using the camera device 50 corresponds to the operation data 14 of the first embodiment. The voice data indicating the voice recorded by using the camera device 50 corresponds to the voice data 13 of the first embodiment.

FIG. 3 is a block diagram showing a hardware example of the conversation analysis device.
The conversation analysis device 100 includes a CPU 101, a RAM 102, an HDD 103, an image signal processing unit 104, input signal processing units 105 and 106, a medium reader 107, and a communication interface 108 which are connected to a bus. The CPU 101 corresponds to the processing unit 12 of the first embodiment. The RAM 102 or the HDD 103 corresponds to the storage unit 11 of the first embodiment. The management device 41 can also be realized using similar hardware.

  The CPU 101 is a processor that executes program instructions. The CPU 101 loads at least a part of the programs and data stored in the HDD 103 into the RAM 102 and executes the programs. The CPU 101 may include a plurality of processor cores, and the conversation analysis device 100 may include a plurality of processors. A set of multiple processors may be referred to as a "multiprocessor" or simply "processor."

  The RAM 102 is a volatile semiconductor memory that temporarily stores a program executed by the CPU 101 and data used by the CPU 101 for calculation. Note that the conversation analysis device 100 may include a memory of a type other than the RAM, or may include a plurality of memories.

  The HDD 103 is a non-volatile storage that stores an OS (Operating System), software programs such as middleware and application software, and data. The conversation analysis device 100 may include another type of storage such as a flash memory or an SSD (Solid State Drive), or may include a plurality of storages.

  The image signal processing unit 104 outputs an image to the display 111 connected to the conversation analysis device 100 according to an instruction from the CPU 101. As the display 111, a CRT (Cathode Ray Tube) display, a liquid crystal display (LCD: Liquid Crystal Display), an organic EL (OEL: Organic Electro-Luminescence) display, or any other type of display can be used.

  The input signal processing unit 105 receives an input signal from the input device 112 connected to the conversation analysis device 100. As the input device 112, any type of input device such as a mouse, a touch panel, a touch pad, a keyboard can be used. Moreover, a plurality of types of input devices may be connected to the conversation analysis device 100.

  The input signal processing unit 106 receives image signals and audio signals from the camera device 50 connected to the conversation analysis device 100. The camera device 50 has an image sensor 51 and a microphone 52. The image sensor 51 is an image sensor that converts light into an electric signal (image signal). As the image sensor 51, any type of image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor can be used. The microphone 52 converts sound into an electric signal (voice signal). Any type of microphone such as a dynamic microphone or a condenser microphone can be used as the microphone 52.

  The medium reader 107 is a reading device that reads programs and data recorded on the recording medium 113. As the recording medium 113, for example, a magnetic disk such as a flexible disk (FD: Flexible Disk) or an HDD, an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), a magneto-optical disk (MO: Magneto-Optical disk), A semiconductor memory or the like can be used. The medium reader 107 stores, for example, the program or data read from the recording medium 113 in the RAM 102 or the HDD 103.

  The communication interface 108 is connected to the network 40 and communicates with the management device 41 via the network 40. The communication interface 108 may be a wired communication interface connected to a wired communication device such as a switch or a router, or a wireless communication interface connected to a wireless communication device such as a base station or an access point.

  Next, a keyword extracting method according to the second embodiment will be described. The conversation analysis device 100 according to the second embodiment estimates a preferable keyword that has the sympathy of the customer from the keywords that appear in the conversation between the customer and the person in charge of customer reception, and extracts the keyword as an important keyword. Further, the conversation analysis device 100 according to the second embodiment estimates important keywords that cannot be sympathized with the customer from the keywords that appear in the conversation between the customer and the person in charge of hospitality. Extract as a keyword. As a result, it is possible to improve the customer service skill of the person in charge of customer service so that the customer can feel more sympathy with the customer after the next time.

FIG. 4 is a diagram showing an example of keyword extraction according to the second embodiment.
In order to evaluate whether the conversation between the customer and the customer contact person is lively and the customer is sympathetic to the story of the customer contact person, in the second embodiment, the behavior of the customer and the behavior of the customer contact person are evaluated. Detect synchronization between and. The synchronization in the second embodiment is a chain of operations in which an operation is started by a person in charge of customer service and immediately after that, the customer performs the same kind of operation. In this case, it is presumed that the customer is carefully watching the customer contact person and listens to the customer contact person positively. Therefore, the keyword issued by the customer or the person in charge of customer service during such synchronization may be a preferable keyword. On the other hand, a keyword issued by a customer or a person in charge of customer service when such synchronization has not occurred may be a keyword requiring attention.

For example, consider scenes 71-73 described below.
In the scene 71, the customer service representative performs an operation of laughing, and immediately after that, the customer performs an operation of laughing. The customer does not perform the operation within the time S1 immediately before the operation of the customer service representative, and the customer operates within the time S2 immediately after the operation of the customer service representative. In scene 71, since the customer's action and the action of the customer contact person are synchronized with each other starting from the customer service person, it is determined that the customer's sympathy is high. Then, the evaluation value of the keyword "response" issued by the customer or the person in charge of serving the customer around the scene 71 becomes high.

  In the scene 72, the customer service representative performed an action of laughing, but the customer did not operate within the time S1 immediately before the operation of the customer service representative, and within the time S2 immediately after the operation of the customer service representative. Customer is not taking action. In the scene 72, the customer's behavior and the customer service representative's behavior are not synchronized, so it is determined that the customer's sympathy is small. Then, the evaluation value of the keyword “new function” issued by the customer or the person in charge of serving the customer around the scene 72 becomes low.

  In the scene 73, the customer makes a nod operation, and immediately after that, the customer service person nods. The customer is operating within the time S1 immediately before the operation of the customer service representative, and the customer is not performing within the time S2 immediately after the operation of the customer service representative. In the scene 73, the action of the customer and the action of the customer are not synchronized with each other starting from the person in charge of the customer, and it is determined that the sensitivity of the customer is small. Then, the evaluation value of the keyword “image quality” issued by the customer or the person in charge of customer service around the scene 73 becomes low.

  The conversation analysis device 100 estimates a keyword having an evaluation value larger than the threshold value T1 as a preferable keyword and extracts it as an important keyword. Also, the conversation analysis device 100 estimates a keyword having an evaluation value smaller than the threshold value T2 as a keyword requiring attention, and similarly extracts it as an important keyword. The threshold values T1 and T2 are predetermined and T1> T2. In the above example, “response” may be extracted as an important keyword.

  Here, the conversation analysis device 100 detects the actions of the customer and the person in charge of serving the customer from the image data. The motion to be detected is a visible physical motion such as a change in facial expression, a gesture of the head, arms, or legs, a change of the line of sight, a change of the posture. The change in facial expression includes laughing. Head gestures include nodding. Changing the line of sight includes looking at the opponent's head. Changing the posture includes getting leaned forward.

  The conversation analysis device 100 recognizes the customer and the customer service representative from the image data by image recognition. For example, in the conversation analysis device 100, the characteristic information of the appearance of the person in charge of customer service is registered in advance, and the person in charge of customer service is recognized based on the characteristic information. In that case, a person other than the person in charge of customer service is recognized as a customer. Further, the conversation analysis device 100 recognizes the types of actions of the customer and the customer service representative from the image data by image recognition. At this time, the size of the motion such as the size of the nod or the size of the arm gesture may be recognized together.

  The technique described in Patent Document 2 (JP-A-2015-64827) may be used to detect the motion. For example, regarding facial expressions, the conversation analysis device 100 extracts the contours of the eyes and mouth from each frame of the image data, and determines the change of facial expressions from the change in the contours between the frames. Further, for example, for nodding, the conversation analysis device 100 extracts the positions of the eyes, the nose, and the mouth from each frame of the image data, and determines the nod from the change in the positions of the eyes, the nose, and the mouth between the frames. The magnitude of the motion can be determined from the amount of change.

  Further, the conversation analysis device 100 recognizes a keyword from voice data by voice recognition. At this time, the conversation analysis device 100 may recognize the utterance of the customer and the utterance of the person in charge of serving the customer separately, or may recognize them without distinguishing both. As a method of distinguishing between the utterance of the customer and the utterance of the customer service representative, for example, the characteristic information of the voice quality of the customer service representative is registered in the conversation analysis device 100 in advance, and the utterance of the customer service representative of the utterances of two different voice qualities is registered. A method in which the utterance is determined first and the other utterance is regarded as the utterance of the customer can be considered. Further, as a method of distinguishing between the utterance of the customer and the utterance of the person in charge of customer service, for example, a method of determining from the arrival direction of the voice at the time of recording can be considered.

  The technique described in Patent Document 1 (Japanese Patent Laid-Open No. 2010-224715) may be used for detecting the keyword. For example, in the conversation analysis device 100, search target keywords that may affect the customer's sympathy are registered in advance. The conversation analysis device 100 converts the voice waveform indicated by the voice data into voice feature information by Fourier transform or the like, inputs the voice feature information into a voice recognition model prepared in advance, converts the voice feature information into a word string, and retrieves from the word string. Search for the target keyword. However, the conversation analysis device 100 may directly detect only the search target keyword by word spotting without converting the entire utterance into a word string.

Next, a method of calculating the keyword evaluation value will be described.
FIG. 5 is a diagram illustrating a calculation example of the keyword evaluation value.
First, the conversation analysis device 100 calculates a scene evaluation value at a predetermined time interval using the image data recording the customer and the person in charge of serving the customer. In the second embodiment, the scene evaluation value indicates the excitement of the conversation between the customer and the customer service representative at that time, and corresponds to the customer's empathy. It is estimated that the greater the scene evaluation value, the greater the excitement of conversation and the greater the sensitivity. It is estimated that the smaller the scene evaluation value, the smaller the excitement of conversation and the smaller the sensitivity.

  Then, the conversation analysis device 100 calculates the keyword evaluation value for the keyword extracted from the voice data, using the scene evaluation value around the time when the keyword is issued. A keyword having a larger keyword evaluation value is more likely to be a keyword that has the sympathy of the customer. The smaller the keyword evaluation value, the higher the possibility that the keyword has not gained the customer's sympathy. The conversation analysis device 100 extracts a keyword whose keyword evaluation value exceeds the threshold value T1 as an important keyword. The conversation analysis device 100 also extracts keywords having a keyword evaluation value less than the threshold value T2 as important keywords.

  In the calculation of the scene evaluation value, the conversation analysis device 100 sets, as a sliding window 81, a section from before the time S0 to after the time S0 (a section before and after the time S0) around a certain time. The position of the sliding window 81 will be shifted by the time Δt. The time Δt is, for example, about 1 frame time to 1 second. The time S0 is, for example, about 1 minute to 2 minutes. Since one scene evaluation value is calculated with respect to the center time of the sliding window 81, the scene evaluation value is calculated at time Δt intervals.

  In the sliding window 81, the time when the person in charge of customer service performed an action is F (x), and the weight of the action at the time F (x) is w (x). The weight w (x) is determined based on the behavior of the customer during the time S1 immediately before the time F (x). The time S1 is, for example, about 1 second to 2 seconds. Weight w (x) = w1 when the customer is not operating during the immediately preceding time S1, and weight w (x) = w2 when the customer is performing during the immediately preceding time S1. . However, the magnitude relationship between the weight w1 and the weight w2 is w1> w2.

  Further, each operation belonging to the sliding window 81 is classified into sets r1 and r2 depending on whether or not the customer is synchronized. When the customer is performing the same type of operation during the time S2 immediately after the time F (x), that is, when there is synchronization, the operation of the customer service representative is classified into the set r1. When the customer does not perform the same type of action during the time S2 immediately after, that is, when there is no synchronization, the action of the customer service representative is classified into the set r2. The time S2 is, for example, about 1 second to 2 seconds and may be the same as the time S1. Further, the value of the coefficient a used below is determined in advance. The value of the coefficient a is a real number less than 1 (a <1), and may be a negative value.

  From such a sliding window 81, for example, the scene evaluation value Vt at the central time is calculated as in Expression (1). That is, the scene evaluation value is the average value of the weights of the motions belonging to the set r1 and the weights of the motions belonging to the set r2 multiplied by the coefficient a. Since the value of the coefficient a is less than 1, the scene evaluation value increases as the proportion of the actions of the customer service personnel who belong to the sliding window 81 that are synchronized with the action of the customer increases.

  However, when the action of the customer service representative and the action of the customer are synchronized, the scene evaluation value can be calculated by further considering the delay time from the action of the customer service representative to the action of the customer. The delay time from time F (x) is E (x), and the value of the coefficient b is predetermined. The value of the coefficient b is a positive real number. In this case, for example, the scene evaluation value Vt at the central time is calculated as in Expression (2). In Expression (2), the weight of the operation belonging to the set r1 is corrected by the delay time E (x) and the coefficient b. In the sliding window 81, the shorter the delay time from the operation of the customer service person to the operation of the customer, the larger the scene evaluation value.

  Further, when the action of the customer service person and the action of the customer are synchronized, the scene evaluation value can be calculated by further considering the magnitude of the action of the customer. The value of the coefficient c is determined in advance, where D (x) is the magnitude of the customer's action synchronized with the action of the customer service representative. The size D (x) indicates the amount of change in the position per unit time that is recognized from the image data, such as the amount of change in the position of the head or the amount of movement of the arm. Takes a value. The value of the coefficient c is a positive real number. In this case, for example, the scene evaluation value Vt at the central time is calculated as in Expression (3). In Expression (3), the weight of the motion belonging to the set r1 is corrected by the magnitude D (x) and the coefficient c. In the sliding window 81, the larger the movement of the customer synchronized with the movement of the customer service person, the larger the scene evaluation value. Although the delay time E (x) is also taken into consideration in the equation (3), the delay time E (x) may not be taken into consideration.

  In the calculation of the keyword evaluation value, the conversation analysis device 100 sets, as the window 82, a section from before the time S3 to after the time S3 (section before and after the time S3) centered on the time G (y) at which the keyword is issued. To do. The time S3 is, for example, about 2 minutes to several minutes. The same keyword may be issued multiple times in a single segment of a conversation (for example, a series of conversations from the start to the end of customer service for one customer). Here, focusing on one keyword, it is assumed that the keyword is uttered Y + 1 times in a single conversation. The window 82 is set for each of y = 0, 1, ..., Y.

  Let H (y) be the average value of the scene evaluation values within the range of the window 82 for one appearance of the keyword of interest. Then, for example, the keyword evaluation value Vk of the keyword in the conversation of one segment is calculated as in Expression (4). That is, the average value of the keyword evaluation values around each time when the keyword is issued is averaged over a plurality of appearances of the keyword to obtain the keyword evaluation value of the keyword.

  When the keyword evaluation value exceeds the threshold value T1, the keyword is an important keyword. When the keyword evaluation value is less than the threshold value T2, the keyword is an important keyword. The conversation analysis device 100 transmits the extracted important keyword to the management device 41. The management device 41 displays all or some of the important keywords received from the conversation analysis device 100 on the display. For example, the management device 41 displays N (top N) important keywords from the received important keywords with the largest keyword evaluation value. The management device 41 also displays N (lower N) important keywords from the received important keywords with the smallest keyword evaluation value. N is a predetermined integer of 1 or more. However, the conversation analysis device 100 may narrow down the important keywords to upper N cases and lower N cases.

Next, the function and processing procedure of the conversation analysis device 100 will be described.
FIG. 6 is a block diagram showing a functional example of the conversation analysis device according to the second embodiment.
The conversation analysis device 100 includes a voice storage unit 121, an image storage unit 122, a keyword storage unit 123, and an evaluation result storage unit 124. These storage units are realized by using the storage area of the RAM 102 or the HDD 103, for example. The conversation analysis device 100 also includes a keyword detection unit 125, a motion detection unit 126, a scene evaluation unit 127, and a keyword evaluation unit 128. These processing units are realized by using a program, for example.

  The voice storage unit 121 stores voice data including a voice signal received from the camera device 50. The image storage unit 122 stores image data including the image signal received from the camera device 50. The keyword storage unit 123 stores a search target keyword. The search target keyword is designated in advance. The administrator may be allowed to add or delete the search target keyword. The evaluation result storage unit 124 stores the evaluation result of the keyword. The evaluation result includes the important keywords extracted from the voice data and their ranks.

  The keyword detection unit 125 converts the voice data stored in the voice storage unit 121 into a word string by voice recognition. The keyword detection unit 125 detects the search target keyword stored in the keyword storage unit 123 from the word string. The keyword detection result includes the detected keyword and the time at which the keyword appears.

  The motion detection unit 126 recognizes a customer and a person in charge of customer service by image recognition from each frame included in the image data stored in the image storage unit 122. The motion detection unit 126 extracts the characteristic information of the area where the person in charge of the customer is reflected from each frame, and detects the motion of the person in charge of the customer based on the change in the characteristic information between the frames. In addition, the motion detection unit 126 extracts the characteristic information of the area where the customer appears from each frame, and detects the motion of the customer based on the change in the characteristic information between the frames. The motion detection result includes time, motion subject, motion type, and motion size.

  The scene evaluation unit 127 uses the detection result of the motion by the motion detection unit 126 to calculate the scene evaluation value at time Δt intervals. As described above, the scene evaluation unit 127 determines, based on the operation time of the customer service representative, whether or not the customer's operation is occurring during the time S1 immediately before, and during the time S2 immediately thereafter. It is determined whether the same type of customer action is occurring. The scene evaluation unit 127 calculates a scene evaluation value based on the synchronization status between the operation of the customer service representative and the operation of the customer. In the evaluation of the synchronization status, the delay time from the action of the customer service person to the action of the customer and the magnitude of the action of the customer may be further considered. The scene evaluation result includes a plurality of times and a plurality of scene evaluation values corresponding to the plurality of times.

  The keyword evaluation unit 128 calculates a keyword evaluation value for each of the detected keywords using the keyword detection result by the keyword detection unit 125 and the scene evaluation result by the scene evaluation unit 127. As described above, the keyword evaluation unit 128 extracts, for each keyword, one or more appearance times of the keyword, averages the scene evaluation values of the peripheral time for each appearance time, and calculates the appearance evaluation time between the one or more appearance times. Further averaged to obtain a keyword evaluation value.

  The keyword evaluation unit 128 extracts keywords having a keyword evaluation value exceeding the threshold T1 and keywords having a keyword evaluation value less than the threshold T2 as important keywords. Then, the keyword evaluation unit 128 stores the ranking (best 1, best 2, worst 1, worst 2, etc.) determined by the extracted important keyword and the keyword evaluation value in the evaluation result storage unit 124 as the evaluation result. The keyword evaluation unit 128 transmits the evaluation result to the management device 41.

FIG. 7 is a first diagram showing an example of a table held by the conversation analysis device.
The keyword table 131 is stored in the keyword storage unit 123. In the keyword table 131, the character string of the keyword designated as the search target keyword is registered. Only the keywords registered in the keyword table 131 are extracted from the utterance indicated by the voice data, and the other words are not extracted.

  The keyword detection table 132 is generated by the keyword detection unit 125. The keyword detection table 132 may be stored in the RAM 102 or the HDD 103. The keyword detection table 132 includes items of time and keyword. In the item of time, the time when any of the keywords registered in the keyword table 131 is issued is registered. The issued keyword is registered in the keyword field. In the case of distinguishing between the utterance by the customer and the utterance by the customer service representative, the keyword detection table 132 may further include an item indicating the speaker. The speaker is a customer or a contact person.

  The motion detection table 133 is generated by the motion detection unit 126. The motion detection table 133 may be stored in the RAM 102 or the HDD 103. The motion detection table 133 includes items of time, motion subject, type, and size. The time of operation is registered in the item of time. In the item of the action subject, “customer” or “contact person in charge” is registered as the subject who performed the action. In the type item, the type of action such as “laugh” or “nod” is registered. In the size item, a numerical value indicating the size of the motion is registered.

FIG. 8 is a second diagram showing an example of a table held by the conversation analysis device.
The scene evaluation table 134 is generated by the scene evaluation unit 127. The scene evaluation table 134 may be stored in the RAM 102 or the HDD 103. The scene evaluation table 134 includes items of time and evaluation value. In the item of time, the time when the degree of excitement of the conversation is evaluated, that is, the time when the customer's sympathy is evaluated is registered. The calculated scene evaluation value is registered in the evaluation value item.

  The keyword evaluation table 135 is generated by the keyword evaluation unit 128. The keyword evaluation table 135 may be stored in the RAM 102 or the HDD 103. The keyword evaluation table 135 includes keywords and evaluation value items. Keywords appearing in the keyword detection table 132 are registered in the keyword field. The calculated keyword evaluation value is registered in the evaluation value item.

  The important keyword table 136 is stored in the evaluation result storage unit 124. The important keyword table 136 includes items of ranking and keywords. In the order item, the order of important keywords such as best 1, best 2, worst 1, worst 2, which are determined by the keyword evaluation value, is registered. An important keyword selected based on the keyword evaluation value among the keywords registered in the keyword evaluation table 135 is registered in the keyword field. The contents of the important keyword table 136 are transmitted to the management device 41.

FIG. 9 is a flowchart showing a procedure example of conversation analysis according to the second embodiment.
(S10) The operation detection unit 126 reads out image data from the image storage unit 122. The image data to be read is image data for a fixed time period subsequent to the processed image data. Further, the keyword detection unit 125 reads out voice data from the voice storage unit 121. The audio data to be read is audio data for a certain time period following the processed audio data.

  (S11) The keyword detection unit 125 converts the voice data read in step S10 into a word string by voice recognition. The keyword detection unit 125 searches the search target keyword registered in the keyword table 131 from the converted word string, and generates the keyword detection table 132 indicating the searched keyword and the appearance time of the keyword.

  (S12) The operation detection unit 126 recognizes the area in which the customer is reflected and the area in which the customer service representative is reflected by image recognition from each frame included in the image data read in step S10. The motion detection unit 126 detects the type of motion, the time of motion, and the size of motion of the customer from the position change between frames. In addition, the motion detection unit 126 detects the type of motion, the time of motion, and the size of motion of the person in charge of customer service from the position change between frames. The motion detection unit 126 generates a motion detection table 133 including the detected information.

(S13) The scene evaluation unit 127 extracts the operation time of the customer service representative from the operation detection table 133 generated in step S12.
(S14) The scene evaluation unit 127 searches the action detection table 133 for the immediately preceding customer action for each action time of the person in charge of service extracted in step S13, determines whether there is a customer action, and determines whether there is a customer action. Determine the weight. Specifically, the scene evaluation unit 127 selects the weight w1 when there is no customer motion during the immediately preceding time S1, and a weight smaller than the weight w1 when there is a customer motion during the immediately preceding time S1. Select w2.

  (S15) The scene evaluation unit 127 searches the action detection table 133 for the immediately following customer action for each action time of the customer service person extracted in step S13, determines whether there is synchronization by the same type of customer action, and synchronizes. Determine the coefficient according to the presence or absence of. Specifically, if there is synchronization, that is, if there is the same type of customer action during the immediately following time S2, the scene evaluation unit 127 selects coefficient = 1. On the other hand, when there is no synchronization, that is, when there is no customer action of the same type during the time S2 immediately after, the scene evaluation unit 127 selects the coefficient = a. These coefficients are values by which the weight is multiplied, and a <1.

  (S16) The scene evaluation unit 127 sets a sliding window having a time width of time S0 × 2. The scene evaluation unit 127 calculates the scene evaluation value at the center time of the sliding window by using the weights and the coefficients of steps S14 and S15 calculated for the operation of the customer service person who belongs to the sliding window. This scene evaluation value represents the customer's sensitivity. The scene evaluation unit 127 calculates a scene evaluation value at time Δt intervals by sliding the sliding window by time Δt. The scene evaluation unit 127 generates a scene evaluation table 134 indicating scene evaluation values at each of a plurality of times.

  (S17) The operation detection unit 126 determines whether the image data has ended. The keyword detection unit 125 determines whether the audio data has ended. For example, the image data and the audio data are finished when the customer service for one customer is finished. If the image data and the audio data have been completed, the process proceeds to step S18, and if not completed, the process proceeds to step S10.

  (S18) The keyword evaluation unit 128 extracts the keyword appearance time from the keyword detection table 132 generated in step S11. For each appearance time of the keyword, the keyword evaluation unit 128 searches the scene evaluation table 134 generated in step S16 for surrounding scene evaluation values belonging to the time S3 immediately before and the time S3 immediately after the appearance time. The keyword evaluation unit 128 calculates the average value of the surrounding scene evaluation values.

  (S19) The keyword evaluation unit 128 classifies the average value of the scene evaluation values calculated in step S18 according to the identity of the keywords. The keyword evaluation unit 128 further averages the average value of the scene evaluation values for each keyword to calculate a keyword evaluation value. The keyword evaluation unit 128 generates a keyword evaluation table 135 indicating the keyword evaluation value.

  (S20) The keyword evaluation unit 128 extracts keywords having a keyword evaluation value exceeding the threshold T1 and keywords having a keyword evaluation value less than the threshold T2 as important keywords from the keyword evaluation table 135 generated in step S19. The keyword evaluation unit 128 generates an important keyword table 136 indicating the extracted important keywords and their ranks, and stores the important keyword table 136 in the evaluation result storage unit 124. The keyword evaluation unit 128 transmits the contents of the important keyword table 136 to the management device 41. The management device 41 displays the top N and bottom N important keywords based on the contents of the important keyword table 136.

  According to the information processing system of the second embodiment, the keyword is detected from the voice data, and the motion of the customer and the motion of the customer service representative are detected from the image data. The customer service person first performed the action, and immediately after that, the synchronization that the customer performed the same type of action was detected, and the scene evaluation value indicating the customer's empathy was calculated based on the action synchronization, and the vicinity of the keyword was calculated. A keyword evaluation value is calculated from the time scene evaluation value. Then, the preferred keyword having a high keyword evaluation value and the caution keyword having a low keyword evaluation value are extracted and reported to the administrator.

  This makes it possible to estimate important keywords that are likely to have had a positive impact on the customer's psychological state and important keywords that are likely to have had a negative impact on the customer's psychological state. Can improve the customer service skill of the person. In addition, since the keywords are evaluated based on the synchronization of the behavior of the customer service representative and the behavior of the customer, it is possible to extract the important keywords that reflect the psychological state of the customer more accurately than the method of evaluating the appearance frequency of the keywords. it can. In addition, the customer's psychological state of the customer who receives the service is determined in order to determine the condition that the customer did not perform the operation immediately before the operation of the customer service representative and the customer performed the same type of operation immediately after the operation of the service representative. Can be accurately estimated.

[Third Embodiment]
Next, a third embodiment will be described. The description will focus on the differences from the second embodiment, and the description of the same contents as those of the second embodiment may be omitted. The information processing system of the third embodiment is different from that of the second embodiment in the arrangement of the conversation analysis device 100.

FIG. 10 is a diagram illustrating an example of the information processing system according to the third embodiment.
The information processing system according to the third embodiment includes a management device 41, a camera device 50, and a conversation analysis device 100, as in the second embodiment. However, in the third embodiment, the camera device 50 is connected to the network 40, and the conversation analysis device 100 operates as a server device that communicates with the camera device 50 via the network 40. The camera device 50 transmits the audio signal and the image signal to the conversation analysis device 100 via the network 40. According to the information processing system of the third embodiment, the same effect as that of the second embodiment can be obtained.

[Fourth Embodiment]
Next, a fourth embodiment will be described. The description will focus on the differences from the second embodiment, and the description of the same contents as those of the second embodiment may be omitted. The information processing system according to the fourth embodiment is applied to a business where a customer service representative remotely serves a customer.

FIG. 11 is a diagram illustrating an example of the information processing system according to the fourth embodiment.
The information processing system according to the fourth embodiment includes a user device 42, camera devices 50 and 60, and a conversation analysis device 100. A camera device 60 is connected to the user device 42. A camera device 50 is connected to the conversation analysis device 100. The user device 42 and the conversation analysis device 100 are connected to the network 40.

  The user device 42 is a terminal device installed in a place different from the conversation analysis device 100, such as the customer's home, and used by the customer. The conversation analysis device 100 is a terminal device installed in an office or the like and used by a customer service representative. The camera devices 50 and 60 are device devices having a moving image shooting function using an image sensor and a voice recording function using a microphone. The camera device 50 is set to capture an image of the customer service representative and record the speech of the customer service representative. The camera device 60 is installed so as to photograph the customer and record the customer's speech.

  The user device 42 collects the image data of the image of the customer and the voice data of the voice of the customer, and transmits it to the conversation analysis device 100 via the network 40. Further, the user device 42 receives, from the conversation analysis device 100 via the network 40, the image data obtained by photographing the customer service representative and the voice data obtained by recording the voice of the customer service representative. The user device 42 displays the image of the customer service representative on the display based on the received image data, and reproduces the speech of the customer service representative from the speaker based on the received audio data.

  The conversation analysis device 100 collects the image data of the customer service representative and the voice data of the customer service representative, and transmits it to the user device 42 via the network 40. Further, the conversation analysis device 100 receives the image data of the customer and the voice data of the customer from the user device 42 via the network 40. The conversation analysis device 100 displays the video of the customer on the display based on the received image data, and reproduces the utterance of the customer from the speaker based on the received voice data. As a result, the customer and the person in charge of hospitality can talk in a video conference system.

  Further, the conversation analysis device 100, based on the image data of the customer, the image data of the customer service representative, the voice data of the customer, and the voice data of the customer service representative, identifies the customer and the customer service representative as in the second embodiment. Analyze the conversation between. That is, the conversation analysis device 100 detects a keyword uttered by a customer or a customer service representative from the voice data, and detects a customer action and a customer serviceperson's action during the conversation from the image data. The conversation analysis device 100 evaluates the keywords from the actions of the customer and the person in charge of serving the customer and extracts the important keywords. The conversation analysis device 100 displays the upper N and lower N important keywords among the extracted important keywords on the display.

  However, the conversation analysis device 100 may transmit the important keyword to the management device 41 via the network 40. Alternatively, the customer service representative may separate the terminal device used for customer service from the conversation analysis device 100. According to the information processing system of the fourth embodiment, the same effect as that of the second embodiment can be obtained with respect to remote customer service.

[Fifth Embodiment]
Next, a fifth embodiment will be described. The description will focus on the differences from the second embodiment, and the description of the same contents as those of the second embodiment may be omitted. In the second embodiment, the customer's co-sensitivity is estimated from the synchronization of the customer's behavior and the customer service representative's behavior, and the importance of the keyword is evaluated according to the customer's co-sensitivity. On the other hand, in the fifth embodiment, the degree of customer service of the customer service representative is estimated from the synchronization status of the behavior of the customer and the operation of the customer service representative, and the importance of the keyword is evaluated according to the degree of customer service of the customer service representative. To do. The degree of customer service represents a customer service attitude that includes customer service aggressiveness, enthusiasm, and politeness. The scene evaluation value calculated in the fifth embodiment corresponds to the degree of customer service of the customer service representative, and the keyword evaluation value calculated in the fifth embodiment is when the customer or the customer service engineer utters a keyword. It reflects the degree of customer service. Therefore, the important keywords extracted in the fifth embodiment are keywords that are estimated to be highly related to good customer service and keywords that are estimated to be highly related to bad customer service. It can be said that the important keyword extracted in the fifth embodiment reflects the psychological state of the customer service representative.

  The information processing system of the fifth embodiment can be realized by the same configuration as the information processing system of the second embodiment shown in FIGS. Therefore, in the following, the fifth embodiment may be described using the same reference numerals as those in FIGS. 2, 3, 6 to 8. The information processing system according to the fifth embodiment may have the same system configuration as the information processing system according to the third embodiment shown in FIG. 10, and the fourth embodiment shown in FIG. It is also possible to adopt a system configuration similar to that of the information processing system of the embodiment.

FIG. 12 is a diagram showing an example of keyword extraction according to the fifth embodiment.
In the fifth embodiment, the synchronization between the movement of the customer and the movement of the customer service person is detected. The synchronization according to the fifth embodiment is a chain of operations in which an operation is started by a customer, and immediately after that, a customer service person performs the same kind of operation. The synchronization detected in the fifth embodiment differs from the synchronization in the second embodiment in that the order of operations is different. In this case, it is presumed that the customer service representative watches the customer carefully and listens carefully to the customer. Therefore, the keyword issued by the customer or the person in charge of customer service during such synchronization may be a keyword associated with good customer service. On the other hand, a keyword issued by a customer or a person in charge of customer service when such synchronization has not occurred may be a keyword associated with bad customer service.

For example, consider scenes 74-76 described below.
In the scene 74, the customer service representative performs an operation of laughing, and immediately after that, the customer performs an operation of laughing. Scene 74 corresponds to scene 71 of FIG. The customer service representative is operating within the time S1 immediately before the customer's operation, and the customer service representative is not performing within the time S2 immediately after the customer's operation. In the scene 74, the behavior of the customer and the behavior of the customer contact person are not synchronized with each other starting from the customer, and it is determined that the degree of customer service is small. Then, the evaluation value of the keyword “quick” issued by the customer or the person in charge of customer service around the scene 74 becomes low.

  In the scene 75, although the customer laughed, the customer service representative did not operate within the time S1 immediately before the customer's operation, and the customer service representative did not perform within the time S2 immediately after the customer's operation. Not working. In the scene 75, the customer's action and the action of the person in charge of customer service are not synchronized, so it is determined that the degree of customer service is small. Then, the evaluation value of the keyword “interesting” issued by the customer or the person in charge of serving the customer around the scene 75 becomes low.

  In the scene 76, the customer makes a nod operation, and immediately after that, the customer service person makes a nod operation. Scene 76 corresponds to scene 73 of FIG. The customer service person does not perform the operation within the time S1 immediately before the customer's operation, and the customer service person performs the same type of operation within the time S2 immediately after the customer operation. Since the operation of the customer and the operation of the person in charge of customer service are synchronized with each other starting from the customer, it is determined that the degree of customer service is high. Then, the evaluation value of the keyword "beautiful" issued by the customer or the person in charge of serving the customer around the scene 76 becomes high.

  When the keyword evaluation value is calculated, as in the second embodiment, the conversation analysis device 100 estimates a keyword having a keyword evaluation value larger than the threshold value T1 as a preferable keyword and extracts it as an important keyword. Further, the conversation analysis device 100 estimates a keyword having a keyword evaluation value smaller than the threshold value T2 as a keyword requiring attention and extracts it as an important keyword. In the above example, “beautiful” may be extracted as an important keyword.

FIG. 13 is a flowchart showing a procedure example of conversation analysis according to the fifth embodiment.
(S30) The operation detection unit 126 reads image data from the image storage unit 122. Further, the keyword detection unit 125 reads out voice data from the voice storage unit 121.

  (S31) The keyword detecting unit 125 converts the voice data read in step S30 into a word string by voice recognition. The keyword detection unit 125 searches the search target keyword registered in the keyword table 131 from the converted word string, and generates the keyword detection table 132 indicating the searched keyword and the appearance time of the keyword.

  (S32) The motion detection unit 126 recognizes, from each frame included in the image data read in step S30, the area in which the customer is reflected and the area in which the customer service representative is reflected by image recognition. The motion detection unit 126 detects the type of motion, the time of motion, and the size of motion of the customer from the position change between frames. In addition, the motion detection unit 126 detects the type of motion, the time of motion, and the size of motion of the person in charge of customer service from the position change between frames. The motion detection unit 126 generates a motion detection table 133 including the detected information.

(S33) The scene evaluation section 127 extracts the customer's motion time from the motion detection table 133 generated in step S32.
(S34) The scene evaluation unit 127 searches the action detection table 133 for the last action of the customer service representative for each of the customer operation times extracted in step S33, determines whether or not there is a service of the customer service representative, and determines the action of the customer service representative. The weight is determined according to the presence or absence of. Specifically, the scene evaluation unit 127 selects the weight w1 when there is no action of the customer service representative during the immediately preceding time S1, and selects the weight w1 when there is action of the customer service representative during the immediately preceding time S1. Select w2.

  (S35) The scene evaluation unit 127 searches the motion detection table 133 for the immediately following service operation of the customer for each of the customer operation times extracted in step S33, and determines whether or not there is synchronization by the same type of customer operation. , Determine a coefficient according to the presence or absence of synchronization. Specifically, when there is synchronization, that is, when there is the same type of customer contact person operation during the immediately following time S2, the scene evaluation unit 127 selects coefficient = 1. On the other hand, when there is no synchronization, that is, when there is no same type of customer contact person operation during the immediately following time S2, the scene evaluation unit 127 selects the coefficient = a. These coefficients are values by which the weight is multiplied, and a <1.

  (S36) The scene evaluation unit 127 sets a sliding window having a time width of time S0 × 2. The scene evaluation unit 127 calculates the scene evaluation value at the center time of the sliding window by using the weights and coefficients of steps S34 and S35 calculated for the movement of the customer who belongs to the sliding window. This scene evaluation value represents the degree of customer service of the customer service representative. The scene evaluation unit 127 calculates a scene evaluation value at time Δt intervals by sliding the sliding window by time Δt. The scene evaluation unit 127 generates a scene evaluation table 134 indicating scene evaluation values at each of a plurality of times.

  (S37) The operation detection unit 126 determines whether the image data has ended. The keyword detection unit 125 determines whether the audio data has ended. If the image data and the audio data have been completed, the process proceeds to step S38, and if not completed, the process proceeds to step S30.

  (S38) The keyword evaluation unit 128 extracts the appearance time of the keyword from the keyword detection table 132 generated in step S31. For each appearance time of the keyword, the keyword evaluation unit 128 searches the scene evaluation table 134 generated in step S36 for surrounding scene evaluation values belonging to the time S3 immediately before and the time S3 immediately after the appearance time. The keyword evaluation unit 128 calculates the average value of the surrounding scene evaluation values.

  (S39) The keyword evaluation unit 128 classifies the average value of the scene evaluation values calculated in step S38 according to the identity of the keywords. The keyword evaluation unit 128 further averages the average value of the scene evaluation values for each keyword to calculate a keyword evaluation value. The keyword evaluation unit 128 generates a keyword evaluation table 135 indicating the keyword evaluation value.

  (S40) The keyword evaluation unit 128 extracts, from the keyword evaluation table 135 generated in step S39, keywords whose keyword evaluation value exceeds the threshold T1 and keywords whose keyword evaluation value is below the threshold T2 as important keywords. The keyword evaluation unit 128 generates an important keyword table 136 indicating the extracted important keywords and their ranks, and stores the important keyword table 136 in the evaluation result storage unit 124. The keyword evaluation unit 128 transmits the contents of the important keyword table 136 to the management device 41. The management device 41 displays the top N and bottom N important keywords based on the contents of the important keyword table 136.

  According to the information processing system of the fifth embodiment, the keyword is detected from the voice data, and the action of the customer and the action of the person in charge of customer service are detected from the image data. The synchronization that the customer performed the action first and the customer service representative performed the same type of action immediately after that was detected, and the scene evaluation value indicating the degree of customer service was calculated based on the synchronization of the action and the time around the keyword was calculated. A keyword evaluation value is calculated from the scene evaluation value. Then, the preferred keyword having a high keyword evaluation value and the caution keyword having a low keyword evaluation value are extracted and reported to the administrator.

  This makes it possible to estimate important keywords that are likely to be related to the good customer service attitude of the customer service staff and important keywords that are likely to be related to the bad customer service attitude of the customer service staff. Can help improve. Further, since the keyword is evaluated from the synchronization of the customer's action and the action of the customer service representative, the important keyword reflecting the psychological state of the customer service provider is extracted more accurately than the method of evaluating the keyword occurrence frequency. be able to. In addition, since the service representative does not perform the action immediately before the customer's action and the condition that the service representative has performed the same type of action immediately after the customer's action, the service representative in the service attitude is determined. Can be accurately estimated.

[Sixth Embodiment]
Next, a sixth embodiment will be described. The description will focus on the differences from the second embodiment, and the description of the same contents as those of the second embodiment may be omitted. In the second embodiment, the search target keyword to be detected from the voice data is designated in advance. On the other hand, in the sixth embodiment, the search target keyword is automatically added through the conversation analysis, and the burden of manually specifying the search target keyword is reduced.

  The information processing system of the sixth embodiment can be realized by the same system configuration as the information processing system of the second embodiment shown in FIG. The information processing system according to the sixth embodiment may have the same system configuration as the information processing system according to the third embodiment shown in FIG. 10, and the information processing system according to the fourth embodiment shown in FIG. It is also possible to have a system configuration similar to that of the information processing system of the embodiment. However, instead of the conversation analysis device 100, a conversation analysis device 200 described later is used. The conversation analysis device 200 according to the sixth embodiment can be realized with the same hardware configuration as the information processing system according to the second embodiment shown in FIG. Note that, as in the fifth embodiment, it is possible to extract an important keyword that reflects the degree of customer service.

FIG. 14 is a block diagram showing a functional example of the conversation analysis device according to the sixth embodiment.
The conversation analysis device 200 includes a voice storage unit 221, an image storage unit 222, a keyword storage unit 223, and an evaluation result storage unit 224. These storage units are realized by using a storage area of a RAM or a HDD, for example. The conversation analysis device 200 also includes a keyword detection unit 225, a motion detection unit 226, a scene evaluation unit 227, a word extraction unit 228, and a keyword evaluation unit 229. These processing units are realized by using a program, for example.

  The voice storage unit 221 stores voice data including a voice signal received from the camera device 50. The image storage unit 222 stores the image data including the image signal received from the camera device 50. The keyword storage unit 223 stores the keyword table 131 shown in FIG. 7. In the keyword table 131, search target keywords designated in advance by an administrator or the like are registered. Further, in the keyword table 131, the search target keywords automatically added by the keyword evaluation unit 229 are registered. The evaluation result storage unit 224 stores the important keyword table 136 shown in FIG.

  The keyword detection unit 225 converts the voice data stored in the voice storage unit 221 into a word string by voice recognition. The keyword detection unit 225 detects the search target keyword indicated by the keyword table 131 stored in the keyword storage unit 223 from the word string, and generates the keyword detection table 132 shown in FIG. 7.

  The motion detection unit 226 recognizes a customer and a person in charge of customer service by image recognition from each frame included in the image data stored in the image storage unit 222. The motion detection unit 226 extracts the characteristic information of the area in which the customer service representative is reflected from each frame, and detects the motion of the customer service representative based on the change in the characteristic information between the frames. In addition, the motion detection unit 226 extracts the characteristic information of the area in which the customer appears from each frame, and detects the customer's motion based on the change in the characteristic information between the frames. The motion detector 226 generates the motion detection table 133 shown in FIG. 7.

  The scene evaluation unit 227 calculates a scene evaluation value at intervals of time Δt based on the motion detection table 133. As described above, the scene evaluation unit 227 determines whether or not the customer's motion is occurring during the time S1 immediately before that based on the operation time of the customer service representative, and during the time S2 immediately thereafter. It is determined whether the same type of customer action is occurring. The scene evaluation unit 227 calculates a scene evaluation value based on such a synchronization situation between the operation of the customer service representative and the operation of the customer, and generates the scene evaluation table 134 shown in FIG. 8.

  The word extraction unit 228 converts the voice data stored in the voice storage unit 221 into a word string by voice recognition. The word extraction unit 228 extracts an unregistered word that is not registered in the keyword table 131 from the word string. However, general-purpose words (stop words) such as Japanese particles and auxiliary verbs that can occur in large numbers during utterance are excluded. For extracting unregistered words, the technique described in Non-Patent Document 1 (“Speech summary sentence generation method by word extraction and its evaluation”) may be used. The word extraction unit 228 notifies the keyword evaluation unit 229 of the extraction result of the unregistered word. The extraction result includes the extracted unregistered word and the time when the unregistered word appears.

  The keyword evaluation unit 229 calculates a word evaluation value for each of the keyword and the unregistered word based on the keyword detection table 132, the scene evaluation table 134, and the extraction result of the unregistered word. The method of calculating the word evaluation value is the same as the keyword evaluation value of the second embodiment. That is, the word evaluation value for the keyword is the same as the keyword evaluation value, and the word evaluation value for the unregistered word is calculated by the same method as the keyword evaluation value.

  Therefore, the keyword evaluation unit 229 extracts, for each keyword, one or more appearance times of the keyword, averages the scene evaluation value of the peripheral time for each appearance time, and further averages between the one or more appearance times. The word evaluation value. In addition, the keyword evaluation unit 229 extracts, for each unregistered word, one or more appearance times of the unregistered word, averages the scene evaluation values of the peripheral time for each appearance time, and performs one or more appearance times. Is further averaged to obtain a word evaluation value. Then, the keyword evaluation unit 229 generates a word evaluation table having the same data structure as the keyword evaluation table 135 shown in FIG. In the word evaluation table, both the word evaluation value for the keyword and the word evaluation value for the unregistered word are described.

  The keyword evaluation unit 229 extracts, from the keywords, keywords whose word evaluation value exceeds the threshold value T1 and keywords whose threshold value is less than the threshold value T2 as important keywords. Then, the keyword evaluation unit 229 generates the important keyword table 136 shown in FIG. 8 and stores it in the evaluation result storage unit 224, and transmits the contents of the important keyword table 136 to the management device 41. The keyword evaluation unit 229 also extracts unregistered words whose word evaluation value exceeds the threshold value T1 and unregistered words whose threshold value is less than the threshold value T2 among the unregistered words. The keyword evaluation unit 229 adds the extracted unregistered word to the keyword table 131 as a search target keyword.

FIG. 15 is a flowchart showing a procedure example of conversation analysis according to the sixth embodiment.
(S50) The operation detection unit 226 reads the image data from the image storage unit 222. Further, the keyword detection unit 225 reads the voice data from the voice storage unit 221. Further, the word extraction unit 228 reads the same voice data from the voice storage unit 221.

  (S51) The keyword detection unit 225 converts the voice data read out in step S50 into a word string by voice recognition, searches the search target keyword registered in the keyword table 131 from the word string, and generates the keyword detection table 132. . The word extracting unit 228 also converts the voice data read in step S50 into a word string by voice recognition, and extracts unregistered words that are not registered in the keyword table 131 from the word string.

  (S52) The motion detection unit 226 recognizes the area in which the customer is in the image and the area in which the customer service representative is in the image by image recognition from each frame included in the image data read in step S50. The motion detection unit 226 detects the type of motion, the time of motion, and the size of motion of the customer from the position change between frames. In addition, the motion detection unit 226 detects the type of motion, the motion time, and the size of the motion of the customer service representative from the position change between the frames. The motion detection unit 226 generates a motion detection table 133 including the detected information.

(S53) The scene evaluation unit 227 extracts the operation time of the customer service representative from the operation detection table 133 generated in step S52.
(S54) The scene evaluation unit 227 searches the action detection table 133 for the immediately preceding customer action for each action time of the person in charge of service extracted in step S53, determines whether there is a customer action, and determines whether there is a customer action. Determine the weight.

  (S55) The scene evaluation unit 227 searches the action detection table 133 for the immediately following customer action for each action time of the customer service person extracted in step S53, determines whether there is synchronization by the same type of customer action, and synchronizes. Determine the coefficient according to the presence or absence of.

  (S56) The scene evaluation unit 227 sets a sliding window having a predetermined time width. The scene evaluation unit 227 calculates the scene evaluation value at the center time of the sliding window by using the weights and coefficients of steps S54 and S55 calculated for the operation of the customer service person who belongs to the sliding window. The scene evaluation unit 227 generates a scene evaluation table 134 by sliding the sliding window by time Δt.

  (S57) The operation detection unit 226 determines whether the image data has ended. The keyword detection unit 225 determines whether the voice data has ended. The word extraction unit 228 determines whether the voice data has ended. If the image data and the audio data have been completed, the process proceeds to step S58, and if not completed, the process proceeds to step S50.

  (S58) The keyword evaluation unit 229 extracts the appearance time of the keyword from the keyword detection table 132 generated in step S51. For each appearance time of the keyword, the keyword evaluation unit 229 retrieves the scene evaluation value around the appearance time from the scene evaluation table 134 generated in step S56, and calculates the average value of the surrounding scene evaluation values. Similarly, the keyword evaluation unit 229 extracts the appearance time of the unregistered word from the extraction result of the unregistered word in step S51. For each appearance time of an unregistered word, the keyword evaluation unit 229 searches the scene evaluation table 134 for a scene evaluation value around the appearance time, and calculates an average value of the surrounding scene evaluation values.

  (S59) The keyword evaluation unit 229 classifies the average value of the scene evaluation values calculated in step S58 according to the identity of the keyword for the keyword, and further averages the average value of the scene evaluation values for each keyword to obtain the word. Calculate the evaluation value. Similarly, the keyword evaluation unit 229 classifies the average value of the scene evaluation values calculated in step S58 for the unregistered words according to the identity of the unregistered words, and the average value of the scene evaluation values for each unregistered word. Is further averaged to calculate a word evaluation value. The keyword evaluation unit 229 generates a word evaluation table showing the word evaluation value of the keyword and the unregistered word.

  (S60) The keyword evaluation unit 229 extracts, from the word evaluation table generated in step S59, keywords having a word evaluation value exceeding the threshold T1 and keywords having a word evaluation value less than the threshold T2 as important keywords. The keyword evaluation section 229 generates an important keyword table 136 showing the extracted important keywords and their ranks, and stores them in the evaluation result storage section 224. The keyword evaluation unit 229 transmits the contents of the important keyword table 136 to the management device 41. The management device 41 displays the top N and bottom N important keywords based on the contents of the important keyword table 136.

  (S61) The keyword evaluation unit 229 extracts an unregistered word whose word evaluation value exceeds the threshold T1 and an unregistered word whose word evaluation value is less than the threshold T2 from the word evaluation table generated in step S59. The keyword evaluation unit 229 adds the extracted unregistered word to the keyword table 131 as a new search target keyword.

  According to the information processing system of the sixth embodiment, the same effect as that of the second embodiment can be obtained. Further, in the sixth embodiment, search target keywords that may be highly related to customer service are automatically added through conversation analysis. Therefore, it is not necessary to comprehensively specify the search target keywords in advance, and the burden of the work of specifying the search target keywords can be reduced. Further, since the search target keyword is automatically learned, the accuracy of the important keyword extracted from the conversation is improved and the usefulness of the conversation analysis result is improved.

10 Keyword Extractor 11 Storage Unit 12 Processing Unit 13 Voice Data 14 Motion Data 15 Keywords 16, 17 Motion 18 Evaluation Value

Claims (8)

On the computer,
Voice data indicating an utterance made by at least one of the first user and the second user in a conversation between a first user who is a service providing side and a second user who is a service receiving side Detect keywords from
From the motion data indicating the motion performed by the first user and the motion performed by the second user in the conversation, the timing of the first motion performed by the first user and the second motion performed by the second user. Detects the timing of movement,
An evaluation value indicating the importance of the keyword is calculated based on the relationship between the timing of the first operation and the timing of the second operation.
A keyword extraction program that executes processing. The operation data is image data obtained by capturing the first user and the second user during the conversation.
The keyword extraction program according to claim 1. The second operation is the same type of operation that is performed after the first operation,
In the calculation of the evaluation value, when the timing of the second operation is within a predetermined time from the timing of the first operation, the timing of the second operation is less than that when the timing of the second operation is not within the predetermined time. Highly value the importance,
The keyword extraction program according to claim 1. The second operation is the same type of operation that is performed after the first operation,
In the calculation of the evaluation value, the shorter the elapsed time from the timing of the first operation to the timing of the second operation, the higher the importance is evaluated.
The keyword extraction program according to claim 1. In the calculation of the evaluation value, as a predetermined condition in which the timing of the second operation is later than the timing of the first operation, the second user has a predetermined time immediately before the first operation. The psychological state of the second user is determined according to whether or not the operation is not performed and the timing of the second operation exists within the predetermined time immediately after the first operation. Then, the evaluation value is calculated based on the determination result of the psychological state,
The keyword extraction program according to claim 1. In the detection of the keyword, a predetermined search target keyword is searched from the voice data,
Further on the computer,
Extracting words other than the search target keyword from the voice data,
Based on the relationship, to calculate another evaluation value indicating the importance of the extracted word,
When the other evaluation value satisfies a predetermined condition, a process of adding the extracted word to the search target keyword is executed,
The keyword extraction program according to claim 1. Computer
Voice data indicating an utterance made by at least one of the first user and the second user in a conversation between a first user who is a service providing side and a second user who is a service receiving side Detect keywords from
From the motion data indicating the motion performed by the first user and the motion performed by the second user in the conversation, the timing of the first motion performed by the first user and the second motion performed by the second user. Detects the timing of movement,
An evaluation value indicating the importance of the keyword is calculated based on the relationship between the timing of the first operation and the timing of the second operation.
Keyword extraction method. Voice data indicating an utterance made by at least one of the first user and the second user in a conversation between a first user who is a service providing side and a second user who is a service receiving side And a storage unit that stores operation data indicating an operation performed by the first user and an operation performed by the second user in the conversation,
A keyword is detected from the voice data, the timing of the first operation by the first user and the timing of the second operation by the second user are detected from the operation data, and the timing of the first operation is detected. A processing unit that calculates an evaluation value indicating the importance of the keyword based on the relationship with the timing of the second operation;
A keyword extraction device having.

Images