JP6400445B2 - Conversation analyzer, conversation analysis system, conversation analysis method, and conversation analysis program - Google Patents

Description

  The present invention analyzes the situation of real communication between humans, makes it possible to evaluate the preference of information transmission between real humans, and further enables users (influencers) who have a great influence on others on a specific topic. The present invention relates to a conversation analysis apparatus, a conversation analysis system, a conversation analysis method, and a conversation analysis program that can be extracted.

For example, techniques disclosed in Patent Documents 1 to 3 are disclosed as techniques for analyzing the situation of real communication between humans.
In Patent Document 1, there is SNS (Social Networking Service) that can build a virtual human relationship on the network, and which SNS user has influence on the network from the human relationship registered in this SNS and its posting history Is performed.

  Patent Document 2 discloses a technique for accurately recognizing the start and end of a conversation by a plurality of general users and recording only the necessary conversation. Specifically, a conversation management system is provided in which a conversation partner is identified, the start and end of a conversation are recorded by a wearable device, the recorded conversation is converted into text by a server device, and can be searched later. Yes.

  Patent Document 3 includes a plurality of sensors attached to an operator, a base station that receives data transmitted from the plurality of sensors, and a management computer that receives the data from the base station via a network. In the sensor network system, the identifier assigned to the sensor is detected by each other sensor by infrared communication, and the detected time is measured to group the workers as if they are facing each other. A system for holding information to be associated is provided.

For communication between humans, not only verbal conversation but also non-verbal information such as facial expressions and gestures is an element that activates communication, which is related to the listening skill regardless of the speaking skill. As a listening skill, attention has been paid to facial expressions and gestures as non-verbal information other than utterances. Specifically, facial expressions include smiles and facial expressions as elements that promote communication. “Golding” is known (see Non-Patent Document 1 and Non-Patent Document 2).
Furthermore, “kindness and gratitude” is known as a good communication that gives people happiness (see Non-Patent Document 3).

Japanese Patent No. 5499924 JP 2011-180645 A Japanese Patent No. 5417627

“Dialogue flow and whispering pattern change” (HAI Symposium 2010, Masafumi Inoue et al.) “The role of facial expression and body movement in communication” (Sachiko Takagi) https://dspace.wul.waseda.ac.jp/dspace/handle/2065/27539 “Effects of kindness and gratitude on happiness” (Mizuho Kitamura)

  However, the technique described in Patent Document 1 is not limited to extracting influencers from real human relationships, but only to efficiently find influencers from user relationships on SNS.

  According to the technique of Patent Document 2, it is possible to guess the start and end of a real conversation with a wearable device, and record and manage the conversation. However, it is not possible to analyze the skill of communication only by recording conversations.

  In Patent Document 3, a real face-to-face recording was inferred with a wearable device, and human relations were drawn as a community. On the other hand, a reference to whether the person shows the role of an influencer in a certain topic is Absent.

As described above, in the related art described above, although there is a technique for analyzing human relationships using big data such as SNS, it has been impossible to analyze real human relationships and extract influencers on a certain topic.
Although there was a technique for recording conversation and feelings at that time, there was no technique for analyzing the skill of conversation.

  The present invention has been proposed in view of the above circumstances, and records and analyzes a real conversation using a wearable terminal, analyzes the user's conversation partner acceptance and the user's conversation skill, It is an object of the present invention to provide a conversation analysis device, a conversation analysis system, a conversation analysis method, and a conversation analysis program that extract users (influencers) who have a great influence on other people about a topic.

In order to achieve the above object, the conversation analyzer according to claim 1 is a wearable terminal that is worn on a user's head to detect a conversation between people,
An important word database storing a plurality of important words for extracting important words uttered in the conversation;
A microphone for collecting voice information of the conversation;
A voice evaluation unit for recognizing important words from the voice information;
A conversation analysis unit that analyzes a partner's acceptability for a conversation with a predetermined weighting from the recognition result of the voice evaluation unit;
A notification unit that feeds back the acceptance to the user;
It is characterized by having.

Claim 2 is the conversation analyzer of claim 1 ,
A camera that collects the facial expression of the conversation partner and the movement of the head as a video,
A video evaluation unit for recognizing the smile time and number of whispering of the conversation partner per unit conversation time from the video,
The conversation analysis unit is characterized in that it receives a certain weight from the recognition results of the voice evaluation unit and the video evaluation unit, and analyzes the acceptability of the other party for the conversation.

Claim 3 is the conversation analysis device according to claim 1 or claim 2 ,
The notification unit is configured by a voice reproduction unit that can recognize the acceptance level by hearing.

Claim 4 is the conversation analyzer according to claim 1 or claim 2 ,
The notification unit is configured by a display device that can visually recognize the acceptability.
Claim 5 is the conversation analysis apparatus according to claim 1 or claim 2 ,
The voice evaluation unit has a function of analyzing the tone and / or tempo of conversation,
The conversation analysis unit has a function of analyzing the skill of the conversation by summing up the difference in tone and / or tempo of the conversation between the other party and the user and giving a certain weight.
The notification unit has a function of feeding back the skill of the analyzed conversation to a user.

Claim 6 is the conversation analyzer of claim 5 ,
An acceleration sensor for recognizing the movement of the user's head;
A whisker recognition unit that analyzes the number of whistling per unit conversation time of the user from the acceleration sensor,
The conversation analysis unit analyzes the skill of the conversation by giving a predetermined weight to the recognition result by the whisper recognition unit.

Claim 7 is the conversation analysis device according to claim 5 or claim 6 ,
The conversation analysis unit is characterized by analyzing the skill of the conversation with a function of counting and repeating the words in the conversation and performing a constant weighting.

Claim 8 is the conversation analysis apparatus according to any one of claims 1 to 7,
The analysis by the conversation analysis unit analyzes a feature word using an index (TF-IDF) for identifying how characteristic the word that appears in the conversation is from the speech recognition result of one conversation. A storage unit is provided for recording each conversation together with the degree of acceptance and / or the skill of the conversation.

A conversation analysis system according to claim 9 includes the conversation analysis apparatus according to any one of claims 1 to 8 , and a conversation evaluation server apparatus that collects and manages conversation logs of the conversation analysis apparatus,
The conversation evaluation server device
A control unit that analyzes a conversation log of all users of the conversation analyzer and evaluates a conversation;
A storage unit for storing the conversation log in a conversation log database;
It is characterized by having.
Claim 10 is the conversation analysis system of claim 9 ,
The conversation analysis device comprises a topic word database for storing a plurality of topic words for influencer extraction for a specific topic uttered during the conversation,
The voice evaluation unit further recognizes a topic word from the voice information,
The storage unit records the recognized topic word for each conversation in a conversation log,
The control unit of the conversation evaluation server device extracts a conversation log including a topic word to be evaluated when analyzing a conversation log of a plurality of users and performing a conversation evaluation on the topic word.
It is characterized in that users who have an influence on the topic word are extracted as influencers when the conversation acceptance or skill of conversation is higher than the average value from a plurality of users of the extracted conversation logs. .

Claim 11 is the conversation analysis system of claim 10,
The conversation analysis device comprises a communication unit for exchanging an identifier of a user of the user at the start of conversation and an identifier of a user who is a conversation partner,
The control unit of the conversation evaluation server analyzes the conversation log, further records the identifier of the user who is a conversation partner and the conversation time, and the total conversation time for the conversation log between the identifiers assigned to the user. It is equipped with a human relationship management function that tabulates the degree of connection between users and the degree of acceptance, and manages the magnitude relationship between users as the user's human relationship,
The storage unit of the conversation evaluation server includes a human relationship database that stores the human relationships of the users in association with each other.

Claim 12 is a method for analyzing and evaluating a conversation by information processing by computer software for a conversation analysis system including a conversation analysis device and a conversation evaluation server,
A procedure for recognizing preset topic words and important words from voice information generated during conversation between a user and a conversation partner;
Acquiring the facial expression of the conversation partner and the movement of the head as a video, and recognizing the smile time and number of whispering of the conversation partner per unit conversation time from the video;
A procedure for analyzing the degree of acceptance of the conversation partner with respect to the topic of conversation from each recognition result;
Feeding back the acceptance to the user;
It is characterized by including.

Claim 13 is a method for analyzing and evaluating a conversation by information processing by computer software for a conversation analysis system including a conversation analysis device and a conversation evaluation server,
A procedure for recognizing preset topic words and important words from voice information generated during conversation between a user and a conversation partner;
Acquiring the facial expression of the conversation partner and the movement of the head as a video, and recognizing the smile time and number of whispering of the conversation partner per unit conversation time from the video;
A procedure for counting the difference in tone and / or tempo of conversation between the conversation partner and the user;
A procedure for analyzing the skill of the conversation with respect to the topic of conversation by performing a predetermined weighting from each recognition result and the total result,
A procedure for feeding back the skill of the conversation to the user;
It is characterized by including.

  Claim 14 is the conversation analysis method of claim 12 or claim 13,
  Conversation of the conversation partner with respect to the topic of the analyzed conversation of the identifier for identifying the user given in advance and the recognized topic word and important word for the conversation between the plurality of users and each conversation partner Alternatively, the skill of conversation is stored as a conversation log, and the plurality of conversation logs are aggregated and managed by the conversation evaluation server device, and
  When analyzing the conversation log of the plurality of users and performing conversation evaluation on the topic word, a procedure for extracting a conversation log including the topic word to be evaluated;
  A procedure for extracting a user who has an influence on the topic word as an influencer when the conversation acceptance or skill of the conversation is higher than an average value from a plurality of users of the extracted conversation log;
It is characterized by including.
  Claim 15 is the conversation analysis method of claim 14,
  A procedure for exchanging the identifier of the user and the identifier of the user who is the conversation partner at the start of the conversation;
  In the conversation log of a plurality of users that are aggregated and managed by the conversation evaluation server, a procedure for further recording an identifier and a conversation time of a user who is a conversation partner;
  For the conversation log between the identifiers given to the user, a procedure for storing the degree of connection between the users who totaled the total conversation time, and the degree of acceptance and the magnitude relationship between the users as a human relationship database;
  A procedure for extracting information of the human relation database by a topic to be evaluated;
  A procedure for displaying the extracted human relationship as a human relationship map;
It is characterized by including.

The conversation analysis program according to claim 16 comprises:
Each procedure of the conversation analysis method according to any one of claims 12 to 15 is constructed on a computer.

According to the conversation analysis apparatus of claim 1, an important word is detected from the voice of the conversation with the other party, the other party's acceptance level for the conversation is calculated from the voice analysis result, and the other party's degree of acceptance (acceptance level) regarding the conversation is real-time You can give feedback to users.
In addition, it is possible to determine from the important word database that the other party has accepted the conversation, and it is possible to set an important word for confirming whether the conversation is meaningful.

According to the conversation analysis apparatus of claim 2 , it is detected from the other person's whispering video and smiley video whether the conversation topic is transmitted to the other party, and the other party's acceptability for the conversation is calculated from the voice analysis result and the video analysis result. It is possible to feed back the acceptability (acceptance level) of the other party to the user in real time.

According to the conversation analysis device of claim 3 , by configuring the notification unit with the voice reproduction unit, it is possible to grasp the degree of acceptance of the partner during the conversation by hearing.

According to the conversation analysis device of the fourth aspect , by configuring the notification unit with the display unit, it is possible to visually grasp the acceptability of the partner during the conversation.

According to the conversation analysis device of claim 5 , by analyzing the tone of the conversation (tone and / or tempo), it is possible to determine in real time the skill of the conversation, such as whether or not the conversation is proceeding comfortably in synchronization with the other party. Can provide feedback.

According to the conversation analysis device of claim 6 , by analyzing the number of whispering per unit conversation time of the user himself, the conversation can be advanced while showing the partner that the user has received the word of the partner once. It is possible to provide feedback to the user in real time about the skill of the conversation.

According to the conversation analysis device of claim 7 , by repeatedly speaking a word in the exchange of words between the user and the conversation partner, the user once accepts and firmly understands the partner's word. It is possible to provide feedback to the user in real time about the skill of the conversation as to whether the conversation is progressing while showing to the other party.

According to the conversation analysis apparatus of claim 8 , the analysis result of the exchange of words between the user and the conversation partner is recorded for each conversation, and the characteristic words of the conversation contents are further recorded. When looking back, it is possible to make the memory of conversation easier.

According to the conversation analysis system of claim 9 , the conversation log of all users of the conversation analysis apparatus is analyzed and managed, and the system is used for acceptability of conversation performed by the user and skill of conversation. In addition to being able to evaluate for each user whether it is superior to the entire user, it is possible to evaluate which user is accepted by others regarding a specific topic and whether the conversation is skillful.
According to the conversation analysis system of claim 10, when calculating the acceptability of the other party from the voice analysis result, the acceptability of the conversation regarding the specific topic by the user is evaluated by using a system that recognizes the topic word. Can do. Further, topic words relating to a specific topic for which conversation is to be evaluated can be set by the topic word database.

  According to the conversation analysis system of claim 11, by analyzing the conversation logs of all users of the conversation analysis apparatus and associating the human relations, which user is central in which human relations in which topics ( It can be a system that can manage whether an influencer is extracted or accepted for a topic.

  According to the conversation analysis method of claim 12, the topic word and the important word are detected from the voice of the conversation with the other party, and whether the topic during the conversation is transmitted to the other party is detected from the other person's whispering video and the smiling face video, and the voice. From the analysis result and the video analysis result, it is possible to calculate the acceptability of the other party with respect to the topic, and the other party can feed back the acceptability (acceptance degree) regarding the topic to the user in real time.

  According to the conversation analysis method of the thirteenth aspect, by summing up and analyzing the difference in the tone and / or tempo of the conversation between the conversation partner and the user, it is possible to feed back the skill of the conversation to the topic in real time to the user. it can.

According to the conversation analysis method of claim 14, the conversation log of a plurality of users and each conversation partner is collected and managed by the conversation evaluation server device, and the conversation log of the plurality of users is analyzed, thereby realizing a realistic human relationship. It is possible to extract influencers related to certain topics.
According to the conversation analysis method of claim 15, by analyzing the conversation log, the degree of connection between users as a human relation between users and the magnitude relation of the acceptability between users are obtained. It can be stored as a human relationship database, can be displayed as a human relationship map from the information in this human relationship database, and can also be displayed as a human relationship map extracted on the topic to be evaluated.

According to the conversation analysis program of the sixteenth aspect , each procedure of the conversation analysis method can be performed on a computer .

1 is a configuration explanatory diagram of a conversation analysis system. FIG. It is a model figure which shows the mounting state to the user of a conversation analyzer (wearable terminal). It is a block diagram which shows an example of a structure of a conversation analyzer (wearable terminal). It is a block diagram which shows an example of a structure of a conversation evaluation server apparatus. It is a flowchart which shows the procedure of the conversation analysis method. (A) (b) is a front face facial expression explanatory drawing in the case of detecting a whisper of a conversation partner from a camera image. (A) and (b) are front face facial expression explanatory drawings in the case of detecting the smile of the conversation partner from the camera video. (A) and (b) are side face facial expression explanatory drawings in the case of detecting the user's own whisper from the camera video. It is a record table of the conversation log database which the memory | storage part of the conversation evaluation server apparatus of FIG. 4 memorize | stores. It is a block diagram which shows the other example of a structure of a conversation analyzer (wearable terminal). It is a block diagram which shows the other example of a structure of a conversation evaluation server apparatus. It is a record table of the conversation log database which the memory | storage part of the conversation evaluation server apparatus of FIG. 11 memorize | stores. (A) (b) is a model figure for demonstrating the communicable range of the users 1 and 2 during the operation | movement of a conversation analysis system, and the camera image | video which the user 2 acquires. It is a model figure of the human relationship map which shows the human relationship between users.

A conversation analysis system according to an embodiment of the present invention will be described with reference to FIGS.
As shown in FIG. 1, the conversation analysis system includes a conversation analysis apparatus 1 composed of a wearable terminal and a conversation evaluation server apparatus 3 connected to the wearable terminal via a network 2 such as the Internet. ing.
The conversation analysis apparatus (wearable terminal) 1 and the conversation evaluation server 3 are mainly composed of a ROM that stores basic devices, a hard disk drive (HDD) that stores various programs and data, a CPU that executes the programs, and the like. A conversation analysis system is constructed by including a computer and storing a conversation analysis program in the HDD.

  As shown in FIG. 2, the appearance of the conversation analysis device (wearable terminal) 1 is configured so that it can be worn on the front of the user's face by laying the temple portions 5 on both ears of the user. The collecting microphone 11a and the camera 12 to be collected are mounted near the height of the user's eyes, and the user microphone 11b is mounted at a position near the user's mouth. Further, the wearable terminal 1 is provided with a display unit 40 on which a virtual image is formed in front of the user's vision.

  As shown in FIG. 3, the conversation analysis device (wearable terminal) 1 includes a sensor unit 10 that acquires various types of information such as voice information and video information when the user has a conversation with the other party, and a control that analyzes the various types of information. Conversation analyzer (wearable terminal) comprising: unit 20, storage unit 30 that stores analysis results, display unit 40 that displays analysis results, and first communication unit 50 that transmits the analysis results to the outside as information 1 is provided.

The sensor unit 10 includes a microphone 11 that acquires surrounding voice information, a camera 12 that acquires facial expressions and head movements of conversation partners as video information, and a gyro / acceleration sensor that acquires user (user) head movements. 13 and a mounting sensor 14.
The microphone 11 includes two microphones 11a and 11b, and each uses a directional microphone.

The counterpart microphone 11 a is composed of a receiving microphone having directivity in the imaging direction of the camera 12.
The user microphone 11b is composed of a double microphone (two microphones for transmitting / environmental sound) having two microphones, one for transmitting to the mouth side of the user and the other for environmental microphones on the opposite side. A signal with a higher S / N ratio is obtained than when a spoken voice signal is acquired by a single microphone.

  As a microphone configuration that replaces the double microphone of the user microphone 11b, as a microphone that directly contacts the user's skin, a vocal cord (throat) microphone, a cheek, a jawbone, or a mastoid behind the ear, etc. The user may recognize that the user is speaking using a bone conduction microphone or the like that contacts the skin.

  The camera 12 can be imaged in front of the user's face by being mounted at an appropriate position of the wearable terminal 1 such as the front end of the temple unit 5.

The gyro / acceleration sensor 13 is for detecting the inclination of the user's head in order to detect the user's own whispering.
The gyro / acceleration sensor 13 may use a triaxial (X, Y, Z direction) acceleration sensor, but detects a triaxial gyro in order to accurately detect the user's “whit”. It is preferable to use a six-axis sensor that detects acceleration in three axes (X, Y, and Z directions).

  The wearing sensor 14 is a sensor that detects that the user has arrived at the head of the wearable terminal 1, and has a mechanism in which the sensor is switched on by, for example, expanding the temple unit 5.

  The control unit 20 includes a voice evaluation unit 21 that detects an utterance from the microphone 11, a video evaluation unit 22 that recognizes the facial expression (smile or whisper) of the conversation partner from the video information of the camera 12, and the gyro / acceleration sensor 13. A whisker recognizing unit 23 that recognizes a user's whistling, a wear recognizing unit 24 that detects wearing by a user, and a conversation analyzing unit 25 that comprehensively evaluates detection results from each unit and performs conversation analysis. ing.

The voice evaluation unit 21 has a speech detection function, a speaker recognition function, a voice recognition function, a morphological analysis function, and a tone / tempo measurement function.
The utterance detection function detects the start and end of a conversation from voice information detected from the microphone 11 (the partner microphone 11a and the user microphone 11b).
The voice recognition function converts the detected conversation into text.
The morphological analysis function morphizes the text and divides it into words.
The tone / tempo measurement function measures the consistency of conversation tones (size, height) and conversation tempo in a conversation.

  The speaker recognition function recognizes that the user of the wearable terminal 1 is speaking. That is, based on the spoken voice, it is determined whether the user has spoken by speaker recognition (speaker determination). Speaker recognition requires a speaker model for collating speakers. This speaker model is created in advance by making the user speak before using the wearable terminal, and is registered as data for speaker recognition in a user registration data unit 31 described later. Further, while using the wearable terminal 1, the speaker model may be updated as needed from the spoken voice.

  The video evaluation unit 22 has a smile recognition function that detects the other party's smile by detecting the other party's face information from the video information from the camera 12, and a whip recognition function that detects the other party's whisper.

  The whispering recognition unit 23 detects the whispering of the user based on information from the gyro / acceleration sensor 13.

  The wearing recognition unit 24 detects wearing of the wearable terminal 1 to the user based on a signal from the wearing sensor 14.

  The conversation analysis unit 25 is analysis data based on voice information of the microphone 11 from the voice evaluation unit 21, analysis data based on video information of the camera 12 from the video evaluation unit 22, and analysis data based on the gyro / acceleration sensor 13 from the whisper recognition unit 23. Are comprehensively evaluated and conversation analysis is performed, and the analysis result is stored in the conversation log 34 of the storage unit 30.

The storage unit 30 includes a user registration data unit 31 in which speaker recognition data is registered, an important word database 32, and a topic word database 33. The analysis result of the conversation analysis from the control unit 20 is a conversation log 34. Is recorded.
The speaker recognition data registered in the user registration data unit 31 enables the recognition of the speaker of the conversation.

  In the important word database 32, important words (important words for confirming whether the conversation is meaningful) are registered in advance in order to promote smooth conversation in order to evaluate the skill of the conversation. Important words registered in the important word database are words of thanks such as “Thank you”, “Thank you”, “Thank you”, “Terrible” “Good” “Cool” “Want” “Cute” “Great” “Cool” "," "OK", "OK", "Yes", "Yes", "Yes", and "Yes" are registered.

  In the topic word database 33, words (topic words) for analyzing a conversation of a certain topic and finding a user (influencer) having a strong influence on the topic are registered as a data list. For example, in the case of a communication carrier, “mobile phone”, “smart phone”, “network”, “radio wave”, “communication company name”, “service name”, etc. are registered as topic words. It is preferable to register topic words in various fields in advance.

The display unit 40 is attached to the wearable terminal 1 so that advice based on characters, icons, and the like corresponding to the analysis result by the control unit 20 is displayed as a virtual image in front of the user's field of view. Specifically, characters, icons, and the like such as “more slowly”, “larger”, and “smile” are displayed on the display unit 40 as advice.
In this example, the notification is visually notified by display, but the advice may be notified through hearing by sound generated by a sound reproduction unit such as a speaker.

  The first communication unit 50 performs data transmission and reception for recording in the conversation evaluation server device 3 to be described later.

Next, the configuration of the conversation evaluation server device 3 will be described.
The conversation evaluation server device 3 includes a first communication unit 60 that receives data of a conversation analysis result from the wearable terminal 1, a control unit 70 that performs conversation evaluation, and a recording unit 80 that records the received data. .
The control unit 70 includes a conversation evaluation unit 71 that evaluates information of a conversation analysis result in the wearable terminal 1 via the first communication unit 60.
The recording unit 80 records and manages the information of the analysis evaluation result in the conversation log database 81. Further, by using the conversation log database 81, various inquiries can be handled. Details of various inquiries will be described later.

Next, a procedure for performing conversation analysis and conversation evaluation with the conversation analysis apparatus (wearable terminal) 1 using the conversation analysis system described above will be described with reference to the flowchart of FIG.
First, the wear recognition unit 24 determines whether or not the wearable terminal 1 is worn on the user's head (step 101).
If the wearable terminal 1 is worn, the following process is performed. If it is determined that the wearable terminal 1 is not worn, the process is terminated (step 102).

When the wearable terminal 1 is worn, it is determined whether or not the conversation start is detected (step 103). Specifically, voices are collected from the user microphone (two microphones for transmitting microphone / microphone for environmental sound) 11b, and the noise level is evaluated every 10 seconds for the environment around the user (environmental noise calibration). ) Considering that the environmental sound (environmental noise) is generated from various directions, it is considered that the sound is diffusive. Therefore, it is determined that the difference in the noise level detected by the two microphones is small.
When the user speaks, a voice signal larger than the environmental sound microphone is input to the transmission microphone of the user microphone 11b. Measure the voice difference between the microphone for sending and the microphone for environmental sound, and if there is a difference of 6dB or more, start conversation analysis as if there was a signal (the user uttered).

In the case of “start conversation analysis”, the following processing is performed.
・ Memorize conversation start time and start conversation time measurement.
・ Start the conversation end detection timer to detect when the conversation ends.
・ Turn on the other party's microphone 11a and start acquiring the voice of the other party's voice.
・ Turn on camera 12 and start video acquisition.
・ Turn on the gyro / acceleration sensor 13 and start collecting data from the sensor.

Subsequently, the control unit 20 of the wearable terminal 1 performs the conversation analysis by acquiring the above-described information (step 104).
Conversation analysis will be described later.

The end of the conversation is determined (step 105).
For example, when the conversation end timer is set to 30 seconds and no user utterance is detected for 30 seconds, the conversation end is determined to be ended.
・ Confirm conversation time.
・ Turn off the partner microphone 11a.
・ Turn off the camera 12.
・ Turn off the gyro / acceleration sensor 13.

If it is determined that the conversation has ended, the conversation is evaluated and recorded (step 106).
Conversation evaluation and recording will be described later.

The conversation analysis (step 104) is performed according to the following procedure.
“Speech of speech, analysis of conversation tempo”
First, the voice collected by the user microphone 11b and the voice collected by the counterpart microphone 11a are converted into text (dictation) by voice recognition.
In addition, the speed of the voice-recognized text (speaking speed) is analyzed for each utterance of the user and the partner.
Here, “the conversation tempo matches” means that the speech is once phonemeized by the speech recognition function, and the ratio between the number of phonemes and the time is defined as the tempo. This tempo is calculated for each utterance, and the difference from the tempo of the previous utterance is taken, and the difference is continuously added until the end of one conversation. The number of additions is also measured.

"Analysis of conversation text, number of repetitions of the same word"
The text is morphologically analyzed and divided into words and stored until the conversation ends.
Here, “the number of repetitions of the same word” is defined as the noun in the text that is voice-recognized for each utterance and the noun in the text that is voice-recognized by the previous utterance. If there is the same word, “+1” is added as a repetition of the same word.

"Analysis of conversation tones"
Here, “the conversation tone matches” means that the voice volume of the other party and the voice volume of the other party are almost the same.
Measure the average value of the A-weighted sound pressure level for each utterance of the sound recorded by the microphone 11, take the difference from the sound pressure level of the previous partner's utterance, and add the difference until the end of one conversation . Also count the number of additions.

"Calculation of opponent's face detection time from camera video"
An image is acquired from the camera 12, and a position where a face is present in the image is detected by pattern recognition.
The feature points for pattern recognition are the positions of both eyes relative to the head contour and the nose and mouth between them.
Accumulate the time during which the face was detected.

"Calculating the number of whispering of the other party from the camera video"
The position of the face detected from the image of the camera 12 is further analyzed, and it is determined whether or not it has been whispered.
For each video slice, the ratio of the vertical and horizontal dimensions of the rectangle surrounding the eyes and mouth is calculated and estimated from the time series change of the ratio.
That is, as shown in FIG. 6B, from the state where the square (the area surrounded by the dotted line) surrounding the eyes and mouth is close to a square (longitudinal: horizontal ratio is large) as shown in FIG. The square surrounding the eyes and mouth (the area surrounded by the dotted line) gradually changes to a crushed rectangle (vertical: horizontal ratio is small), and then the original size (FIG. 6 (a)). The change that returns to is judged as one “whisper”. From the video of the camera 12, the time and number of times determined as “whispering” are recorded.

"Calculating the smile time of the other party from the camera video"
The position of the face detected from the image of the camera 12 is further analyzed to determine whether it is a smile.
As shown in FIG. 7, the smile is a normal expression state (FIG. 7 (a)) and a smile state (FIG. 7 (b)). ) Is acquired and compared with a predetermined threshold value to estimate a smile. Accumulate the time that was determined to be a smile.

"Calculation of whispering from gyroscope and acceleration sensor"
The user's own “whispering” is determined based on data from the gyro / acceleration sensor 13.
That is, as shown in FIG. 8, the one-time whispering is from a state where the user is facing the front (FIG. 8A) to a state where the neck is tilted forward (FIG. 8B), and again. The operation returns to the state of facing the front (FIG. 8A).
For example, when the 6-axis gyro / acceleration sensor 13 is arranged in the temple portion (vine) 5 of the glasses, each axis is defined as follows.
X axis: An axis that penetrates the user in the horizontal direction. Rotation around the X axis is called “pitch”.
Y axis: An axis that penetrates the user in the longitudinal direction. The rotation around the Y axis is called “roll”.
Z axis: An axis that penetrates the user in the vertical direction. The rotation around the Z axis is called “Yaw”.
First, the gravity vector can be recognized downward from the triaxial acceleration sensor by gravity. That is, there is a constant acceleration in the downward direction of the Z axis.
When the user tilts the whispered neck forward, the state shown in FIG. The pitch angle of the gyro sensor falls below the estimated position. In addition, the acceleration downward in the Z axis temporarily decreases, and a gravity component appears in the forward direction of the Y axis.
When the user pulls the neck down, the state returns to the state of FIG.
The pitch angle of the gyro sensor returns to the horizontal position. Further, the acceleration downward in the Z axis temporarily increases, and the gravity component decreases in the forward direction of the Y axis.
This series of changes is pattern-recognized, and the number of times of singing is one.

  Based on the analysis result data (each data of the sensor unit 10) of the control unit 20 of the wearable terminal (conversation analysis device) 1, the conversation analysis unit 25 calculates an index for evaluating the acceptance of the conversation and the skill of the conversation. It is recorded in the conversation log 34 and updated while the conversation continues. Details of the index calculation will be described later.

Then, the calculation method of each parameter | index which evaluates the acceptance degree of conversation and the skill of a conversation is demonstrated.
In order to obtain these indices, the following behaviors per unit time are measured.
For example, the index is recalculated and updated every 15 seconds without waiting for the conversation to end.
・ Number of beatings (number of whispering of opponent)
・ Number of whispers (number of whispers)
・ Number of important words (utterances by yourself and the other party)
・ Laughter time, opposite time, conversation tone consistency, conversation tempo consistency, number of repetitions

Number of whirlings: Nnr and Nnd
Whispering is an action that recognizes and affirms the other party. A person who is often asked about the unit time of the conversation has an easy-to-understand way of speaking to the other party and is considered accepted by the other party.
Nnr = (number of whisperings of the other party) / (conversation time)
Nnd = (Number of whispers) / (Conversation time)

Here, the number of hits (the number of times the other party whispered) is the number of times that the other party's face was reflected on the camera 12 and was estimated to be “whited” by the whisper detection function.
Here, the number of times of whispering (the number of times of whispering) is the number of times that the whispering is estimated by the whispering detection function from the gyro acceleration sensor 13.
When Nnr is lower than a predetermined threshold during the conversation, the display unit 40 displays that the other party's whisper is insufficient.
If Nnd is lower than a predetermined threshold value during the conversation, the display unit 40 displays that it is not enough.

Laughed time: Tsr
Laughter is an action to facilitate conversation.
A person who is often laughed per conversation time unit shows that the conversation is proceeding happily and smoothly.
Tsr = (time when the other party laughed) / (time when the other party's face was detected)

Here, “the time when the other party laughed” is the time when the other party's face was reflected on the camera and the smile was detected by the smile detection function.
If Tsr is lower than a predetermined threshold during the conversation, the display unit displays that the other party is not smiling.

Number of important words: Ntr and Ntd
The important word is recorded as the number of matches with the word in the important word database. When saying important after a conversation unit time, it shows that the content of the conversation is useful and positive for the person who speaks the important word.
Ntr = (number of utterances of the partner's important words) / (conversation time)
Ntd = (Number of utterances of own important words) / (Conversation time)

Here, the “number of utterances of important words” is a number in which the speech is converted into text by the speech recognition function and the important words are included in the text.
When Ntr is lower than a predetermined threshold during the conversation, the display unit 40 displays that the conversation content is to be changed.

Opposite time: Ted
When talking, you can gain confidence by looking at the other person's eyes.
A person who is looking at the other person per unit time of conversation shows that he / she is concentrating or talking with interest.
Ted = (Time spent watching the other person's face) / (Conversation time)

Here, “the time when the other party's face was being seen” is the time when the other party's face was reflected on the camera and the face detection function was able to detect the eyes and mouth.
When Ted is lower than a predetermined threshold during the conversation, the display unit 40 displays that the other party is seen more.

Time when conversation tone matches: Rst
The tone of conversation is the tone of the voice. Conversations with different voice tensions are psychologically insensitive to each other.
Rst = (sum of conversation tone differences) / (number of additions)

Divide the sum by the number of additions to get the average difference.
The larger Rst, the greater the difference in voice volume, indicating that the conversation is not skillful.
When Rst is larger than a predetermined threshold during the conversation, the display unit 40 displays that the other party's voice is imitated.

Time when conversation tempo matches: Rss
The conversation tempo is the speed of speaking. Conversations with different utterance speeds are not psychologically synchronized.

Rss = (sum of differences in conversation tempo) / (number of additions)
The larger Rss, the more tempo is shifted, indicating that the conversation is not skillful.
When Rss is larger than a predetermined threshold during the conversation, the display unit 40 displays that the other party's voice is imitated.

Repeated occurrences of the same word: Nrd
When a word that is a conversation is repeated, if the same word is uttered from both, it is felt that the word has been understood.
Nrd = (Number of utterances of the same word) / (Conversation time)
The more the same word is spoken, the more the listener listens and feels like they are once.
In other words, the larger the Nrd, the better the conversation.
When Nrd is smaller than a predetermined threshold during the conversation, the display unit 40 is displayed to repeat.

These values are calculated by the following formula, and the other party's acceptance (Rc) and skill of the conversation (Cc) are quantified.
A higher Rc indicates that the conversation is accepted, and a higher Cc indicates that the conversation is more skillful.
Conversation acceptability of conversation
Rc = a1 × Nnr + a2 × Ntr + a3 × Tsr
Skill of conversation
Cc = a1 × Nnr + a2 × Ntr + a3 × Tsr + a4 × Nnd + a5 × Ntd + a6 × Ted-a7 × Rst-a8 × Rss + a9 × Nrd
A1 to a9 of each term are predetermined weighting constants (values of 0 or more) of each term.

The voice evaluation unit 21 performs morphological analysis on the text of the voice recognition result, and stores all words classified as nouns in the conversation log 34 as documents.
The above processing is continued until it is determined that the conversation has ended.

The conversation evaluation / recording (step 106) is performed in the following procedure.
When it is determined that the conversation has ended, the conversation analysis unit 25 of the control unit 20 of the wearable terminal (conversation analysis device) 1 updates the conversation acceptance and the conversation skill index with the determined conversation time, Using all the conversation documents in the conversation log 34, the characteristic words of the conversation are obtained by the following method.

  The TF (term frequency) value of all the words in the document may be calculated, and the word having the largest TF value may be used as the feature word of the conversation. The TF value represents the frequency of occurrence of the word in the sentence.

Also, instead of the most frequently occurring word, calculate the TF-IDF value for all the words in the user's conversation document and all the conversation documents for the period you want to investigate, and select the word with the highest TF-IDF value. Record conversations as feature words.
The appearance frequency may be a TF-IDF (Term Frequency-Inverse Document Frequency: word appearance frequency-inverse appearance frequency) value. TF-IDF is an index for identifying how characteristic a word that appears in a document is. TF (term frequency) represents the frequency of occurrence of the word in the document.
IDF (inverse document frequency) is represented by the natural logarithm of the number of documents including the word in all documents, and the higher this value, the more frequently the word is a word that does not appear frequently in all documents.
A word having a higher value of TF value × IDF value is more important.

When a topic word registered in advance in the topic word database 33 is included, the topic word is recorded in the conversation log 34.
The above is the procedure for focusing on only one conversation and evaluating the acceptance of the conversation and the skill of the conversation.
Since the generation of the conversation log for one conversation is completed, the conversation analysis unit 25 of the wearable terminal 1 transfers the conversation log to the voice evaluation server 3 via the first communication unit 50.
When receiving the conversation log from the first communication unit 60, the control unit 70 of the conversation evaluation server 3 records the conversation log in the conversation log database 81 of the storage unit 80.

FIG. 9 shows record values of the conversation log database 81 that records the analysis result of one conversation.
The conversation log is managed in one conversation for each user ID, and each index described above is stored together with the conversation start time and conversation time.
In addition, in order to calculate the TF value and the like, a list of all nouns in the conversation is stored, and a list of feature words and a list of topic words that existed in the conversation are stored.

  The conversation evaluation unit 71 of the conversation evaluation server apparatus 3 further analyzes this conversation log, analyzes all the conversations of the users, and compares (averages and standard deviations) the Rc and Cc of all the users. A user who has a high conversation acceptance index Rc and a conversation skill index Cc can accept that the conversation is accepted and that the conversation is skillful.

Further, by analyzing each topic word of the conversation log, it is possible to evaluate what conversation topic has been accepted by the other party.
For example, referring to a conversation record including a topic word, the skill level Cc of the conversation is evaluated.
All users whose topic words are included in the conversation are searched, and the statistical value (average value or standard deviation) of Cc is calculated by the total number of conversations in which the topic words appear for each user. If conversation skill level Cc is relatively high, it can be said that the user has an influence on the topic.

According to the conversation analysis system described above, the skill of the conversation performed by the user can be evaluated, so that it can be used in the following cases, for example.
Example of use 1: Used to evaluate cleverness of a store clerk during customer service. The store clerk and customer conversations are accepted by the customer, and the store clerk can check in real time by using a wearable device whether the store clerk's conversation is clever and skillful. It can be confirmed by inquiring to the conversation evaluation server whether it was skillful.
Usage example 2: Even when wearable terminals are not yet widespread among general users, it is possible to estimate the influence of a wearable terminal on another topic word for a user using the wearable terminal.

Next, another example of the embodiment of the conversation analysis system according to the present invention will be described with reference to FIGS.
In this conversation analysis system, influencers (users who have a great influence on others) are extracted by using the conversation analysis apparatus (wearable terminal) of FIG. 10 and the conversation evaluation server apparatus 3 of FIG.
That is, in this conversation analysis system, it is possible to automatically identify realistic human relationships by specifying a conversation partner. In addition to the user himself / herself, the conversation partner is also identified in order to know the human relationship with the conversation partner.

  In this conversation analysis system, it is assumed that the conversation analysis apparatus (wearable terminal) 1 is used not only by the user itself but also by a conversation partner. By using the wearable terminal 1 for both conversations, a unique user identifier (user ID) is given to each user in the system. The wearable terminal 1 knows the user ID of the other party during the conversation, and specifically knows from which user the degree of “swing” and “smile” on the other side in the conversation analysis system of FIGS. Can do.

In the conversation analysis system using the conversation analysis apparatus (wearable terminal) 1 in FIG. 10 and the conversation evaluation server apparatus 3 in FIG. 11, parts having the same configurations as those in FIG. 3 and FIG. In the following, a description will be given focusing on a configuration different from the conversation analysis system of FIGS.
The wearable terminal 1 of this conversation analysis system includes a second communication unit 51 for exchanging user IDs between wearable terminals.
Specifically, IrSS (registered trademark), which is an IrDA standard for IR (infrared) communication, may be used. For example, by controlling the directivity of infrared light, for example, 30 ° left and right (total 60 °) directivity from the front center. Send and receive. Also, ultrasonic communication may be used instead of infrared communication, and radio wave communication having directivity of GHz band or higher may be used.

Further, in the record value (FIG. 12) of the conversation log database corresponding to FIG. 9, a partner user ID for identifying the partner user is added.
The user ID may be recorded and used in advance on the wearable terminal 1, or an arbitrary code that is unique in the system at the time of user registration may be registered.
By using the second communication unit 51, as shown in FIG. 13, when facing each other, direct communication within the communicable range with the wearable terminal 1 on the other side is possible, and ID exchange is performed to automatically communicate with the other party. Can be identified.

The conversation evaluation server device 3 of this conversation analysis system includes a human relationship management function 72 for the control unit 10 to analyze a conversation log from a user ID and manage a user's human relationship. The storage unit 30 also includes a human relationship database 82 that stores the user's human relationships in association with each other.
In the human relationship management function, each time a conversation log is recorded, an ID for calculating the skill level Cc of the conversation measured by each wearable terminal by specifying the conversation partner, its own ID and the other party's user ID If the conversation logs match each other from the conversation start time, their own smile time (Tsd: Fig. 12) can also be known from the conversation log of the measurement result of the other wearable terminal, and the guideline data Can be complemented as

Further, a real human relationship map between users as shown in FIG. 14 is automatically constructed.
In the human relationship map of FIG. 14, the degree of connection between the ID of the conversation log and the ID of the other party is indicated by the total conversation time between users in the conversation log for an arbitrary period to be analyzed, and the link direction. Is shown by the difference of total Rc. In addition, it means that information is flowing toward the person with the higher acceptance level (Rc).
For example, if the user of ID: W1 and ID: W2 in FIG. 14 is described as an example, since the link between the users is thicker than others, there is a strong connection, and the direction of the arrow is larger in the direction of the user W2, This shows that the user W2 has more accepted the user W1 (higher acceptance). Further, it can be estimated that the information flow is in the direction of the user W2.

In addition, in the human relationship map, it is possible to provide a bird's-eye view by filtering for each topic.
Thus, by mapping the human relationships of all users as shown in FIG. 14 and filtering on a certain topic, it is possible to find a central user on that topic. That is, an influencer related to a specific topic in a real human relationship can be found.

  According to the conversation analysis device described above, it is possible to analyze a real conversation between people and extract influencers in a certain field, so it is related to topic words by direct mail etc. for users who are regarded as influencers By sending the latest information and advertisements to be transmitted, information transmission can be expected through real human relations around the user.

  In addition, by using a conversation analysis device at the store customer service, it is possible to measure how much the store clerk recognizes and convinces with the customer about the realistic communication between the store clerk and the customer. Can determine the customer service.

  DESCRIPTION OF SYMBOLS 1 ... Conversation analysis apparatus (wearable terminal), 2 ... Internet, 3 ... Conversation evaluation server apparatus, 5 ... Temple part, 10 ... Sensor part, 11 ... Microphone, 11a ... Microphone for other party, 11b ... Microphone for user, 12 ... Camera 13 ... Gyro / acceleration sensor, 14 ... Mounting sensor, 20 ... Control unit, 21 ... Speech evaluation unit, 22 ... Video evaluation unit, 23 ... Glow recognition unit, 24 ... Mounting recognition unit, 25 ... Conversation analysis unit, 30 ... Storage unit 31 User registration data unit 32 Key word database 33 Topic word database 34 Conversation log 40 Display unit 50 First communication unit 51 Second communication unit 60 ... 1st communication part, 70 ... Control part, 71 ... Conversation evaluation part, 72 ... Human relationship management part, 80 ... Memory | storage part, 81 ... Conversation log database 82 ... human relations database.

Claims (16)

A wearable device that is worn on the user's head to detect human conversations,
An important word database storing a plurality of important words for extracting important words uttered in the conversation;
A microphone for collecting voice information of the conversation;
A voice evaluation unit for recognizing important words from the voice information;
A conversation analysis unit that analyzes a partner's acceptability for a conversation with a predetermined weighting from the recognition result of the voice evaluation unit;
A notification unit that feeds back the acceptance to the user;
Conversation analyzer characterized by comprising. A camera that collects the facial expression of the conversation partner and the movement of the head as a video,
A video evaluation unit for recognizing the smile time and number of whispering of the conversation partner per unit conversation time from the video,
The conversation analysis unit performs a predetermined weighting from the recognition results of the voice evaluation unit and the video evaluation unit, and analyzes a partner's acceptability for the conversation.
The conversation analysis device according to claim 1 . The notification unit, conversation analysis apparatus according to claim 1 or claim 2 which is constituted by the audio reproduction unit that can be recognized by hearing the acceptance. The notification unit, conversation analysis apparatus according to claim 1 or claim 2 which is composed of a display device capable of recognizing the acceptance visually. The voice evaluation unit has a function of analyzing the tone and / or tempo of conversation,
The conversation analysis unit has a function of analyzing the skill of the conversation by summing up the difference in tone and / or tempo of the conversation between the other party and the user and giving a certain weight.
The conversation analysis apparatus according to claim 1 , wherein the notification unit has a function of feeding back the skill of the analyzed conversation to a user. An acceleration sensor for recognizing the movement of the user's head;
A whisker recognition unit that analyzes the number of whistling per unit conversation time of the user from the acceleration sensor,
The conversation analysis apparatus according to claim 5 , wherein the conversation analysis unit analyzes the skill of the conversation by applying a predetermined weight to the recognition by the whisper recognition unit. The conversation analysis unit, the user and the mating of both claim 5 or analyzing the skillfulness of the conversation with the ability to analyze by a constant weighting aggregates the repetition of the words uttered in the conversation Item 7. The conversation analysis device according to item 6 . The analysis by the conversation analysis unit analyzes a feature word using an index (TF-IDF) for identifying how characteristic the word that appears in the conversation is from the speech recognition result of one conversation. The conversation analysis apparatus according to any one of claims 1 to 7 , further comprising a storage unit that records each conversation together with the acceptance level and / or skill of the conversation. A conversation analysis apparatus according to any one of claims 1 to 8 , and a conversation evaluation server apparatus that collects and manages conversation logs of the conversation analysis apparatus,
The conversation evaluation server device
A control unit that analyzes a conversation log of all users of the conversation analyzer and evaluates a conversation;
A storage unit for storing the conversation log in a conversation log database;
A conversation analysis system characterized by comprising:   The conversation analysis device comprises a topic word database for storing a plurality of topic words for influencer extraction for a specific topic uttered during the conversation,
  The voice evaluation unit further recognizes a topic word from the voice information,
  The storage unit records the recognized topic word for each conversation in a conversation log,
  The control unit of the conversation evaluation server device extracts a conversation log including a topic word to be evaluated when analyzing a conversation log of a plurality of users and performing a conversation evaluation on the topic word.
  Extract users who have an influence on the topic word as influencers when the acceptance or skill of the conversation is higher than the average value from the multiple users of the extracted conversation logs
The conversation analysis system according to claim 9.   The conversation analysis device comprises a communication unit for exchanging an identifier of a user of the user at the start of conversation and an identifier of a user who is a conversation partner,
  The control unit of the conversation evaluation server analyzes the conversation log, further records the identifier of the user who is a conversation partner and the conversation time, and the total conversation time for the conversation log between the identifiers assigned to the user. It is equipped with a human relationship management function that tabulates the degree of connection between users and the degree of acceptance, and manages the magnitude relationship between users as the user's human relationship,
  The conversation analysis system according to claim 10, wherein the storage unit of the conversation evaluation server includes a human relationship database that stores the human relationships of the users in association with each other. A method for analyzing and evaluating a conversation by information processing using computer software for a conversation analysis system including a conversation analysis device and a conversation evaluation server,
A procedure for recognizing preset topic words and important words from voice information generated during conversation between a user and a conversation partner;
Acquiring the facial expression of the conversation partner and the movement of the head as a video, and recognizing the smile time and number of whispering of the conversation partner per unit conversation time from the video;
A procedure for analyzing the degree of acceptance of the conversation partner with respect to the topic of conversation from each recognition result;
Feeding back the acceptance to the user;
Conversation analysis method characterized by including. A method for analyzing and evaluating a conversation by information processing using computer software for a conversation analysis system including a conversation analysis device and a conversation evaluation server,
A procedure for recognizing preset topic words and important words from voice information generated during conversation between a user and a conversation partner;
Acquiring the facial expression of the conversation partner and the movement of the head as a video, and recognizing the smile time and number of whispering of the conversation partner per unit conversation time from the video;
A procedure for counting the difference in tone and / or tempo of conversation between the conversation partner and the user;
A procedure for analyzing the skill of the conversation with respect to the topic of conversation by performing a predetermined weighting from each recognition result and the total result,
A procedure for feeding back the skill of the conversation to the user;
Conversation analysis method characterized by including. Conversation of the conversation partner with respect to the topic of the analyzed conversation of the identifier for identifying the user given in advance and the recognized topic word and important word for the conversation between the plurality of users and each conversation partner Alternatively, the skill of conversation is stored as a conversation log, and the plurality of conversation logs are aggregated and managed by the conversation evaluation server device, and
  When analyzing the conversation log of the plurality of users and performing conversation evaluation on the topic word, a procedure for extracting a conversation log including the topic word to be evaluated;
  A procedure for extracting a user who has an influence on the topic word as an influencer when the conversation acceptance or skill of the conversation is higher than an average value from a plurality of users of the extracted conversation log;
The conversation analysis method according to claim 12 or 13, characterized by comprising:   A procedure for exchanging the identifier of the user and the identifier of the user who is the conversation partner at the start of the conversation;
  In the conversation log of a plurality of users that are aggregated and managed by the conversation evaluation server, a procedure for further recording an identifier and a conversation time of a user who is a conversation partner;
  For the conversation log between the identifiers given to the user, a procedure for storing the degree of connection between the users who totaled the total conversation time, and the degree of acceptance and the magnitude relationship between the users as a human relationship database;
  A procedure for extracting information of the human relation database by a topic to be evaluated;
  A procedure for displaying the extracted human relationship as a human relationship map;
The conversation analysis method according to claim 14, further comprising: The conversation analysis program characterized by constructing each procedure of the conversation analysis method of any one of Claims 12 thru | or 15 on a computer.