JP6715410B2 - Evaluation method, evaluation device, evaluation program, and evaluation system - Google Patents

Description

The present invention relates to a communication action of a person, accurately records the content thereof, analyzes the content using an artificial intelligence technique, and makes it possible to evaluate a person's communication skill and communication content. Equipment, evaluation program, and evaluation system.

As related techniques and methods, there are related technical information in Patent Documents 1 to 6, for example.

Patent Document 1 proposes a method of measuring nonverbal behavior, outputting information on the psychological state of a subject, and displaying the result of false detection regarding the behavior of the suspect.

Patent Document 2 proposes a method of utilizing a biometric sensor to measure mainly walking, running, sleep, and the like, and displaying the progress and result of the action goal.

In Patent Document 3, a conversation is recorded and analyzed using a wearable terminal, the acceptability of the conversation partner of the user and the skill of the conversation of the speaker are analyzed based on a smile determination from a camera image, and a certain topic to another person is analyzed. A method has been proposed that makes it possible to extract users (influencers) who have a large influence.

In Patent Document 4, when the context meaning of a user's utterance is understood, an utterance meaning analysis program that can improve the understanding accuracy of the context meaning according to the emotion classification of the user is used to correct the misunderstanding of the positive meaning/negative meaning. Methods have been proposed to prevent and prevent the progression of dialogue scenarios for computer dialogue systems to the user.

Patent Document 5 proposes a method of evaluating the communication skill by analyzing the speech contents of the dialogue from the clustering means based on the positive/negative evaluation axis.

Patent Document 6 proposes a dialogue system or a method for evaluating a listening dialogue in a person.

Japanese Patent Publication No. 2004-527313 JP, 2017-188089, A JP, 2016-103081, A JP, 2017-156854, A JP, 2016-157388, A JP, 2011-44060, A

However, the technique described in Patent Document 1 is intended to analyze nonverbal behavior, mainly for detecting falsehood, and does not refer to measurement or evaluation of communication skills including conversation and facial expressions.

The technique described in Patent Document 2 is limited to behavior analysis for walking, running, and sleep, and does not refer to evaluation of communication skills including conversation and facial expressions.

The technique described in Patent Document 3 aims at extracting a user (influencer) who has a large influence on another person about a certain topic, conversation analysis using limited words registered in a predetermined data list, and predetermined data. It provides a mechanism to limit smiles based on the definition and a data set of a predetermined index, but analyzes and evaluates communication based on other emotional elements such as anger and sadness, and analyzes and evaluates communication that includes words not in the data list. Not something to do.

The technique described in Patent Document 4 provides a mechanism for measuring nonverbal behavior or verbal behavior regarding an arbitrary individual or outputting from observation for the purpose of providing a dialogue program having a consistent dialogue. It does not refer to the method of analysis and evaluation of communication between multiple speakers, evaluation of communication technology based on it, or provision of results for improving the communication method of speakers.

Although the technology described in Patent Document 5 is aimed at improving communication skills, it is limited to a method of using only text information from uttered linguistic information and outputting communication skill evaluation based on a positive or negative evaluation axis. However, it does not have a method of interactive communication analysis including facial expression elements and listening parts.

In Patent Document 6, the purpose is to evaluate a dialogue system or a listening dialogue in a person, not to evaluate the whole communication. Further, in order to perform dialogue evaluation only based on dialogue such as chat, a facial expression element is used. No mention of evaluation.

By the way, various services are known for skill improvement training and education methods related to communication. The methods used in such services are generally classroom lectures by teachers, role-playing, individual interviews with coaches and coaching.

In such a method, it is often the case that the good or bad is judged by questionnaires, hearings and subjective judgments of the instructor and the students themselves regarding the current grasp of communication skills and improvement results.

Other than such subjective judgment, there is no means for grasping the current state of skill and determining the result of skill improvement, so that essential skill improvement may or may not be achieved.

Therefore, there is a need for a mechanism that can grasp the current state of communication skills and visualize the improvement results based on more objective data, and that leads to efficient communication skill improvement.

In business, there are similar problems in fields including human communication, such as customer service, medical care, education, entertainment, and service.

Therefore, in the present invention, regarding the communication, through the acquisition of objective data, the analysis method and the analysis result of the data, by a more effective means, the visualization of the current situation of the skill, and the visualization of the result of the skill improvement, An object of the present invention is to provide an evaluation method, an evaluation device, an evaluation program, and an evaluation system that can realize more reliable skill improvement. In addition, although the conventional method often uses a wearable terminal or a dedicated device to increase the system cost for implementation, the present invention does not need to purchase an expensive device or the like and can be implemented at a relatively low cost.

Briefly, various aspects of the subject matter described in the present specification, in order to solve the above-mentioned problems, capture video data and audio data of human communication, and convert them into numerical data using artificial intelligence technology, An object of the present invention is to provide an evaluation method, an evaluation device, an evaluation program, and an evaluation system, which analyze based on numerical data and evaluate the communication content and communication skill of a person.

(1) An evaluation method that supports evaluation of communication between users, comprising: a data capturing step of capturing communication video data and audio data; and the video data captured by the data capturing step, An image analysis step of converting into first numerical data using an artificial intelligence technology corresponding to the data, and the voice data captured by the data capturing step into text data using the artificial intelligence technology corresponding to the voice data. And a voice analysis step of converting the text data into second numerical data by using an artificial intelligence technique corresponding to the text data, and combining the first numerical data, the text data and the second numerical data. And a consolidation step of consolidating and consolidating in a time series to obtain consolidation data; a difference data acquiring step of acquiring difference data corresponding to the difference in reaction of the other user at different times from the consolidation integrated data; An evaluation step of calculating an evaluation result including at least one of the evaluation of the content of the communication based on the difference data and the evaluation of the communication skill of the user with reference to a predetermined evaluation axis according to the communication purpose, and the evaluation. An output step of outputting the evaluation result obtained in step.

(2) The integrating step includes a step of defining a communication event as a minimum unit for evaluating a communication action performed by a user at an arbitrary time point based on the consolidated integrated data, and a step of defining a communication event at a different time point for each user. The method according to (1), further comprising a step of defining, as a communication whole, data in which the communication events included from the start time to the end time of the communication are collected and integrated in a time series based on the communication event. ..

(3) The evaluation method according to (2), wherein the difference data acquisition step includes a step of acquiring the difference data of the partner user for each communication event.

(4) The evaluation step includes a step of calculating the evaluation result for each user based on the consolidation integrated data and the difference data, and the content of the communication whole body based on the consolidation integrated data and the difference data. The evaluation method according to (2) above, which includes a step of calculating the evaluation result regarding.

(5) The evaluation axis includes a communication evaluation standard defined from the purpose of communication given in the business, and the evaluation step includes the communication event and the communication total, which are given with reference to the evaluation axis. Assigning an evaluation value to at least one of them, assigning an evaluation value relating to a predetermined analysis item according to the communication purpose in the business, and assigning an evaluation value to the business result obtained as a result of the communication, The evaluation method according to (2) above, including a step of calculating at least one of the above.

(6) An evaluation device that supports evaluation of communication between users, comprising a data capturing function for capturing video data and audio data for communication, and the video data captured by the data capturing function for the video. An image analysis function for converting into first numerical data using an artificial intelligence technology corresponding to the data, and the voice data captured by the data capturing function is converted into text data using the artificial intelligence technology corresponding to the voice data. And a voice analysis function for converting the text data into second numerical data by using an artificial intelligence technology corresponding to the text data, and combining the first numerical data, the text data and the second numerical data. And a difference data acquisition function for acquiring difference data corresponding to the difference in the reaction of the other user at different times from the consolidation integrated data, An evaluation function and an evaluation function for calculating an evaluation result including at least one of the evaluation of the content of the communication based on the difference data and the evaluation of the communication skill of the user with reference to a predetermined evaluation axis according to the communication purpose. An evaluation device having an output function of outputting the evaluation result obtained in 1.

(7) The integration function includes a function of defining a communication event as a minimum unit for evaluating a communication action performed by a user at an arbitrary time point based on the organized integration data, and a function of defining a communication event at a different time point for each user. The evaluation device according to (6), having a function of defining, as a communication whole, data obtained by time-sequentially integrating the communication events included from the start time to the end time of the communication based on the communication event. ..

(8) The evaluation device according to (7), wherein the difference data acquisition function has a function of acquiring the difference data of the opponent user for each communication event.

(9) The evaluation function has a function of calculating the evaluation result for each user based on the consolidation integrated data and the difference data, and the content of the communication whole body based on the consolidation integrated data and the difference data. The evaluation device according to (7) above, which has a function of calculating the evaluation result regarding.

(10) The evaluation axis includes a communication evaluation standard that is defined from the purpose of communication given in business, and the evaluation function includes the communication event and the communication total that are given with reference to the evaluation axis. Assigning an evaluation value to at least one of them, assigning an evaluation value relating to a predetermined analysis item according to the communication purpose in the business, and assigning an evaluation value to the business result obtained as a result of the communication, The evaluation device according to (7), which includes at least one of the above and has a function of calculating the evaluation result.

(11) An evaluation program which, when incorporated in an information terminal, controls and operates the information terminal as the device according to any one of (6) to (10).

(12) An evaluation system that supports evaluation of communication between users, comprising a data capturing function for capturing video data and audio data of communication, and the video data captured by the data capturing function for the video. An image analysis function for converting into first numerical data using an artificial intelligence technology corresponding to the data, and the voice data captured by the data capturing function is converted into text data using the artificial intelligence technology corresponding to the voice data. And a voice analysis function for converting the text data into second numerical data by using an artificial intelligence technology corresponding to the text data, and combining the first numerical data, the text data and the second numerical data. And a difference data acquisition function for acquiring difference data corresponding to the difference in the reaction of the other user at different times from the consolidation integrated data, An evaluation function and an evaluation function for calculating an evaluation result including at least one of the evaluation of the content of the communication based on the difference data and the evaluation of the communication skill of the user with reference to a predetermined evaluation axis according to the communication purpose. And an output function for outputting the evaluation result obtained in step 1.

According to the present invention, by quantifying and analyzing various emotional elements such as anger, joy, and sadness, and a method of analyzing and evaluating communication among a plurality of users through conversation analysis including various words and silent periods, It provides suggestions and methods for evaluating communication skills of users or improving communication methods.

According to the present invention, regarding communication, data is collected, numerical data obtained by analyzing the data using artificial intelligence technology is utilized, and evaluation is performed based on objective data. Through the evaluation, more effective means can be used to visualize the current communication skill of the user to be evaluated and the result of the improvement of the communication skill, thereby realizing more reliable skill improvement. In addition, although the conventional method often uses a wearable terminal or a dedicated device to increase the system cost for implementation, the present invention does not need to purchase an expensive device or the like and can be implemented at a relatively low cost.

According to the present invention, in the evaluation step, the evaluation result regarding the communication content of the evaluation target user or the communication skill of the evaluation target user is calculated. The evaluation result includes a method calculated based on the difference data acquired from the consolidated integrated data. The acquired difference data is configured by comprehensively using the reactions of the conversation partners obtained through the communication as a whole. Therefore, it is possible to obtain a more realistic evaluation result than the absolute evaluation using a uniform threshold value that is subjectively determined by the instructor, which supports evaluation of substantial communication. This makes it possible to obtain a substantial and quantitative evaluation of communication in various conversation situations. From the above, in the present invention, through the objective acquisition of data regarding communication and the provision of evaluation results based on the data, the visualization of the current state of skills and the visualization of the results of skill improvement can be achieved by more effective means, and more A sure skill improvement can be realized.

It is a block diagram which shows an example of the evaluation system in this embodiment. FIG. 3 is a block diagram showing an example of a system of a data collection device, a data analysis device, an analysis unit and a display device using artificial intelligence, and a functional unit included in them in the present embodiment. It is a flowchart which illustrates the step regarding the capture of the video data and audio data of communication in this embodiment as an example. In the present embodiment, a video analysis step of captured video data, a voice analysis step of captured audio data, an integration step thereof, a differential data acquisition step based on consolidation integrated data, an evaluation step using differential data, and the evaluation thereof. FIG. 6 is a flow diagram illustrating steps as an example regarding output steps. It is a schematic diagram which shows an example of the method of defining the communication event in the integration step in this embodiment. It is a schematic diagram which shows an example of the method of defining the communication whole in the integration step in this embodiment. It is a schematic diagram which shows an example of the method of the difference data acquisition in the difference data acquisition step using the said consolidation integrated data in this embodiment. It is a flow figure showing an example of the whole outline of the flow included in the evaluation step in this embodiment. It is the flowchart which showed the outline and the specific example of the evaluation data value provision flow in the communication event unit contained in the evaluation step in this embodiment. It is the flowchart which showed the outline|summary and the specific example of the evaluation data value provision flow to the communication whole contained in the evaluation step in this embodiment. It is the figure which showed as an example the outline and specific example of the data value calculation based on the consolidation integrated data contained in the evaluation step in this embodiment. It is the figure which showed as an example the outline|summary of the evaluation data value provision based on the work result obtained after the fact included in the evaluation step, and its specific example in this embodiment. It is the flowchart which showed the outline|summary of the provision flow of the final evaluation included in the evaluation step, and its specific example as an example in this embodiment. 6 is a table showing an example of data items that can be used in the communication evaluation method in the present embodiment.

The voice data of the present invention is data that can be collected through a microphone, and as an element of voice related to human communication, collects and converts human speech voice, physiological voice such as laughter, cough, and throat into data, and performs communication analysis. Use this as an element. When there is no speech of a certain user and the other user speaks at the same time, the time when there is no speech of a certain user is classified as listening and this is used as a communication analysis element.

The image data of the present invention is data that can be collected through a camera, and is a user's facial expression, gesture, line of sight, head, body movement, clothes, etc. as elements of images related to communication of a person (hereinafter referred to as a user). The video of is collected and made into data, and this is used as a communication analysis element.

The artificial intelligence technology of the present invention is a method including a known AI program represented by machine learning or deep learning, an AI library, or an AI platform (total service), and is used as an artificial intelligence technology or video data corresponding to voice data. The artificial intelligence technology corresponding to and the artificial intelligence technology corresponding to text data include known Markov models, convolutional neural networks (CNN), recurrent neural networks (RNN), LSTM (Long short-term memory), etc. Models and methods are used. Since it is known, its description is omitted here. As for the artificial intelligence technology to be used, it is preferable to use a method suitable for a predetermined business purpose in communication evaluation.

The first numerical data of the present invention includes numerical data obtained from a result of analyzing the video data. For example, it includes numerical data obtained by analyzing the video data using the artificial intelligence technology and numerically expressing the facial expression of the user for each emotion category. Since the user's emotion classification used here is well known, the description thereof is omitted here. According to the business purpose, it is also preferable that the communication element obtained from the video data is discriminated as to the user's action from the video data by using an artificial intelligence technique and the action is meaningfully used as data. It is also preferable that the evaluator includes numerical value addition, flagging, and meaning addition to the obtained video data according to the business purpose. For example, gestures of hands, flagging of arms, etc., and meaning are given.

The second numerical data of the present invention includes numerical data obtained from the result of analyzing the voice data. For example, using the data obtained by converting the voice data into text by the artificial intelligence technology, the sentiment analysis is performed on the words or sentences included in the text, and numerical data numerically converted according to the degree of positive or negative (hereinafter , Expressed as a positive or negative analysis value). Since it is known, its description is omitted here. According to the business purpose, it is also preferable that the communication element obtained from the voice data is distinguished from the voice data by using the artificial intelligence technology and the user's utterance action is determined and the utterance action is meaningfully used as data. According to the business purpose, it is also preferable to use a known artificial intelligence technique for the text data to determine the intention included in the text data, to make meaning based on the discrimination, and to use the data as data. It is also preferable that the evaluator includes numerical value addition, flagging, and meaning addition to the obtained voice data according to the business purpose. For example, coughing, whispering, etc. may be flagged or given meaning.

The text data of the present invention includes, with respect to the voice data, data obtained by converting the user's voice into text data by using the artificial intelligence technology. For example, a sentence of a spoken conversation or a word is converted into text data. Contains data.

The communication event of the present invention is a minimum unit of communication evaluation defined on the basis of consolidated integrated data divided for each user at an arbitrary time point. For example, a utterance action, a listening action, or a communication action not including an utterance of any user is performed. Including. Communication actions that do not include speech include actions such as smiling faces.

The communication whole of the present invention is a communication evaluation target defined from the start time to the end time of the communication action of the user who is the evaluation target, and a plurality of the communication whole is continuously generated along the time series. , The communication event is included.

The time-series consolidation of the present invention means that the first numerical data, the text data, and the second numerical data obtained by the analysis using the artificial intelligence technology are classified by the user and the video data and the audio data are collected respectively. It includes a step of arranging in time series along with the time information at the time point, integrating as data regarding the same user, and obtaining consolidated data.

The difference data of the present invention is data used to evaluate any communication event of the evaluation target user, for example, when calculating the reaction of the other user, at the time before and after the communication event of the evaluation target user, It includes data obtained by comparing the first numerical value data regarding the facial expression of the other user with respect to the degree of change in the reaction of the other user of the communication based on the above-mentioned consolidated integrated data, and calculating the degree of change as a difference. Further, for example, for any communication event of the evaluation target user, in order to quantify the facial expression intended by the evaluation target user himself/herself in the communication event, the change regarding the facial expression of the evaluation target user at the time immediately before the communication event and the communication event is described above. It is also desirable to include data that is compared based on the consolidated integrated data and that the degree of change of the first numerical data is calculated as a difference.

The predetermined evaluation axis of the present invention includes a communication evaluation standard defined by the purpose of communication given in business. For example, an evaluation item and an evaluation reference value are selected along the evaluation axis, and an evaluation table regarding the evaluation item is determined. By referring to the evaluation table based on the evaluation axis, the content of communication and communication skill performed by the user to be evaluated are evaluated. Along the evaluation axis, for example, determine which data of communication is used as an evaluation item, what value is used as the numerical standard of the evaluation item, and the degree of whether the observed data is better or worse than the numerical standard. The numerical value to be set in the evaluation table is determined.

The work result of the present invention includes items obtained as a result of the communication as a whole. The work result is used to determine how much the communication purpose defined by the predetermined evaluation axis has been achieved by the communication as a whole, and is one of the elements for evaluating the communication content and the communication skill of the user. It becomes one. The business result is, for example, a questionnaire result obtained after the communication with the other users of the evaluation target user or a result of the business (user) who is the evaluation target performing business negotiation communication with the business customer in the context of the business sales of the business user. It includes result information such as whether the customer has information on needs, issues, budget amount, authorizer, desired delivery date, etc.

The evaluation value of the present invention is given to at least one of the communication event, the communication overall, an analysis item predetermined according to the communication purpose in the business, and the business result obtained after the communication overall. And the values used to visualize their degree of evaluation. For example, the evaluation value is given by at least one of the communication event, the communication overall, an analysis item predetermined according to the communication purpose in the business, and the work result obtained after the communication overall. With respect to the evaluation data value of the item, the evaluation value is given by referring to the evaluation table based on the evaluation axis.

The evaluation of the present invention is a concept including a step of judging the communication skill of the user to be evaluated and the degree of good or bad of the communication content, and expressing the degree by a numerical value based on the consolidated integrated data and the difference data obtained by the present invention. Is. The above definition and standard of good and bad are determined from the communication purpose given in the business. For example, based on the knowledge obtained from work, we give a positive meaning to the data that is estimated to increase the possibility of achieving the purpose of the work, and the data that is estimated to reduce the possibility of achieving the purpose of the work. Techniques that give a low rating are included. It is also preferable that such estimation includes a method using known knowledge.

The evaluation result of the present invention is a concept including the final evaluation regarding the communication skill of the evaluation target user or the communication content of the evaluation target user. For example, the final evaluation is predetermined based on the evaluation data value for the communication event, the evaluation data value for the communication whole, and the communication purpose in the business, which are obtained based on the consolidated integrated data and the difference data. It is calculated comprehensively based on the evaluation data value concerning the analysis item and the evaluation data value based on the work result obtained after the communication as a whole. The comprehensive calculation, for each of the evaluation data values, with reference to the evaluation table based on a predetermined evaluation axis, to give the corresponding evaluation value, for each of the evaluation values obtained therefrom, It is calculated by summing the values calculated using the corresponding weighting factors. A method of giving the summed calculation result as the final evaluation and giving it as the evaluation result regarding the communication skill of the evaluation target user or the communication content of the evaluation target user is included. In addition to the above, a method including an evaluation value given by a third party such as an evaluator is also preferable depending on the purpose of the work.

The present invention includes a data acquisition method for acquiring and acquiring audio information and video information from communication performed by a user to be evaluated and its counterpart user, and a method of analyzing each acquired data by artificial intelligence technology. The analyzed voice data and facial expression data are sorted according to the user and the communication event and organized and integrated in time series, the reaction of the other user is analyzed based on the difference data, and communication is performed based on the evaluation axis predetermined in the business. The present invention relates to a method for providing an evaluation regarding content and/or an evaluation regarding communication skills of a user.

Examples of utilization of the present invention include sales customer communication, retail store sales communication, coaching communication between supervisors and subordinates, teaching communication between teachers and students, medical institution staff such as doctors and nurses at medical sites. And patients, communication between multiple users using sign language, presentation communication for the audience, communication to see the reaction and evaluation of customers who eat meals and drinks presented at restaurants etc., entertainment events such as music and dance Communication to see the reaction and evaluation of the audience to performance and dance expression, Communication to see the reaction and evaluation of the customer with the service that provides aroma at the aroma salon, Communication to see the reaction and evaluation of the customer when giving a toy at the toy department, See the reaction and evaluation of the other party when showing traffic and commercial signs such as signs and signboards to the customer Communication, reaction and evaluation of the other party when showing pictures, ceramics and digital art at the exhibition Can be suitably used in evaluating and improving the communication content and technology of a person who issues communication necessary for work in various communication areas, or the expression and method of communication presented to the other party.

Hereinafter, embodiments of the present invention will be described. The present invention is not limited to the following forms and business types. Including the following forms, the inventors of the present invention may make appropriate changes within the obvious range and use in business forms. That is, it is not limited to any particular embodiment, aspect, concept, structure, functionality or illustration described herein. Rather, any of the embodiments, aspects, concepts, structures, functionalities or illustrations described herein are non-limiting, and the present invention generally quantifies video and audio data from communication to improve communication content. Can be used in a variety of ways to provide benefits and benefits in assessment and assessment of a person's communication skills

FIG. 1 shows an example of the overall configuration as a conceptual diagram of a system 100 according to the present invention. For example, as shown in FIG. 1, the system 100 includes a data collection device 1a, a data collection device 1b, a data analysis device 2, and a database device 103 used via the Internet 102. In this example, the data collection device 1 a and the data collection device 1 b have a function of capturing video data and audio data of the communication of the user to be evaluated and transmitting them to the data analysis device 2. The data analysis device 2 converts the received video data into first numerical data using an artificial intelligence technique corresponding to the video data. The data analysis device 2 converts the received voice data into text data using an artificial intelligence technique corresponding to the voice data, and converts it into second numerical data using an artificial intelligence technique corresponding to the text data. In addition, the data analysis device 2 combines the first numerical data, the text data, and the second numerical data to consolidate them in a time series, and consolidates them in a time series. A difference data acquisition function that acquires difference data corresponding to the difference in user reaction, and an evaluation axis that is predetermined according to the purpose of communication in business is referred to, and the content of the communication is evaluated based on the difference data, and the user. It has an evaluation function of calculating an evaluation result including at least one of the communication skill evaluations. In addition, the data analysis device 2 has a function of using artificial intelligence technology on the database 103 via the Internet 102 in order to analyze and convert the captured video data and audio data. The database 103 also preferably includes a cloud service utilized via the Internet 102. The configuration, function, and steps of each of these devices will be specifically described using examples.

As shown as an example in FIG. 2, the data collection device 1 a has a function of capturing video data through the camera device 3 and a function of capturing audio data through the microphone device 10. The data collection device 1a includes a camera device 3, a video data acquisition unit 4, a video data processing unit 5, a video data storage unit 6, and a video data transmission unit 7 in order to realize the function of capturing video data. .. The data collecting device 1a includes a microphone device 10, a voice data capturing unit 11, a voice data processing unit 12, a voice data storage unit 13, and a voice data transmitting unit 14 in order to realize a function of capturing voice data. .. The data collection device 1a includes a data collection time recording unit 15 that records the time when the video data and the audio data of the communication to be evaluated are recorded. The data collection device 1b preferably has at least the same function or device as the function of the data collection device 1a described above as an example.

FIG. 3 shows an example of the video data capturing step and the audio data capturing step in the data collection device 1a. As shown in the example of FIG. 3, in the video data capturing step D1 in the data collecting device 1a, a video image such as the facial expression of the target user who communicates is captured through the camera device 3 and captured as video data. It is provided with a step of converting the image data into a data format suitable for analysis and transmitting the processed data to the storage/storage and data analysis device 2. The voice data capturing step D2 in the data collection device 1a captures the voice or the like of the target user who communicates with the microphone device 10 as collected voice data, converts the voice data into a data format suitable for analysis, and processes the data. The method further comprises the step of storing the stored voice data and transmitting it to the data analysis and storage device 2. The step including the above-mentioned video data acquisition step D1 and audio data acquisition step D2 is referred to as a data acquisition step. In the data acquisition step, in the data collection device 1a, the common time setting is preset, and the data collection device 1a has a function of recording the time when the video data and the audio data of the communication to be evaluated are recorded. Each recorded time data includes the step of being transmitted to the data analysis device 2 along with the video data and the audio data. As for the functions of the data collection device 1a described above as an example, it is desirable that the data collection device 1b has at least the same functions and devices.

Each of the functional units realized by the data collection device 1a and the data collection device 1b is a web camera or a video camera internally or externally connected with an image capturing function, and an internal or external connection having an audio collecting function. It is basically realized by a built-in microphone, input/output IF, a processor such as CPU, RAM, and GPU, and hardware parts for data storage.

The data collecting device 1a and the data collecting device 1b fetch moving image data such as facial expressions and data such as voices of one or more users to be evaluated. It is also preferable to increase the number of data collection devices having the same function as the number of users to be evaluated increases. At that time, the data collection device 1a and the data collection device 1b use well-known computers, tablet terminals, mobile terminals such as smartphones, wearable terminals or computers with a simple credit card size, programmable logic device devices such as FPGAs, etc. Good. As the camera device and the microphone device, those provided in the data collection device 1a and the data collection device 1b may be used, or a device known as an external device or a device having both functions may be used as the data. You may attach them to the collection device 1a and the data collection device 1b, and use them.

The data collection device 1a and the data collection device 1b appropriately convert the captured moving image data and audio data into a format suitable for processing by the data analysis device 2. The acquired data and the converted data are accumulated in the data collecting device 1a and the data collecting device 1b. In the conversion, the data collection device 1a and the data collection device 1b convert the moving image data and the audio data into a known format such as MP3, WAV, WMA, WMV, MP2 or MP4, and the converted data is a wired network medium. Alternatively, it is sent to the data analysis device 2 via a non-limiting network configured by a wireless network medium or the like.

The data collection device 1b shown as an example in FIG. 1 has the same role as the above-described data collection device 1a, and at least the same function or higher, and has the function of realizing the data acquisition step. It is preferable to use at least one such data collection device, but depending on the number of target users who collect communication data, the number of devices having the same role as the data collection device 1a and at least the same or more functions. It is also preferable to increase and use.

As shown in FIG. 2 as an example, the data analysis device 2 analyzes the video data and the audio data collected from the data collection device 1a and the data collection device 1b by the corresponding artificial intelligence technology to obtain the video data. It has a functional unit that converts voice data into numerical data and text data. The data analysis device 2 obtains the converted data results, consolidates them in time series, and analyzes the consolidated data, and analyzes the difference data acquired based on the consolidated data. And a functional unit that calculates an evaluation result from the data obtained from the analysis along an evaluation axis that is predetermined according to the business purpose, a functional unit that outputs the evaluation result, and a functional unit that accumulates. The output of the evaluation result is displayed on the display device. Hereinafter, each of these functions and the device configuration thereof will be specifically described.

The data analysis device 2 includes the video data receiving unit 20, the audio data receiving unit 21, the video data analyzing unit 22, the audio data analyzing unit 30, the video data and audio data processing unit (time series arrangement and integration) 40, the video data. And a voice data analysis (difference data acquisition, analysis and evaluation) unit 41, an analysis result output (evaluation result output) unit 42, and a data storage unit 43. Each of these functional units is basically realized by an input/output IF, a processor such as CPU, RAM, and GPU, and hardware parts for data storage. The data analysis device 2 may have a function and a role of at least one of the data collection device 1a and the data collection device 1b, and may be, for example, a device including a program that causes each of the functional units described above to function as means.

As shown as an example in FIG. 2, the video data and the audio data transmitted from the data collection device 1a and the data collection device 1b to the data analysis device 2 and the time data when the respective data are collected are the data analysis device 2 The video data analysis unit 22 and the audio data analysis unit 30 are analyzed using the AI program analysis unit 50, respectively.

The AI program analysis unit 50 shown as an example in FIG. 2 is an AI program that is called at that time, and these AI programs include at least one or a plurality of known artificial intelligence technologies according to the analysis purpose. AI library or AI platform (comprehensive service) is used. The AI program, AI library or AI platform (general service) calls the AI program, AI library or AI platform (general service) arranged on the external database 103 with an API via the web server of the Internet 102, The command and data to be analyzed may be transmitted, and the analysis result may be received. Alternatively, at least one or a plurality of AI programs, AI libraries, and AI platforms (general services) are arranged in the data analysis device 2 or the data collection device 1a or the data collection device 1b, and are called to analyze the commands. For example, a device including a program that causes each of the functional units described above to function as each unit, such as data transmission and analysis result reception, may be used.

As shown in FIG. 2 and as an example, the analysis result output unit 42 makes it easy to visually see the evaluation results generated by the video data and audio data analysis unit 41, and based on those evaluation results, a desired result for business is displayed. It is converted into an expression that is easy to use and output in order to obtain it.

As shown as an example in FIG. 2, the display device 44 displays the evaluation result output by the analysis result output unit 42 of the data analysis device 2. At this time, a display device provided in the data analysis device 2, a display device connected to the data analysis device 2, a display device on a computer terminal wired or wirelessly connected to the data analysis device 2, a data analysis device 2 wired or Display may be performed using a physical device having a display, such as a display device that is either wirelessly connected to a tablet terminal or a mobile phone (smartphone), or a large display device that is connected to the data analysis device 2 by wire or wirelessly. .. Alternatively, the evaluation result data output by the analysis result output unit 42 of the data analysis device 2 is transmitted from the data analysis device 2 to a Web server via the Internet, and the user can access the mobile terminal, computer terminal, or tablet terminal via the Internet. The result may be displayed on a physical device having a display such as. The output on the left is an example of the process of the output step S6 described as an example in FIG.

As shown as an example in FIG. 2, the data storage unit 43 includes a video data analysis unit 22, a voice data analysis unit 30, an AI program analysis unit 50, a video data and voice data processing unit 40, a video in the data analysis device 2. The data and voice data analysis unit 41 and the data obtained by the analysis result output unit 42 are accumulated. At this time, the data storage unit 43 may utilize a storage device such as a hard disk or SSD for data storage provided in the data analysis device 2, or an external storage device connected to the data analysis device 2. For example, an electronic storage medium such as a USB memory or an SD card may be used. Alternatively, an electronic storage medium such as a cloud storage may be used as the storage medium on the Internet that is connected to the data analysis device 2 by wire or wirelessly. It is also preferable to perform similar data analysis on a plurality of conversation cases of an arbitrary evaluation target user from the accumulated data and compare the evaluation results generated from the same data analysis.

As shown as an example in FIG. 4, the video data analysis unit 22 included in the data analysis device 2 performs a video analysis step S1. In the video analysis step S1, the AI calling unit 23 calls the AI program video analysis unit 51 (artificial intelligence technology corresponding to video data) for analyzing the video data received from the data collection device 1a or the data collection device 1b, The analysis command and data transmission unit 24 transmits the analysis command and the received video data to the AI program video analysis unit 51. The analysis command used at that time includes, for example, an analysis command for analyzing the facial expression of the user or the emotion of the user from the video data according to the business purpose. The AI program image analysis unit 51 receives the analysis instruction and the image data used for the analysis, executes the instructed analysis process, and generates a result converted into (first numerical data). As for the artificial intelligence technology used here, in communication evaluation, it is preferable to use an optimum technology or method according to a predetermined business purpose. The result data analyzed and generated by the AI program image analysis unit 51 as described on the left is received by the analysis result acquisition unit 25, and the acquired data analysis unit 26 further performs data analysis from the viewpoint necessary for the business. The result (first numerical data) obtained from the analysis by the AI program image analysis unit 51 is, for example, the pleasure, sorrow, anger, and startleness shown in FIG. 4 regarding the facial expression of the user to be evaluated from the obtained image data. Outputs numerical values for each facial expression category such as neutral, dislike, disrespect, fear, etc. As the type of facial expression category used at this time, a classification based on known knowledge is used.

As described above, the analysis result output from the AI program video analysis unit 51 and received by the analysis result acquisition unit 25 is analyzed by the acquired data analysis unit 26 in accordance with a predetermined viewpoint necessary for work and data analysis. Be implemented. As the analysis in the acquired data analysis unit 26, for example, the following analysis is executed. It is preferable to select the most appropriate analysis method according to a predetermined business purpose, and it is also preferable to select a method such as the analysis of only the following (1) and (3).
(1) The obtained numerical values for each facial expression category are classified for each user who speaks.
(2) The numerical values for each facial expression category, which are classified according to the user who speaks, are converted into a relative value. For example, the (first numerical value data) obtained for the user A has a numerical value of 5.0 for (expression category: pleasure), 3.0 for (expression category: neutral), and 2. for (expression category: anger). In the case of 0, each value is divided by the total value (5.0+3.0+2.0=10.0) and converted into a ratio value (%) (in the above example, it is converted into the numerical value on the right: pleasure=50% , Neutral = 30%, anger = 20%).
(3) The numerical values for each facial expression category obtained in (2) are arranged in a time series based on the data collection time.
It is preferable to use the optimum time division of the video data analyzed here according to the business purpose. For example, a method is included in which the video data collected by the data collection device 1a is cut out every tenth of a second along the time of collection and the facial expression data is digitized by an artificial intelligence technique. In addition, it is preferable to select an optimum processing method and content for these processings according to a predetermined business purpose. The data analyzed by the acquired data analysis unit 26 is transmitted to the video data and audio data processing unit 40.

As shown as an example in FIG. 4, the voice data analysis unit 30 included in the data analysis device 2 performs a voice analysis step S2. In the voice analysis step S2, the AI calling unit 31 is provided, and the AI program voice analysis unit 52 (artificial intelligence technology corresponding to voice data) for analyzing the received voice data is called, and the analysis command and data transmission unit 32 is used. The data together with the analysis command is transmitted to (artificial intelligence technology corresponding to the voice data) included in the AI program voice analysis unit 52. The AI program voice analysis unit 52 (artificial intelligence technology corresponding to voice data) receives these instructions and data, executes analysis, and generates a result. For the analysis command used at that time, a process of extracting, for example, spoken words from voice data and converting them into text data according to the business purpose is performed. Next, with respect to the text data, for example, a process of analyzing the emotion level of the text content and digitizing it by (artificial intelligence technology corresponding to the text data) included in the AI program voice analysis unit 52 is performed. The above-mentioned analysis of the emotional degree includes, for example, a process of analyzing the degree of the affirmation degree and the degree of denial, and digitizing the degree (acceptability degree analysis). As for the artificial intelligence technology used here, in communication evaluation, it is preferable to use an optimal method or technology according to a predetermined business purpose. An AI program for analyzing the text data (artificial intelligence technology corresponding to the text data) is called, and the data is included in the AI program voice analysis unit 52 together with the analysis command in the analysis command and data transmission unit 32 (text data). (Artificial intelligence technology corresponding to the). The AI program voice analysis unit 52 (artificial intelligence technology corresponding to text data) receives these instructions and data, executes analysis and generates a result, and obtains an analysis result (second numerical data) of the AI program. .. The analysis result (second numerical data) of the AI program obtained above is received by the analysis result acquisition unit 33, analyzed by the acquired data analysis unit 34, and then transmitted to the video data and audio data processing unit 40. The analysis result (second numerical data) obtained as described above includes data (quantity/acceptability analysis value) in which the emotional degree of the uttered text contents is digitized. Alternatively, it is also desirable to extract the keyword from the data (text data) converted into text from the utterance and perform the keyword analysis. For these analysis methods, it is preferable to select and use the most suitable method or technique according to the business purpose.

The processing included in the analysis command to (artificial intelligence technology corresponding to voice data) and (artificial intelligence technology corresponding to text data) included in the AI program voice analysis unit 52 of the data analysis device 2 is, for example, as follows. At least one such process is included. As for the processing included in these analysis commands, it is preferable to select the optimum processing according to a predetermined business purpose and include it in the command.
(A) A process of extracting spoken words or words from voice data and converting them into text data.
(B) A process of extracting a keyword from the data converted into text from the utterance.
(C) Regarding the extracted words and words in the text data, we analyze whether they have a positive meaning or a negative meaning from the emotional side of the user, and determine the positive or negative degree. A process of outputting the indicated numerical value (second numerical data (acceptability analysis value)).

The analysis result acquisition unit 33 included in the data analysis device 2 is analyzed and generated from (artificial intelligence technology corresponding to voice data) and (artificial intelligence technology corresponding to text data) included in the AI program voice analysis unit 52. The acquired data is received, and the acquired data analysis unit 34 analyzes the data in accordance with a predetermined viewpoint necessary for the business. As for the method of data analysis performed here and the method of data definition used for the analysis, it is preferable to select an optimum processing method and contents according to a predetermined business purpose. The data analyzed by the acquired data analysis unit 34 is transmitted to the video data and audio data processing unit 40. In the analysis by the acquired data analysis unit 34, for example, at least one of the following analyzes is executed.
(1) The obtained text data is classified according to the main subject of the utterance.
(2) The number of characters included in the obtained text data is calculated.
(3) The utterance speed (the number of seconds required for each character) per one character of the uttered text is calculated and output.
(4) Based on the collected time data of the voice data, the time zone in which the speech is uttered and the time zone in which the speech data is not uttered are classified, and a process of flagging each is performed. Flagging, for example, gives an utterance flag during the time of utterance, and when one user is uttering and the other user is silent about the time when not uttering, the user who is not uttering "listens" It includes processing such as giving a listening flag indicating that the The aforementioned flagging includes a method that is automatically added according to the definition on the left. Alternatively, it is also preferable that the evaluator performs it according to the business purpose.
(5) An utterance in which the uttered text includes the intent of the question is flagged as a question. For this, a known method such as determining based on a predetermined question vocabulary DB (database) whether the uttered text includes the question vocabulary is used. Alternatively, or according to the business purpose, it is also preferable that the evaluator discriminates the question and flags it.

As shown as an example in FIG. 4, the video data and audio data processing unit 40 performs an integration step S3. In the integration step S3, the analysis result data sent from the video data analysis unit 22 and the audio data analysis unit 30 is received by the video data and audio data processing unit 40, and the video data and the audio data are collected by the camera and the microphone by the data collection device. Data consolidation and data integration for each user is performed according to the obtained time data, and processing for obtaining consolidation integrated data is performed. In FIG. 5, as an example, from the processing result performed by the video data and audio data processing unit 40, data is sorted and integrated for any user according to the time data collected on the data collection device. The results obtained with integrated data are shown. The data processed by the video data/audio data processing unit 40 is transmitted to the video data/audio data analysis unit 41. The video data and audio data analysis unit 41 executes the difference data acquisition step S4 and the evaluation step S5, and the evaluation result obtained therefrom is sent to the analysis result output unit 42 and output (output step S6). The evaluation result is stored in the data storage unit 43 and displayed on the display device 44. The data processing performed by the video data and audio data processing unit 40 is not limited to the above-described processing method, and it is preferable to select the optimum processing method and content according to the business purpose.

From the integration step S3, the integrated data for each user is obtained, and then the communication skill of the evaluation target user is evaluated, so that the evaluation target user himself/herself visualizes his/her own communication, and the own communication is performed. It will be easier to look back on the implementation results and your communication skills, and it will be easier for you to work on communication improvement based on that. In addition, if there is another user who has a high communication skill evaluation, the communication skill of the highly evaluated user is objectively visualized, so it is positioned as a best practice, and it is expected that it will be useful for helping another user improve communication. ..

As shown in FIG. 5 as an example, the integration step S3 includes a step of defining the communication event E1 based on the above-mentioned consolidated integration data. The step is included in the processing performed by the video data and audio data processing unit 40. For example, in the communication of the evaluation target, one action such as utterance or listening performed by any user at any time is performed. The communication event E1 is defined as the minimum unit for evaluating communication.

As described above, by calculating the evaluation for each communication event E1 performed by the user as the minimum unit for evaluating communication, which communication action the user to be evaluated should improve, or no improvement is necessary. It becomes a supporting material for recognizing whether it is a proper element.

As shown as an example in FIG. 6, the integration step S3 performed by the video data and audio data processing unit 40 includes a step of defining the communication whole E2. The communication whole E2 is a target of communication evaluation determined from the start time to the end time of the communication action to be evaluated, and the communication whole E2 is continuously generated in time series, and a plurality of the communication are generated. Event E1 is included.

FIG. 7 shows an example of the processing of the difference data acquisition step S4 performed by the video data and audio data analysis unit 41. In the example of FIG. 7, in order to evaluate an arbitrary communication event performed by the user A, as an example of acquiring difference data corresponding to the difference in reaction of the other user (user B in the example of FIG. 7) at different times, the integration Based on the consolidated integrated data obtained in step S3, the reaction of the user B who is a communication partner is acquired at different time points (time), and the amount of change between the time points is acquired as a difference. In this example, the following processing is performed in order to obtain and visualize as differential data what kind of reaction an arbitrary communication event performed by the user A has elicited from the user B.
(1) Defining a communication event to be evaluated: In the example of FIG. 7, the communication event of the user A (utterance “What are you looking for today?” is based on the consolidated integrated data obtained in the integration step S3. And the facial expression of user A at the time of the utterance) is evaluated.
(2) Determine the data that will be the reaction to the communication event defined in (1): In the example of FIG. 7, for the communication event of the corresponding user A based on the consolidated integrated data obtained in the integrating step S3. , The data at the time when the user B reacts is determined and used for the calculation of the difference data acquisition.
(3) Defining the time point for obtaining the difference data: In the example of FIG. 7, the time point of the user A communication event (9:00:27 to 9:00:32) is set to T2 time point, and the time point immediately before that is (9 :00:22 to 9:00:27) is set as T1. Regarding the selection and method of the time point determined here, it is preferable to select the optimum processing method and content according to the predetermined business purpose. For example, when the change as the reaction of the other user is calculated, it is also preferable to use the time point immediately after the communication event and compare with the time point of the communication event. Alternatively, for example, when the change as the reaction of the other user is calculated, it is also desirable to compare the time immediately before the communication event and the time immediately after the communication event.
(4) Obtaining difference data: In the example of FIG. 7, the amount of change from the time point T1 to the time point T2 is calculated based on the consolidated integrated data obtained in the integrating step S3, with respect to the numerical values of each user's facial expression category. It is calculated and acquired as difference data. The calculation formula is as follows. The calculation formula used here is preferably an optimum calculation formula according to a predetermined business purpose. In order to calculate the amount of change, based on the consolidated integrated data obtained in the integration step S3, for example, the facial expression numerical values of both users are vectorized, and the amount of change of the vector and the vector angle are compared to determine that It is also desirable to estimate the degree of reaction.
Example of calculation formula: (calculated by the formula on the right for each facial expression category) (Numerical value at T2)-(Numerical value at T1) = (Facial expression change amount)
Here, as an example, the numerical value at the time point T2 and the numerical value at the time point T1 are determined by the following method. In addition, it is preferable to use an optimal time segment according to a business purpose for the time segment segment used for the facial expression category data of the target user. For example, the video data collected by the data collection device 1a is cut out every 1/10 second along the time of collection, and for each of the cut-out video data, data obtained by digitizing facial expression data by artificial intelligence technology. The method of using is included.
-Numerical value at time T2: Regarding the data for each facial expression category of the target user collected within the target time (9:00:27 to 9:00:32), the arithmetic mean value is used to determine the numerical value at the time T2.
-Numerical value at time point T1: Regarding the data by target user's facial expression category collected within the target time (9:00:22 to 9:00:27), the arithmetic mean value is used and determined as the numerical value at time point T2.
The method of determining the numerical value is not limited to the above method, and it is preferable to select the optimum method and content according to the business purpose.
In the example of FIG. 7, as an example of obtaining difference data corresponding to the difference in the reaction of the other user at different times using the above calculation formula, the facial expression change for each user is as follows from the change at time T2 and time T1. The amount difference data is calculated for each facial expression category. :
・User A
Pleasant: 10.0%-5.0%=5.0%, sorrow: 2.0%-1.0%=1.0%, anger: 1.0%-1.0%=0%, startle: 3.0%-1.0%=2.0%, neutral: 75.0%-80.0 %=(-5.0%), Aversion: 3.0%-5.0%=(-2.0%), Minority: 1.0%-1.0%=0%, Fear: 5.0%-6.0%=(-1.0%)
(The above facial expression value and its change amount are estimated as the facial expression intended by User A for User B at the relevant communication event)
-User B
Pleasant: 1.0%-1.0%=0%, sorrow: 10.0%-1.0%=9.0%, anger: 1.0%-1.0%=0%, startle: 2.0%-1.0%=1.0%, neutral: 64.0%-85.0 %=(−21.0%), Disgust: 3.0%−5.0%=(−2.0%), Minority: 4.0%−1.0%=3.0%, Fear: 15.0%−5.0%=10.0%
(The above facial expression value and its change amount are estimated as the reaction facial expression of user B to the communication event of user A)
As shown as an example above, in order to evaluate the communication event of the user to be evaluated, difference data corresponding to the difference in reaction of the other user at different times is acquired. Regarding all or some of the communication events included in the total communication of the user to be evaluated, difference data corresponding to the difference in reaction of the other user at different times is similarly acquired. It is also preferable to obtain the difference data corresponding to the difference in the reaction of the other user at different times by the method even in the case where there are three or more users to be evaluated and the other users regardless of the above example.

As shown as an example in FIG. 7, for an arbitrary communication event performed by the evaluation target user, the amount of change in the reaction of the other user is analyzed and calculated in time series around the time of the communication event. By doing so, the reaction of the other party can be visualized. In such a form, the evaluation target user can be visualized by visualizing how the other user reacts or does not react to an arbitrary communication event performed by the evaluation target user. It is possible to objectively evaluate the degree of whether the person got the expected reaction from the other party or did not get the expected reaction. Based on that objective evaluation, the effect of the communication event that one has done can be made objective from the "other's point of view", and it can be used to improve one's communication skills.

In the above example, regarding the continuous communication events performed between the user A and the user B, the difference data as described above is calculated for each communication event performed by the user A, and regarding the communication whole including the continuous communication events. , Use it as an element to give a comprehensive evaluation. That is, it is used as one element reflected in the comprehensive evaluation of the communication of the user A, which is the evaluation target.

As described above, by considering the communication as a whole and making the communication contents comprehensively evaluated, it is possible to comprehensively understand the contents of the communication implementation and its results from the perspective of the communication purpose in the business. It enables the evaluator to make an evaluation from the viewpoint of producing the effect of achieving the purpose. As a result, the user to be evaluated can objectively make the communication performed by himself/herself from a holistic viewpoint from the viewpoint of obtaining the communication result in line with the business purpose, and can be useful for improving his/her communication skill.

FIG. 8 exemplarily shows the overall outline of the flow included in the evaluation step S5, which is carried out by the video data and audio data analysis unit 41. Using the data obtained from the integration step S3 and the difference data acquisition step S4, the evaluation data value (Val) assignment process in the following specific example is performed. At this time, a work result obtained after the fact as a result of the communication overall is separately given by, for example, an evaluator, and a method of using the evaluation data value (Val) is included. The items for communication evaluation and the evaluation reference values used in the evaluation step S5, and the evaluation table for the evaluation items are determined according to the communication purpose given in the business. A specific example of the flow included in the evaluation step S5 is shown below.
(1)
(1-1) An evaluation data value (Val) is assigned in communication event units (1001).
(1-2) Using the evaluation data value (Val) given to each communication event unit of 1001, the evaluation data value (Val) is given to the communication whole composed of those communication events (1002).
(2) The evaluation data value (Val) is calculated and assigned to the data of the predetermined analysis item (1010).
(3) The evaluation data value (Val) is given as the communication result based on the work result obtained after the fact (1020).
(4) Regarding the evaluation data value (Val) acquired in each of the above flows (1002) (1010) (1020), the evaluation value (T) corresponding to it based on the evaluation reference value and the evaluation table given along with the work in advance. ) Are respectively calculated, and the evaluation value (Y) is calculated by multiplying a predetermined weighting coefficient (W) according to the type of each evaluation data value (Val) (1030).
(5) The final evaluation (Z) is calculated by summing the evaluation value (Y) calculated in 1030 above for all the evaluation data value (Val) types, and the communication skill of the evaluation target user or the evaluation target user It is given as an evaluation result regarding communication contents (1040).

FIG. 9 exemplarily shows a flow (1001) of adding an evaluation data value (Val) in communication event unit, which is included in the evaluation step S5. Using the data obtained from the integration step S3 and the difference data acquisition step S4, the following evaluation data value (Val) assignment processing is performed in parallel.
(1) The communication event of the user to be evaluated ((A) in the following example) and the reaction data of the other user to that ((B) in the following example) are defined (1001a).
(Data defined in the example of FIG. 9)
(A) Evaluation target data (user A communication event): A communication event including the utterance of user A "What are you looking for today?" and the facial expression of user A (the amount of facial expression change (user A)).
(B) Response data of the other user B to (A) (user B communication event): A communication event including the user B utterance “(no utterance)” and the user B facial expression (the facial expression change amount (user B)).
(2) The integrated value is calculated based on the difference data between the evaluation target data and the reaction data of the other user against it (1001b).
When calculating the utterance integrated value, the following formulas (1) and (2) are used as an example. As the calculation formula used here, it is preferable to select an optimum formula or calculation method according to a predetermined business purpose.
Calculation formula (1) Calculation of utterance integrated value: ((acceptability analysis value)-(median value *1)) x 100 = utterance integrated value *1: The median value used here is It is desirable to use the median value that is determined according to the method of the corresponding artificial intelligence technology (in this example, the affirmative or negative analysis including the sentiment analysis of the text), or the most neutral value in the degree scale included in the method. ..
Calculation formula (2)
(2-1) Calculation of facial expression integrated value:
(A) Total change in positive emotions (expression) = pleasure (%) change/1 *2
(A) Total change in negative emotion (facial expression) = (sorrow (%) change + anger (%) change + disgust (%) change + neglect (%) change + fear (%) change) / 5 *3
*2: The divisor used here uses the number of items determined in advance according to the business purpose as the positive emotion (in this example, Kiraku is the positive emotion, and the number of items is 1).
*3: The divisor used here is the negative emotion divided by the number of predetermined items according to the business purpose (in this example, sadness, anger, disgust, neglect, and fear are negative emotions, and the number of items is 5). Is used)
(2-2)
Facial expression integrated value=((A)-(I))×100
(Example of integrated value calculation in the example of FIG. 9)
Calculation example (user A in FIG. 9) Calculation of utterance integrated value: (0.52 (acceptability analysis value)−0.50 (median value))×100=2.0
(User A in FIG. 9) Calculation of facial expression integrated value:
(A) 5.0%/1=5.0% (please change)
(B) (1.0% (change of sadness) + 0% (change of anger) + (-2% (change of disgust)) + 0% (change of neglect) + (-1% (change of fear))/5 = (-0.4%)
Facial expression integrated value = ((A)-(I)) x 100 = 5.4
(User B in FIG. 9) Calculation of integrated utterance value: 0.0 (in this example, 0 is given because there is no utterance by user B)
(User B in FIG. 9) Calculation of facial expression integrated value:
(C) 0.0%/1=0.0% (please change amount)
(D) (9.0% (change of sadness) +0% (change of anger) + (-2.0% (change of disgust)) +3.0% (change of neglect) + 10.0% (change of fear)/5 = 4.0%
Facial expression integrated value = ((c)-(d)) x 100 = (-4.0)
(3) The utterance integrated value and the facial expression integrated value obtained in 1001b are totaled for each user, and the total value is compared between the users to calculate a difference (1001c).
(Calculation example in the example of FIG. 9)
(3-1) Sum of utterance integrated value and facial expression integrated value for each user:
(User A in FIG. 9) 2.0 (utterance integrated value) + 5.4 (facial expression integrated value) = 7.4
(User B in FIG. 9) 0.0 (utterance integrated value) + (-4.0 (facial expression integrated value)) = (-4.0)
(3-2) The total value is compared between users and the difference between users is calculated: 7.4−(−4.0)=11.4
(4) Using the value obtained by squaring the difference calculated in 1001c, the evaluation data value (Val) is added to the communication event of the user, which is the evaluation target data (1001d).
(Example of calculating the evaluation data value (Val) by the squared value of the difference in the example of FIG. 9)
Squared value of 11.4 = 130.0 (Here, the value rounded to the first decimal place is used as an example)
The above 130.0 is given as the evaluation data value (Val) for (A) evaluation target data (user A communication event).

FIG. 10 exemplarily shows a flow of giving an evaluation data value (Val) (1002) to the entire communication, which is included in the evaluation step S5. The following processing is performed using the evaluation data value (Val) in communication event units obtained in (1001) above.
(1) With respect to the communication event of the evaluation target user, which is included from the communication start time to the communication end time, the evaluation data value (Val) is calculated by the method of (1001), and the calculated evaluation data value (Val) is calculated. Sum up the total communication (1002a).
(2) The total number of communication events of the evaluation target user from the communication start time to the communication end time is calculated (1002b).
(3) Divide the total value obtained in 1002a by the total number of times obtained in 1002b (1002c).
(4) The result value of the division obtained in 1002c is given as the evaluation data value (Val) regarding the communication whole of the user A (1002d).

FIG. 11 shows an example of calculation (1010) of the evaluation data value (Val) based on the consolidated integrated data and the difference data included in the evaluation step S5. In the integration step S3 and the difference data acquisition step S4, an evaluation data value (Val) is calculated for a predetermined evaluation item for the communication body of the evaluation target user based on the obtained data.
The following items are examples of items of the calculated evaluation data value (Val). Regarding the selection of the items to be calculated and the calculation method thereof, it is preferable to select the optimum item selection and calculation method according to the predetermined business purpose.
(A) Total conversation time (a1) Total utterance time of user A (a2) Total utterance time of the other party (user B) (a3) Total listening time of user A (b) Total utterance time for each user and total time Ratio
(B1) Ratio of user A's utterance (a1)/((a1)+(a2))
(B2) Ratio of user B's utterance (a2)/((a1)+(a2))
(C) User A's utterance time (a1)/User B's utterance time (a2)
(D) User A listening time ratio (User A listening time (a3)/total utterance time ((a1) + (a2))
(E) Total number of communication events (utterances) by user A (f) Number of question utterances by user A (g) Question rate in utterances (number of questions for user A/total number of communication events for user A (utterances))
(H) Difference in utterance speed from the other user (*Number of seconds spent per utterance character)
(H1) Average speed of user A's utterance (h2) Average speed of user B's utterance (i) Positive by user *Average facial expression (%)
(I1) Positive of User A *Average facial expression (%)
(I2) Positive for User B *Facial expression average (%)
(J) Negative by user *Facial expression average (%)
(J1) Negative of User A *Average facial expression (%)
(J2) Negative of User B *Average facial expression (%)
(K) User B's positivity when user A speaks *Facial expression average (%)
(L) Negative of User B when speaking to User A *Average facial expression (%)
(M) User B's positive when listening to user A *Average facial expression (%)
(N) Negative of user B when listening to user A *Average facial expression (%)

FIG. 12 shows an example of the evaluation data value (Val) assignment (1020) based on the work result obtained after the fact, which is included in the evaluation step S5. As the communication result, an evaluation data value (Val) obtained from at least one work result obtained after the whole communication is given. The work result obtained after the fact as a result of the communication overall is, for example, separately given by the evaluator and includes a method of using the evaluation data value (Val). Examples of the work results obtained after the communication overall include the following results. The selection of the business result used here and the calculation method are the selection of the optimum business result according to the predetermined business purpose, the definition of the evaluation data (Val) corresponding to it, and the evaluation data (Val) according to the definition. It is preferable to select the calculation method of ). It is also preferable to use a plurality of business results depending on the business purpose.
・Is the satisfaction level of the other party obtained in order to achieve the business purpose?
(Example) In a company's internal communication, as a result of the coaching communication with his subordinates as a result of the supervisor's coaching, ex-post evaluation etc. obtained from his subordinates regarding the same session.
(Example) As a result of performing role play communication in role play during training, ex-post evaluation etc. obtained by the user to be evaluated from the role play partner regarding the same session.
・Has the original communication purpose been achieved?
(Example) As a result of a salesclerk communicating with a customer at a retail store, whether the customer has purchased the product or not.
・Is the necessary information obtained to achieve the business purpose?
(Example) In corporate sales, as a result of business negotiations with corporate customers, it was possible to obtain information about the corporate customers' needs, budget amount, authorizers, and desired delivery dates.
(Example of assigning evaluation data value (Val) based on business results used in the example of FIG. 12)
(Example) Give the evaluation of the questionnaire result obtained from the conversation partner (user B).
With the result of the questionnaire from the user B obtained after the communication aggregate, the evaluation regarding the communication aggregate of the user A is obtained. Based on the result of the questionnaire, the satisfaction level of the user B is calculated as NSAT (Net Satisfaction: Satisfaction level) by the following calculation formula, and the NSAT value is given as the evaluation data value (Val) of the communication aggregate of the user A.
NSAT calculation formula * (example): NSAT = P-N + 100
P = (number of answers with the highest rating ("5" in the example of Fig. 12)) / total number of responses (6 total responses are used in the example of Fig. 12)
(Example of calculating P in the example of FIG. 12) P=0/6=0
N = (the lowest and the second lowest from the evaluation (the sum of the numbers of responses "1" and "2" in the example of Fig. 12))/total number of responses (in the example of Fig. 12, the total number of responses is 6)
(Example of calculating N in the example of FIG. 12) N=2/6=0.33 (Here, the value rounded to the third decimal place is used as an example)
(Example of NSAT calculation in the example of FIG. 12) NSAT=0-0.33+100=99.7 (Here, the value rounded to the second decimal place is used as an example.)
*The NSAT calculation formula is not limited to the above example, but it is preferable to use an optimum method according to the method of the questionnaire result and the number of items.

When there is a business communication purpose, it is preferable to evaluate the result of the communication overall according to the degree to which the corresponding communication purpose has been achieved. Therefore, in the method of the present application, by using the work result separately obtained after the communication as a whole as one of the evaluation factors, it is possible to improve the communication skill of the user to be evaluated from the viewpoint of improving the work effect. .. As an example, after the communication is finished, the evaluation of the overall communication or the evaluation of the communication skill of the evaluation target user, which the other user provides to the evaluation target user, is an objective viewpoint for improving the communication. As a result, it becomes possible for the users to be evaluated to improve their communication skills, which leads to essential communication improvements along the line of sight of the other person.

FIG. 13 shows an example of the flow of giving the final evaluation, which is included in the evaluation step S5. This flow includes the following procedures.
(1)
(1-1) Regarding the evaluation data value (Val) obtained from the above (1002, 1010 and 1020), the evaluation value (T ) Is calculated (1030a).
(1-2) Multiply (multiply) the evaluation value (T) of each analysis item obtained in 1030a by a weighting coefficient (W) corresponding to the type of evaluation data value (Val) given in advance according to the work. ), and the evaluation value (Y) is calculated (1030b).
(2) The evaluation value (Y) calculated in 1030b above is summed for all of the above evaluation items to calculate the final evaluation (Z), and the evaluation result regarding the communication skill of the evaluation target user or the communication content of the evaluation target user. (1040).
As in the above example, using the evaluation data value (Val) obtained in the procedure of (1002, 1010 and 1020) for the items selected in the most suitable manner according to the predetermined business purpose, By calculating the evaluation value (Y) by using the weighting coefficient determined according to the predetermined business purpose by referring to the evaluation table corresponding to, the objectiveness is high and the business purpose is high based on a wider perspective. It is possible to give an evaluation according to. As a result, it is possible to improve communication skills by increasing the productivity of the work of the user to be evaluated. The selection of the evaluation reference value used here, the selection of the value of the weighting coefficient, the configuration of the values and scales included in the evaluation table, the calculation method, the selection of the optimum evaluation reference value according to the predetermined business purpose, weighting It is preferable to select the value of the coefficient, the configuration of the value and the scale in the evaluation table, and the calculation method.

In order to compare the changes in the communication skills of the users to be evaluated in the above example, the above-mentioned final evaluation (Z) is obtained multiple times at any given time for the relevant users, and the evaluations regarding the communication skills of the relevant users are compared in time series. By doing so, it is also preferable to evaluate the time change of the communication skill of the corresponding user.

In the above-mentioned comparison of user evaluation data at multiple time points, regarding the result of learning about the communication improvement method that the evaluation target user learns, the communication result before learning the improvement method and the learning result after learning the improvement method By comparing the communication results at the time point, it becomes possible to visualize the degree to which the evaluation target user has learned the communication improvement method and has acquired the method. For example, the following methods can be used to support more objective and effective communication evaluation.
・By comparing the final evaluations of multiple users, the communication skills of users who have a high evaluation of communication skills and users who do not have a high evaluation of communication skills are compared from the viewpoint of a predetermined work, and a high communication evaluation is made. By visualizing the connected difference elements, it is possible for the other user to incorporate the skill of the user with a high evaluation from a more specific viewpoint.
-For the users to be evaluated, by comparing the communication evaluation (final evaluation) before learning the communication method by the arbitrary knowledge and the communication evaluation (final evaluation) after learning the communication method by the aforementioned knowledge, the corresponding user is It is possible to visualize the degree to which a communication method based on knowledge has been acquired.
-For a plurality of arbitrary communication learning methods, it is possible to select a more effective communication learning method by visualizing the degree of the effect for each communication learning method. To that end, which communication learning method improves the user's communication skills by comparing the changes in the communication evaluation (final evaluation) before and after learning for multiple users who learn the corresponding communication learning method. It is possible to visualize and utilize the degree to which it has been done. As in the above example, it is preferable to use the method of utilizing the framework of the present invention by using an optimal method according to a predetermined business purpose.

As described in the above example, regarding the user communication, the image data mainly obtained from the camera device and the voice data obtained from the microphone device are analyzed by using the corresponding artificial intelligence technology, and based on the analysis result, etc. An example was given in which communication was evaluated by analyzing the evaluation axis according to the business purpose. Not limited to the above example, which data should be selected from the user's communication and collected and analyzed should be selected and collected optimally according to the predetermined business purpose and analyzed according to the data. It is preferable to choose a method. FIG. 14 shows an example of data items that can be used in the communication evaluation method, and includes the following items as examples.
(1) Example of data item obtained from camera as example of device (example of data item visually obtained):
・Facial expression/gesture (physiological gesture)
・Gesture (gesturing)
・Movement of eyes ・Movement of head (nod, sideways, etc.)
-Body posture and limb movements (eg, limb assembly, front center of gravity, rear center of gravity)
-Example of data item obtained from microphone as an example of clothing (2) device (example of data item obtained by hearing):
・Voice tone, pitch ・Voice volume ・Voice rhythm (speed, breathing)
・Silence (listening, silence, etc.)
・Physiological phenomena such as laughter, cough, and sore throat (3) Examples of data items obtained from devices including temperature sensors and vibration sensors (eg, smartphones, wearable devices, etc.):
・Temperature (air temperature, body temperature, etc.)
・Shake (body)
(4) As an example of a device, an example of data items obtained from an electronic device (eg, a data collection device included in the present invention):
・Time information (collection time of video data, audio data, etc.)

In addition, according to the above-described embodiment, for example, the data analysis device 2 acquires data corresponding to the data acquisition step, the video analysis step, the audio analysis step, the integration step, the difference data acquisition step, the evaluation step, and the output step. The evaluation device may include a function, a video analysis function, a voice analysis function, an integration function, a difference data acquisition function, an evaluation function, and an output function. Further, a computer, a tablet terminal, a programmable logic device such as an FPGA, or the like may be an apparatus including an evaluation program that causes each unit of the information terminal such as the evaluation apparatus described above to function.

The embodiments of the present invention have been described above with reference to the drawings. It should be noted that the present invention is not limited to the above-mentioned forms and business types. The present invention also includes use in a form or business mode in which the inventor of the present case appropriately changes within the obvious range.

1a...Data collection device 1b...Data collection device
2... Data analysis device 3... Camera
4... Video data acquisition unit 5... Video data processing unit 6... Video data storage unit 7... Video data transmission unit
10... Microphone 11... Audio data capturing unit 12... Audio data processing unit 13... Audio data storage unit 14... Audio data transmitting unit 15... Data collection time recording unit 20... Video data receiving unit 21... Audio data receiving unit 22... Video Data analysis unit 23...AI calling unit 24...Analysis command and data transmission unit 25...Analysis result acquisition unit 26...Acquired data analysis unit 30...Voice data analysis unit 31...AI calling unit 32...Analysis command and data transmission unit 33...Analysis Result acquisition unit 34... Acquired data analysis unit 40... Video data and audio data processing unit 41... Video data and audio data analysis unit 42... Analysis result output unit 43... Data storage unit 44... Display device 50... AI program analysis unit 51... AI program video analysis unit 52... AI program audio analysis unit
100... Communication evaluation system 101... Communication network 102... Internet 103... Database 200-250... Data acquisition steps 302-306... Video analysis step S1
310-319... Voice analysis step S2
330... Integration step S3
331... Difference data acquisition step S4
332... Evaluation step S5
340... Output step S6
1000...1050 Steps included in evaluation step S5

Claims (12)

An evaluation method that supports evaluation of communication between users, wherein a data capturing step of capturing communication video data and audio data, and the video data captured by the data capturing step correspond to the video data. The video analysis step of converting into the first numerical value data by using the artificial intelligence technology, and the voice data captured by the data capturing step are converted into text data using the artificial intelligence technology corresponding to the voice data. , A voice analysis step of converting the text data into second numerical data using an artificial intelligence technique corresponding to the text data, and a time series by combining the first numerical data, the text data and the second numerical data. And a consolidation data for collecting consolidation data, and a difference data acquisition step for obtaining difference data corresponding to the difference in the reaction of the other user at different times from the consolidation integrated data, for the purpose of business communication. Along with the evaluation axis determined in advance along with the evaluation step of calculating the evaluation result including at least one of the evaluation of the content of the communication based on the difference data and the evaluation of the communication skill of the user, and the evaluation step. An output step of outputting the evaluation result, wherein the data acquisition step, the video analysis step, the audio analysis step, the integration step, the difference data acquisition step, the evaluation step and the output step are executed by a computer. Evaluation method.
The integrating step defines a communication event as a minimum unit for evaluating a communication action performed by a user at an arbitrary time point based on the consolidated integrated data; a step of defining a communication event at a different time point for each of the users; The method according to claim 1, further comprising the step of defining, as a communication whole, data obtained by chronologically integrating and integrating the communication events included from the start time to the end time of the communication based on the event. The evaluation method according to claim 2, wherein the difference data acquisition step includes a step of acquiring the difference data of the partner user for each communication event. The evaluation step includes a step of calculating the evaluation result for each user based on the consolidation integrated data and the difference data, and the evaluation regarding the content of the communication whole body based on the consolidation integrated data and the difference data. The evaluation method according to claim 2, comprising a step of calculating a result. The evaluation axis includes a communication evaluation standard defined from a communication purpose given in business, and the evaluation step is given with reference to the evaluation axis, and is at least one of the communication event and the communication whole body. At least one of assigning an evaluation value to the evaluation result, assigning an evaluation value regarding a predetermined analysis item according to the communication purpose in the business, and assigning an evaluation value to the business result obtained as a result of the communication. The evaluation method according to claim 2, further comprising: a step of calculating the evaluation result including. An evaluation device supporting evaluation of communication between users, wherein a data capturing function for capturing video data and audio data of communication and the video data captured by the data capturing function correspond to the video data. The video analysis function for converting into the first numerical data by using the artificial intelligence technology and the voice data captured by the data capturing function are converted into the text data by using the artificial intelligence technology corresponding to the voice data. , A time-series combination of a voice analysis function for converting the text data into second numerical data using an artificial intelligence technology corresponding to the text data, and the first numerical data, the text data, and the second numerical data. The consolidation function for consolidating and consolidating into and the consolidation data, and the difference data acquisition function for acquiring the difference data corresponding to the difference in the reaction of the other user at different times from the consolidation integrated data, and for the purpose of business communication. Along with a predetermined evaluation axis, the evaluation function for calculating the evaluation result including at least one of the evaluation of the content of the communication based on the difference data and the evaluation of the communication skill of the user and the evaluation function are obtained. An evaluation device having an output function of outputting the evaluation result. The integration function includes a function of defining a communication event as a minimum unit for evaluating a communication action performed by a user at an arbitrary time point based on the consolidated integration data, a function of defining a communication event at a different time point for each of the users, and the communication The evaluation device according to claim 6, further comprising a function of defining, as a communication whole, data obtained by chronologically integrating and integrating the communication events included from the start time to the end time of the communication based on the event. The evaluation device according to claim 7, wherein the difference data acquisition function has a function of acquiring the difference data of the opponent user for each communication event. The evaluation function has a function of calculating the evaluation result for each user based on the consolidation integrated data and the difference data. Based on the consolidation integrated data and the difference data, the evaluation regarding the contents of the communication whole body is performed. The evaluation device according to claim 7, which has a function of calculating a result. The evaluation axis includes a communication evaluation standard defined from the purpose of communication given in business, and the evaluation function is given with reference to the evaluation axis, and is at least one of the communication event and the communication whole body. At least one of assigning an evaluation value to the evaluation result, assigning an evaluation value regarding a predetermined analysis item according to the communication purpose in the business, and assigning an evaluation value to the business result obtained as a result of the communication. The evaluation device according to claim 7, further comprising a function of calculating the evaluation result. An evaluation program which, when incorporated into an information terminal, controls and operates the information terminal as the device according to any one of claims 6 to 10. An evaluation system that supports evaluation of communication between users, wherein a data capturing function for capturing video data and audio data for communication and the video data captured by the data capturing function correspond to the video data. The video analysis function for converting into the first numerical data by using the artificial intelligence technology and the voice data captured by the data capturing function are converted into the text data by using the artificial intelligence technology corresponding to the voice data. , A time-series combination of a voice analysis function for converting the text data into second numerical data using an artificial intelligence technology corresponding to the text data, and the first numerical data, the text data, and the second numerical data. The consolidation function for consolidating and consolidating into and the consolidation data, and the difference data acquisition function for acquiring the difference data corresponding to the difference in the reaction of the other user at different times from the consolidation integrated data, and for the purpose of business communication. Along with a predetermined evaluation axis, the evaluation function for calculating the evaluation result including at least one of the evaluation of the content of the communication based on the difference data and the evaluation of the communication skill of the user and the evaluation function are obtained. An evaluation system having an output function of outputting the evaluation result.