JP2017054307A - Utterance content evaluation system and utterance content evaluation method - Google Patents

Abstract

The present invention automatically evaluates the utterance content of an operator with high accuracy without depending on the ability of an evaluator.
For each word, a value that minimizes the difference between a weighted average frequency, an evaluation formula of a normal distribution based on an average frequency standard deviation, and a sample score over all utterance content samples is represented as an appearance frequency score. For each word, for each word, a value that minimizes the difference between the weighted relative appearance position and the normal distribution evaluation formula based on the appearance position standard deviation and the sample score is used for each word. For each word of utterance content calculated and evaluated as a coefficient, the appearance frequency evaluation value of the normal distribution evaluation formula multiplied by the appearance frequency score coefficient and the appearance position of the normal distribution evaluation expression multiplied by the appearance position score coefficient And the sum of the words of the utterance content to be evaluated is taken as the evaluation value of the utterance content.
[Selection] Figure 5

Description

  The present invention relates to an utterance content evaluation system and an utterance content evaluation method, and more particularly, to an utterance content evaluation system and an utterance content evaluation method suitable for use in diagnosing reception skills by evaluating an operator's reception in a call center. .

  A call center is a business establishment / department that specializes in telephone support for customers in a company. Some call centers include mail order / service / manufacturing companies for general consumers that accept complaints, various inquiries and orders, and others that teach device operations such as computer operations over the phone.

  In any case, the response of the call center operator greatly affects the corporate image of the customer, customer satisfaction, etc., so it is important for the corporate side to secure a high-quality operator.

  Conventionally, the operator's response evaluation by such a call center has been performed manually. However, evaluating a large amount of utterances requires a great deal of cost, so the target of the evaluation is limited, and the evaluation tends to be subjective due to the preconceptions of the evaluator. It was difficult. In order to solve this problem, Patent Document 1 proposes a technique for automatically evaluating the utterance content using voice recognition using an information processing apparatus. In the “telephone response diagnosis device” of Patent Document 1, the telephone response of the operator is recognized as voice, converted into text, the text is grouped for each item, and then matched with the expression pattern, so that the speaker (creator) Evaluating the telephone response.

JP-A-2005-318101 Japanese Patent Laid-Open No. 1-156866

  As in the technique described in Patent Document 1, when evaluating the response of an operator in a call center, it is possible to automatically evaluate the utterance content using voice recognition in order to reduce costs. However, in Patent Document 1 of the prior art, as an evaluation method, it is determined whether or not a pattern using a predetermined word is matched. It is not an automatic evaluation. In addition, there is room for including the evaluator's subjectivity when the evaluation standard is created, and there is a problem that it is difficult to say objective evaluation and the evaluation accuracy depends on the ability of the evaluator.

  The present invention has been made to solve the above problems, and its purpose is not to depend on the evaluator's ability to evaluate the difference between a plurality of evaluated and scored utterance contents. Another object of the present invention is to provide an utterance content evaluation system that can automatically evaluate the utterance content of an operator with high accuracy.

  The utterance content evaluation system of the present invention is an utterance content evaluation system that evaluates utterance content based on one or more utterance content samples each scored, and the appearance frequency of each word in the utterance content sample and Calculate the relative appearance position of each word in the utterance content sample, and for each word, the weighted average frequency of the appearance frequency weighted by the utterance content score and the appearance weighted by the utterance content score Calculate the average frequency standard deviation of the frequency.

  Next, for each word, calculate the weighted average appearance position of the relative appearance position weighted by the utterance content score, and the appearance position standard deviation of the relative appearance position weighted by the utterance content score, A value that minimizes the difference between the weighted average frequency, the normal distribution evaluation formula based on the average frequency standard deviation, and the score of the sample is calculated as an appearance frequency score coefficient. Then, for each word, over the sample of all utterance contents, a value that minimizes the difference between the weighted relative appearance position, the normal distribution evaluation formula based on the appearance position standard deviation, and the score of the sample is represented as an appearance position score coefficient. For each word of the utterance content to be calculated and evaluated, the weighted average frequency multiplied by the appearance frequency score coefficient and the normal distribution evaluation formula by the average frequency standard deviation are used as the evaluation expression of the appearance frequency at the time of evaluation. The evaluation value of the appearance frequency for the word is obtained. When evaluating the utterance content, for each word in the utterance content to be evaluated, the weighted average appearance position multiplied by the appearance position score coefficient and the normal distribution evaluation formula based on the appearance position standard deviation are evaluated. As an expression, the evaluation value of the appearance position for the word is obtained, and the evaluation value of the appearance frequency and the evaluation value of the utterance content to be evaluated are obtained by taking the sum of the evaluation value of the appearance frequency and the evaluation value of the appearance position over each word of the utterance content to be evaluated. To do.

  According to the present invention, it is possible to automatically evaluate an utterance content of an operator with high accuracy without depending on an evaluator's ability, by evaluating a difference from a plurality of evaluated and scored utterance content. It is possible to provide an utterance content evaluation system that can

It is a block diagram of the utterance content evaluation system concerning one embodiment of the present invention. It is a hardware block diagram of the utterance content evaluation system which concerns on one Embodiment of this invention. It is a detailed functional block diagram of a past utterance content accumulation unit and an evaluation criterion generation unit. It is a detailed functional block diagram of an utterance content evaluation part. It is a figure explaining the outline | summary of the production | generation of utterance content evaluation criteria, and the evaluation of the utterance content based on it. It is a figure which shows the content of the utterance content A. FIG. It is a figure which shows the content of the utterance content B. FIG. It is a figure which shows the content of the utterance content C. FIG. It is a figure which shows an example of the score table which shows the score of each utterance. It is a flowchart which shows the process to normalization frequency calculation and relative position calculation. It is a flowchart which shows the process until the production | generation of an evaluation formula. It is a figure which shows the frequency table of the utterance content A. It is a figure which shows the frequency table of the utterance content B. FIG. It is a figure which shows the frequency table of the content of speech. It is a figure which shows an example of a user definition category table. It is a figure which shows an example of a meaningless word table. It is a figure which shows an example of the total character number table of each speaker. It is a normalization frequency table of the utterance content A after a meaningless word removal. It is a normalization frequency table of the utterance content B after a meaningless word removal. It is a normalization frequency table of the utterance content C after a meaningless word removal. It is a relative appearance position table of the utterance content A after a meaningless word removal. It is a relative appearance position table of the utterance content B after a meaningless word removal. It is a relative appearance position table of the utterance content C after a meaningless word removal. It is a figure which shows an example of the weighted average frequency table after a meaningless and low frequency word removal. It is a weighted relative appearance position table after meaningless and low frequency word removal. It is a figure which shows an example of the average frequency standard deviation table after a meaningless low frequency word removal. It is an example of an appearance position standard deviation table after meaningless, low-frequency word removal It is a figure which shows an example of an appearance frequency score coefficient table. It is a figure which shows an example of an appearance position score coefficient table. It is a flowchart which shows the process which evaluates the utterance content X with the produced | generated evaluation formula.

  Hereinafter, an embodiment according to the present invention will be described with reference to FIGS. 1 to 18B.

  First, the configuration of an utterance content evaluation system according to an embodiment of the present invention will be described with reference to FIGS. 1 to 4. FIG. 1 is a block diagram of an utterance content evaluation system according to an embodiment of the present invention. FIG. 2 is a hardware configuration diagram of the utterance content evaluation system according to the embodiment of the present invention. FIG. 3 is a detailed functional block diagram of the past utterance content accumulation unit and the evaluation criterion generation unit. FIG. 4 is a detailed functional block diagram of the utterance content evaluation unit.

  As shown in FIG. 1, the utterance content evaluation system according to the present embodiment includes a speech recognition unit 101, an evaluation reference generation unit 102, an utterance content evaluation unit 103, an evaluation result storage unit 104, and a past utterance content accumulation unit 105.

The speech recognition unit 101 converts the utterance data to be evaluated into text data, and transmits it as the utterance content 106 to the utterance content evaluation unit 103.
The evaluation reference generation unit 102 receives the scores corresponding to the utterance contents stored in the past utterance content storage unit 105 as the past utterance contents 108 and the past utterance score 109, respectively, automatically generates the evaluation reference, and generates the generation evaluation reference 107. To the utterance content evaluation unit 103.

The utterance content evaluation unit 103 calculates the utterance score based on the generation evaluation criterion 107 received from the evaluation criterion generation unit 102 and transmits the utterance score as the utterance content evaluation result 110 to the evaluation result storage unit 104.
The evaluation result storage unit 104 stores the utterance content evaluation result 110 received from the utterance content evaluation unit 103.

  The past utterance content accumulation unit 105 stores a score corresponding to the utterance content as past utterance content.

Next, the hardware configuration of the utterance content evaluation system of this embodiment will be described with reference to FIG.
The hardware configuration of the utterance content evaluation system is realized by, for example, a general information processing apparatus 400 as shown in FIG.
The information processing apparatus 400 is configured such that a CPU (Central Processing Unit) 402, a main storage device 404, a network I / F 406, a display I / F 408, an input / output I / F 410, and an auxiliary storage I / F 412 are coupled by a bus. ing.

  The CPU 402 controls each unit of the information processing device 400 and loads and executes a necessary program in the main storage device 404.

  The main storage device 404 is generally composed of a volatile memory such as a RAM, and stores a program executed by the CPU 402 and data to be referred to.

  A network I / F 406 is an interface for connecting to a network. A display I / F 408 is an interface for connecting a display device 420 such as an LCD (Liquid Crystal Display).

  The input / output I / F 410 is an interface for connecting an input / output device. In the example of FIG. 5, a keyboard 430 and a pointing device mouse 432 are connected.

The auxiliary storage I / F 412 is an interface for connecting an auxiliary storage device such as an HDD (Hard Disk Drive) 450 or a DVD drive.
The HDD 450 has a large storage capacity, and stores a program for executing this embodiment. A speech recognition program 460, an evaluation reference generation program 462, and an utterance content evaluation program 464 are installed in the information processing apparatus 400 of this embodiment. In addition, a past utterance content DB 470, an evaluation reference generation DB 472, and an utterance content evaluation DB 474 are held as databases.

  The speech recognition program 460 converts the utterance data to be evaluated into text data, and passes it to the utterance content evaluation program 464 as the utterance content. The evaluation criterion generation program 462 refers to the scores corresponding to the utterance contents stored in the past utterance content storage DB 470 as the past utterance contents and the past utterance scores, respectively, automatically generates an evaluation criterion, and evaluates the utterance contents as the generation evaluation criterion. Hand it over to the program.

  The utterance content evaluation program 464 calculates the utterance score based on the generation evaluation criterion generated by the evaluation criterion generation program, and stores it in the evaluation result storage DB 472 as the utterance content evaluation result.

  The past utterance content accumulation DB 470 is a database that stores past utterance content and the corresponding score as past utterance content.

  The evaluation reference generation DB 472 is a database that stores tables created at the time of evaluation reference.

  The evaluation result storage DB 474 is a database that stores tables created at the time of utterance content evaluation and utterance content evaluation results.

  Next, detailed functional configurations of the past utterance content accumulation unit and the evaluation criterion generation unit will be described with reference to FIG.

The utterance content storage unit 201 transmits the utterance content of the past utterance content 108 to the word list extraction unit 208 and the total character count unit 203.
The total character count unit 203 counts the total number of characters for each utterance from the past utterance content 108 received from the utterance content storage unit 201 and transmits the total number of characters as total character number information 204 to the total character number storage unit 205.

  The total character number storage unit 205 stores the total character number information 204 received from the total character number count unit 203, and transmits the total character number information 204 to the normalization frequency calculation unit 215 and the relative position calculation unit 224.

  The word list extraction unit 208 extracts a word list from the past utterance content 108 received from the utterance content storage unit 201, and transmits it as a word list 209 to the word frequency count unit 211 and the word / frequency list storage unit 213. In addition, the technique etc. which are described in patent document 2 can be utilized for extraction of a word.

  The word frequency counting unit 211 counts the appearance frequency of words included in the word list 209 received from the word list extraction unit 208 during utterance, and transmits the frequency as a frequency list 212 to the word / frequency list storage unit 214.

  The word / frequency list storage unit 213 stores the word list 209 received from the word list extraction unit 208 as an extracted word list, and stores the frequency list 212 received from the word frequency count unit 211 as an extracted word frequency list. It is transmitted to the normalization frequency calculation unit 215 as 214.

  The normalized frequency calculation unit 215 adds the user-defined category information 216 received from the user-defined category table storage unit 217 to the word / frequency list 214 received from the word / frequency list storage unit 213, and receives it from the total character number storage unit 205. The normalization frequency is calculated from the total number of characters data 206. The meaning of the normalization frequency will be described later. Then, with reference to the user-defined category information 216, the normalized frequency data 218 is processed and transmitted to the meaningless word removal unit 219.

  The meaningless word removal unit 219 removes words included in the meaningless word list 220 received from the meaningless word table storage unit 221 from the normalized frequency data 218 received from the normalization frequency calculation unit 215, and the meaningless word The removed normalized frequency information 222 is transmitted to the normalized frequency storage unit 223 and the relative position calculation unit 224.

  The normalized frequency storage unit 223 stores the meaningless word removed normalized frequency information 222 received from the meaningless word removal unit 218, and the weighted average frequency calculation unit 232 as the meaningless word removed normalized frequency list 226; The weighted average appearance position calculation unit 236, the average frequency standard deviation calculation unit 240, and the appearance position standard deviation calculation unit 244 are transmitted.

  The user-defined category table storage unit 217 stores words defined to be handled in the same way by the user, and transmits them as user-defined category information 216 to the normalization frequency calculation unit.

  The meaningless word table 221 stores meaningless words to be excluded from the generation of evaluation criteria, and transmits the meaningless word list 220 to the meaningless word removal unit 219.

  The relative position calculation unit 224 calculates the relative position from the meaningless word removed normalized frequency data 222 received from the meaningless word removal unit 219 and the total number of characters data 206. Then, the relative position information 227 is transmitted to the relative position storage unit 228.

  The relative position storage unit 228 stores the relative position information 227 received from the relative position calculation unit 224 and transmits the relative position information list 229 to the weighted average appearance position calculation unit 236 and the appearance position standard deviation calculation unit 244.

  The score storage unit 230 stores a score corresponding to each utterance and calculates a weighted average frequency calculation unit 232, a weighted average appearance position calculation unit 236, an average frequency standard deviation calculation unit 240, and an appearance position standard deviation as the past utterance score 109. To the unit 244.

  The weighted average frequency calculation unit 232 calculates an average frequency weighted by a score from the meaningless word removed normalized frequency list 226 received from the normalized frequency storage unit 223. The weighted average frequency will be described later. The weighted average frequency information 233 is transmitted to the weighted average frequency storage unit 234.

  The weighted average frequency storage unit 234 stores the weighted average frequency information 233 received from the weighted average frequency calculation unit 232 and transmits the weighted average frequency information list 235 to the score coefficient calculation unit 248.

  The average frequency standard deviation calculator 240 calculates the standard deviation from the meaningless word removed normalized frequency list 226 received from the normalized frequency storage 223. Then, the average frequency standard deviation information 241 is transmitted to the average frequency standard deviation storage unit 242.

  The average frequency standard deviation storage unit 242 stores the weighted average frequency standard deviation information 241 received from the average frequency standard deviation calculation unit 240, and transmits the weighted average frequency standard deviation information list 243 to the score coefficient calculation unit 248.

  The weighted average appearance position calculation unit 236 calculates the average appearance position weighted by a score from the normalized frequency list 226 with meaningless word removal received from the normalization frequency storage unit 223 and the relative position information list 229 received from the relative position storage unit 228. Calculate The weighted average appearance position will be described later. Then, the weighted average appearance position information 237 is transmitted to the weighted average appearance position storage unit 238.

  The weighted average appearance position storage unit 238 stores the weighted average appearance position information 237 received from the weighted average appearance position calculation unit 236 and transmits the weighted average appearance position information list 239 to the score coefficient calculation unit 248.

  The appearance position standard deviation calculation unit 244 calculates the standard deviation from the meaningless word removed normalized frequency list 226 received from the normalization frequency storage unit 223 and the relative position information list 229 received from the relative position storage unit 228. Then, the appearance position standard deviation information 245 is transmitted to the appearance position standard deviation storage unit 246.

  The appearance position standard deviation storage unit 246 stores the weighted appearance position standard deviation information 245 received from the appearance position standard deviation calculation unit 244 and transmits it to the score coefficient calculation unit 248 as the appearance position standard deviation information list 247.

  The score coefficient calculation unit 248 includes a weighted average frequency information list 235 received from the weighted average frequency storage unit 234, an average frequency standard deviation information list 243 received from the average frequency standard deviation storage unit 242, and a weighted average appearance position storage unit 238. From the weighted average appearance position information list 239 received from, and the weighted appearance position standard deviation information list 247 received from the appearance position standard deviation storage unit 246, the appearance frequency, the weighted average value of the appearance position, and the standard deviation are extracted for each word. . Based on them, a score coefficient is calculated for each word, and the score coefficient information 225 is transmitted to the score coefficient storage unit 210. A method for calculating the score coefficient will be described later.

  The score coefficient storage unit 210 stores the score coefficient information 225 received from the score coefficient calculation unit 248.

Next, a detailed functional configuration of the utterance evaluation reference generation unit will be described with reference to FIG.
As shown in FIG. 4, the functional configuration of the utterance evaluation reference generation unit 103 includes a total character count unit 203, a total character count storage unit 205, a word list extraction unit 208, a word frequency count unit 211, and a word / frequency list storage unit 213. The normalized frequency calculation unit 215, meaningless word removal unit 219, normalization frequency storage unit 223, user-defined category table storage unit 217, meaningless word table 221, relative position calculation unit 224, and relative position storage unit 228 are: This is the same as the evaluation reference generation unit 102 shown in FIG.

  Further, it has a score calculation unit 250 and a score storage unit 260. The score calculation unit 250 includes a frequency score calculation unit 252 that calculates a score related to frequency and an appearance position score calculation unit 254.

  The frequency score calculation unit 252 receives the weighted average frequency for each word and the frequency score coefficient from the evaluation reference generation unit 102, and calculates a score related to the frequency.

  The appearance position score calculation unit 254 receives the weighted appearance position and the appearance position score coefficient for each word from the evaluation reference generation unit 102, and calculates a score related to the appearance position. Then, both scores are added and finally stored in the score coefficient unit 210 as a score related to the utterance content.

  Next, the outline | summary of the production | generation of the speech content evaluation criteria of this embodiment and evaluation of the speech content based on it is demonstrated using FIG. 5, FIG. 6A thru | or FIG. 6C, FIG. FIG. 5 is a diagram for explaining the outline of the generation of the utterance content evaluation criteria and the evaluation of the utterance content based thereon. FIG. 6A is a diagram showing the content of the utterance content A. FIG. 6B is a diagram showing the content of the utterance content B. FIG. 6C is a diagram showing the content of the utterance content C. FIG. 7 is a diagram illustrating an example of a score table indicating the score of each utterance.

  As shown in FIG. 5, the utterance content evaluation system according to the present embodiment scores a sample of past utterance content, thereby creating an evaluation criterion (generating an evaluation formula), thereby generating a target. The content of the utterance is evaluated. In the example of the present embodiment, as shown in FIGS. 6A to 6C, there are utterance content A, utterance content B, and utterance content C. As shown in FIG. It shall be attached. And an evaluation formula is produced | generated by it and the utterance content X is evaluated by the evaluation formula.

  The higher the score of each utterance content, the better the score, and the call center operator's expert, manager, etc. shall score in consideration of the speaker's history and skills.

  Next, the procedure up to the generation of the evaluation formula of this embodiment will be described with reference to FIGS. 8 and 9 and FIGS. 10A to R3. FIG. 8 is a flowchart showing processing up to normalization frequency calculation and relative position calculation. FIG. 9 is a flowchart showing processing up to generation of the evaluation formula. FIG. 10A is a diagram showing a frequency table of utterance content A. FIG. FIG. 10B is a diagram showing a frequency table of the utterance content B. FIG. 10C is a diagram showing a frequency table of utterance contents C. FIG. 11 is a diagram illustrating an example of a user-defined category table. FIG. 12 is a diagram illustrating an example of a meaningless word table. FIG. 13 is a diagram illustrating an example of a total character count table for each speaker. FIG. 14A is a normalization frequency table of utterance content A after meaningless word removal. FIG. 14B is a normalization frequency table of utterance content B after meaningless word removal. FIG. 14C is a normalization frequency table of utterance content C after meaningless word removal. FIG. 15A is a relative appearance position table of the utterance content A after the meaningless word is removed. FIG. 15B is a relative appearance position table of the utterance content B after the meaningless word is removed. FIG. 15C is a relative appearance position table of the utterance content C after the meaningless word is removed. FIG. 16A is a diagram illustrating an example of a weighted average frequency table after meaningless, low-frequency word removal. FIG. 16B is a weighted relative appearance position table after meaningless, low-frequency word removal. FIG. 17A is a diagram illustrating an example of an average frequency standard deviation table after meaningless, low-frequency word removal. FIG. 17B shows an example of an appearance position standard deviation table after meaningless and low-frequency word removal. FIG. 18A is a diagram illustrating an example of an appearance frequency score coefficient table. FIG. 18B is a diagram illustrating an example of an appearance position score coefficient table.

  First, processing up to normalization frequency calculation will be described with reference to FIG.

First, one of the utterances in the utterance content storage unit 302 is selected (S301).
Next, the word list extraction unit 208 extracts words from the selected utterance (S302).
Next, the appearance frequency is counted by the word frequency counting unit 211 (S303). As a result, the frequency tables shown in FIGS. 10A to 10C are generated and stored in the word / frequency storage unit 213. Here, as shown in FIGS. 10A to 10C, the format of the frequency table includes the extracted word list 801 of the utterance content A, the word appearance frequency 802 of the utterance content A, and the extracted word list 901 of the utterance content B, respectively. The word appearance frequency 902 of the utterance content B, the extracted word list 1001 of the utterance content C, and the word appearance frequency 1002 of the utterance content C are paired.

  Then, the total number of characters of each utterance is counted by the total character number counting unit 203 (S304). As a result, the total number of characters table shown in FIG. 13 is created.

  Next, with reference to the user-defined category table shown in FIG. 11 of the user-defined category table storage unit 217, the normalized frequency calculation unit 215 normalizes the defined user-defined category and each word. The frequency is calculated (S305). A user-defined category is a unit that handles words belonging to the category as one. For example, as shown in FIG. 11, as “recommended word categories”, “only this time”, “only this time”, and “only this time” are treated as substantially the same word. The normalization frequency is a quotient obtained by dividing the frequency of the word by the total number of utterances.

  Next, the meaningless word removal unit 219 refers to the meaningless word table shown in FIG. 12 stored in the meaningless word table storage unit 221 and removes the meaningless word from the word list. Then, the normalized frequency table after the meaningless word removal shown in FIGS. 14A to 14C is stored in the normalized frequency storage unit 223 (S306). Here, the meaningless word is a word that is excluded as not contributing to the evaluation of the utterance content. Here, as shown in FIGS. 14A to 14C, the format of the normalization frequency table includes a word list 1401 after removal of meaningless words of utterance content A, a normalization frequency list 1402 of utterance content A, and utterance content, respectively. A word list 1501 after removal of B meaningless words, a normalization frequency list 1502 of utterance contents B, a word list 1601 after removal of meaningless words of utterance contents C, and a normalization frequency list 1602 of utterance contents C are paired. It has become.

  Next, with reference to the user-defined category table shown in FIG. 11 of the user-defined category table storage unit 217, the relative position calculation unit 224 determines the relative position for the defined user-defined category and each word. Calculate (S307). Further, the relative position is a numerical value assigned as a scale with the beginning of 0 and the end of 1 for the position of the word for all characters in the utterance.

  Next, the meaningless word removal unit 219 refers to the meaningless word table shown in FIG. 12 stored in the meaningless word table storage unit 221 and removes the meaningless word from the word list. Then, the relative appearance position table after the meaningless word removal shown in FIGS. 15A to 15C is stored in the relative appearance position storage unit 228 (S308). When a word appears in multiple places, the arithmetic average is taken. Here, as shown in FIGS. 15A to 15C, the format of the relative appearance position table includes the word list 2001 of the utterance content A after removal of the meaningless word, the relative appearance position list 2002 of the utterance content A, and the meaningless, respectively. The word list 2101 of the utterance content B after the word removal, the relative appearance position list 2102 of the utterance content B, the utterance content C2201 after removal of the meaningless word, and the relative appearance position list 2202 of the utterance content C are paired.

  Then, it is determined whether or not there is an unprocessed utterance (S309). If there is an unprocessed utterance (S309: NO), the process returns to S301, and if there is no unprocessed utterance (S309: YES), the process is performed. finish.

  Next, processing from normalization frequency calculation to evaluation formula generation will be described with reference to FIG.

First, one word in the normalization frequency table stored in the normalization frequency storage unit 223 is selected (S401).
Then, it is determined whether or not the normalization frequency of the corresponding word in at least one utterance is higher than a predetermined threshold (S402). When the normalization frequency is not higher than a predetermined threshold for all utterances (S402: NO), the processing returns to the processing of S402 without processing the word, and the frequency is predetermined for at least one utterance. If the word is higher than the threshold, the process proceeds to S404. This means that a word whose normalization frequency is too low is not subject to generation of an evaluation formula. If it is desired to set all words as targets for generating an evaluation expression, the predetermined threshold value may be set to zero.

Next, the average value of the normalized frequencies weighted by the scores in the score table of the score storage unit 230 is calculated by the weighted average frequency calculation unit 232 as the weighted average frequency (S403). The weighted average frequency m _F is calculated by the following (Formula 1).

  Here, Σ in the equation is taken over the utterance content A, utterance content B, and utterance content C of the scored sample, and F (i) is the utterance content i stored in the normalization frequency table. The normalization frequency of the word, and S (i) is the score of the utterance content i stored in the score table.

  The weighted average frequency of each word of the utterance content A, the utterance content B, and the utterance content C is stored in the weighted average frequency table as shown in FIG. 16A.

Next, the standard deviation of the normalized frequency weighted by the score in the score table of the score storage unit 230 is calculated by the average frequency standard deviation calculation unit 40 as the average frequency standard deviation σ _F. The average frequency standard deviation σ _F is calculated by the following (Formula 2).

  The average frequency standard deviation of each word of the utterance content A, the utterance content B, and the utterance content C is stored in the average frequency standard deviation table as shown in FIG. 17A.

  Next, the average value of the appearance positions weighted by the scores in the score table of the score storage unit 230 is calculated by the weighted average appearance position calculation unit 236 as a weighted average appearance position (S405).

Weighted average appearance position m _P is calculated by the following equation (3).

Here, P (i) is the relative appearance position of the word in the utterance content i stored in the relative appearance position table.
The relative appearance positions of the words of the utterance contents A, the utterance contents B, and the utterance contents C are stored in the relative appearance position table as shown in FIG. 16B.

Next, the appearance position standard deviation calculating unit 244 calculates the standard deviation of the relative appearance position weighted by the score in the score table of the score storage unit 230 as the appearance position standard deviation σ _P (S406). The appearance position standard deviation σ _P is calculated by the following (formula 4).

  The appearance position standard deviation of each word of the utterance content A, the utterance content B, and the utterance content C is stored in the appearance position standard deviation table as shown in FIG. 17B.

  Next, the appearance frequency score coefficient of each word is calculated by the score coefficient calculation unit 248 (S407).

Hereinafter, a method of calculating the appearance frequency score coefficient will be described.
First, the appearance frequency normal distribution evaluation expression E _{i, F} in the utterance content i (i = A, B, C) is defined as the following (Expression 5).

Next, an appearance frequency normal distribution evaluation formula in the utterance content i and an evaluation function Gi _{, F} for evaluating the difference between the scores are defined by the following (formula 6).

Note that X is a variable that moves a real number, and the evaluation functions Gi _{, F} are quadratic expressions of X.

Then, the evaluation function G _F is defined as the sum of the utterance contents i (i = A, B, C) of the evaluation functions G _{i, F by} the following (Equation 7).

Finding is given, the evaluation function G _F is the value of X which takes the minimum value. The evaluation function G _F is a quadratic convex downward, as is known in elementary mathematics, X is, when the following Equation (8), takes the minimum value. In the present embodiment, the value of X that takes the minimum value of (Equation 8) is defined as the appearance frequency score coefficient C _F that minimizes the difference in scores over the utterance contents i.

  The appearance frequency score coefficient of each word is stored in the appearance frequency score coefficient table as shown in FIG. 18A.

  Next, an appearance position score coefficient of each word is calculated by the score coefficient calculation unit 248 (S408).

  Hereinafter, a method of calculating the appearance position score coefficient will be described. The method of calculating the score coefficient of the appearance position is the same as the method of obtaining the appearance frequency score coefficient.

First, the appearance position normal distribution evaluation expression E _{i, P} in the utterance content i (i = A, B, C) is defined as the following (Expression 9).

Next, an appearance position normal distribution evaluation formula in the utterance content i and an evaluation function G _{i, P} for evaluating the difference between the scores are defined by the following (formula 10).

Note that X is a variable that moves a real number, and the evaluation function Gi _{, P} is a quadratic expression of X.

The evaluation function _GP is defined as the sum of the utterance contents i (i = A, B, C) of the evaluation functions G _{i, P by} the following (Equation 11).

What is calculated is the minimum value of the evaluation function _GP . The evaluation function _GP is a quadratic expression convex downward, and, as is known in elementary mathematics, takes a minimum value when X is the following (Expression 12). In the present embodiment, the value of X that takes the minimum value of (Equation 12) is assumed to be the appearance position score coefficient C _F that minimizes the difference in score over each utterance content i.

  The appearance position score coefficient of each word is stored in the appearance position score coefficient table as shown in FIG. 18B.

  This is performed over words of all utterance contents i (i = A, B, C) (S409), and an appearance frequency score coefficient and an appearance position score coefficient are obtained.

Next, processing for evaluating the utterance content X using the evaluation formula generated with reference to FIG. 19 will be described.
FIG. 19 is a flowchart illustrating a process of evaluating the utterance content X using the generated evaluation formula.

Here, the evaluation value (score) for the utterance content X is represented by V (X).
First, 0 is substituted to initialize V (X) (S500).
Then, one word of the utterance content X is selected (S501).

  Next, it is searched whether or not the appearance frequency score of the selected word is in the appearance frequency score coefficient table shown in FIG. 18A (S502), and when it is in the appearance frequency score coefficient table (S502: YES), it is selected. The score of the evaluation formula of the word appearance frequency is calculated (S503).

Hereinafter, a method of calculating the score of the appearance frequency evaluation formula will be described.
Here, since the appearance frequency score coefficient C _F obtained by (Equation 8) is different for each word, the appearance frequency score coefficient C _F corresponding to a certain word w is described as C _F (w). It shall be. Similarly, with the weighted average frequency m _F obtained by (Equation 1), the weighted average frequency corresponding to a certain word w is m _F (w), and the average frequency standard deviation σ _F obtained by (Equation 2) is used. The average frequency standard deviation corresponding to a certain word w is described as m _F (w). A normalization frequency corresponding to a certain word w is described as F (w). Here, it is assumed that the normalization frequency after removal of meaningless words is obtained for the words of the evaluation utterance content X by the processing of the flowchart shown in FIG.

At this time, the evaluation formula V _F (w) of the appearance frequency of a certain word w is expressed by the following (formula 13).

  Next, it is searched whether or not the appearance position score of the selected word is in the appearance position score coefficient table shown in FIG. 18B (S504), and when it is in the appearance position score coefficient table (S502: YES), it is selected. The score of the evaluation formula of the word appearance position is calculated (S503).

Hereinafter, a method of calculating the score of the evaluation formula for the appearance position will be described. As with the frequency of the evaluation formula, the appearance position score coefficients C _P obtained by equation (12), since to differ by each word, the occurrence position score coefficients C _P corresponding to a word w , C _P (w). Similarly, weighting appearance position m _P obtained by the (Equation 3), the weighting appearance position corresponding to a word w, and m P _(w), the occurrence position standard deviation sigma _P obtained by (Equation 4) The appearance position standard deviation corresponding to a certain word w is described as m _P (w). A relative appearance position corresponding to a certain word w is described as P (w). Here, it is assumed that the relative position is obtained for the word of the evaluation utterance content X by the processing shown in the flowchart of FIG.

At this time, the evaluation expression V _P (w) of the appearance position of a certain word w is expressed by the following (Expression 14).

Then, the score of the evaluation formula for the appearance frequency in S503 and the score of the evaluation formula for the appearance position in S504 are added to V (X).
Then, the processing from S501 to S506 is performed for all words (S507).
V (X) is finally obtained by the following (Formula 15).

  Here, the word w covers all the words in the evaluation utterance content X. However, the appearance frequency score of a word not in the appearance frequency score coefficient table and the appearance position score of a word not in the appearance position score coefficient table are excluded.

  As described above, the utterance content evaluation system of the present embodiment prepares a plurality of scored utterance samples, generates an evaluation expression for evaluating the relationship between the appearance frequency of each word unit and the relative position thereof, Based on this, the utterance content is evaluated. Therefore, it is possible to automatically and objectively evaluate utterances by speakers with little experience by comparing the scores of utterances of experienced veteran speakers with high scores. The value can also be used as a basis for the progress of the speaker's speech skills.

DESCRIPTION OF SYMBOLS 101 ... Speech recognition part 102 ... Evaluation criteria production | generation part 103 ... Utterance content evaluation part 104 ... Evaluation result storage part 105 ... Past utterance content storage part 106 ... Utterance content 107 ... Generation evaluation criteria 108 ... Past utterance content 109 ... Past utterance score 201 ... utterance content storage section 203 ... total character count section 204 ... total character count information 205 ... total character count storage section 206 ... total character count data 208 ... word list extraction section 209 ... word list 210 ... score coefficient storage section 211 ... word frequency count section 212 ... frequency list 213 ... word / frequency list storage unit 214 ... word / frequency list 215 ... normalized frequency calculation unit 216 ... user-defined category information 217 ... user-defined category table storage unit 218 ... normalized frequency data 219 ... meaningless word removal Unit 220 ... meaningless word list 221 ... meaningless word table storage unit 22 ... meaningless word removed normalized frequency information 223 ... normalized frequency storage section 224 ... relative position calculation section 225 ... score coefficient information 226 ... meaningless word removed normalized frequency list 227 ... relative position information 228 ... relative position storage section 229 ... Relative position information list 230 ... Score storage section 232 ... Weighted average frequency calculation section 233 ... Weighted average frequency information 234 ... Weighted average frequency storage section 235 ... Weighted average frequency information list 236 ... Weighted average appearance position calculation section 237 ... Weighted average Appearance position information 238 ... Weighted average appearance position storage section 239 ... Weighted average appearance position information list 240 ... Average frequency standard deviation calculation section 241 ... Weighted average frequency standard deviation information 242 ... Average frequency standard deviation storage section 243 ... Average frequency standard deviation information List 244 ... Appearance position standard deviation calculator 245 ... Weighting Current position standard deviation information 246 ... appearance position standard deviation storage unit 247 ... weighted appearance position standard deviation information list 248 ... score coefficient calculation unit 801 ... extracted word list utterance content A 802 ... word appearance frequency 901 of utterance content A ... utterance content B extracted word list 902 ... utterance content B word appearance frequency 1001 ... utterance content C extracted word list 1002 ... utterance content C word appearance frequency 1101 ... product name category word list 1102 ... recommended word category word list 1103 ... noise category Word list 1201 ... meaningless word list 1301 ... utterance list 1302 ... total character number list 1401 ... word list after removal of meaningless words of utterance content A 1402 ... normalization frequency list 1501 of utterance content A ... removal of meaningless words from utterance content B Later word list 1502 ... Normalization frequency list 16 of utterance content B 01 ... Word list 1602 after removal of meaningless words of utterance content C ... Normalization frequency list 1701 of utterance content C ... Speech list 1702 ... Score list 1801 ... Meaningless, low-frequency word removal degree word list 1802 ... Weighted average frequency list 1901 ... meaningless, low-frequency word removal degree word list 1902 ... weighted average frequency standard deviation list 2001 ... utterance content A word list 2002 after meaningless word removal ... utterance content A relative appearance position table.
2101 ... Word list 2 of utterance contents B after removal of meaningless words 2102 ... Relative appearance position table of utterance contents B.
2201 ... Word list 2202 of utterance contents C after meaningless word removal ... Relative appearance position table of utterance contents C.
2301 ... Word list 2301 after removal of meaningless and low-frequency words 2301 Mean relative position 2401 weighted by score ... Word list 2402 after removal of meaningless and low-frequency words 2402 Standard deviation 2801 of average relative position weighted by score ... None Meaning, word list after removal of low-frequency words 2802 ... appearance frequency score coefficient 2901 ... word list after removal of meaningless, low-frequency words 2902 ... appearance position score coefficient

Claims (4)

An utterance content evaluation system for evaluating utterance content based on one or more utterance content samples each scored,
Calculating the appearance frequency of each word in the utterance content sample and the relative appearance position of each word in the utterance content sample;
For each word, calculate a weighted average frequency of the appearance frequency weighted by the score of the utterance content, and an average frequency standard deviation of the appearance frequency weighted by the score of the utterance content,
For each word, calculate a weighted average appearance position of the relative appearance position weighted by the score of the utterance content, and an appearance position standard deviation of the relative appearance position weighted by the score of the utterance content,
For each word, the appearance frequency score coefficient is a value that minimizes the difference between the weighted average frequency, the normal distribution evaluation formula based on the average frequency standard deviation, and the score of the sample over all utterance content samples. Calculate as
For each word, the appearance position score is a value that minimizes the difference between the weighted relative appearance position, the normal distribution evaluation formula based on the appearance position standard deviation, and the score of the sample over all utterance content samples. Calculated as a coefficient,
For each word of the utterance content to be evaluated, the weighted average frequency multiplied by the appearance frequency score coefficient, and the normal distribution evaluation formula by the average frequency standard deviation, as the evaluation formula of the appearance frequency at the time of evaluation, Find the evaluation value of the appearance frequency for the word,
For each word of the utterance content to be evaluated, the weighted average appearance position multiplied by the appearance position score coefficient and the normal distribution evaluation formula by the appearance position standard deviation are used as the evaluation expression of the appearance position at the time of evaluation. , Find the evaluation value of the appearance position for the word,
An utterance content evaluation system characterized by taking the sum of the evaluation value of the appearance frequency and the evaluation value of the appearance position over each word of the utterance content to be evaluated to obtain the evaluation value of the utterance content to be evaluated .   A meaningless word table for storing meaningless words is retained, and the meaningless words stored in the meaningless word table are excluded when calculating the appearance frequency of each word in the utterance content sample. The utterance content evaluation system according to claim 1. Holds a user-defined category table that defines words in the same category,
When calculating the appearance frequency of each word in the utterance content sample, for a plurality of words defined as the same category by the user-defined category table, the frequency is calculated as a single word. The utterance content evaluation system according to claim 1. An utterance content evaluation method for evaluating utterance content based on one or more utterance content samples each scored,
Calculating the appearance frequency of each word in the utterance content sample and the relative appearance position of each word in the utterance content sample;
For each word, calculating a weighted average frequency of appearance frequency weighted by the score of the utterance content and an average frequency standard deviation of appearance frequency weighted by the score of the utterance content;
For each word, calculating a weighted average appearance position of relative appearance weights weighted by the utterance content score, and an appearance position standard deviation of relative appearance positions weighted by the utterance content score;
For each word, the appearance frequency score coefficient is a value that minimizes the difference between the weighted average frequency, the normal distribution evaluation formula based on the average frequency standard deviation, and the score of the sample over all utterance content samples. Step to calculate as
For each word, the appearance position score is a value that minimizes the difference between the weighted relative appearance position, the normal distribution evaluation formula based on the appearance position standard deviation, and the score of the sample over all utterance content samples. Calculating as a coefficient;
For each word of the utterance content to be evaluated, the weighted average frequency multiplied by the appearance frequency score coefficient, and the normal distribution evaluation formula by the average frequency standard deviation, as the evaluation formula of the appearance frequency at the time of evaluation, Calculating an appearance frequency evaluation value for the word;
For each word of the utterance content to be evaluated, the weighted average appearance position multiplied by the appearance position score coefficient and the normal distribution evaluation formula by the appearance position standard deviation are used as the evaluation expression of the appearance position at the time of evaluation. , Obtaining an evaluation value of the appearance position for the word;
A step of taking the sum of the evaluation value of the appearance frequency and the evaluation value of the appearance position over each word of the utterance content to be evaluated to obtain the evaluation value of the utterance content to be evaluated. Utterance content evaluation method.

Images