JP6259377B2 - Dialog system evaluation method, dialog system evaluation apparatus, and program - Google Patents

Description

  The present invention relates to a technique for automatically evaluating an utterance sentence generated by a dialog system in a system that interacts with a user using a natural language (hereinafter referred to as a dialog system).

  In recent years, there is an increasing need for a chat dialogue system that performs open domain chat without specific tasks. One of the challenges in improving the chat dialogue system is the evaluation of the constructed system. In a dialogue system for performing a task, since there are clear evaluation indexes such as the achievement rate of the task and the time taken to achieve the task, the evaluation of the system is relatively easy. However, in the chat dialogue system, the correct answer to be output by the system is not always obvious. For this reason, conventionally, the method of manually assigning an evaluation value of an order scale such as the Likert scale to the output sentence of the system and taking the average value has been the mainstream.

  However, since the value given by the order scale is a relative value, although the order relationship is consistent, the average value may be different for each evaluation. That is, in order to compare the conventional system and the proposed system with the average value of the given evaluation values, it is necessary to re-implement the conventional system to be compared and to perform an experiment simultaneously with the proposed system. Thus, since it is not easy to compare with existing research, a mechanism that can automatically assign evaluation values in a reproducible form is necessary.

  As an attempt to automatically evaluate a task dialogue system, there is a method called PARADISE proposed in Non-Patent Document 1. This is a method for evaluating the quality of an already-conversed dialogue based on features such as the length of the spoken sentence and the number of utterances obtained from the dialogue. Further, as a framework for automatic evaluation of a sentence output by the system, a technique described in Non-Patent Document 2 can be cited.

Marilyn Walker, Candace Kamm, Diane Litman, “Towards developing general models of usability with PARADISE”, Natural Language Engineering, vol. 6, no. 3-4, pp. 363-377, 2000. Alan Ritter, Colin Cherry, Bill Dolan, “Data-Driven Response Generation in Social Media”, In proceedings of EMNLP, 2011.

  However, in the method described in Non-Patent Document 1, it is necessary to perform a dialogue every time in order to evaluate the dialogue system. For that purpose, a partner who interacts with the dialogue system is required. When the conversation partner is a human, the problem arises that the evaluation value varies with each experiment as in the case of manually assigning the evaluation value. If the other party is a different dialogue system, there is no dialogue system that can respond at the same time as a human being, and the quality of the dialogue may deteriorate depending on the dialogue system that is the other party. From this point of view, a method of performing evaluation in a format that does not involve actual dialogue is desirable.

  The framework for automatic evaluation of sentences output by the system has been actively researched in the field of machine translation. For example, the system outputs a sentence for the input sentence, calculates the distance between the system output sentence and one reference sentence based on a special function such as BLEU score or ROUGE score, and outputs it as an evaluation value An automatic rating scale has been developed. When this is used for chat conversation, the range of sentences that should be correct in the chat conversation is wider than machine translation, so a single reference sentence cannot be covered. Thus, automatic evaluation based on the distance from the reference sentence is difficult in the chat conversation.

  In view of such a technical background, an object of the present invention is to provide a dialogue system evaluation technique for automatically evaluating a chat dialogue system having no task without human intervention.

  In order to solve the above-mentioned problem, in the dialog system evaluation method of the present invention, an output sentence acquisition unit inputs an input sentence to a dialog system having no specific task and obtains a system output sentence from the dialog system. The sentence acquisition step and the evaluation value calculation unit include an evaluation value calculation step of calculating a system evaluation value for evaluating the system output sentence based on a reference sentence predetermined for the input sentence.

  According to the dialogue system evaluation technique of the present invention, in a chat dialogue system having no task, the dialogue system can be automatically evaluated without any manual intervention. As a result, the dialog system can be evaluated at high speed and at low cost, and the dialog system can be improved efficiently.

FIG. 1 is a diagram illustrating a functional configuration of the dialogue system evaluation apparatus according to the first embodiment. FIG. 2 is a diagram illustrating a processing flow of the interactive system evaluation method according to the first embodiment. FIG. 3 is a diagram illustrating a functional configuration of the dialogue system evaluation apparatus according to the second embodiment and the third embodiment. FIG. 4 is a diagram illustrating a processing flow of the interactive system evaluation method according to the second embodiment and the third embodiment. FIG. 5 is a diagram illustrating a functional configuration of the dialogue system evaluation apparatus according to the fourth embodiment. FIG. 6 is a diagram illustrating a processing flow of the interactive system evaluation method according to the fourth embodiment. FIG. 7 is a diagram illustrating a functional configuration of the dialogue system evaluation apparatus according to the fifth embodiment. FIG. 8 is a diagram illustrating a processing flow of the interactive system evaluation method according to the fifth embodiment.

  The present invention is a dialog system evaluation apparatus and method for calculating an evaluation value by inputting a sentence to a dialog system and comparing a sentence output from the dialog system with a reference sentence. In the present invention, the problem of the conventional technique is solved by enlarging the reference sentence to about several tens to several hundred sentences and covering the range of utterances that are correct. Further, an evaluation value may be given to the reference sentence in advance, and this may be used for estimation of an evaluation value of the system output sentence (hereinafter referred to as a system evaluation value). The evaluation value to be assigned to the reference sentence is obtained by a method of directly assigning a value manually, a method of comparing which is better for each pair of reference sentences, and using the winning percentage as an evaluation value.

  In the following, reference sentences include positive examples such as sentences created manually to be correct, as well as negative examples such as sentences prepared to be incorrect and sentences automatically extracted from the corpus. Assume and explain. However, the dialog system evaluation apparatus and method of the present invention operate even if only one of them is excluded and only a positive example or only a negative example is configured.

  The following description is divided into five embodiments based on the type of data used.

  The first embodiment is the simplest form and uses only a reference sentence. Between the sentence output from the dialogue system and each reference sentence, between sentences such as BLEU score, ROUGE score, tf-idf weighted cosine distance, and Word Error Rate (WER) used in automatic evaluation of machine translation This is a method for calculating a scale representing the degree of similarity and using the average value of the top N (N is a natural number of about 1 to 7) as a system evaluation value. For details on the BLUE score, see Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu, “BLEU: a method for Automatic Evaluation of Machine Translation”, ACL '02, pp. 311-318, 2002. 1) ”, for details on the ROUGE score, see“ Lin, Chin-Yew, and Eduard Hovy. “Automatic evaluation of summaries using n-gram co-occurrence statistics”, NAACL '03, vol. 1, pp. 71- 78, 2003. (Reference 2) ”.

  In the method of the first embodiment, when a reference sentence with a low evaluation is included, there is a possibility that the system evaluation value will be high even if the system output sentence should originally have a low system evaluation value. For this reason, when the average value is acquired, if the evaluation value of the reference sentence falls below the threshold value, it may be excluded from the system evaluation value. The system evaluation value obtained by this method does not match the evaluation value assigned to the reference sentence and the scale cannot be compared.

  The second to fourth embodiments are methods using an evaluation value assigned to each reference sentence in addition to the reference sentence. Here, as for the evaluation value to be assigned to the reference sentence, a method of directly assigning a value manually or a method of manually evaluating which one is appropriate for each pair of reference sentences and assigning the winning percentage as an evaluation value can be considered. . The following three methods can be considered as a method for calculating the system evaluation value using this type of data.

  The second embodiment is a method of calculating a scale representing the similarity between sentences as in the first embodiment, and limiting the evaluation value only to a threshold value or higher when taking the average of the similarity. In the first embodiment, when a reference sentence with a low evaluation is included, there is a possibility that the system evaluation value will be high even if the system output sentence should have a low system evaluation value. This method is considered to be able to calculate the system evaluation value more appropriately, excluding such reference sentences.

  The third embodiment is a method of weighting and adding the evaluation values with the obtained similarity. At this time, instead of adding all, only the top N (N is a natural number of about 1 to 7) may be added. Since the evaluation value is directly used as compared with the second embodiment, it is expected that an appropriately low system evaluation value can be given particularly when the degree of similarity with a reference sentence having a low evaluation is large. Moreover, since the system evaluation value with respect to the obtained system output sentence corresponds with the evaluation value provided to the reference sentence, the scale can be compared.

  The fourth embodiment is a method for directly estimating a system evaluation value using a regression model such as Support Vector Regression (SVR). For more information on SVR, see “Smola, Alex J., and Bernhard Scholkopf.“ A tutorial on support vector regression ”, Statistics and computing, Vol. 14 (3), pp 199-222, 2004. (reference 3)” Please refer to. A regression model is a model that gives a pair of input features and output values (system evaluation values in this case) as correct answers in advance, saves the correspondences as parameters, and outputs corresponding to the input of unknown features. This is a method for estimating the value. As the feature amount of the regression model, a word included in a reference sentence or a system output sentence, or a similarity such as a BLEU score for each reference sentence can be considered.

  The fifth mode of realization is a method in which only winning or losing for each pair of reference sentences is used as an evaluation value. In this method, the winning or losing of each given reference sentence pair is estimated by a classification model such as Support Vector Machine (SVM), and the winning ratio for the reference sentence is calculated again to obtain a system evaluation value. For details about SVM, see "Cortes, Corinna, and Vladimir Vapnik," Support-vector networks ", Machine learning, vol. 20 (3), pp. 273-297, 1995. (reference 4). .

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the component which has the same function in drawing, and duplication description is abbreviate | omitted.

[First embodiment]
As shown in FIG. 1, the dialogue system evaluation device 1 according to the first embodiment includes, for example, a reference sentence database 10, an output sentence acquisition unit 11, an inter-sentence similarity calculation unit 12, and an evaluation value calculation unit 13.

  The dialogue system evaluation device is, for example, a special program configured by reading a special program into a known or dedicated computer having a central processing unit (CPU), a main storage device (RAM: Random Access Memory), and the like. Device. For example, the dialogue system evaluation apparatus executes each process under the control of the central processing unit. Data input to the dialogue system evaluation device and data obtained in each process are stored in, for example, the main storage device, and the data stored in the main storage device is read out as necessary and used for other processing. Is done. At least a part of each processing unit of the dialogue system evaluation apparatus may be configured by hardware such as an integrated circuit.

  Each storage unit included in the interactive system evaluation device includes, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device configured by a semiconductor memory element such as a hard disk, an optical disk, or a flash memory, or It can be configured with middleware such as a relational database or key-value store. Each storage unit included in the dialog system evaluation device may be logically divided, and may be stored in one physical storage device.

  The dialogue system evaluation apparatus 1 is configured to be accessible to an external dialogue system 9. The dialogue system 9 is a chat dialogue system that does not have a specific task. FIG. 1 shows an example in which the dialogue system evaluation device and the dialogue system are configured separately. However, the dialogue system evaluation device and the dialogue system may be configured as a single device that has both the functions that the dialogue system evaluation device and the dialogue system should have. I do not care.

  The reference sentence database 10 stores a reference sentence database including an input sentence and a plurality of reference sentences corresponding to each input sentence.

  Hereinafter, a method for creating a reference sentence database will be described.

  First, an arbitrary input sentence is prepared. The input sentence may be created by manually describing it, or it may be a transcript of an actual dialogue, or extracted from an article published on a web service such as Twitter (registered trademark) or a blog. May be.

  Next, a reference sentence for each input sentence is created. The reference text can be created by hand. The reference sentence is a sentence created to be correct with respect to the input sentence (hereinafter referred to as the correct example), but it is incorrect by hiding a part of the input sentence for the purpose of expanding the coverage of the reference sentence. A negative example such as a sentence created as described above or a sentence automatically extracted from a corpus may be added. However, even if one of them is excluded, the dialogue system evaluation apparatus 1 operates. Here, it is not necessary to attach a label indicating whether the reference sentence is a positive example or a negative example. A set of collected input sentences and a reference sentence set corresponding to each input sentence is stored in the reference sentence database 10.

  With reference to FIG. 2, the interactive system evaluation method of the first embodiment will be described.

  In step S <b> 11, the output sentence acquisition unit 11 inputs the input sentence acquired from the reference sentence database 10 to the dialog system 9 and obtains a system output sentence from the dialog system 9. A system output sentence obtained from the dialogue system 9 and an input sentence and reference sentence set obtained from the reference sentence database 10 are sent to the inter-sentence similarity calculation unit 12.

  In step S12, the inter-sentence similarity calculation unit 12 calculates the similarity between each reference sentence included in the reference sentence set corresponding to the input sentence and the system output sentence. This similarity may be a general similarity such as cosine similarity or word error rate (WER) weighted by tf-idf, or a word such as BLEU score or ROUGE score. You may use the similarity which considered the combination of. In addition, it is possible to use the words included in each sentence as they are, such as “Supervised by NTT Communication Science Laboratories, edited by Ikehara et al.,“ Japanese Vocabulary System ”, Iwanami Shoten (Reference 5)”. The word concept may be abstracted using a dictionary, and the similarity may be calculated. The obtained similarity, the system output sentence, and the combination of the input sentence and the reference sentence set are sent to the evaluation value calculation unit 13.

  In step S13, the evaluation value calculation unit 13 calculates a system evaluation value based on the system output sentence, the combination of the input sentence and the reference sentence set, and all or part of the similarity of the reference sentence. Specifically, the top N similarity sentences (N is a natural number of about 1 to 7) are selected for each system output sentence, and an average value of the N similarity degrees is calculated as a system evaluation value.

[Second Embodiment]
As shown in FIG. 3, the dialogue system evaluation apparatus 2 of the second embodiment includes an output sentence acquisition unit 11 and an inter-sentence similarity calculation unit 12 as in the first embodiment, and includes a reference sentence database 20 and an evaluation value calculation. Furthermore, the unit 23 is further included.

  The dialogue system evaluation device 2 is configured to be accessible to an external dialogue system 9 as in the first embodiment.

  The reference sentence database 20 stores a reference sentence database including an input sentence, a plurality of reference sentences corresponding to each input sentence, and an evaluation value corresponding to each reference sentence. That is, the difference from the reference sentence database 10 is that an evaluation value is assigned to each reference sentence.

  Hereinafter, a method for creating a reference sentence database according to the second embodiment will be described. Since the method for creating the input sentence and the reference sentence is the same as in the first embodiment, the description thereof is omitted here.

  As a method for assigning an evaluation value, for example, a method of directly assigning a value manually, a method of comparing which is better for each pair of reference sentences, and using the winning percentage as an evaluation value can be used. In the latter case, it is not necessary to save the winning or losing of each pair. The obtained evaluation value is stored in the reference sentence database 20 as a pair with the input sentence and the reference sentence set.

  With reference to FIG. 4, the interactive system evaluation method of 2nd embodiment is demonstrated. Below, it demonstrates centering on difference with the above-mentioned 1st embodiment.

  In step S23, the evaluation value calculation unit 23 calculates a system evaluation value based on a system output sentence, a set of an input sentence, a reference sentence set, an evaluation value set, and all or part of the similarity of the reference sentence. . Specifically, by excluding reference sentences whose evaluation values are equal to or less than a predetermined threshold, for each system output sentence, a reference sentence having the highest N similarity (N is a natural number of about 1 to 7) is selected, and N A system evaluation value is calculated by calculating the average of individual similarities or evaluation values.

[Third embodiment]
As shown in FIG. 3, the dialogue system evaluation device 3 of the third embodiment includes a reference sentence database 20, an output sentence acquisition unit 11, and an inter-sentence similarity calculation unit 12 as in the second embodiment, and calculates an evaluation value. A portion 33 is further included.

  The dialogue system evaluation apparatus 3 is configured to be accessible to an external dialogue system 9 as in the above-described embodiment.

  With reference to FIG. 4, the interactive system evaluation method of 3rd embodiment is demonstrated. Below, it demonstrates centering on difference with the above-mentioned 2nd embodiment.

  In step S33, the evaluation value calculation unit 33 calculates a system evaluation value based on a system output sentence, a set of an input sentence, a reference sentence set, an evaluation value set, and all or part of the similarity of the reference sentence. . Specifically, by excluding reference sentences whose evaluation values are equal to or lower than a predetermined threshold, for each system output sentence, a reference sentence having the highest N similarity (N is a natural number of about 1 to 7) is selected, and each An average value obtained by weighting the evaluation value of the reference sentence by the similarity is calculated and used as a system evaluation value.

[Fourth embodiment]
As shown in FIG. 5, the dialogue system evaluation device 4 of the fourth embodiment includes a reference sentence database 20, an output sentence acquisition unit 11, and an inter-sentence similarity calculation unit 12 as in the third embodiment, and stores learning data. Further included is a unit 40, a regression model learning unit 41, a regression model parameter storage unit 42, a feature amount extraction unit 43, and an evaluation value calculation unit 44.

  The dialog system evaluation device 4 is configured to be accessible to an external dialog system 9 as in the above-described embodiment.

  The learning data storage unit 40 stores the feature amount of each reference sentence stored in the reference sentence database 20 and the evaluation value given to each reference sentence in association with each other. As the feature amount, a word included in a reference sentence or a system output sentence, or a similarity such as a BLEU score for each reference sentence can be used.

  The regression model parameter storage unit 42 stores regression model parameters. As the parameters of the regression model, a set of feature values and evaluation values stored in the learning data storage unit 40 is input to the regression model learning unit 41, and when a certain feature value is input, a corresponding evaluation value is output. adjust. As the regression model, for example, the above-described SVR can be used.

  With reference to FIG. 6, the interactive system evaluation method of the fourth embodiment will be described. Below, it demonstrates centering on difference with the above-mentioned 3rd embodiment.

  In step S <b> 43, the feature amount extraction unit 43 extracts a feature amount from the system output sentence of the dialogue system 9. The feature amount to be extracted is the same as the feature amount of the reference sentence stored in the learning data storage unit 40. The extracted feature amount is sent to the evaluation value calculation unit 44.

  In step S44, the evaluation value calculation unit 44 uses the regression model parameters acquired from the regression model parameter storage unit 42 to predict an evaluation value for the feature quantity of the system output sentence, and sets it as a system evaluation value.

[Fifth embodiment]
As shown in FIG. 7, the dialogue system evaluation apparatus 5 of the fifth embodiment includes an output sentence acquisition unit 11, an inter-sentence similarity calculation unit 12, a learning data storage unit 30, and a feature amount extraction unit 43 as compared with the fourth embodiment. Similarly, a reference sentence database 50, a classification model learning unit 51, a classification model parameter storage unit 52, and an evaluation value calculation unit 54 are further included.

  The dialogue system evaluation apparatus 5 is configured to be accessible to an external dialogue system 9 as in the above-described embodiment.

  The reference sentence database 50 stores a reference sentence database including an input sentence, a plurality of reference sentences corresponding to each input sentence, and an evaluation value corresponding to each reference sentence. However, the evaluation values of the fifth embodiment are different in calculation method. The evaluation values of the fifth embodiment are limited to those given to the reference sentence pair by manually determining which is the more appropriate response as a win or a loss.

  The classification model parameter storage unit 52 stores classification model parameters. As the parameters of the classification model, a feature value and an evaluation value (for example, winning: 1, losing: 0, etc.) for each reference sentence pair are input to the classification model learning unit 51, and the corresponding winning and losing when two certain feature values are input. Adjust to output an evaluation value indicating. As the classification model, for example, the above-described SVM can be used.

  With reference to FIG. 8, the dialogue system evaluation method of the fifth embodiment will be described. Below, it demonstrates centering around difference with the above-mentioned 4th embodiment.

  In step S54, the evaluation value calculation unit 54 uses the classification model parameters acquired from the classification model parameter storage unit 52 to predict an evaluation value indicating a win or loss from the feature amount of the system output sentence and the feature amount of each reference sentence. The predicted winning / losing win rate is calculated as a system evaluation value.

  As described above, according to the dialogue system evaluation apparatus and method of the present invention, a reference sentence is prepared on a large scale, and the evaluation value is used together. A system evaluation value can be appropriately assigned even to a chat dialogue system having a wide variety of conversations. Since the interactive system can be evaluated at high speed and at low cost, the interactive system can be improved efficiently.

  The present invention is not limited to the above-described embodiment, and it goes without saying that modifications can be made as appropriate without departing from the spirit of the present invention. The various processes described in the above embodiment may be executed not only in time series according to the order of description, but also in parallel or individually as required by the processing capability of the apparatus that executes the processes or as necessary.

[Program, recording medium]
When various processing functions in each device described in the above embodiment are realized by a computer, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, the present apparatus is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

1, 2, 3, 4, 5 Dialog system evaluation device 9 Dialog system 10, 20, 50 Reference sentence database 11 Output sentence acquisition unit 12 Inter-sentence similarity calculation unit 13, 23, 33, 44, 54 Evaluation value calculation unit 40 Learning data storage unit 41 Regression model learning unit 42 Regression model parameter storage unit 43 Feature quantity extraction unit 51 Classification model learning unit 52 Classification model parameter storage unit

Claims (7)

In the reference sentence database, an input sentence, a plurality of reference sentences predetermined for each of the input sentences, and an evaluation value assigned to each of the reference sentences are stored,
An output sentence acquisition step, wherein the output sentence acquisition unit inputs the input sentence to a dialog system having no specific task, and obtains a system output sentence from the dialog system;
An inter-sentence similarity calculation unit calculates an inter-sentence similarity between the system output sentence and each of the reference sentences;
A system evaluation in which the evaluation value calculator selects the reference sentence based on the evaluation value and the similarity, and evaluates the system output sentence using at least one of the similarity or evaluation value of the selected reference sentence An evaluation value calculating step for calculating a value;
Dialog system evaluation method including In the reference sentence database, an input sentence, a plurality of reference sentences predetermined for each of the input sentences, and an evaluation value assigned to each of the reference sentences are stored,
The regression model parameter storage unit stores the parameters of the regression model that outputs the evaluation value corresponding to the feature value when the feature value is learned using the feature value extracted from the reference sentence and the evaluation value. And
An output sentence acquisition step, wherein the output sentence acquisition unit inputs the input sentence to a dialog system having no specific task, and obtains a system output sentence from the dialog system;
The evaluation value calculation unit inputs the feature amount extracted from the system output sentence to the regression model, and uses the evaluation value output from the regression model as a system evaluation value for evaluating the system output sentence. When,
Dialog system evaluation method including In the reference sentence database, an input sentence, a plurality of reference sentences predetermined for each of the input sentences, and an evaluation value representing a win or loss representing an appropriate one for each pair of the reference sentences are stored,
The classification model parameter storage unit stores the parameters of the classification model that outputs the winning or losing when the two feature quantities learned using the feature quantity extracted from the reference sentence and the evaluation value are input,
An output sentence acquisition step, wherein the output sentence acquisition unit inputs the input sentence to a dialog system having no specific task, and obtains a system output sentence from the dialog system;
The evaluation value calculation unit inputs the feature quantity extracted from the system output sentence and the feature quantity extracted from the reference sentence to the classification model, and calculates the winning percentage calculated from the winning or losing output from the classification model. An evaluation value calculation step which is a system evaluation value for evaluating a sentence;
Dialog system evaluation method including A reference sentence database that stores an input sentence, a plurality of reference sentences predetermined for each of the input sentences, and an evaluation value assigned to each of the reference sentences;
An input sentence is input to an interactive system that does not have a specific task, and an output sentence acquisition unit that obtains a system output sentence from the interactive system;
A sentence similarity calculator that calculates the similarity between sentences between the system output sentence and the reference sentence;
Evaluation value for selecting a reference sentence based on the evaluation value and the similarity, and calculating a system evaluation value for evaluating the system output sentence using at least one of the similarity or evaluation value of the selected reference sentence A calculation unit;
An interactive system evaluation device. A reference sentence database that stores an input sentence, a plurality of reference sentences predetermined for each of the input sentences, and an evaluation value assigned to each of the reference sentences;
A regression model parameter storage unit that stores parameters of a regression model that outputs an evaluation value corresponding to the feature value when the feature value is input, learned using the feature value extracted from the reference sentence and the evaluation value;
An input sentence acquisition unit that inputs the input sentence to a dialog system having no specific task and obtains a system output sentence from the dialog system;
An evaluation value calculation unit that inputs the feature amount extracted from the system output sentence to the regression model and sets the evaluation value output from the regression model as a system evaluation value for evaluating the system output sentence;
An interactive system evaluation device. A reference sentence database for storing an input sentence, a plurality of reference sentences predetermined for each of the input sentences, and an evaluation value representing a win or loss representing an appropriate one for each pair of the reference sentences;
A classification model parameter storage unit that stores parameters of a classification model that outputs the winning or losing when inputting two feature quantities learned using the feature quantity extracted from the reference sentence and the evaluation value;
An input sentence acquisition unit that inputs the input sentence to a dialog system having no specific task and obtains a system output sentence from the dialog system;
A system evaluation that evaluates the system output sentence by inputting the feature quantity extracted from the system output sentence and the feature quantity extracted from the reference sentence to the classification model, and calculating the winning percentage calculated from the winning or losing output from the classification model. An evaluation value calculation unit as a value,
An interactive system evaluation device.   The program for making a computer perform each step of the dialog system evaluation method in any one of Claim 1 to 3.

Images