JP7160571B2 - Evaluation device, evaluation method and evaluation program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Law Proceedings https://www2018. the webconf. org/proceedings/ Submitted paper 1 https://dl. acm. org/citation. cfm? id=3185992 TheWebConference2018 https://www2018. the webconf. org/ The 24th Annual Conference of the Association for Natural Language Processing (NLP2018) http://www. anlp. jp/nlp2018/ Submission paper 2

The present invention relates to an evaluation device, an evaluation method, and an evaluation program.

2. Description of the Related Art Conventionally, a dialogue system is known that receives a voice or text message from a user and provides the user with a message corresponding to the received message. In such a dialogue system, a user sends messages (hereinafter collectively referred to as "responses") with various contents corresponding to messages received from users (hereinafter collectively referred to as "utterances"). The technology provided for is applied. Also, in such a dialogue system, a technique is known that evaluates the dialogue based on the flow of utterances and responses.

JP 2011-253053 A

However, it may not be possible to say that the above-described conventional technology appropriately evaluates dialogue.

For example, in the conventional technology described above, in order to estimate the degree of satisfaction for each utterance, the state of the dialogue is estimated from the flow of the dialogue, and the evaluation is performed according to the estimated state. Therefore, it may not always be possible to precisely determine whether the user is satisfied with the entire dialogue.

The present application has been made in view of the above, and an object thereof is to improve the accuracy of evaluation of dialogue.

An evaluation device according to the present application includes an acquisition unit that acquires a session including a user's utterance and a response to the utterance in an interactive service, and the acquisition unit that acquires the and an evaluation unit for evaluating the session.

According to one aspect of the embodiments, the accuracy of evaluation for interactions can be improved.

FIG. 1 is a diagram illustrating an example of processing executed by an information providing device according to an embodiment. FIG. 2 is a diagram illustrating a configuration example of an information providing device according to the embodiment; FIG. 3 is a diagram illustrating an example of information registered in a session log database according to the embodiment; FIG. 4 is a diagram illustrating an example of information registered in a session evaluation database according to the embodiment; FIG. 5 is a diagram showing an example of information registered in an indicator utterance database according to the embodiment. FIG. 6 is a diagram illustrating an example of information registered in an engagement database according to the embodiment; FIG. 7 is a diagram illustrating an example of information registered in a model database according to the embodiment; FIG. 8 is a diagram showing the concept of processing for extracting new index utterances from seed index utterances. FIG. 9 is a diagram showing the concept of processing for extracting new index utterances based on usage patterns in interactive services. FIG. 10 is a diagram showing the concept of processing for extracting a new index utterance based on the session evaluation results. 11 is a diagram illustrating an example of a functional configuration of an indicator utterance extraction processing unit according to the embodiment; FIG. FIG. 12 is a diagram showing the concept of processing for evaluating a session based on index utterances. FIG. 13 is a diagram showing the concept of processing for evaluating a session based on index utterances. FIG. 14 is a diagram showing an example of processing for evaluating a session using a ternary evaluation model that has learned features of three types of dialogue. FIG. 15 is a diagram illustrating an example of processing for evaluating sessions based on engagement. FIG. 16 is a diagram showing the concept of processing for evaluating a session based on repeated utterances. FIG. 17 is a diagram showing the concept of processing for evaluating a session using a model that has learned features of automatically generated data. FIG. 18 is a diagram showing the concept of evaluation of repeated utterances using images. FIG. 19 is a diagram illustrating the concept of processing for evaluating sessions based on apology responses. FIG. 20 is a diagram showing the concept of processing for evaluating a session by combining multiple evaluation methods. FIG. 21 is a diagram showing the concept of processing for automatically extending a labeled session. 22 is a diagram illustrating an example of a functional configuration of a session evaluation processing unit according to the embodiment; FIG. FIG. 23 is a diagram showing the concept of processing for predicting future usage patterns based on indicator utterances. FIG. 24 is a diagram showing the concept of processing for predicting future usage patterns based on evaluation results. 25 is a diagram illustrating an example of a functional configuration of an engagement prediction processing unit according to the embodiment; FIG. FIG. 26 is a diagram illustrating an example of relationships between processes executed by the information providing apparatus according to the embodiment; FIG. 27 is a diagram illustrating the concept of processing for executing reinforcement learning of a dialogue model. 28 is a diagram illustrating an example of a functional configuration of a reinforcement learning processing unit according to the embodiment; FIG. FIG. 29 is a flowchart illustrating an example of the flow of processing executed by the information providing device according to the embodiment based on the usage mode; FIG. 30 is a flowchart showing an example of the flow of processing for extracting index utterances by the information providing device according to the embodiment. 31 is a flowchart illustrating an example of the flow of processing for evaluating a session by the information providing device according to the embodiment; FIG. 32 is a flowchart illustrating an example of the flow of processing for evaluating a session based on an image by the information providing device according to the embodiment; FIG. FIG. 33 is a flowchart illustrating an example of the flow of processing in which the information providing device according to the embodiment evaluates a session based on repeated utterances; FIG. 34 is a flowchart illustrating an example of the flow of reinforcement learning in which the information providing device according to the embodiment executes utterances as rewards. 35 is a flowchart illustrating an example of a flow of processing for extracting indicator utterances based on co-occurrence by the information providing device according to the embodiment; FIG. FIG. 36 is a flowchart illustrating an example of the flow of processing in which the information providing device according to the embodiment evaluates a session using a ternary evaluation model; 37 is a flowchart illustrating an example of the flow of processing in which the information providing device according to the embodiment evaluates sessions using a plurality of models; FIG. FIG. 38 is a diagram illustrating an example of a hardware configuration; FIG. 39 is a diagram showing an example of labeling results for session data. FIG. 40 shows the results of the first experiment. FIG. 41 shows the results of the second experiment. FIG. 42 is a diagram showing the results of Prop_D in the third experiment. FIG. 43 is a diagram showing the results of Prop_S in the fourth experiment. FIG. 44 shows the results of Prop_E in the fifth experiment.

Embodiments (hereinafter referred to as "embodiments") for implementing the evaluation apparatus, evaluation method, and evaluation program according to the present application will be described in detail below with reference to the drawings. Note that the evaluation device, evaluation method, and evaluation program according to the present application are not limited to this embodiment. Further, in each of the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[Embodiment]
[1. Overview of the information providing device]
First, with reference to FIG. 1, an example of processing executed by an information providing device, which is an example of an extraction device, an evaluation device, and a learning device, will be described. FIG. 1 is a diagram illustrating an example of processing executed by an information providing device according to an embodiment.

In FIG. 1, the information providing device 10 can communicate with the interactive device 200 via a predetermined network N such as the Internet (see FIG. 2, for example). The information providing device 10 may be capable of communicating with the terminal device 100 used by the user. In addition to the terminal device 100 and the dialogue device 200, the information providing device 10 is capable of mutual communication with various external servers.

The terminal device 100 is a smart device such as a smartphone or a tablet, and is a mobile terminal device capable of communicating with an arbitrary server device via a wireless communication network such as 3G (3rd Generation) or LTE (Long Term Evolution). be. The terminal device 100 may be not only a smart device but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC. Note that when the terminal device 100 acquires a voice (so-called “utterance”) generated by the user, it transmits the acquired utterance voice to the dialogue device 200, and when it acquires a response transmitted from the dialogue device 200, the acquired Any device such as a smart speaker can be used as long as it outputs a response as voice.

The dialogue device 200 is an information processing device that executes dialogue processing for realizing dialogue with a user, and is realized by, for example, a server device, a cloud system, or the like. For example, upon receiving an utterance from the terminal device 100 , the dialogue device 200 generates a response according to the received utterance, and transmits data for outputting the generated response as voice to the terminal device 100 .

Here, an example of interactive processing realized by the interactive device 200 will be described. Note that the interactive processing described below is merely an example and does not limit the embodiments. For example, the terminal device 100 acquires the voice uttered by the user as an utterance (step S1). As a more specific example, the terminal device 100 acquires voice data of speech using various voice extraction techniques such as beamforming. In such a case, the terminal device 100 transmits the acquired voice data to a predetermined conversion server (not shown). In such a case, the conversion server generates speech text data from the voice data and transmits the generated text data to the terminal device 100 . Then, the terminal device 100 transmits the text data as an utterance to the dialogue device 200 (step S2).

On the other hand, the dialogue device 200 analyzes the text data received as an utterance using various intention analysis techniques, and estimates the user's intention indicated by the utterance. The interactive device 200 then executes various processes according to the estimated intention, and generates text data indicating the execution results. That is, the interactive device 200 generates a response (step S3). The interactive device 200 then transmits the text data generated as a response to the terminal device 100 (step S4). As a result, the terminal device 100 provides the user with a response to the utterance by reading out the text data or displaying the text data on the screen (step S5).

In the interactive process realized by the interactive device 200, it is important to evaluate whether the user is satisfied with the content of the response. For example, information indicating whether or not the user is satisfied with a certain response can be used as information for improving the accuracy of interactive processing executed by the interactive device 200 . However, asking the user whether they were satisfied with the response may damage the user experience.

Here, in order to appropriately evaluate whether or not the user is satisfied with the dialogue, we prepare labeled dialogue data with a label indicating whether the user is satisfied or not, and apply it to various models. A conceivable method is to learn the features of labeled dialogue data by supervised or unsupervised learning, and evaluate the dialogue using the trained model. However, learning such a model requires a relatively large amount of labeled dialogue data, but creating labeled dialogue data is time-consuming.

For example, creating data for labeled interactions may be created manually, such as by crowdsourcing. However, since some dialogues include many utterances and responses (that is, long dialogues), it may be difficult to assign labels manually. Therefore, if it is possible to automatically acquire labeled dialogue data used for learning, learning can be efficiently performed, and the accuracy of dialogue processing can be improved.

Therefore, the information providing apparatus 10 uses the user's utterances and responses in the dialogue processing to perform evaluation processing for automatically evaluating whether or not the user is satisfied with the dialogue processing. The information providing device 10 also executes an extraction process for extracting an utterance used when performing the evaluation process. The information providing device 10 also executes a learning device that learns a model for dialogue. Then, the information providing device 10 improves the accuracy of dialogue processing by the dialogue device 200 based on the extraction device, the evaluation device, and the learning device.

[1-1. Example of processing executed by information providing device]
An example of processing executed by the information providing apparatus 10 will be described below. First, the information providing device 10 acquires a dialogue history, which is a history of utterances and responses in dialogue processing, from the dialogue device 200 (step S6). Then, the information providing device 10 evaluates the dialogue based on the dialogue history (step S7).

More specifically, the information providing apparatus 10 stores, as a session, a dialogue history including a series of multiple utterances and multiple responses, such as a user's utterance, a response to the utterance, and a user's utterance to the response. Extract. Here, the information providing apparatus 10 may employ a plurality of utterances and responses defined by arbitrary criteria as one session.

For example, when there is no utterance from the user for a predetermined period of time (for example, 30 minutes), the information providing apparatus 10 changes the utterance received from the user and the response provided to the user to 1 may be utterances and responses contained in one session. In addition, the information providing apparatus 10 may treat as one session the utterances and responses obtained after the user activates an application for interactive processing until the application is put to sleep or terminated. In other words, a session may include utterances and responses organized in arbitrary units as long as it is a series of dialogues that can be evaluated by the user.

The information providing device 10 then evaluates the dialogue in the session. More specifically, the information providing device 10 estimates whether or not the user is satisfied with the dialogue in the session, based on the responses and utterances in the session. For example, the information providing apparatus 10 evaluates a session using the evaluation results of a plurality of models that evaluate a session based on different types of features from among a plurality of features that serve as indicators of user evaluation of a session. conduct.

For example, when utterances included in one session include utterances of the same or similar content, it is presumed that the user is not satisfied with the dialogue in that session. For example, when the user intends to look for a job at an inn and utters "work at an inn", the dialogue device 200 erroneously determines that the user intends to search for an accommodation facility from the word "inn". , may return a response such as "I will guide you to the accommodation facility". When such an unintentional response is output, the user presumes that the speech recognition did not go well, and utters again "work at the inn", and the dialogue device 200 again says "I will guide you to the accommodation facility". It may return a response. Thus, repetition of similar utterances in one session is considered to suggest that the user is not very satisfied with the session.

Therefore, the information providing device 10 generates a model for evaluating sessions based on repetition of the same or similar utterances (step S7-1). For example, when an utterance or a character string of a response included in a session is input, the information providing apparatus 10 determines how dissatisfied the user is with the session based on the repetition of the utterance included in the session. That is, it generates a repeated evaluation model M1 that outputs a first degree of dissatisfaction, which is a score indicating the degree of dissatisfaction of the user. For example, in such a repeated evaluation model, when a session containing the same utterance (or the same character string) or a session containing a similar utterance (or a similar character string) is input, the same utterance (or the same character string) column) or a session that does not contain similar utterances (or similar character strings) is input, a higher value of the first dissatisfaction level is output. Here, "similar utterances" are utterances with similar meanings, such as "work at an inn" and "work at a hotel." Note that the repeated evaluation model M1 evaluates a session based on features of an image generated from a character string rather than the character string itself of the session. Details of such processing will be described later.

Also, for example, an apology response such as "I'm sorry" or "I'm sorry" may be provided in response to an utterance such as "No". If a session includes such an apology response (hereinafter sometimes collectively referred to as an "apology response"), it is presumed that the user is not very satisfied with the session. Therefore, the information providing device 10 generates a model for evaluating a session based on whether or not an apology response is included (step S7-2). For example, the information providing apparatus 10 outputs a higher value of the second dissatisfaction level for sessions containing more apology responses, and outputs a lower second dissatisfaction level for sessions containing less apology responses. Generate an apology evaluation model M2 that outputs a second degree of dissatisfaction of the value.

In addition, user utterances include utterances suggesting the user's impression of the session. can be adopted as For example, a user's utterance such as "cannot use" or "uninstall" is considered to be an utterance indicating the user's unfavorable evaluation of the response, that is, unfavorable feedback. It is presumed that a user's satisfaction level is low in a session that includes many utterances indicating such unfavorable evaluation (hereinafter referred to as "unfavorable indicator utterances"). In addition, the user's utterances such as "wonderful" and "well done" are considered to be utterances indicating the user's favorable evaluation of the response, that is, favorable feedback. A session that includes many utterances indicating such favorable evaluation (hereinafter referred to as "favorable index utterances") is presumed to have high user satisfaction.

That is, such favorable indicator utterances and unfavorable indicator utterances (hereinafter sometimes collectively referred to as "index utterances") can serve as indicators of the user's degree of satisfaction with the session. Therefore, the information providing apparatus 10 generates a model for evaluating a session based on various index utterances such as favorable index utterances and unfavorable index utterances.

For example, the information providing device 10 generates a model for evaluating sessions based on unfavorable indicator utterances (step S7-3). For example, the information providing apparatus 10 assigns a higher value to a session in which a larger number of preset unfavorable indicator utterances such as "unusable" and "uninstall" are included in the utterances included in the session. An unfavorable indicator utterance evaluation model M3 is generated that outputs a third dissatisfaction level and outputs a lower value of the third dissatisfaction level for a session that contains less unfavorable indicator utterances.

The information providing device 10 also generates a model for evaluating the session based on the favorable indicator utterances (step S7-4). For example, the information providing apparatus 10 assigns a higher value to a session in which a greater number of preset favorable indicator utterances such as "wow" and "well done" are included in the utterances included in the session. , and a favorable indicator utterance evaluation model M4 that outputs a lower value of the first satisfaction level for a session containing fewer favorable indicator utterances. Note that the unfavorable indicator utterances and the favorable indicator utterances described above may be utterances automatically extracted from the session as appropriate by a process described later.

Note that the information providing device 10 may learn each model using any known technique. For example, the information providing device 10 may learn each model using techniques such as back propagation, reinforcement learning, and GAN (Generative Adversarial Network).

Then, the information providing apparatus 10 evaluates the input session based on each dissatisfaction level and satisfaction level output by each model (step S7-5). For example, the information providing device 10 extracts a session to be evaluated, and repeats the extracted session for each of the evaluation model M1, the apology evaluation model M2, the unfavorable index utterance evaluation model M3, and the favorable index utterance evaluation model M4. to enter. Then, the information providing apparatus 10 calculates a dissatisfaction score by subtracting the first dissatisfaction from the sum of the first dissatisfaction, the second dissatisfaction, and the third dissatisfaction. That is, the information providing apparatus 10 calculates a dissatisfaction score that increases as the user is less satisfied with the session. Note that "dissatisfied" means not only the case where the user is frustrated with the response in the session, but also the case where the user is flabbergasted. It is a concept that includes any state that has

Further, the information providing apparatus 10 calculates a dissatisfaction score from each session extracted from the dialogue history. Then, the information providing apparatus 10 extracts a predetermined number of sessions from among the sessions with the highest dissatisfaction score, and sets the extracted sessions as low-satisfaction sessions, which are sessions with a low user's satisfaction. In addition, the information providing apparatus 10 extracts a predetermined number of sessions in descending order of dissatisfaction score, and sets the extracted sessions as high-satisfaction sessions, which are sessions with high user satisfaction. Note that the information providing apparatus 10 may treat a session with a dissatisfaction score exceeding a predetermined threshold as a low-satisfaction session, and a session with a dissatisfaction score below a predetermined threshold as a high-satisfaction session. By executing such processing, the information providing apparatus 10 realizes processing for evaluating whether or not the user is satisfied with the session.

Then, the information providing device 10 updates the dialogue model based on the evaluation result (step S8). For example, when an utterance and a response to the utterance are included in a low-satisfaction session, the information providing apparatus 10 reduces the probability of outputting the response to the utterance, If included in a high-satisfaction session, update the dialogue model for generating responses to increase the probability of outputting that response to that utterance. That is, the information providing device 10 updates the interaction model so that the user is more satisfied.

In this way, the information providing apparatus 10 acquires a session including the user's utterances and responses in the interactive service, and based on different types of features among a plurality of features that serve as indicators of the user's evaluation of the session, A session is evaluated using evaluation results of a plurality of models M1 to M4 for evaluating a session. As a result, the information providing apparatus 10 comprehensively evaluates the session from a plurality of viewpoints, so it is possible to improve the accuracy of the evaluation. In addition, the information providing apparatus 10 can automatically evaluate the session without manually evaluating the session, such as crowdsourcing, so that the evaluation of the dialogue can be facilitated.

[1-2. Regarding the processing executed by the information providing device]
In the above-described example, as an example of the processing executed by the information providing apparatus 10, depending on whether or not the utterance included in the session is repeated, whether or not there is an apology response, and whether or not there is a favorable indicator utterance or an unfavorable indicator utterance, An example of processing for evaluating a session has been described. However, embodiments are not so limited.

For example, as described below, the information providing device 10 may evaluate sessions from various viewpoints. Further, the information providing apparatus 10 may automatically extract various index utterances from the utterances included in the session in order to evaluate such a session. Further, the information providing apparatus 10 may estimate the period and frequency (hereinafter sometimes referred to as "engagement") of the user's use of the interactive service in order to evaluate such a session. Further, the information providing apparatus 10 may learn a dialogue model for generating a response from an utterance based on the history of dialogue.

In other words, the information providing apparatus 10 may implement the session evaluation shown in FIG. 1 by combining various types of processing described below. Various types of processing described below may be performed in any combination.

[1-3. User's subjectivity]
In the above description, the information providing apparatus 10 evaluated the session using various indicator utterances that indicate the user's favorable or unfavorable impression of the response. Here, "favorable impression" and "unfavorable impression" refer not only to the user's impression of whether the user likes or dislikes it, but also to the user's positive impression of the response, such as the impression that the response is correct or incorrect. It is a concept that includes various negative impressions as well as various negative impressions.

Also, in the above description, the information providing apparatus 10 evaluated whether or not the user was satisfied with the session. Here, the information providing device 10 may perform not only a binary evaluation as to whether or not the customer is satisfied, but also an arbitrary form of evaluation. For example, the information providing device 10 may use the dissatisfaction score itself as the evaluation result. In addition, the information providing apparatus 10 was trained to calculate a score, not dissatisfaction, in which the higher the user's satisfaction, the higher the value, and the lower the user's satisfaction, the lower the value, that is, the satisfaction. Models M1-M3 may be utilized. Further, the information providing apparatus 10 may calculate, for each session, a satisfaction score indicating the degree of satisfaction of the user instead of the dissatisfaction score.

In the above description, the information providing apparatus 10 evaluated whether the user was "satisfied" or "unsatisfied" with the session. Here, the concepts of "satisfaction" and "dissatisfaction" include various favorable or unfavorable impressions of the session by the user, such as whether or not the user is irritated by the session. is.

In other words, the above descriptions such as "favorable" and "satisfactory" indicate various favorable impressions by users, and descriptions such as "unfavorable" and "unsatisfactory" indicate unfavorable It shows various impressions. If the session is evaluated based on various index utterances that indicate the user's impression of the response, the information providing apparatus 10 can perform arbitrary It may be possible to realize an evaluation centered on the impression of

[1-4. Evaluation based on index utterance]
In the above description, the information providing apparatus 10 evaluates whether or not the user is satisfied with the session based on various indicator utterances such as favorable indicator utterances and unfavorable indicator utterances. Here, the various indicator utterances are utterances that indicate whether or not the user has a favorable impression of the target, such as the dialogue and response until the indicator utterance appears, and are indicators of the evaluation of the session. It is an utterance that can be

Here, the index utterance can be an utterance that indicates the user's impression of any target as long as the utterance can serve as an index for session evaluation, regardless of the target of the user's impression indicated by the index utterance. For example, the information providing apparatus 10 may use an utterance indicating the user's impression of a single response (that is, an utterance serving as feedback) as the index utterance, and may use an utterance indicating the user's impression of the series of dialogues or the entire session. You may employ|adopt as an index utterance. That is, if an utterance indicating the user's evaluation of the interactive service in response to the response is used as the index utterance, the information providing apparatus 10 adopts the utterance indicating the user's impression of an arbitrary object in the interactive service as the index utterance. good too.

[2. Configuration of Information Providing Device]
An example of the functional configuration of the information providing device 10 that implements various processes executed by the information providing device 10 will be described below. FIG. 2 is a diagram illustrating a configuration example of an information providing device according to the embodiment; As shown in FIG. 2, the information providing device 10 has a communication section 20, a storage section 30, and a control section 40. FIG.

[2-1. About the communication part]
The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 20 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the terminal device 100 and the interactive device 200 .

[2-2. About the memory part]
The storage unit 30 is implemented by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 30 also stores a session log database 31 , a session evaluation database 32 , an indicator utterance database 33 , an engagement database 34 and a model database 35 .

An example of each of the databases 31 to 35 stored in the storage unit 30 will be described below with reference to FIGS. 3 to 7. FIG. A dialogue history is registered in the session log database 31 for each session. For example, FIG. 3 is a diagram showing an example of information registered in the session log database according to the embodiment. As shown in FIG. 3, in the session log database 31, information having items such as "session ID (identifier)", "interaction time", "user ID", and "conversation content" is registered. Note that any information other than the information shown in FIG. 3 may be registered in the session log database 31 as long as it is information related to utterances and responses in interactive processing.

Here, "session ID" is information for identifying a session. Further, the “dialogue time” is information indicating the date and time when the dialogue included in the session indicated by the “session ID” took place, for example, information indicating the date and time when the first utterance included in the session was obtained. . A “user ID” is an identifier that identifies a user who has made an utterance included in the session indicated by the “session ID”. "Dialogue content" is information in which the content of utterances and responses included in the session indicated by the "session ID" is arranged in chronological order.

For example, in the example shown in FIG. 3, the session ID "session #1", the dialogue time "dialogue time #1", the user ID "user #1", the dialogue content "Usr: work at the inn, sys: information on accommodation facilities I will... (omitted)..." is registered. Such information is such that the date and time when the dialogue included in the session indicated by the session ID “session #1” was performed is “dialogue time #1”, and the user who spoke in the session was “user #1”. Indicates that you are the user who indicates Also, the session indicated by the session ID "session #1" includes the user's utterance "work at the inn", indicating that the response to such utterance was "I will guide you to the accommodation facility".

Note that in the example shown in FIG. 3, conceptual values such as "session #1", "interaction time #1", and "user #1" are described, but in reality, character strings and numerical values that identify sessions are used. , numerical values indicating date and time, character strings and numerical values identifying users, and the like are registered in the session log database 31 .

Evaluation results for sessions are registered in the session evaluation database 32 . For example, FIG. 4 is a diagram showing an example of information registered in the session evaluation database according to the embodiment. As shown in FIG. 4, in the session evaluation database 32, information having items such as "session ID", "session type", and "dissatisfaction score" is registered. Note that the information registered in the session evaluation database 32 is not limited to this, and arbitrary information may be registered as long as it is information related to evaluation of a session. Also, the session evaluation database 32 may be integrated with the session log database 31, for example.

Here, the "session type" is information indicating whether it is estimated that the user is satisfied with the interaction in the session indicated by the associated "session ID". This is the label to be assigned. For example, if the "session type" is estimated that the user is not satisfied (for example, the dissatisfaction score is high), the "low-satisfaction session" indicates that the user was satisfied (for example, the dissatisfaction score is low), a “high satisfaction session” is adopted, and a “neutral session” is adopted if neither is estimated (e.g., if the dissatisfaction score falls within a predetermined range) be. The “dissatisfaction score” is the value of the dissatisfaction score given to the session indicated by the associated “session ID”.

For example, in the example shown in FIG. 4, the session evaluation database 32 registers a session ID "session #1", a session type "low satisfaction session", and a dissatisfaction score "150" in association with each other. Such information indicates that the session type of the session indicated by the session ID "session #1" was "low satisfaction session" and the dissatisfaction score was "150". When the session has not been evaluated, the session type is "not evaluated", and when the dissatisfaction score is not calculated, the dissatisfaction score is "not evaluated".

Various index utterances are registered in the index utterance database 33 . For example, FIG. 5 is a diagram showing an example of information registered in the indicator utterance database according to the embodiment. As shown in FIG. 5, in the index utterance database 33, information having items such as "index utterance", "index type", "index score" and "registration source" is registered. In addition to the information shown in FIG. 5 , various types of information regarding index utterances may be registered in the index utterance database 33 .

Here, the "index utterance" is a character string of utterances that serve as an indicator of whether or not the user is satisfied with the dialogue in the session that includes the utterance, such as a non-favorable index utterance or a favorable index utterance. is. Further, the "index type" is information indicating whether the associated index utterance is a non-favorable index utterance or a favorable index utterance. Also, the "index score" is a score that indicates the degree to which the user is dissatisfied with the dialogue in the session that includes the associated "index utterance". Also, the "registration source" means whether the associated index utterance is a preset index utterance, that is, whether it is a seed index utterance or an index utterance automatically extracted from the session by the extraction process. , and if the index utterance is extracted from a session, the session ID of the extraction source session is registered.

For example, in the example shown in FIG. 5, the index utterance “good”, the index type “favorable”, the index score “−150”, and the registration source “pre-registered” are registered in association with each other in the index utterance database 33. It is In such information, the character string "well done" is registered as the index utterance, the type of the index utterance is "favorable", that is, the favorable index utterance, and the index score is "-150". indicates that it is In addition, such information indicates that the favorable indicator utterance "well done" is a pre-registered indicator utterance, that is, a seed indicator utterance.

In the engagement database 34, information indicating how the user uses the interactive service is registered. More specifically, in the engagement database 34, information such as usage period and usage frequency, that is, engagement information indicating how long the user uses the interactive service is registered. For example, FIG. 6 is a diagram showing an example of information registered in the engagement database according to the embodiment. As shown in FIG. 6, in the engagement database 34, information having items such as "user ID", "engagement history", and "predicted engagement" is registered. In addition to the information shown in FIG. 6, arbitrary information may be registered in the engagement database 34 as long as it is information related to engagement.

Here, the "engagement history" is a history period, which is a history of the period of use of the interactive service by the user indicated by the associated "user ID", and a history of the frequency of use of the interactive service by the user. This is information indicating "history frequency". In addition, "predicted engagement" indicates the "predicted period" that is the future usage period by the user indicated by the associated "user ID", and the "predicted frequency" that is the future usage frequency by the user. Information.

In addition, the "history period" is, for example, information indicating a period during which the user continuously used the interactive service during a predetermined period in the past. This is information indicating the longest period during which the user has used the interactive service. "History frequency" is information indicating how often the user has used the interactive service during the historical period. For example, the number of times the user has used the interactive service during the historical period number) divided by the historical period. Also, the "prediction period" is information indicating a period during which it is estimated that the user will continue to use the interactive service in a predetermined period in the future. Also, the "predicted frequency" is information indicating the estimated frequency that the user will use the interactive service in a predetermined period in the future.

For example, in the example shown in FIG. 6, the user ID "user #1", the engagement history "history period #1 history frequency #1", and the predictive engagement "prediction period #1 prediction frequency #1" are associated and registered. ing. Such information is such that the engagement history of the user indicated by the user ID "user #1" is "history period #1" and "history frequency #1", and the predictive engagement is "prediction period #1" and " "predicted frequency #1".

Note that in the example shown in FIG. 6, conceptual values such as "historical period #1", "historical frequency #1", "predicted period #1", and "predicted frequency #1" are described. Numerical values indicating various periods and frequencies are registered.

Various models are registered in the model database 35 . For example, FIG. 7 is a diagram showing an example of information registered in the model database according to the embodiment. As shown in FIG. 7, in the model database 35, information having items such as "model ID", "model type", and "model data" is registered. In addition to the information shown in FIG. 7, the model database 35 may register various types of information regarding models.

Here, "model ID" is an identifier for identifying a model. "Model type" is information indicating what kind of model the model is. "Model data" is model data. In the example shown in FIG. 7, a conceptual value such as "model data #1" is described, but in reality, information such as various parameters that make up the model is registered.

For example, in the example shown in FIG. 7, the model database 35 includes a “repetition evaluation model” that evaluates a session based on whether the session contains repetition, "Apology evaluation model" that evaluates a session based on, "Unfavorable indicator utterance evaluation model" that evaluates a session based on whether or not the session contains unfavorable indicator utterances, and Favorable indicator utterances in the session A "favorable indicator utterance evaluation model" is registered that evaluates a session based on whether or not it contains .

The model database 35 also stores a "three-value evaluation model" that learns the characteristics of dialogues that are estimated to satisfy the user, dialogues that are estimated to dissatisfy the user, and dialogues that are neither of these. is registered. The model database 35 also registers an "engagement session evaluation model" that learns the characteristics of a session that includes user utterances whose utilization mode of the interactive service satisfies a predetermined condition. The model database 35 also registers a “repetitive speech evaluation model” that is a pseudo-generated session that has learned the characteristics of sessions that include the same or similar speech.

In the model database 35, a “dialogue model”, which is a model used by the dialogue device 200 and is used to generate a response to an utterance, is registered. Further, in the model database 35, a "user model model", which is a model that simulates a user and outputs an utterance in response to an input response, is registered.

In the model database 35, model information in any format is registered as model data of each model. For example, each model may be realized by SVM (Support Vector Machine) or DNN (Deep Neural Network). Here, the DNN may be a CNN (Convolutional Neural Network) or an RNN (Recurrent Neural Network). Also, the RNN may be LSTM (Long short-term memory) or the like. That is, each model can adopt any type of model. Also, each model may be a model realized by combining a plurality of models, such as a model combining CNN and RNN, for example.

[2-3. About the control part]
Returning to FIG. 2, the description is continued. The control unit 40 is a controller, and various programs stored in a storage device inside the information providing apparatus 10 are transferred to a RAM or the like by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). is executed as a work area. Also, the control unit 40 is a controller, and may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 2, the control unit 40 has a session acquisition unit 41, a usage mode identification unit 42, an indicator utterance extraction processing unit 50, a session evaluation processing unit 60, an engagement prediction processing unit 70, and a reinforcement learning processing unit 80. .

The session acquisition unit 41 acquires a session. For example, when acquiring the dialogue history from the dialogue device 200, the session acquisition unit 41 divides the acquired dialogue history into sessions. For example, the session acquisition unit 41 selects a user to be processed, and extracts the speech history including the speech of the selected user. Subsequently, the session acquisition unit 41 identifies the time between each response and the utterance in the extracted utterance history. Then, if the time between a certain response and the next utterance is longer than a predetermined period, the session acquisition unit 41 determines that the response is the last response in the session, and based on the determination result, Divide your interactions with other people into sessions. Then, the session acquisition unit 41 assigns a session ID and a user ID of the user to each session and registers them in the session log database 31 .

The usage pattern identification unit 42 identifies the usage pattern of the user. For example, the usage pattern identification unit 42 selects a user to be processed, and identifies the user's usage pattern of the interactive service within a predetermined period (for example, the past six months). More specifically, the usage pattern identification unit 42 identifies a history period and a history frequency within a predetermined period. Then, the usage pattern identification unit 42 registers the identified history period and history frequency as an engagement history in the engagement database 34 in association with the user ID of the user.

[2-4. About extraction processing]
The index utterance extraction processing unit 50 executes extraction processing for extracting index utterances used for session evaluation from a session. An example of the concept of the processing executed by the index utterance extraction processing unit 50 and the functional configuration of the index utterance extraction processing unit 50 will be described below with reference to FIGS. 8 to 11. FIG.

[2-4-1. Extraction processing based on co-occurrence]
For example, the index utterance extraction processing unit 50 extracts a new index utterance based on co-occurrence with the index utterance. For example, FIG. 8 is a diagram showing the concept of processing for extracting new index utterances from seed index utterances. For example, if a certain session includes a favorable indicator utterance, the same session is likely to contain the favorable indicator utterance. Also, when a certain session includes unfavorable indicator utterances, it is estimated that the same session is likely to include unfavorable indicator utterances. That is, within the same session, favorable indicator utterances tend to co-occur, and unfavorable indicator utterances tend to co-occur.

Therefore, the index utterance extraction processing unit 50 identifies a preset index utterance, which is an utterance that serves as an index for the user's evaluation of the dialogue service, from the history of the user's utterances and responses to the utterances in the dialogue service. Then, new index utterances are extracted from the user's utterances based on the co-occurrence with the specified index utterances. For example, the index utterance extraction processing unit 50 extracts a new index utterance from each utterance based on co-occurrence with pre-registered seed unfavorable index utterances and seed favorable index utterances, and extracts a new index utterance from each utterance. to register.

For example, if the character string "wow" is registered in the index utterance database 33 as the seed favorable indicator utterance, the index utterance extraction processing unit 50 extracts the character string "wow" from the session log database 31. Extract all sessions. For each utterance included in the extracted session, the index utterance extraction processing unit 50 counts the number of sessions including that utterance. That is, the index utterance extraction processing unit 50 counts the number of times each utterance co-occurs with the seed favorable index utterance.

Further, for example, when the character string "fool" is registered in the index utterance database 33 as the seed unfavorable index utterance, the index utterance extraction processing unit 50 extracts the character string "fool" from the session log database 31. extracts all sessions that include in the utterance. For each utterance included in the extracted session, the index utterance extraction processing unit 50 counts the number of sessions including that utterance. That is, the index utterance extraction processing unit 50 counts the number of times each utterance co-occurs with the seed unfavorable index utterance.

Then, the index utterance extraction processing unit 50 calculates the ratio of the counted number of co-occurrences for each utterance. More specifically, the index utterance extraction processing unit 50 calculates for each utterance a value obtained by dividing the number of co-occurrences with the seed unfavorable index utterance by the number of co-occurrences with the seed favorable index utterance. Then, the index utterance extraction processing unit 50 extracts utterances with the highest N ratios (N is an arbitrary number) as new unfavorable index utterances, and extracts utterances with the lowest N ratios as new favorable indexes. Extract as an utterance. The index utterance extraction processing unit 50 also registers the extracted unfavorable index utterances and favorable index utterances in the index utterance database 33 .

Here, the index utterance extraction processing unit 50 may extract new index utterances from the seed index utterances based on activity propagation on the co-occurrence network. For example, the index utterance extraction processing unit 50 generates a co-occurrence network of utterances included in the same session as the seed index utterance. For example, the index utterance extraction processing unit 50 generates a network in which each utterance is set as a node and utterances included in the same session are connected by links. Any well-known technique can be adopted for generating the co-occurrence network.

Subsequently, the index utterance extraction processing unit 50 identifies the nodes corresponding to the seed index utterances from the co-occurrence network, and for each node, calculates the distance between the seed index utterances and the corresponding nodes (for example, the hops following the link). number). Then, the index utterance extraction processing unit 50 extracts the utterances corresponding to the nodes in which the distance between the seed unfavorable index utterance and the corresponding node is short and the distance between the seed favorable index utterance and the node corresponding to the seed unfavorable index utterance is long. be an indicator utterance. In addition, the index utterance extraction processing unit 50 extracts utterances corresponding to nodes in which the distance between the seed favorable index utterance and the corresponding node is short and the distance between the seed unfavorable index utterance and the node corresponding to the node is long. Let it be an indicator utterance.

For example, the index utterance extraction processing unit 50 identifies and identifies a node whose distance from the seed favorable index utterance is less than a predetermined threshold and whose distance from the seed unfavorable index utterance is equal to or greater than a predetermined threshold. An utterance corresponding to the node is extracted as a new favorable indicator utterance. In addition, the index utterance extraction processing unit 50 identifies and identifies nodes whose distance from the seed favorable index utterance is equal to or greater than a predetermined threshold and whose distance from the seed unfavorable index utterance is less than a predetermined threshold. An utterance corresponding to the node is extracted as a new unfavorable index utterance. Then, the index utterance extraction processing unit 50 registers each extracted index utterance in the index utterance database 33 .

Note that the index utterance extraction processing unit 50 sets the index utterance extracted by the above-described extraction processing as a new seed index utterance, and executes the above-described extraction processing using the new seed index utterance, thereby producing a new index utterance. may be extracted.

[2-4-2. Extraction processing based on usage mode]
Also, for example, the index utterance extraction processing unit 50 extracts a new index utterance based on the usage mode of the user's interactive service. For example, FIG. 9 is a diagram showing the concept of processing for extracting new index utterances based on usage patterns in interactive services.

For example, users who continue to use the interactive service, that is, users who use the interactive service for a long period of time, and users who frequently use the interactive service are presumed to be satisfied with the interactive service. Therefore, it is presumed that a user with a long period of use or a user with a high frequency of use utters many favorable (favorable) statements, that is, favorable indicator utterances. On the other hand, users with a short period of use and users with a short use frequency are presumed to utter many unfavorable (unfavorable) utterances, that is, unfavorable indicator utterances.

Therefore, the index utterance extraction processing unit 50 acquires a session including a user's utterance whose utilization mode of the dialogue service satisfies a predetermined condition, and based on the appearance frequency in the acquired session, the user's evaluation of the dialogue service. Extract indicator utterances that are indicators of For example, the index utterance extraction processing unit 50 refers to the engagement database 34 and identifies users whose engagement history satisfies a predetermined condition.

For example, the index utterance extraction processing unit 50 identifies users whose history period and history frequency exceed predetermined thresholds as favorable users, and users whose history period and history frequency each exceed predetermined thresholds. identify as unfavorable users. In addition, the index utterance extraction processing unit 50 identifies a session including a favorable user's utterance and a session including an unfavorable user's utterance, and extracts the utterances included in the identified sessions.

Then, the index utterance extraction processing unit 50 calculates, for each extracted utterance, the frequency of appearance in sessions of favorable users and the frequency of appearance in sessions of unfavorable users. For example, the index utterance extraction processing unit 50 divides the number of times the utterance appears in sessions containing utterances of favorable users by the total number of sessions containing utterances of favorable users, Calculated as frequency of occurrence. Further, for example, the index utterance extraction processing unit 50 divides the number of times the utterance appears in sessions including utterances of unfavorable users by the total number of sessions including utterances of unfavorable users, Calculated as the appearance frequency in the user's session.

The index utterance extraction processing unit 50 also calculates a value obtained by dividing the appearance frequency in the session of the unfavorable user by the appearance frequency in the session of the favorable user, that is, the ratio of the appearance frequency for each utterance. Then, the index utterance extraction processing unit 50 extracts utterances with the highest N calculated ratios as new unfavorable index utterances, and extracts utterances with the lowest N calculated ratios as new favorable index utterances.

[2-4-3. Extraction processing based on evaluation results]
Also, for example, the index utterance extraction processing unit 50 extracts a new index utterance based on the evaluation result of the session. For example, FIG. 10 is a diagram showing the concept of processing for extracting new index utterances based on session evaluation results.

For example, a session may be given an evaluation result by an evaluation process, which will be described later. In addition, a session may be provided with a manual evaluation result such as crowdsourcing. An evaluation result for such a session can serve as an index for extracting index utterances from utterances included in the session.

For example, it is estimated that a session with a high degree of user satisfaction (that is, a highly satisfying session) includes many favorable indicator utterances. On the other hand, it is presumed that many unfavorable index utterances are included in sessions with low user satisfaction (that is, low-satisfaction sessions).

Therefore, the index utterance extraction processing unit 50 selects an utterance whose appearance frequency satisfies a predetermined condition among utterances included in a session whose evaluation satisfies a predetermined condition, and determines that the user's evaluation of the session including the utterance is favorable. It is extracted as an indicator utterance that serves as an indicator of whether or not the user is satisfied with the session. For example, the index utterance extraction processing unit 50 extracts utterances included in a high-satisfaction session and a low-satisfaction session, and calculates the appearance frequency in the low-satisfaction session and the appearance frequency in the high-satisfaction session for each extracted utterance.

Then, the index utterance extraction processing unit 50 calculates a value obtained by dividing the appearance frequency in the low-satisfaction session by the appearance frequency in the high-satisfaction session as the appearance frequency ratio. Then, the index utterance extraction processing unit 50 extracts utterances with the highest N calculated ratios as new unfavorable index utterances, and extracts utterances with the lowest N calculated ratios as new favorable index utterances.

[2-4-4. Regarding the functional configuration of the index utterance extraction processing unit]
Next, an example of the functional configuration of the index utterance extraction processing unit 50 will be described with reference to FIG. 11 . 11 is a diagram illustrating an example of a functional configuration of an indicator utterance extraction processing unit according to the embodiment; FIG. As shown in FIG. 11 , the index utterance extraction processing unit 50 has an index utterance identification unit 51 , an utterance extraction unit 52 and a network generation unit 53 .

The index utterance identifying unit 51 identifies a preset index utterance, which is an utterance that serves as an index for the user's evaluation of the dialogue service, from the history of the user's utterances and responses in the dialogue service. For example, the index utterance identification unit 51 refers to the index utterance database 33 to identify the index utterance. Note that the index utterance identifying unit 51 may identify an index utterance registered in advance, or may identify an index utterance extracted from a session.

The index utterance identifying unit 51 then refers to the session log database 31 to search for a session including the identified index utterance. That is, the index utterance identifying unit 51 identifies a session in which the user's utterance includes a favorable index utterance or a non-favorable index utterance in the session history. Then, the index utterance identifying unit 51 notifies the utterance extracting unit 52 of the identified session, that is, the session including the index utterance.

When identifying an index utterance from the co-occurrence network shown in FIG. 8, index utterance identifying section 51 notifies network generating section 53 of a session including the index utterance. Then, the index utterance identifying unit 51 selects a node corresponding to the identified index utterance among the nodes included in the co-occurrence network generated by the network generation unit 53, that is, the favorable index registered in the index utterance database 33. Identify nodes that correspond to utterances and unfavorable indicator utterances. Then, the index utterance identifying unit 51 notifies the utterance extracting unit 52 of the co-occurrence network and the identified node.

The utterance extraction unit 52 extracts index utterances from each session. For example, as shown in FIG. 8, the utterance extraction unit 52 extracts new index utterances from utterances included in the session history based on the co-occurrence with seed index utterances.

For example, the utterance extraction unit 52 extracts from the session log database 31 the session identified by the index utterance identification unit 51, that is, the session including the index utterance. Then, the utterance extraction unit 52 identifies co-occurrence with the favorable indicator utterance for each utterance, and extracts utterances whose co-occurrence with the favorable indicator utterance exceeds a predetermined threshold as new favorable indicator utterances. do. In addition, the utterance extraction unit 52 identifies co-occurrence with the unfavorable index utterance for each utterance, and extracts an utterance for which the identified co-occurrence exceeds a predetermined threshold as a new unfavorable index utterance.

For example, the utterance extraction unit 52 counts the number of appearances in sessions containing favorable indicator utterances and the number of appearances in sessions containing unfavorable indicator utterances for each utterance contained in the extracted session. That is, the utterance extraction unit 52 counts the number of times a favorable indicator utterance is included in the same session and the number of times a non-favorable indicator utterance is included in the same session for each utterance. Then, the utterance extraction unit 52 extracts an utterance that satisfies a predetermined condition for the ratio of the counted times as a new index utterance. For example, the utterance extraction unit 52 divides the number of appearances in sessions containing unfavorable indicator utterances by the number of appearances in sessions containing favorable indicator utterances, and calculates the value as a ratio. The utterance is treated as a new unfavorable indicator utterance, and the utterances with the lowest N ratios are treated as new favorable indicator utterances.

Note that the utterance extraction unit 52 may simply set an utterance whose number of appearances in a session including the favorable indicator utterance exceeds a predetermined threshold as a new favorable indicator utterance. Further, the utterance extracting unit 52 may set, as a new unfavorable indicator utterance, an utterance whose appearance count exceeds a predetermined threshold in a session including unfavorable indicator utterances.

Further, when receiving notification of a co-occurrence network and a node corresponding to an index utterance, utterance extraction section 52 extracts an utterance whose co-occurrence with the index utterance satisfies a predetermined condition based on the co-occurrence network as the index utterance. Extract as That is, the utterance extracting unit 52 extracts, as a new index utterance, an utterance whose distance from the index utterance on the co-occurrence network satisfies a predetermined condition.

For example, the utterance extraction unit 52 may extract, as a new favorable indicator utterance, an utterance whose distance from the favorable indicator utterance on the co-occurrence network is equal to or less than a predetermined threshold. Further, for example, the utterance extraction unit 52 may extract, as the unfavorable indicator utterance, an utterance whose distance from the unfavorable indicator utterance on the co-occurrence network is equal to or less than a predetermined threshold. Further, for example, the utterance extraction unit 52 selects, as a new favorable indicator utterance, an utterance in which the distance from the favorable indicator utterance is equal to or less than a predetermined threshold and the distance from the unfavorable indicator utterance is equal to or greater than a predetermined threshold. may be extracted. Further, for example, the utterance extracting unit 52 selects an utterance whose distance from the unfavorable indicator utterance is equal to or less than a predetermined threshold and whose distance from the favorable indicator utterance is equal to or greater than a predetermined threshold as a new unfavorable indicator utterance. can be extracted as

For example, the utterance extraction unit 52 identifies the distance between the favorable indicator utterance and the corresponding node and the distance between the unfavorable indicator utterance and the corresponding node for each node included in the co-occurrence network. Then, the utterance extracting unit 52 corresponds to a node in which the distance between the unfavorable index utterance and the corresponding node is equal to or less than a predetermined threshold and the distance between the favorable index utterance and the corresponding node exceeds the predetermined threshold. Let the utterance be a new unfavorable index utterance. Further, the utterance extracting unit 52 corresponds to a node in which the distance between the favorable indicator utterance and the corresponding node is equal to or less than a predetermined threshold and the distance between the unfavorable indicator utterance and the corresponding node exceeds the predetermined threshold. Let the utterance be a new favorable indicator utterance.

When executing the process shown in FIG. 9, the utterance extraction unit 52 acquires a session including the user's utterance and response whose use mode of the interactive service satisfies a predetermined condition, and sets the appearance frequency in the acquired session to the session. Based on this, the index utterance is extracted. For example, the utterance extraction unit 52 refers to the engagement database 34 and identifies the user's engagement history within a predetermined period. To give a more specific example, the utterance extraction unit 52 identifies the processing date and time, the history period and the history frequency in the last several months.

Then, the utterance extraction unit 52 identifies users whose history period is longer than a predetermined threshold and whose history frequency is higher than a predetermined threshold as favorable users. In addition, the utterance extraction unit 52 identifies users whose history period is shorter than a predetermined threshold and whose history frequency is lower than a predetermined threshold as unfavorable users. Note that the utterance extraction unit 52 may identify favorable users and unfavorable users based on whether the prediction period and prediction frequency registered as prediction engagement exceed a predetermined threshold.

Then, the utterance extraction unit 52 calculates, for each utterance, a value obtained by dividing the frequency of appearance in the session of the unfavorable user by the frequency of appearance in the session of the favorable user. Then, the utterance extracting unit 52 extracts utterances with the highest N calculated values as new unfavorable index utterances, and extracts utterances with the lowest N calculated values as new favorable index utterances.

That is, the utterance extracting unit 52 extracts, as new favorable indicator utterances, utterances whose frequency of appearance in a session including utterances of users whose usage mode satisfies a predetermined condition, that is, utterances of favorable users, exceeds a predetermined threshold. do. In addition, the utterance extraction unit 52 extracts an utterance whose appearance frequency exceeds a predetermined threshold value in a session including an utterance of a user whose usage mode does not satisfy a predetermined condition, that is, an utterance of an unfavorable user, as a new unfavorable index. Extract as an utterance.

When executing the process shown in FIG. 10, the utterance extraction unit 52 extracts, as index utterances, utterances whose appearance frequency satisfies a predetermined condition from sessions whose evaluation satisfies a predetermined condition. For example, the utterance extraction unit 52 refers to the session evaluation database 32 and identifies a highly satisfying session and a lowly satisfying session. In addition, the utterance extraction unit 52 refers to the session log database 31 and extracts a highly satisfied session and a less satisfied session.

Then, the utterance extraction unit 52 calculates the ratio of the appearance frequency in the low-satisfaction session and the appearance frequency in the high-satisfaction session for each utterance, and extracts utterances whose ratio satisfies a predetermined condition as new index utterances. For example, the utterance extraction unit 52 calculates the value obtained by dividing the appearance frequency in the low-satisfaction session by the appearance frequency in the high-satisfaction session, and the calculated value is the top N utterances as new unfavorable index utterances. Utterances with the lowest N values are extracted as new favorable indicator utterances.

Note that the utterance extraction unit 52 registers index utterances extracted by the various processes described above in the index utterance database 33 . In this way, index utterances registered in the index utterance database 33 are used as seed index utterances and index utterances for evaluating sessions in processing after registration.

The network generator 53 generates a co-occurrence network that connects co-occurring utterances or responses from the history of utterances and responses. For example, the network generator 53 refers to the session log database 31 and identifies each utterance included in the session. Then, the network generation unit 53 sets nodes corresponding to each utterance, and generates a co-occurrence network in which nodes corresponding to each utterance are connected based on the co-occurrence of each utterance. Then, the network generation unit 53 provides the index utterance identification unit 51 with the co-occurrence network.

[2-5. Regarding the evaluation process]
Returning to FIG. 2, the description is continued. The session evaluation processing unit 60 executes evaluation processing for evaluating sessions. An example of the concept of evaluation performed by the session evaluation processing unit 60 and an example of the functional configuration of the session evaluation processing unit 60 will be described below with reference to FIGS. 12 to 21. FIG.

[2-5-1. Evaluation processing based on index utterance]
For example, the session evaluation processing unit 60 evaluates the session based on index utterances included in the session. For example, FIG. 12 is a diagram showing the concept of processing for evaluating a session based on index utterances. For example, when a certain session includes many favorable indicator utterances, it is estimated that the user's impression of each response included in the session is good, and thus the user's satisfaction with the session is estimated to be high. Moreover, when a certain session contains many unfavorable indicator utterances, it is estimated that the user's impression of each response included in the session is bad, and thus the user's satisfaction with the session is estimated to be low.

Therefore, the session evaluation processing unit 60 evaluates the session based on the appearance frequency of unfavorable index utterances and the appearance frequency of favorable index utterances included in the session. For example, the session evaluation processing unit 60 refers to the session evaluation database 32 and selects sessions that have not been evaluated as evaluation targets. It should be noted that the session evaluation processing unit 60 may select an evaluated session as an evaluation target. For example, the session evaluation processing unit 60 may reselect a session for which a predetermined period of time has passed since the evaluation was performed as an evaluation target.

Next, the session evaluation processing unit 60 extracts a session to be evaluated from the session log database 31, refers to the index utterance database 33, and identifies index utterances included in the extracted session. Then, the session evaluation processing unit 60 calculates the ratio between the total appearance frequency of the unfavorable indicator utterances and the total appearance frequency of the favorable indicator utterances. For example, the session evaluation processing unit 60 identifies all of the utterances registered as unfavorable indicator utterances among the utterances included in the session to be processed, and calculates the total number of identified utterances as the appearance frequency of the unfavorable indicator utterances. specified as the sum of In addition, the session evaluation processing unit 60 identifies all of the utterances registered as favorable indicator utterances among the utterances included in the session to be processed, and calculates the total number of the identified utterances as the total appearance frequency of the favorable indicator utterances. Identify as

Then, the session evaluation processing unit 60 calculates a value obtained by dividing the total appearance frequency of the unfavorable indicator utterances by the total appearance frequency of the favorable indicator utterances as a ratio. In addition, the session evaluation processing unit 60 similarly calculates the ratio for other sessions, extracts sessions with the highest N calculated values as low-satisfaction sessions, and selects sessions with the lowest N calculated values. Extract as a high-satisfaction session.

Here, it is considered that there is a difference in the user's degree of satisfaction suggested by each index utterance. For example, it is conceivable that a user who uttered "wonderful" is more satisfied with the interactive service than a user who uttered "nice". Therefore, the session evaluation processing unit 60 may evaluate the session in consideration of the degree of satisfaction of the user indicated by the index utterance.

For example, the index utterance database 33 is provided with an index score indicating how satisfied or dissatisfied the user who uttered the corresponding index utterance is with the dialogue service (for example, see FIG. 5). ). Such an index score, for example, may be manually provided by crowdsourcing, for example, automatically calculated from the relationship between the dissatisfaction score given to the session and the appearance frequency of each index utterance. It can be anything. For example, the information providing apparatus 10 may calculate a higher index score for unfavorable index utterances that are frequently included in sessions with higher dissatisfaction scores, and may calculate a higher index score for sessions with lower dissatisfaction scores. A higher index score may be calculated for a more targeted index utterance.

When such an index score is given, the session evaluation processing unit 60 executes the following processing. First, the session evaluation processing unit 60 extracts a session to be evaluated from the session log database 31, refers to the index utterance database 33, and identifies index utterances included in the extracted session.

Then, the session evaluation processing unit 60 calculates the ratio between the total index score of the unfavorable index utterances and the total index score of the favorable index utterances included in the session to be processed. For example, the session evaluation processing unit 60 calculates a value obtained by dividing the total index score of unfavorable index utterances included in the session to be processed by the total index score of favorable index utterances as a ratio. Note that the session evaluation processing unit 60 may calculate the ratio from the sum of the absolute values of the index scores of each index utterance. In addition, the session evaluation processing unit 60 similarly calculates the ratios for other sessions, and treats sessions with the highest N calculated values as low-satisfaction sessions, and the sessions with the lowest N calculated values as high-satisfaction sessions. session. After that, the session evaluation processing unit 60 registers the evaluation result in the session evaluation database 32. FIG.

[2-5-2. Evaluation processing based on index utterance]
Also, for example, the session evaluation processing unit 60 evaluates a session using a model for evaluating a session based on index utterances included in the session. For example, FIG. 13 is a diagram showing the concept of processing for evaluating a session based on index utterances.

For example, the session evaluation processing unit 60 refers to the session log database 31 and extracts the highly satisfying sessions and the sessions other than the highly satisfying sessions as learning data. For example, the session evaluation processing unit 60 extracts sessions evaluated as highly satisfying sessions from the session log database 31 based on the evaluation results obtained by the processing shown in FIG. 12 . Also, the session evaluation processing unit 60 extracts from the session log database 31 sessions other than the sessions evaluated as highly satisfying sessions.

Then, the session evaluation processing unit 60 uses the extracted learning data to learn the favorable index utterance evaluation model M4. For example, the session evaluation processing unit 60 outputs a first satisfaction value higher than a predetermined threshold value when inputting a character string of utterances and responses included in a highly satisfied session, and to sessions other than highly satisfied sessions The favorable index utterance evaluation model M4 is trained so that when the included utterance and response character string are input, a first satisfaction level lower than a predetermined threshold value is output. In addition, the session evaluation processing unit 60 outputs a value indicating that the session is a highly satisfactory session when the character string of the utterances and responses included in the highly satisfactory session is input, and the utterances and responses included in sessions other than the highly satisfactory session The favorable index utterance evaluation model M4 may be trained so that when a response character string is input, a value indicating that the session is not highly satisfactory is output.

In addition, it is presumed that many favorable indicator utterances are included in the highly satisfying session. Therefore, when the above-described learning is performed, the favorable indicator utterance evaluation model M4 outputs a higher value of the first satisfaction level as the target session to be evaluated contains more favorable indicator utterances, and evaluates The less favorable index utterances are included in the target session, the lower the value of the first satisfaction level is output.

The session evaluation processing unit 60 also refers to the session log database 31 and extracts low-satisfaction sessions and sessions other than the low-satisfaction sessions as learning data. Then, the session evaluation processing unit 60 uses the extracted learning data to learn the unfavorable index utterance evaluation model M3. For example, the session evaluation processing unit 60 outputs a third dissatisfaction level higher than a predetermined threshold value when inputting a character string of utterances and responses included in a low-satisfaction session, to a session other than a low-satisfaction session The unfavorable index utterance evaluation model M3 is trained so that when the included utterance and response character string are input, a third degree of dissatisfaction with a value lower than a predetermined threshold is output. It should be noted that the session evaluation processing unit 60, when inputting the character string of the utterances and responses included in the low-satisfaction session, outputs a value indicating that it is a low-satisfaction session, and the utterances and The unfavorable index utterance evaluation model M3 may be trained so that when a response character string is input, a value indicating that the session is not a low-satisfaction session is output.

In addition, it is presumed that many unfavorable index utterances are included in the low-satisfaction session. Therefore, when the above-described learning is performed, the unfavorable indicator utterance evaluation model M3 outputs a higher value of the third dissatisfaction level as the subject session to be evaluated contains more unfavorable indicator utterances. , the less unfavorable index utterances are included in the target session to be evaluated, the lower the value of the third dissatisfaction level is output.

Then, the session evaluation processing unit 60 evaluates the session using the unfavorable index utterance evaluation model M3 and the favorable index utterance evaluation model M4. For example, the session evaluation processing unit 60 inputs the character strings of the utterances and responses of the target session to be evaluated into the unfavorable index utterance evaluation model M3 and the favorable index utterance evaluation model M4, and the third degree of dissatisfaction and the first Get satisfaction. Then, the session evaluation processing unit 60 evaluates the session based on the third degree of dissatisfaction and the first degree of satisfaction.

For example, the session evaluation processing unit 60 may evaluate the target session as a low-satisfaction session when the third degree of dissatisfaction exceeds a predetermined threshold. Moreover, the session evaluation processing unit 60 may evaluate the target session as a highly satisfying session when the first satisfaction level exceeds a predetermined threshold value. Moreover, the session evaluation processing unit 60 may evaluate the target session as a low-satisfaction session when the third degree of dissatisfaction exceeds a predetermined threshold and the first degree of satisfaction is less than a predetermined threshold. Moreover, the session evaluation processing unit 60 may evaluate the target session as a highly satisfactory session when the third degree of dissatisfaction is less than a predetermined threshold and the first degree of satisfaction is equal to or greater than a predetermined threshold. Also, the session evaluation processing unit 60 may evaluate the target session by the processing shown in FIG.

[2-5-3. Evaluation processing based on ternary learning]
Here, in various dialogues, not limited to sessions, the user's satisfaction is high in the dialogue that precedes the favorable indicator utterance, and the dialogue that precedes the unfavorable indicator utterance is high in the user's satisfaction. Satisfaction is estimated to be low. In addition, it is considered that the user's impression indicated by the index utterance is not for the dialogue after the index utterance, but for the dialogue immediately before the index utterance.

Therefore, the session evaluation processing unit 60 may extract a series of dialogues corresponding to the index utterance, and use the extracted dialogues to learn a model for estimating the user's evaluation of the input dialogues. For example, the session evaluation processing unit 60 may extract utterances and responses exchanged prior to the index utterance, have the model learn the features of the extracted utterance, and evaluate the session using the trained model. good. In addition, the session evaluation processing unit 60 allows the model to learn the characteristics of a neutral dialogue that is neither favorable nor unfavorable (that is, neither satisfied nor dissatisfied) by the user. good.

For example, FIG. 14 is a diagram showing an example of processing for evaluating a session using a ternary evaluation model that has learned features of three types of dialogue. For example, the session evaluation processing unit 60 refers to the session log database 31 and identifies a session including unfavorable index utterances. Subsequently, the session evaluation processing unit 60 extracts the unfavorable indicator utterances and a predetermined number of utterances and responses made immediately before the unfavorable indicator utterances from the session as unfavorable dialogue examples UD. For example, the session evaluation processing unit 60 extracts two utterances made immediately before the unfavorable indicator utterance "I don't understand" and the responses to these utterances, and extracts the unfavorable indicator "I don't understand". It is extracted as an unfavorable dialogue example UD together with the utterance.

In addition, the session evaluation processing unit 60 refers to the session log database 31 to identify sessions including favorable index utterances. Subsequently, the session evaluation processing unit 60 extracts the favorable indicator utterances and a predetermined number of utterances and responses made immediately before the favorable indicator utterances from the session as favorable dialogue examples FD. For example, the session evaluation processing unit 60 extracts two utterances made immediately before the favorable indicator utterance "Kisei ne" and the responses to these utterances, Extract as dialogue example FD.

In addition, the session evaluation processing unit 60 selects one utterance that is not an index utterance as a neutral utterance, and selects a predetermined number of utterances and responses made immediately before the neutral utterance from the session including the selected neutral utterance. are extracted from the session as neutral dialogue examples ND. For example, the session evaluation processing unit 60 extracts two utterances made immediately before the neutral utterance "Tomorrow's weather" and the response to these utterances, and extracts the neutral utterance "Tomorrow's weather" together with the neutral utterance Extract as a dialogue example ND.

Then, the session evaluation processing unit 60 causes the ternary evaluation model M5 to learn the features of the unfavorable dialogue example UD, the neutral dialogue example ND, and the favorable dialogue example FD. For example, when the session evaluation processing unit 60 inputs a character string of an utterance and a response included in an unfavorable dialogue example UD, it outputs a value indicating that the input utterance and response are an unfavorable dialogue example. However, when a character string of utterances and responses included in the neutral dialogue example ND is input, a value indicating that the input utterances and responses are neutral dialogue examples is output and included in the favorable dialogue example FD. When a character string of an utterance and a response is input, the ternary evaluation model M5 is trained so as to output a value indicating that the input utterance and response are examples of favorable dialogue. That is, the session evaluation processing unit 60 determines whether the user is satisfied with the input dialogue (whether it is a favorable dialogue), whether it is unsatisfied (whether it is an unfavorable dialogue), or otherwise. It learns a model that classifies into three categories: whether it is a neutral dialogue or not.

Then, the session evaluation processing unit 60 uses the generated ternary evaluation model M5 to evaluate the target session. For example, the session evaluation processing unit 60 inputs character strings of utterances and responses included in the target session to the ternary evaluation model M5, and determines whether the user is satisfied with the target session based on the output of the ternary evaluation model M5. (ie, high satisfaction session), whether the user is dissatisfied (ie, low satisfaction session), or otherwise (ie, neutral session).

When extracting each dialogue example, the session evaluation processing unit 60 may extract any number of utterances and responses made immediately before the index utterance or the neutral utterance. In this way, by having the model learn the features of the utterances and responses made immediately before the indicator utterance or neutral utterance, the accuracy of the model's evaluation of the dialogue and, in turn, the evaluation of the session can be improved.

[2-5-4. Evaluation process based on engagement]
FIG. 15 is a diagram illustrating an example of processing for evaluating sessions based on engagement. For example, a user who is highly satisfied with the dialogue will have a high usage period and a high frequency of use, and a user with a low level of satisfaction with the dialogue will have a low usage period and low frequency of use. Therefore, it is estimated that a session including utterances of users with high engagement such as usage period and usage frequency has high satisfaction, and a session including utterances of users with low engagement has low satisfaction. Therefore, the session evaluation processing unit 60 evaluates the session based on the engagement of the user who made the utterance included in the session.

For example, the session evaluation processing unit 60 refers to the engagement database 34 and identifies users whose history period and history frequency within a predetermined period exceed predetermined thresholds. Subsequently, the session evaluation processing unit 60 extracts a session including the specified user's utterance from the session log database 31, and treats the extracted session as a session of a user with high engagement.

On the other hand, the session evaluation processing unit 60 refers to the engagement database 34 and identifies users whose history period and history frequency within a predetermined period are less than a predetermined threshold. Subsequently, the session evaluation processing unit 60 extracts a session including the specified user's utterance from the session log database 31, and treats the extracted session as a session of a user with low engagement.

Then, the session evaluation processing unit 60 causes the engagement session evaluation model M6 to learn features of sessions of users with high engagement and sessions of users with low engagement. For example, the session evaluation processing unit 60, when inputting an utterance and response included in a session of a user with high engagement, outputs a value indicating that the user's satisfaction with the session is high, and the session of a user with low engagement is input, the engagement session evaluation model M6 is learned so as to output a value indicating that the user's satisfaction with the session is low.

For example, the session evaluation processing unit 60 outputs an evaluation score with a value higher than a predetermined threshold when an utterance and response included in a session of a user with high engagement is input, and is included in a session of a user with low engagement. The engagement session evaluation model M6 is learned so that when an utterance and a response are input, an evaluation score lower than a predetermined threshold value is output. That is, the session evaluation processing unit 60 classifies the input session into a session with high user satisfaction or a session with low user satisfaction using the learning data extracted based on the engagement session. Learn the evaluation model M6.

Then, the session evaluation processing unit 60 evaluates the target session using the engagement session evaluation model M6. For example, the session evaluation processing unit 60 determines whether the user is satisfied with the target session based on the evaluation score output by the engagement session evaluation model M6 when the character strings of the utterances and responses included in the target session are input. Evaluate whether or not

[2-5-5. Regarding Evaluation Processing Based on Repeated Speech]
Also, FIG. 16 is a diagram showing the concept of processing for evaluating a session based on repeated utterances. For example, if the same or similar utterances are repeated in the session, the user's satisfaction with the session is low, and if the same or similar utterances are not repeated in the session, the user's satisfaction with the session presumed to be high. Therefore, the session evaluation processing unit 60 may evaluate the session based on repeated utterances included in the session.

For example, the session evaluation processing unit 60 counts the number of pairs with similar spellings as the number of pairs of similar spellings from the utterances included in the target session. For example, the session evaluation processing unit 60 determines the number of pairs whose similarities, such as the Jaccard coefficient, Dice coefficient, and Simpson coefficient, exceed a predetermined threshold, and similarities based on various edit distances, such as the Levenshtein distance, exceed a predetermined threshold. Count the number of pairs. In addition, the session evaluation processing unit 60 may calculate the number of pairs whose degree of similarity in meaning exceeds a predetermined threshold value as the number of pairs of similar meanings using word2vec, GloVe, or the like.

Then, when the number of similar notational pairs or the number of similar semantic pairs extracted from the target session exceeds a predetermined threshold, the session evaluation processing unit 60 evaluates the target session as a low-satisfaction session, and evaluates the target session as a low-satisfaction session. If the values of the number of similar notational pairs and the number of similar semantic pairs are below a predetermined threshold value, the target session is evaluated as a highly satisfying session.

Also, the session evaluation processing unit 60 may learn the repetitive utterance evaluation model M7 for evaluating the target session by having the model learn the characteristics of the session including the repetitive utterance. For example, the session evaluation processing unit 60, when inputting a session including repeated utterances (for example, a session in which the number of pairs exceeds a predetermined threshold), outputs that the input session is a low-satisfaction session. A repetitive utterance evaluation model M7 may be learned. Also, the session evaluation processing unit 60 may use such a repetitive speech evaluation model M7 to evaluate the target session.

[2-5-6. Automatic generation of repeated utterances]
Here, the session evaluation processing unit 60 may automatically generate a session including repetitive utterances, and allow the repetitive utterance evaluation model M7 to learn the features of the generated session. For example, FIG. 17 is a diagram showing the concept of processing for evaluating a session using a model that has learned features of automatically generated data. For example, the session evaluation processing unit 60 refers to the session log database 31 and extracts any utterance as a seed utterance. For example, the session evaluation processing unit 60 extracts an utterance such as "Today's weather is" as a seed utterance.

Subsequently, the session evaluation processing unit 60 generates a plurality of utterances similar to the seed utterances from the extracted seed utterances. For example, the session evaluation processing unit 60 randomly deletes words and characters included in the character string of the seed utterance such as "today's weather" in order to imitate speech recognition errors, typos, etc., or Generate multiple utterances replaced with words or strings. To give a more specific example, the session evaluation processing unit 60 extracts from the character string of the seed utterance "today's weather", "today's electricity", "business electricity", and "today's electricity". A character string such as is generated as a duplicate utterance.

Also, the session evaluation processing unit 60 extracts one or more utterances different from the seed utterance from the session log database 31 as mixed utterances. For example, the session evaluation processing unit 60 extracts an utterance such as "the fortune of Aries" as a mixed utterance. Then, the session evaluation processing unit 60 randomly rearranges the seed utterance, a plurality of replicated utterances generated from the seed utterance, and the mixed utterance to generate a session composed of pseudo utterances as a repeated session RS. For example, in the example shown in FIG. 17, the session evaluation processing unit 60 includes the seed utterance "Today's weather is" as the user's utterance "Usr: Today's weather is", and the duplicate utterance " The character string "Today's electricity" is included as the user's utterance "Usr: Today's electricity", and the mixed utterance character string "Aries's fortune" and include the seed utterance string "today's weather wa" again as the user's utterance "Usr: today's weather wa", and duplicate utterances the string "work weather wa" and " Generate a recurring session RS containing "Kyo wo Denki wa" as user utterances "Usr: Kyou no Teki wa" and "Usr: Kyou wo Denki wa".

In other words, the session evaluation processing unit 60 generates a session including pseudo repetitive speech. Note that the session evaluation processing unit 60 may extract a plurality of seed utterances and generate duplicate utterances from each seed utterance.

The session evaluation processing unit 60 also refers to the session log database 31 and randomly extracts a plurality of utterances. Then, the session evaluation processing unit 60 generates a session in which the extracted utterances are randomly rearranged as a non-repeated session NRS. Then, the session evaluation processing unit 60 learns the repetitive speech evaluation model M7 using the repetitive session RS and the non-repetitive session NRS as learning data. For example, the session evaluation processing unit 60 outputs a score indicating that the user is not satisfied when the repetitive session RS is input, and outputs a score indicating that the user is not satisfied when the non-repetitive session NRS is input. The repetitive utterance evaluation model M7 is trained so as to output a score indicating that the speech is correct.

Then, the session evaluation processing unit 60 evaluates the target session using the repetitive speech evaluation model M7. For example, when the target session is input, the session evaluation processing unit 60 evaluates whether or not the user is satisfied with the target session based on the score value output by the repetitive speech evaluation model M7.

[2-5-7. Evaluation of repetitive utterances using images]
Here, the session evaluation processing unit 60 may evaluate repeated utterances using images. For example, FIG. 18 is a diagram showing the concept of evaluating repetitive speech using images. For example, a square image is set up with the same number of areas in the vertical and horizontal directions, and each character in the string of utterances included in a certain session is mapped to the vertical and horizontal directions of the image. When an image is generated by filling in an area in which the same character is associated with the direction, the image becomes a distribution image showing the distribution of the same character included in the character string of the utterance. Such a distribution image includes filled areas to form diagonal lines when a string of utterances contains repetitions. In other words, when a session contains similar utterances, diagonal lines appear in the distribution image.

In recent years, due to advances in image analysis technology, the accuracy of determining similar images using a model using DNN or the like has improved. For this reason, rather than simply learning the features of the character strings of utterances included in the session, learning the features of the distribution image is considered to enable more accurate learning of the features of a session that includes repeated utterances.

Therefore, the session evaluation processing unit 60 may make the iterative evaluation model M1 learn the features of the distribution image, and evaluate the target session using the iterative evaluation model M1. For example, as shown in FIG. 16, the session evaluation processing unit 60 extracts from the session log database 31 a session in which the number of similar or similar pairs of expressions or similar pairs of semantics exceeds a predetermined threshold value as a session including repeated utterances. Note that the session evaluation processing unit 60 may acquire the repetitive session RS generated by the method shown in FIG. 17 as a session including repetitive utterances. Then, the session evaluation processing unit 60 generates a distribution image from each utterance of the session including repeated utterances.

On the other hand, as shown in FIG. 16, the session evaluation processing unit 60 extracts from the session log database 31 sessions in which the number of similar or similar pairs of expressions and similar pairs of semantics is less than a predetermined threshold as sessions that do not include repeated utterances. Note that the session evaluation processing unit 60 may acquire the non-repetitive session NRS generated by the method shown in FIG. 17 as a session that does not include repeated utterances. Then, the session evaluation processing unit 60 generates a distribution image from each utterance of the session that does not include repeated utterances.

Then, the session evaluation processing unit 60 makes the evaluation model M1 repeatedly learn the features of the generated distribution image. For example, when a distribution image generated from each utterance in a session including repeated utterances is input, the session evaluation processing unit 60 outputs a score indicating that the user is not satisfied, and outputs a score indicating that the session does not include repeated utterances. When a distribution image generated from each utterance is input, the evaluation model M1 is repeatedly trained so as to output a score indicating that the user is satisfied.

Subsequently, the session evaluation processing unit 60 generates a distribution image from each utterance included in the target session, and repeatedly inputs the generated distribution image to the evaluation model M1. Then, the session evaluation processing unit 60 evaluates whether or not the user is satisfied with the target session based on the score output by the repeated evaluation model M1.

[2-5-8. About the evaluation based on the apology response]
Also, FIG. 19 is a diagram showing the concept of processing for evaluating a session based on an apology response. As shown in FIG. 19, it is estimated that the user's satisfaction level is low with respect to the session including the apology response, and the user's satisfaction level with the session without the apology response is high. Therefore, the session evaluation processing unit 60 may evaluate the session based on the apology response.

For example, the session evaluation processing unit 60 refers to the session log database 31 and extracts a session including a pre-registered predetermined apology response. Also, the session evaluation processing unit 60 extracts sessions that do not include a predetermined apology response. Then, the session evaluation processing unit 60 causes the apology evaluation model M2 to learn the features of the extracted session. For example, if a session including an apology response is input, the session evaluation processing unit 60 outputs a score indicating that the user is not satisfied, and if a session that does not include an apology response is input, the session evaluation processing unit 60 outputs a score indicating that the user is not satisfied. The apology evaluation model M2 is trained so as to output a score indicating that the person is satisfied.

Subsequently, the session evaluation processing unit 60 inputs the target session into the apology evaluation model M2. Then, the session evaluation processing unit 60 evaluates whether or not the user is satisfied with the target session based on the output score of the apology evaluation model M2.

[2-5-9. Regarding the combination of each evaluation criteria]
Note that the session evaluation processing unit 60 may evaluate the target session by combining the evaluation methods described above. For example, FIG. 20 is a diagram showing the concept of processing for evaluating a session by combining multiple evaluation techniques. As shown in FIG. 20, the session evaluation processing unit 60 includes a repetition evaluation model M1, an apology evaluation model M2, an unfavorable indicator utterance evaluation model M3, a favorable indicator utterance evaluation model M4, a ternary evaluation model M5, and an engagement session evaluation. A model M6 and a repetitive speech evaluation model M7 are generated. Then, the session evaluation processing unit 60 inputs the target session and the distribution image generated from the target session to each of the models M1 to M7, and evaluates the target session.

For example, when the distribution image of the target session is similar to the distribution image of the repeated session, that is, when the target session includes repeated utterances, the repeat evaluation model M1 outputs an evaluation indicating that the user's satisfaction level is low. Further, the apology evaluation model M2 outputs an evaluation indicating that the user's satisfaction level is low when the target session includes an apology utterance. Further, the unfavorable index utterance evaluation model M3 outputs an evaluation indicating that the user's satisfaction level is low when the target session includes an unfavorable index utterance. Also, the favorable indicator utterance evaluation model M4 outputs an evaluation indicating that the user's satisfaction level is high when a favorable indicator utterance is included in the target session.

In addition, the ternary evaluation model M5 outputs an evaluation indicating that the user's satisfaction level is high when the target session includes a dialogue whose features are similar to the dialogue that the user is satisfied with. In addition, the engagement session evaluation model M6, when the characteristics of the target session are similar to the session including the user's utterance with high engagement, that is, when the session including the user's utterance with high satisfaction is similar, , outputs an evaluation indicating that the user's satisfaction is high. Further, the repetitive utterance evaluation model M7 outputs an evaluation indicating that the user's satisfaction level is low when the target session includes repetitive utterances.

The session evaluation processing unit 60 evaluates the target session based on a combination of evaluation results from each of these models M1 to M7. For example, the session evaluation processing unit 60 counts the number of models determined to have high user satisfaction with respect to the target session and the number of models determined to have low user satisfaction. If the number of determined models is larger than the number of models determined to have low satisfaction, it may be determined that the user's satisfaction with the target session is high. That is, the session evaluation processing unit 60 may evaluate the target session by majority vote.

Also, the session evaluation processing unit 60 may evaluate the target session using the evaluation results of some of the models M1 to M7. Also, for example, the session evaluation processing unit 60 may integrate the scores output by the models M1 to M7. For example, the session evaluation processing unit 60 calculates a score by subtracting the scores of the models M4 to M6 from the sum of the scores of the models M1, M2, M3, and M7, and evaluates the target session based on the calculated score. You may

Also, the session evaluation processing unit 60 may further learn evaluation results using the models M1 to M7. For example, the session evaluation processing unit 60 uses, as learning data, a session evaluated as having a high degree of user satisfaction and a session evaluated as having a low degree of user satisfaction, and makes the model learn, and uses such a model. may be used to evaluate the target session. Further, the index utterance extraction processing unit 50 may extract index utterances based on the evaluation result by the session evaluation processing unit 60 as described above. For example, the index utterance extraction processing unit 50 extracts a favorable index utterance from a session evaluated as having a high degree of satisfaction by the session evaluation processing unit 60, that is, from a highly satisfied session, and You may extract an unfavorable index utterance from the session evaluated as this, ie, a low satisfaction session.

[2-5-10. Regarding automatic expansion of labeled data]
Here, the session evaluation processing unit 60 may automatically expand session data used for learning each of the models M1 to M7. For example, FIG. 21 is a diagram showing the concept of processing for automatically expanding a labeled session.

For example, the session evaluation processing unit 60 extracts each utterance included in a session with low user satisfaction, that is, each utterance contained in a low-satisfaction session, and from the session log database 31, other utterances similar to each extracted utterance. Search for. The session evaluation processing unit 60 also extracts each response included in the low-satisfaction session, and searches the session log database 31 for other responses similar to each extracted response.

Then, the session evaluation processing unit 60 combines the searched utterances and responses to generate a new session in a pseudo fashion, and treats the generated new session as a low-satisfaction session, and learns during the learning of the models M1 to M7. may be used as data.

[2-5-11. Regarding the functional configuration of the session evaluation processing unit]
Next, an example of the functional configuration of the session evaluation processing unit 60 will be described with reference to FIG. 22 . 22 is a diagram illustrating an example of a functional configuration of a session evaluation processing unit according to the embodiment; FIG. As shown in FIG. 22 , session evaluation processing section 60 has learning data extraction section 61 , learning section 62 , image generation section 63 , utterance group generation section 64 , and evaluation section 65 .

The learning data extraction unit 61 extracts a session that becomes learning data. For example, as shown in FIG. 12, the learning data extracting unit 61 considers a session in which the appearance frequency of favorable indicator utterances and the total index score of favorable indicator utterances meet predetermined conditions as a favorable indicator utterance evaluation model M4. extracted as a highly satisfying session that serves as learning data for In addition, the learning data extraction unit 61 extracts sessions other than highly satisfactory sessions as sessions other than highly satisfactory sessions that serve as learning data for the favorable index utterance evaluation model M4.

Further, for example, as shown in FIG. 12, the learning data extraction unit 61 selects, as unfavorable, a session in which the appearance frequency of unfavorable indicator utterances and the total index score of unfavorable indicator utterances satisfy predetermined conditions. It is extracted as a highly satisfying session that serves as learning data for the index utterance evaluation model M3. In addition, the learning data extraction unit 61 extracts sessions other than the low satisfaction sessions as sessions other than the low satisfaction sessions that serve as learning data for the unfavorable index utterance evaluation model M3.

Note that the learning data extraction unit 61 refers to the session evaluation database 32 to identify a highly satisfied session and a lowly satisfied session, and classifies the identified highly satisfied session and lowly satisfied session as an unfavorable index utterance evaluation model M3 and a favorable index. It may be extracted from the session log database 31 as learning data for the utterance evaluation model M4.

In addition, as shown in FIG. 15, the learning data extracting unit 61 may extract sessions that serve as learning data for the engagement session evaluation model M6 based on the user's engagement. For example, the learning data extraction unit 61 refers to the engagement database 34 and identifies the usage mode of each user, that is, the history period and history frequency. Then, the learning data extracting unit 61 identifies a user whose specified usage mode satisfies a predetermined condition, and extracts from the session log database 31 a session including an utterance of the identified user.

For example, the learning data extraction unit 61 identifies a user whose history period or history frequency exceeds a predetermined threshold, and extracts a session including the identified user's utterance from the session log database 31 as a session of a user with high engagement. Extract. In addition, the learning data extraction unit 61 identifies a user whose history period or history frequency is below a predetermined threshold, and extracts a session including the identified user's utterance from the session log database 31 as a session of a user with low engagement. Extract. Then, the learning data extraction unit 61 may use each extracted session as learning data for the engagement session evaluation model M6.

That is, the learning data extracting unit 61 extracts sessions including user utterances whose utilization mode satisfies the predetermined condition as learning data having the first attribute of high user satisfaction. In addition, the learning data extraction unit 61 extracts a session including a user's utterance whose usage mode does not satisfy a predetermined condition, as learning data having a second attribute of low user satisfaction.

In addition, as shown in FIG. 16, the learning data extraction unit 61 extracts a session with a large number of similar orthographic pairs or a large number of similar semantic pairs from the session log database 31 as a low-satisfaction session. A session with a small number of semantically similar pairs is extracted as a highly satisfying session. Then, the learning data extraction unit 61 may use the extracted session as a session that is the source of learning data for the repetitive evaluation model M1 and the repetitive speech evaluation model M7.

Also, as shown in FIG. 19, the learning data extraction unit 61 extracts sessions including apology responses and sessions not including apology responses from the session log database 31 as learning data for the apology evaluation model M2. .

Note that the learning data extracting unit 61 may extract, as learning data, sessions useful for learning of each of the models M1 to M7, without being limited to the processing described above. For example, the learning data extraction unit 61 may extract high satisfaction sessions and low satisfaction sessions as learning data for each of the models M1 to M7 based on the evaluation results of the sessions registered in the session evaluation database 32.

The learning unit 62 learns each of the models M1 to M7. For example, as shown in FIG. 13, the learning unit 62 learns the unfavorable indicator utterance evaluation model M3 and the favorable indicator utterance evaluation model M4 as the learning data extracted by the learning data extraction unit 61 .

For example, as shown in FIG. 13, the learning unit 62 causes the unfavorable indicator utterance evaluation model M3 and the favorable indicator utterance evaluation model M4 to learn the features of the session whose evaluation satisfies a predetermined condition. For example, the learning unit 62 learns the favorable indicator utterance evaluation model M4 using the highly satisfying session extracted by the learning data extracting unit 61 and sessions other than the highly satisfying session as learning data. That is, the learning unit 62 learns the favorable indicator utterance evaluation model M4 that classifies the input target session into a highly satisfying session and other sessions.

Also, the learning unit 62 learns the unfavorable index utterance evaluation model M3 using the low-satisfaction session extracted by the learning data extraction unit 61 and sessions other than the low-satisfaction session as learning data. That is, the learning unit 62 learns the unfavorable index utterance evaluation model M3 that classifies the input target session into low-satisfaction sessions and other sessions.

The learning unit 62 also learns the ternary evaluation model M5, as shown in FIG. 14, using the dialogue examples generated by the utterance group generation unit 64 as learning data. That is, the learning unit 62 learns the ternary evaluation model M5 using a series of dialogues corresponding to the index utterance as learning data. For example, the utterance group generation unit 64 extracts a predetermined number of utterances and responses performed immediately before the unfavorable indicator utterance as unfavorable dialogue examples UD, A number of utterances and responses are extracted as favorable dialogue examples FD, and a predetermined number of utterances and responses performed immediately before the neutral utterance are extracted as neutral dialogue examples ND. In such a case, the learning unit 62 learns the ternary evaluation model M5 for evaluating the target session using the unfavorable dialogue example UD, the favorable dialogue example FD, and the neutral dialogue example ND as learning data. That is, the learning unit 62 learns a model for estimating the user's evaluation of the dialogue in the target session.

For example, the learning unit 62 uses the unfavorable dialogue example UD as learning data of a dialogue evaluated unfavorably by the user. In addition, the learning unit 62 uses the favorable dialogue example FD as learning data of a dialogue that is evaluated favorably by the user. In addition, the learning unit 62 uses the neutral dialogue example ND as learning data of a dialogue evaluated neutrally by the user. Then, the learning unit 62 causes the ternary evaluation model M5 to learn the features of the learning data, so that the evaluation by the user is input as an unfavorable dialogue, a favorable dialogue, or a neutral dialogue. Implement a model that classifies interactions.

In addition, as shown in FIG. 15, the learning unit 62 learns the engagement session evaluation model M6 using sessions of users with high engagement and sessions of users with low engagement extracted by the learning data extraction unit 61 as learning data. . That is, the learning unit 62 selects the input target session as a session similar to a session by a user with high engagement (hereinafter referred to as a “first attribute session”), or a session similar to a session by a user with low engagement. It learns an engagement session evaluation model M6 for classifying into sessions (hereinafter referred to as “second attribute sessions”).

Here, a session by a user with high engagement is estimated to have high user satisfaction, and a session by a user with low engagement is estimated to have low user satisfaction. For this reason, the session of the first attribute is a session in which the evaluation by the user is favorable, ie, a highly satisfied session, and the session of the second attribute is a session in which the evaluation by the user is unfavorable, ie, low satisfaction It can be said that it is a session. Therefore, the learning unit 62 can learn the engagement session evaluation model M6 that determines whether the user's impression of the target content is favorable or unfavorable.

Further, as shown in FIGS. 17 and 18, the learning unit 62 learns the repetitive utterance evaluation model M7 and the repetitive evaluation model M1 using sessions including repetitive utterances as learning data. For example, when the learning unit 62 acquires the repetitive session RS generated from the seed utterances and mixed utterances by the utterance group generating unit 64 and the non-repetitive session NRS by the method shown in FIG. Using the session NRS as learning data, the repetitive speech evaluation model M7 is learned. That is, the learning unit 62 learns the repetitive utterance evaluation model M7 that classifies the target session into sessions that include repetitive utterances and sessions that do not include repetitive utterances.

Here, it is estimated that a session including repeated utterances gives a negative impression to the user, and a session that does not include repeated utterances gives a favorable impression to the user. Therefore, the learning unit 62 can learn the repetitive utterance evaluation model M7 that determines whether the user's impression of the target content is favorable or unfavorable.

In addition, the learning unit 62 acquires, as learning data, a distribution image generated by the image generation unit 63 from sessions including repeated utterances and a distribution image generated by the image generation unit 63 from sessions not including repeated utterances. In such a case, the learning unit 62 repeatedly learns the evaluation model M1 using these distribution images as learning data. More specifically, the learning unit 62 converts the input image into an image similar to a distribution image generated from a session including repeated utterances, or an image similar to a distribution image generated from a session not including repeated utterances. The repeated evaluation model M1 for classification is learned.

Here, oblique lines tend to appear in the distribution image of sessions that include repeated utterances, and oblique lines are less likely to appear in the distribution images of sessions that do not include repeated utterances. Furthermore, it is presumed that the user's impression is unfavorable in a session that includes repeated utterances, and that the user has a favorable impression in a session that does not include repeated utterances. Therefore, the learning unit 62 determines whether the user's impression of the target session is favorable or unfavorable by classifying the distribution image in which the utterance character strings of the target session are vertically and horizontally associated. The iterative evaluation model M1 can be learned.

Also, as shown in FIG. 19, the learning unit 62 learns the apology evaluation model M2 using the sessions extracted by the learning data extraction unit 61 as learning data. For example, the learning unit 62 learns an apology evaluation model M2 that classifies the input target session into a session similar to a session containing an apology response or a session similar to a session not containing an apology response.

Here, it is presumed that a session including an apology response gives a negative impression to the user, and a session that does not include an apology response gives a favorable impression to the user. Therefore, the learning unit 62 can learn the apology evaluation model M2 that determines whether the user's impression of the target session is favorable or unfavorable based on the apology responses.

In addition, as shown in FIG. 21, the learning unit 62 may generate a new low-satisfaction session from the low-satisfaction session acquired as learning data and add it to the learning data. For example, the learning unit 62 extracts utterances and responses similar to each utterance and each response included in the session acquired as learning data from the session log database 31, and combines the extracted utterances and responses to perform new learning. data may be generated.

Here, the learning unit 62 may assign the same label as that of the learning data from which the new learning data is generated to the new learning data. For example, the learning unit 62 learns the user's impression from dialogue examples in which the user's impression is favorable, such as sessions with many favorable indicator utterances, sessions of users with high engagement, favorable dialogue examples FD, and non-repetitive sessions NRS. It is also possible to generate new examples of favorable dialogues and use them as learning data. In addition, for example, the learning unit 62 learns from examples of dialogues in which the impression of the user is unfavorable, such as sessions with many unfavorable index utterances, sessions of users with low engagement, unfavorable dialogue examples ND, repetitive sessions RS, etc. It is also possible to generate new dialogue examples that give a negative impression to the user and use them as learning data.

The image generator 63 generates a distribution image. For example, the image generating unit 63 acquires a session including repeated utterances extracted by the learning data extracting unit 61 and a session not including repeated utterances. Further, the image generator 63 acquires the repetitive session RS and the non-repetitive session NRS generated by the utterance group generator 64 as shown in FIG. Note that the image generator 63 may generate a new session from the acquired session, as shown in FIG.

Then, the image generator 63 generates a distribution image indicating whether or not the characters of the utterances included in the acquired sessions are the same. That is, the image generator 63 generates an image showing the distribution of identical characters in the session. For example, when the image generator 63 associates a character string of an utterance included in a session with each vertical region and each horizontal region of the image, the same character corresponds to the vertical direction and the horizontal direction. An image is generated in which a predetermined color is added to the marked area.

The utterance group generation unit 64 extracts unfavorable dialogue examples UD, favorable dialogue examples FD, and neutral dialogue examples ND. That is, the utterance group generation unit 64 identifies index utterances from the history of the dialogue service, and extracts a series of dialogues corresponding to the identified index utterances as a dialogue group. For example, the utterance group generation unit 64 extracts a predetermined number of previous responses and utterances from the index utterance as learning data. The number of responses and utterances to be extracted as dialogue examples from the responses and utterances before the index utterance can be arbitrarily set.

For example, the utterance group generation unit 64 refers to the session log database 31 and identifies a session including favorable indicator utterances. Subsequently, the utterance group generation unit 64 extracts a predetermined number of previous responses and utterances from the favorable indicator utterances as favorable dialogue examples FD. Similarly, the utterance group generation unit 64 identifies a session including unfavorable indicator utterances, and extracts a predetermined number of previous responses and utterances from the unfavorable indicator utterances as unfavorable dialogue examples UD. Further, the utterance group generation unit 64 identifies a session including a neutral utterance, and extracts a predetermined number of previous responses and utterances from the neutral utterance as neutral dialogue examples ND.

The utterance group generation unit 64 also generates a repetitive session RS (ie, first utterance group) and a non-repetitive session NRS (ie, second utterance group). For example, the utterance group generation unit 64 selects any utterance from the history of the dialogue service, and generates an utterance group including a plurality of utterances that are the same as or similar to the selected utterance.

To give a more specific example, the utterance group generation unit 64 extracts seed utterances from the session log database 31 and generates multiple duplicate utterances similar to the seed utterances. Also, the utterance group generation unit 64 extracts mixed utterances from the session log database 31 . Then, the utterance group generation unit 64 randomly rearranges the seed utterances, the plurality of duplicate utterances, and the mixed utterances, and generates a session composed of pseudo utterances as a repeated session RS. The utterance group generation unit 64 also refers to the session log database 31, randomly extracts a plurality of utterances, and generates a session in which the extracted utterances are randomly rearranged as a non-repetitive session NRS.

Note that the utterance group generation unit 64 generates utterances and responses similar to utterances and responses included in the unfavorable dialogue example UD, the favorable dialogue example FD, or the neutral dialogue example ND (hereinafter collectively referred to as "dialogue examples"). and combining the extracted utterances and responses to generate new example dialogues. For example, the utterance group generation unit 64 may generate a new unfavorable dialogue example UD from the unfavorable dialogue example UD by the method shown in FIG. An FD may be generated, and a new neutral dialogue example ND may be generated from the neutral dialogue example ND. Also, the utterance group generation unit 64 may generate a new repetitive session RS from the repetitive session RS, or may generate a new non-repetitive session NRS from the non-repetitive session NRS.

The evaluation unit 65 evaluates sessions. For example, the evaluation unit 65 selects a target session to be evaluated from the session log database 31 . Then, the evaluation unit 65 refers to the index utterance database 33 and evaluates the target session based on the frequency of appearance of the index utterances extracted by the index utterance extraction processing unit 50 and the seed index utterances.

For example, the evaluation unit 65 evaluates the target session based on whether or not the non-favorable indicator utterance or the favorable indicator utterance is included. For example, as shown in FIG. 12, the evaluation unit 65 determines whether the target session is a low-satisfaction session based on the ratio of the appearance frequency of unfavorable indicator utterances to the appearance frequency of favorable indicator utterances. Evaluate whether the session is highly satisfying. In addition, the evaluation unit 65 determines whether the target session is a low-satisfaction session or a high-satisfaction session based on the ratio of the total index score of the unfavorable index utterances and the total index score of the favorable index utterances. may be evaluated.

Here, the index utterances used for evaluating the session include index utterances extracted by the index utterance extraction processing unit 50, that is, index utterances extracted by co-occurrence with the seed index utterances. For this reason, the evaluation unit 65 determines, for example, the appearance frequency of an utterance whose co-occurrence with a non-favorable index utterance satisfies a predetermined condition, or the appearance of an utterance whose co-occurrence with a favorable index utterance satisfies a predetermined condition. Subject sessions will be rated based on frequency.

Also, the index utterances used for session evaluation include index utterances extracted by the index utterance extraction processing unit 50 based on user engagement. That is, the index utterance is extracted based on the appearance frequency in a session of a user whose usage pattern satisfies a predetermined condition, or the appearance frequency in a session of a user whose usage pattern does not satisfy a predetermined condition. is included. Therefore, the evaluation unit 65 evaluates the target session based on the user's utterance whose usage mode satisfies the predetermined condition and the user's utterance whose usage mode does not satisfy the predetermined condition.

In addition, index utterances used for session evaluation include index utterances extracted based on the ratio of the frequency of appearance in low-satisfaction sessions to the frequency of appearance in high-satisfaction sessions. Therefore, the evaluation unit 65 evaluates the target session based on an utterance whose appearance frequency satisfies a predetermined condition in a low-satisfaction session and an utterance whose appearance frequency satisfies a predetermined condition in a high-satisfaction session.

In addition, as shown in FIG. 13, the evaluation unit 65 may evaluate the target session using a favorable index utterance evaluation model M4 that has learned the characteristics of the highly satisfying session. Moreover, the evaluation unit 65 may evaluate the target session using the unfavorable index utterance evaluation model M3 that has learned the characteristics of the low-satisfaction session. For example, the evaluation unit 65 reads the unfavorable indicator utterance evaluation model M3 and the favorable indicator utterance evaluation model M4 from the model database 35, and assigns the target session to the unfavorable indicator utterance evaluation model M3 and the favorable indicator utterance evaluation model M4. input. Then, the evaluation unit 65 may evaluate the target session based on the scores output by the unfavorable index utterance evaluation model M3 and the favorable index utterance evaluation model M4. For example, when the unfavorable indicator utterance evaluation model M3 classifies the session as similar to the session including the unfavorable indicator utterance, the evaluation unit 65 evaluates the target session as a low-satisfaction session, and evaluates the favorable indicator utterance. If the model M4 classifies the session as similar to a session containing favorable indicator utterances, the target session may be evaluated as a highly satisfying session.

In addition, as shown in FIG. 14, the evaluation unit 65 uses a ternary evaluation model M5 that has learned the characteristics of the unfavorable dialogue example UD, the neutral dialogue example ND, and the favorable dialogue example FD to evaluate the target session. may be evaluated. For example, the evaluation unit 65 may input the target session into the ternary evaluation model M5 read from the model database 35, and evaluate the target session based on the classification results of the ternary evaluation model M5. For example, when the target session is classified as being similar to the unfavorable dialogue example UD by the ternary evaluation model M5, the evaluation unit 65 evaluates the target session as a low-satisfaction session, and determines that the target session is similar to the favorable dialogue example FD. If classified, the subject session may be rated as a highly satisfying session.

In addition, as shown in FIG. 15, the evaluation unit 65 may evaluate the target session based on the similarity with the session including the user's utterance whose usage mode satisfies a predetermined condition. For example, the evaluation unit 65 inputs the target session to the engagement session evaluation model M6 that has learned the characteristics of the session of the user with high engagement and the session of the user with low engagement, and based on the classification result of the engagement session evaluation model M6 , may rate the subject session. For example, if the engagement session evaluation model M6 classifies the target session as similar to the session of the user with high engagement, the evaluation unit 65 may evaluate the target session as a highly satisfying session.

The evaluation unit 65 may also evaluate the target session based on repeated utterances, as shown in FIGS. For example, when the same or similar character strings appear repeatedly in the target session, the evaluation unit 65 evaluates the target session as a low-satisfaction session, and the same or similar character strings do not appear repeatedly in the target session. If so, the target session may be rated as a highly satisfying session.

The evaluation unit 65 may also evaluate the target session based on the number of pairs of utterances in which the degree of similarity of character strings or the degree of semantic similarity exceeds a predetermined threshold. For example, as shown in FIG. 16, the evaluation unit 65 evaluates the target session as a low-satisfaction session when the number of similar pairs of expressions and the number of similar pairs of meanings included in the target session exceeds a predetermined threshold, When the number of similar pairs or the number of semantically similar pairs is below a predetermined threshold, the target session may be evaluated as a highly satisfying session.

In addition, as shown in FIG. 17, the evaluation unit 65 may evaluate the target session using a repetitive utterance evaluation model M7 that has learned the characteristics of sessions including repetitive utterances. For example, the evaluation unit 65 inputs the target session to the repetitive utterance evaluation model M7, and when the repetitive utterance evaluation model M7 classifies the target session into sessions similar to the repetitive session RS, evaluates the target session as a low-satisfaction session. However, if the session of interest is classified as a session similar to the non-repeating session NRS, then the session of interest may be evaluated as a highly satisfying session.

In addition, as shown in FIG. 18, the evaluation unit 65 uses a repeated evaluation model M1 that learns the characteristics of a session including repeated utterances by learning the characteristics of a distribution image generated from utterances in a session that includes repeated utterances. may be used to rate the target session. For example, the image generator 63 generates a distribution image of utterances included in the target session. Then, the image generator 63 evaluates the target session based on the distribution image.

For example, when the evaluation unit 65 inputs the target image to the repeated evaluation model M1, and the repeated evaluation model M1 classifies the distribution image of the target session into an image similar to the distribution image of the session including the repeated utterance, the target A session may be rated as a low satisfaction session. Further, for example, when the evaluation unit 65 inputs the target image to the repeated evaluation model M1, and the repeated evaluation model M1 classifies the distribution image of the target session into an image similar to the distribution image of the session that does not include repeated utterances. may rate the subject session as a highly satisfying session.

Note that the evaluation unit 65 uses a known image analysis technique to determine whether or not a diagonal line is included in the distribution image of the target session. You may rate it as a satisfactory session. That is, the evaluation unit 65 determines whether or not the target session includes repeated utterances from the distribution image of the target session without using the repeat evaluation model M1, and evaluates the target session based on the determination result. may

The evaluation unit 65 may also evaluate the target session based on whether or not an apology response is included. For example, the evaluation unit 65 may evaluate the target session as a low-satisfaction session when the number of apology responses included in the target session exceeds a predetermined threshold. Further, as shown in FIG. 19, the evaluation unit 65 inputs the target session to the apology evaluation model M2, and when the apology evaluation model M2 classifies the target session into a session similar to the session including the apology response, the target A session may be rated as a low satisfaction session. Further, when the apology evaluation model M2 classifies the target session into a session similar to a session that does not include an apology response, the evaluation unit 65 may evaluate the target session as a highly satisfying session.

Also, the evaluation unit 65 may evaluate the target session using a plurality of models. For example, as shown in FIG. 20, the evaluation unit 65 evaluates a plurality of models M1 to M7 for evaluating a session based on different types of features among a plurality of features that serve as indicators for user evaluation of a session. may be used to rate the subject session.

Note that when a session is evaluated using evaluation results of a plurality of models M1 to M7 in this way, any combination of models can be adopted. For example, as shown in FIG. 1, the evaluation unit 65 assigns a target session (and a target session ) is input, and a value is calculated by subtracting the value of the score output by the favorable index utterance evaluation model M4 from the sum of the scores output by the respective models M1 to M3. Then, the evaluation unit 65 evaluates, among the target sessions, a predetermined number of sessions with the highest calculated value as low-satisfaction sessions, and evaluates a predetermined number of sessions with the lowest calculated value as highly-satisfactory sessions. You may

If the value obtained by subtracting the value of the score output by the favorable index utterance evaluation model M4 from the sum of the scores output by each of the models M1 to M3 is calculated, the user is satisfied with the target session. is calculated as a negative value, and if the user is not satisfied with the target session, a positive value is calculated. Therefore, the evaluation unit 65 may evaluate the target session according to whether the calculated value is a positive value or a negative value.

In addition, the evaluation unit 65, for example, evaluates the session based on whether or not the same or similar utterances are repeatedly included in the utterances included in the session. may be used. Moreover, the evaluation unit 65 may use at least the evaluation result of the apology evaluation model M2 that evaluates the session based on whether or not the response included in the session includes a predetermined response indicating an apology.

In addition, the evaluation unit 65 may use at least the evaluation result of the unfavorable index utterance evaluation model M3 that evaluates the session based on whether or not the utterances included in the session include the unfavorable index utterance. Moreover, the evaluation unit 65 may use at least the evaluation result of the favorable indicator utterance model M4 that evaluates the session based on whether or not the utterances included in the session include a favorable indicator utterance.

Here, for the unfavorable indicator utterance evaluation model M3 and the favorable indicator utterance evaluation model M4, it is necessary to set or extract an indicator utterance in advance when extracting a session that serves as learning data. However, it takes time and effort to set and extract index utterances in advance. Therefore, the evaluation unit 65 may evaluate the target content only based on the evaluation results of the repetition evaluation model M1 and the apology evaluation model M2. For example, the evaluation unit 65 calculates the sum of the score of the repeated evaluation model M1 and the score of the apology evaluation model M2, and evaluates a predetermined number of target sessions in descending order of the sum as low-satisfaction sessions, A predetermined number of target sessions may be evaluated as high-satisfaction sessions in ascending order of the sum value.

When evaluating the target session using a plurality of models M1 to M7, the evaluation unit 65 may set predetermined weights to the evaluation results of each model M1 to M7. For example, the evaluation unit 65 may multiply the score values output from the models M1 to M7 by a predetermined coefficient for each of the models M1 to M7, and integrate the score values obtained by multiplying the coefficients. Note that the evaluation unit 65 registers the evaluation result in the session evaluation database 32 when evaluating the target session. Such evaluation results may be used, for example, to extract index utterances, or may be used to evaluate other sessions.

[2-6. About prediction processing]
Returning to FIG. 2, the description is continued. The engagement prediction processing unit 70 executes prediction processing for predicting how the user will use the interactive service in the future. An example of the concept of processing executed by the engagement prediction processing unit 70 and the functional configuration of the engagement prediction processing unit 70 will be described below with reference to FIGS. 23 to 25. FIG.

[2-6-1. Prediction processing based on indicator utterance]
For example, the engagement prediction processing unit 70 performs prediction processing based on index utterances included in a session of a user (hereinafter referred to as a "target user") who is a target of predicting the manner of use of the interactive service in the future. Run. For example, FIG. 23 is a diagram showing the concept of processing for predicting future usage patterns based on indicator utterances.

As shown in FIG. 23, it is considered that a user who makes many favorable indicator utterances has a favorable impression of the interactive service. In this way, users who have a favorable impression of interactive services naturally use interactive services, and as a result, it is presumed that they will become users with high engagement in terms of usage period, frequency of use, and the like. On the other hand, users who make many unfavorable indicator utterances are considered to have an unfavorable impression of the interactive service. In this way, users who have an unfavorable impression of interactive services are presumed to stop using interactive services and become low-engagement users.

Therefore, the engagement prediction processing unit 70 predicts the user's future engagement based on the appearance frequency of the index utterance in the session including the user's utterance. For example, the engagement prediction processing unit 70 is a session that includes the target user's utterance, and includes a dialogue that took place within a predetermined period from the execution of the prediction process (hereinafter referred to as an "observation period"). Extract session.

Subsequently, the engagement prediction processing unit 70 extracts the favorable indicator utterances and the unfavorable indicator utterances included in the session during the observation period, the total appearance frequency of the favorable indicator utterances during the observation period, and the non-favorable indicator utterances during the observation period. Calculate the total frequency of occurrence of favorable indicator utterances. Then, the engagement prediction processing unit 70 calculates a score by dividing the total appearance frequency of favorable indicator utterances in the observation period by the total appearance frequency of unfavorable indicator utterances in the observation period. It is determined that the user will be highly engaged in the future.

Note that the engagement prediction processing unit 70 may consider the index score values of favorable index utterances and unfavorable index utterances. For example, the engagement prediction processing unit 70 calculates the total index score of favorable index utterances during the observation period and the total index score of unfavorable index utterances during the observation period. Then, the engagement prediction processing unit 70 calculates a score by dividing the total index score of favorable index utterances in the observation period by the total index score of unfavorable index utterances in the observation period. It may be determined that the user is highly engaged in the future. The engagement prediction processing unit 70 then registers the prediction result in the engagement database 34 .

The future engagement predicted by such processing is used to extract index utterances and evaluate sessions. For example, in the process shown in FIG. 9, the index utterance extraction processing unit 50 treats users with high future engagement predicted by the engagement prediction processing unit 70 as favorable users, An index utterance may be extracted from each user's utterance by setting a user with low engagement as an unfavorable user. Also, for example, the session evaluation processing unit 60, in the processing shown in FIG. The model M6 may be trained, and the target session may be evaluated using the trained engagement session evaluation model M6.

[2-6-2. Prediction processing based on evaluation results]
Further, for example, the engagement prediction processing unit 70 may execute prediction processing based on the evaluation result of the session. For example, FIG. 24 is a diagram showing the concept of processing for predicting future usage patterns based on evaluation results.

For example, as shown in FIG. 24, it is presumed that a session evaluated as having a high degree of satisfaction by the evaluation process, that is, a user with many highly satisfied sessions has a favorable impression of the interactive service. Engagement is expected to increase in the future. On the other hand, it is presumed that users with a high number of low-satisfaction sessions will have an unfavorable impression of the interactive service, and their future engagement will be low. be done.

Therefore, the engagement prediction processing unit 70 predicts the future engagement of the user based on the evaluation result for the session including the target user's utterance. For example, the engagement prediction processing unit 70 extracts a session that includes an utterance of the target user and includes a dialogue that took place during the observation period. Subsequently, the engagement prediction processing unit 70 acquires evaluation results for the extracted sessions. For example, the engagement prediction processing unit 70 refers to the session evaluation database 32 and acquires the evaluation result by the session evaluation processing unit 60 .

Then, the engagement prediction processing unit 70 predicts the engagement of the target user in the prediction period based on the statistical value of the evaluation result. For example, the engagement prediction processing unit 70 calculates the ratio of high-satisfaction sessions and the ratio of low-satisfaction sessions among the extracted sessions. Then, the engagement prediction processing unit 70 may determine that the target user is a user with high future engagement as the rate of highly satisfied sessions is higher. Further, for example, when the number of highly satisfied sessions among the extracted sessions exceeds a predetermined threshold, the engagement prediction processing unit 70 may determine that the target user is a user with high future engagement.

Also, for example, the engagement prediction processing unit 70 assigns a positive score to a highly satisfying session and a negative score to a low satisfying session, and sets a larger coefficient to a newer session. Then, the engagement prediction processing unit 70 estimates the future engagement of the target user based on whether or not the total score of the product of the score of each session and the coefficient exceeds a predetermined threshold. good. In addition, the engagement prediction processing unit 70 assigns a negative score to a highly satisfied session and a positive score to a low satisfied session, and sets a larger coefficient for a newer session, and the value of the product of the score and the coefficient for each session The future engagement of the target user may be estimated based on whether or not the total score of is equal to or less than a predetermined threshold.

[2-6-3. Regarding the functional configuration of the engagement prediction processing unit]
Next, an example of the functional configuration of the engagement prediction processing unit 70 will be described with reference to FIG. 25 . 25 is a diagram illustrating an example of a functional configuration of an engagement prediction processing unit according to the embodiment; FIG. As shown in FIG. 25 , the engagement prediction processing section 70 has an estimation section 71 .

The estimating unit 71 estimates the future utilization mode of the interactive service by the target user based on the evaluation result for the session including the target user's utterance. For example, the estimating unit 71 calculates the sum of the index scores of the favorable index utterances during the observation period, that is, the favorable score indicating how favorable the evaluation by the user is. The estimation unit 71 also calculates the total index score of the unfavorable index utterances during the observation period, that is, the unfavorable score indicating how unfavorable the evaluation by the user is. Then, the estimating unit 71 estimates the future engagement of the target user, that is, the usage mode, based on the ratio of the favorable score and the unfavorable score.

In addition, the estimation unit 71 identifies whether each session in the target user's observation period is a high-satisfaction session or a low-satisfaction session. Then, the estimating unit 71 estimates the future usage mode of the target user based on the number, ratio, and the like of the highly satisfying sessions and the lowly satisfying sessions.

[2-7. Regarding the relevance of each process]
Here, with reference to FIG. 26, the relationship between the extraction processing performed by the index utterance extraction processing unit 50, the evaluation processing performed by the session evaluation processing unit 60, and the prediction processing performed by the engagement prediction processing unit 70 will be described. FIG. 26 is a diagram illustrating an example of relationships between processes executed by the information providing apparatus according to the embodiment;

First, the relationship between extraction processing and evaluation processing will be described. For example, it is estimated that a session with a high evaluation tends to include favorable indicator utterances, and a session with a low evaluation tends to include unfavorable indicator utterances. Therefore, the evaluation result of the session can be used for the index utterance extraction process. More specifically, the information providing apparatus 10 can improve the extraction amount and the extraction accuracy of the index utterances by proceeding with the evaluation of the session through the evaluation process described above. On the other hand, a session including favorable indicator utterances is estimated to have a high evaluation, and a session including unfavorable indicator utterances is estimated to have a low evaluation. Therefore, the index utterance extraction result can be used for session evaluation processing. More specifically, the information providing apparatus 10 can improve the evaluation accuracy of the session by extracting the indicator utterance through the extraction process described above.

Next, the relationship between extraction processing and prediction processing will be described. For example, it is estimated that users with high engagement uttered favorable indicator utterances frequently, and users with low engagement uttered unfavorable indicator utterances less frequently. Therefore, the engagement prediction result can be used for index utterance extraction processing. More specifically, the information providing apparatus 10 can improve the extraction amount and extraction accuracy of index utterances by predicting engagement. On the other hand, it is presumed that users who make many favorable indicator utterances have high engagement, and users who make many unfavorable indicator utterances have low engagement. Therefore, the extraction result of the index utterance can be used for engagement prediction processing. More specifically, the information providing apparatus 10 can improve the engagement prediction accuracy by extracting the indicator utterance.

Next, the relationship between evaluation processing and prediction processing will be described. For example, it is estimated that a user with many sessions with high ratings has high engagement, and a user with many sessions with low ratings has low engagement. Therefore, session evaluation results can be used for engagement prediction processing. More specifically, the information providing apparatus 10 can improve the engagement prediction accuracy by evaluating the session. On the other hand, users with high engagement are estimated to have high session evaluations, and users with low engagement are estimated to have low session evaluations. Therefore, the engagement prediction result can be used for session evaluation processing. More specifically, the information providing apparatus 10 can improve the accuracy of session evaluation by predicting engagement.

In this way, the various extraction processes, evaluation processes, and prediction processes described above have a bootstrap relationship with each other. Therefore, the information providing apparatus 10 can gradually improve the accuracy of each process by repeatedly executing each process in order. For example, the information providing device 10 may alternately perform extraction processing and prediction processing, extraction processing and evaluation processing, or evaluation processing and prediction processing. Further, the information providing apparatus 10 may repeatedly execute each process in order.

[2-8. Reinforcement learning processing]
Returning to FIG. 2, the description is continued. The reinforcement learning processing unit 80 executes reinforcement learning of the dialogue model used by the dialogue device 200 .

[2-8-1. An example of reinforcement learning]
For example, FIG. 27 is a diagram illustrating the concept of processing for executing reinforcement learning of a dialogue model. The dialogue device 200 uses the dialogue model M10 to generate a response to the user's utterance. Here, in the interactive service, in response to the response, the user may utter completely new content (sentence different from the previous sentence), or may utter an utterance including an evaluation of the response. For example, the above-mentioned favorable indicator utterances and unfavorable indicator utterances are considered to be evaluations of responses made immediately before. Then, when the dialog model M10 is regarded as an agent in reinforcement learning and the user as an environment in reinforcement learning, various index utterances uttered by the user can be regarded as rewards for the agent.

Therefore, the reinforcement learning processing unit 80 implements reinforcement learning for the dialogue model M10 by setting rewards based on index utterances for the dialogue model M10. More specifically, the reinforcement learning processing unit 80 acquires index utterances serving as an evaluation index for the dialogue service from among the utterances to the response model M10, and sets a reward based on the obtained index utterances, thereby obtaining the response model. Reinforcement learning of M10 is performed.

For example, the reinforcement learning processing unit 80 refers to the session log database 31 and identifies favorable indicator utterances and unfavorable indicator utterances included in the session. Then, when the favorable indicator utterance is identified, the reinforcement learning processing unit 80 sets a positive reward as a reward for the response immediately before the identified favorable indicator utterance, and learns the dialogue model M10. Further, when an unfavorable index utterance is identified, the reinforcement learning processing unit 80 sets a negative reward as a reward for the response immediately before the identified unfavorable index utterance, and learns the dialogue model M10. .

Further, the reinforcement learning processing unit 80 generates a user model model M20 that imitates the user's utterance, causes the user model model M20 and the dialogue model M10 to have a virtual dialogue, and the user model model M20 Reinforcement learning of the dialogue model M10 may be performed by setting a reward according to the output index utterance. For example, the reinforcement learning processing unit 80 generates the user model model M20 using predetermined dialogue rules and sessions registered in the session log database 31 as learning data. Subsequently, the reinforcement learning processing unit 80 inputs the utterance generated by the user model M20 to the dialogue model M10, and inputs the response output by the dialogue model M10 to the user model M20.

Further, the reinforcement learning processing unit 80 repeatedly executes a virtual dialogue between the dialogue model M10 and the user model model M20. Then, when the user model model M20 outputs an index utterance as an utterance, the reinforcement learning processing unit 80 sets a reward corresponding to the index utterance, and performs reinforcement learning of the dialogue model M10 based on the set reward. .

[2-8-2. Example of functional configuration possessed by reinforcement learning processing unit]
Next, an example of the functional configuration of the reinforcement learning processing unit 80 will be described with reference to FIG. 28 . 28 is a diagram illustrating an example of a functional configuration of a reinforcement learning processing unit according to the embodiment; FIG. As shown in FIG. 28 , the reinforcement learning processing section 80 has an indicator utterance acquisition section 81 , a reward setting section 82 and a reinforcement learning section 83 .

The index utterance acquisition unit 81 acquires an index utterance that serves as an index for evaluating the dialogue service among the utterances for the dialogue model M10. More specifically, the index utterance acquisition unit 81 acquires favorable index utterances and unfavorable index utterances from the utterances output by the user and the user model model M20. For example, the index utterance acquisition unit 81 refers to the session log database 31 and acquires an index utterance. Also, the index utterance acquisition unit 81 acquires an index utterance among the utterances output by the user model model M20.

The reward setting unit 82 sets a reward according to the index utterance acquired by the index utterance acquisition unit 18 . For example, the reward setting unit 82 sets a reward according to the index score of the acquired index utterance. More specifically, the reward setting unit 82 sets a positive reward when the obtained index utterance is a favorable index utterance, and sets a positive reward when the obtained index utterance is a non-favorable index utterance. Set negative rewards.

The reinforcement learning unit 83 uses the reward set by the reward setting unit 82 to perform reinforcement learning for the dialogue model M10. For example, when the index utterance acquisition unit 81 extracts a certain index utterance from the session, the reinforcement learning unit 83 identifies the response immediately before the index utterance, and the reward when the dialogue model M10 outputs the identified response is Reinforcement learning of the dialogue model M10 is performed as a reward based on the index utterance.

Further, for example, the reinforcement learning unit 83 uses an utterance model that outputs an index utterance in response to a response generated by the dialogue model M10, that is, a reward based on the index utterance output by the user model model M20, to generate the dialogue model M10. of reinforcement learning. For example, the reinforcement learning unit 83 uses the content of each session registered in the session log database 31 and the dialogue rule to learn the user model model M20. For such a user model model M20, for example, a learning method similar to that for learning the dialogue model M10 can be adopted.

Subsequently, the reinforcement learning unit 83 causes the user model model M20 and the dialogue model M10 to execute a virtual dialogue. Then, when the user model model M20 outputs the index utterance, the reinforcement learning unit 83 executes reinforcement learning of the dialogue model M10 based on the reward set by the reward setting unit .

[3. Example of flow of processing executed by information providing device]
Next, an example of the flow of each process executed by the information providing apparatus 10 will be described with reference to FIGS. 29 to 37. FIG.

[3-1. Example of flow of processing based on usage mode]
First, with reference to FIG. 29, an example of the flow of processing executed by the information providing apparatus 10 based on the usage mode will be described. FIG. 29 is a flowchart illustrating an example of the flow of processing executed by the information providing device according to the embodiment based on the usage mode;

First, the information providing device 10 specifies the manner in which each user uses the interactive service (step S101). Subsequently, the information providing device 10 identifies users whose identification results satisfy predetermined conditions (step S102). Then, the information providing device 10 extracts a session including the specified user's speech (step S103). Further, the information providing device 10 learns the engagement session evaluation model M6 using the sessions thus extracted as learning data (step S104), and ends the process.

Note that the information providing apparatus 10 may evaluate the extracted session (step S105). For example, when extracting sessions of users with high engagement, the information providing apparatus 10 may evaluate those sessions as highly satisfying sessions. Further, the information providing device 10 may extract index utterances from the session thus extracted (step S106). For example, the information providing apparatus 10 extracts an utterance whose frequency of appearance in a session of a user with high engagement satisfies a predetermined condition as a favorable indicator utterance, and the frequency of appearance in a session of a user with low engagement satisfies a predetermined condition. You may extract an utterance as an unfavorable indicator utterance.

[3-2. Example of flow of processing for extracting indicator utterance]
Next, with reference to FIG. 30, an example of the flow of processing in which the information providing device 10 extracts index utterances will be described. FIG. 30 is a flowchart showing an example of the flow of processing for extracting index utterances by the information providing device according to the embodiment.

First, the information providing apparatus 10 acquires a session of a user whose usage mode satisfies a predetermined condition (step S201), and extracts an index utterance based on the appearance frequency in the acquired session (step S202). Then, the information providing device 10 further extracts a new index utterance based on the co-occurrence with the index utterance (step S203), and ends the process.

Note that the information providing device 10 may evaluate the session based on the extracted index utterance (step S204). For example, the information providing apparatus 10 may evaluate a session containing many favorable indicator utterances as a highly satisfying session, and evaluate a session containing many unfavorable indicator utterances as a low-satisfaction session. Further, the information providing apparatus 10 may estimate the usage mode of the user based on the appearance frequency of the indicator utterance (step S205). For example, the information providing apparatus 10 estimates a user who utters many favorable indicator utterances as a user with high engagement, and estimates a user who utters many unfavorable indicator utterances as a user with low engagement. good too.

[3-3. An example of the flow of processing for evaluating a session]
Next, with reference to FIG. 31, an example of the flow of processing in which the information providing apparatus 10 evaluates a session will be described. 31 is a flowchart illustrating an example of the flow of processing for evaluating a session by the information providing device according to the embodiment; FIG.

First, the information providing apparatus 10 acquires a session to be evaluated (step S301), and evaluates the session based on the appearance frequency of index utterances in the acquired session (step S302). For example, the information providing device 10 evaluates a session containing many favorable indicator utterances as a highly satisfying session, and evaluates a session containing many unfavorable indicator utterances as a low-satisfaction session. Then, the information providing apparatus 10 learns the characteristics of the session as learning data to which a label corresponding to the evaluation result is assigned (step S303), and ends the process. For example, the information providing device 10 makes each of the models M3 and M4 learn the characteristics of a highly satisfying session and a less satisfying session.

Note that the information providing apparatus 10 may extract a new index utterance from the session based on the evaluation result of the session (step S304). For example, the information providing apparatus 10 may extract an utterance with a high frequency of appearance in a highly satisfied session as a favorable index utterance, and extract an utterance with a high frequency of appearance in a low-satisfaction session as a non-favorable index utterance. Further, the information providing apparatus 10 may estimate the usage mode of the user based on the evaluation result of the session (step S305). For example, the information providing apparatus 10 may estimate a user with many highly satisfied sessions as a user with high engagement, and a user with many low-satisfied sessions as a user with low engagement.

[3-4. An example of the flow of processing for evaluating a session based on images]
Next, with reference to FIG. 32, an example of the flow of processing in which the information providing apparatus 10 evaluates sessions based on images will be described. 32 is a flowchart illustrating an example of the flow of processing for evaluating a session based on an image by the information providing device according to the embodiment; FIG.

First, the information providing apparatus 10 acquires a session to be evaluated (step S401), and generates an image indicating whether or not characters of utterances included in the acquired session are the same, that is, a distribution image (step S401). S402). The information providing device 10 then evaluates the session based on the generated distribution image (step S403). For example, when the distribution image includes oblique lines, the information providing apparatus 10 evaluates the target session as a low-satisfaction session when classified as including repetition by the repetition evaluation model M1. The information providing apparatus 10 then learns the characteristics of the session as learning data to which a label corresponding to the evaluation result is assigned (step S404), and terminates the process.

Note that the information providing apparatus 10 may extract a new index utterance from the session based on the evaluation result of the session (step S405). Further, the information providing apparatus 10 may estimate the usage mode of the user based on the evaluation result of the session (step S406).

[3-5. An example of the flow of processing for evaluating a session based on repetition of utterance]
Next, with reference to FIG. 33, an example of the flow of processing in which the information providing apparatus 10 evaluates a session based on repetition of utterances will be described. FIG. 33 is a flowchart illustrating an example of the flow of processing in which the information providing device according to the embodiment evaluates a session based on repeated utterances;

First, the information providing apparatus 10 acquires a session to be evaluated (step S501), and evaluates the session based on repetition of character strings of speech included in the acquired session (step S502). For example, the information providing apparatus 10 evaluates the session as a low-satisfaction session when the number of utterances with the same or similar character strings or the number of utterances with the same or similar meanings exceeds a predetermined threshold. Then, the information providing apparatus 10 learns the characteristics of the session as learning data to which a label corresponding to the evaluation result is assigned (step S503), and ends the process.

Note that the information providing apparatus 10 may extract a new index utterance from the session based on the evaluation result of the session (step S504). Further, the information providing apparatus 10 may estimate the usage mode of the user based on the evaluation result of the session (step S505).

[3-6. Example of Reinforcement Learning Processing Flow Using Utterances as Rewards]
Next, an example of the flow of reinforcement learning in which the information providing apparatus 10 executes utterances as rewards will be described with reference to FIG. 34 . FIG. 34 is a flowchart illustrating an example of the flow of reinforcement learning in which the information providing device according to the embodiment executes utterances as rewards.

First, the information providing apparatus 10 inputs an utterance to the dialogue model M10 (step S601), and inputs the response output by the dialogue model M10 to the user model model M20, thereby obtaining an index utterance for the response (step S601). step S602). Then, the information providing apparatus 10 executes reinforcement learning of the dialogue model M10 using the indicator utterance acquired from the user model model M20 as a reward (step S603), and ends the process.

[3-7. An example of the flow of processing for extracting index utterances based on co-occurrence]
Next, with reference to FIG. 35, an example of the flow of processing in which the information providing apparatus 10 extracts indicator utterances based on co-occurrence will be described. 35 is a flowchart illustrating an example of a flow of processing for extracting indicator utterances based on co-occurrence by the information providing device according to the embodiment; FIG.

First, the information providing apparatus 10 identifies index utterances from the history of sessions (step S701), and identifies co-occurrence with index utterances in each session for each utterance (step S702). Then, the information providing device 10 extracts a new index utterance based on the specified co-occurrence (step S703). For example, the information providing apparatus 10 extracts an utterance highly co-occurring with the favorable indicator utterance as a new favorable indicator utterance, and extracts an utterance highly co-occurring with the unfavorable indicator utterance as a new unfavorable indicator utterance. Extract as indicator utterances.

Note that the information providing device 10 may further extract a new index utterance based on the co-occurrence with the newly extracted index utterance (step S704). Further, the information providing apparatus 10 may evaluate the session based on the indicator utterance newly extracted in this way (step S705), and determine the usage mode of the user based on the appearance frequency of the indicator utterance. You may estimate (step S706).

[3-8. An example of a process flow for evaluating a session using a ternary evaluation model]
Next, with reference to FIG. 36, an example of the flow of processing for evaluating sessions using the ternary evaluation model M5 will be described. FIG. 36 is a flowchart illustrating an example of the flow of processing in which the information providing device according to the embodiment evaluates a session using a ternary evaluation model;

First, the information providing apparatus 10 identifies favorable indicator utterances, unfavorable indicator utterances, and neutral utterances from the session history (step S801), and uses utterances and responses a predetermined number before each indicator utterance as learning data. (step S802). That is, the information providing device 10 generates a favorable dialogue example FD, a neutral dialogue example ND, and an unfavorable dialogue example UD. Further, the information providing apparatus 10 assigns labels according to index utterances included in the extracted learning data (step S803), and uses the learning data to learn the ternary evaluation model M5 (step S804). After that, the information providing apparatus 10 evaluates the session using the ternary evaluation model M5 (step S805), and ends the process.

[3-9. An example of the process flow for evaluating a session using multiple models]
Next, with reference to FIG. 37, an example of the flow of processing for evaluating a session using a plurality of models M1-M7 will be described. 37 is a flowchart illustrating an example of the flow of processing in which the information providing device according to the embodiment evaluates sessions using a plurality of models; FIG.

For example, the information providing apparatus 10 acquires a session to be evaluated (step S901), and causes a plurality of models M1 to M7 that have learned different features to evaluate the session (step S902). The information providing apparatus 10 integrates the evaluation results of the models M1 to M7, evaluates the session (step S903), and ends the process.

[4. others〕
Further, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, and conversely, the processes described as being performed manually can be performed manually. can also be performed automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Moreover, each of the embodiments described above can be appropriately combined within a range that does not contradict the processing contents.

[5. program〕
Also, the information providing apparatus 10 according to the above-described embodiment is implemented by a computer 1000 configured as shown in FIG. 38, for example. FIG. 38 is a diagram illustrating an example of a hardware configuration; A computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090. have

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device such as a RAM that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered. It is realized by

The output IF 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various types of information such as a monitor and a printer. It is realized by a connector conforming to a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Also, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, scanner, etc., and is realized by, for example, USB.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Also, the input device 1020 may be an external storage medium such as a USB memory.

Network IF 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

The arithmetic device 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 10 , the arithmetic device 1030 of the computer 1000 implements the functions of the control unit 40 by executing programs loaded on the primary storage device 1040 . Also, for example, when the computer 1000 functions as the terminal device 100 , the arithmetic device 1030 of the computer 1000 realizes the function of the control unit 40 by executing the program loaded on the primary storage device 1040 .

[6. About the experimental results]
Next, an example of experimental results of the information providing apparatus 10 described above will be described with reference to FIGS. 39 to 47. FIG.

[6-1. About data]
First, an example of session data used for experiments on the information providing apparatus 10 will be described. In the following experiments, 4000 sessions containing at least two utterances or responses were labeled by crowdsourcing. Specifically, 2000 training sessions (Training), 1000 adjustment sessions (Validation), and 1000 test sessions (Test) were prepared. In addition, each session is presented to seven judges, and if it is judged that the user is frustrated, "Yes", and if it is judged that the user is not frustrated, , “No” was given to each session.

For example, FIG. 39 is a diagram showing an example of labeling results for session data. For example, in the example shown in FIG. 39, a session labeled "Yes" by at least one judge is FL (Frustration Level) 1, and a session labeled "Yes" by at least two judges is FL (Frustration Level) 1. FL2, a session labeled "Yes" by at least three judges was labeled FL3, and a session labeled "Yes" by at least four judges was labeled FL4. FIG. 39 also shows the number of sessions classified into FL1 to FL4 among training sessions, adjustment sessions, and test sessions.

[6-2. About the first experiment]
Next, the content of the first experiment using the session data described above will be described. In the first experiment, the method indicated by Prop_D and the method indicated by Prop_U were set as experimental targets, and the methods indicated by BL_S1, BL_S2, and BL_SS were set as baselines to be compared.

Prop_D is, as shown in FIG. 1, a determination method using a repetition evaluation model M1, an apology evaluation model M2, an unfavorable index utterance evaluation model M3, and a favorable index utterance evaluation model M4. For example, in Prop_D, a test session was input into each model M1-M4 to calculate a dissatisfaction score. In Prop_D, the 500,000 sessions with the highest dissatisfaction score calculated are determined to be low-satisfaction sessions (that is, the label is "Yes"), and the dissatisfaction score Half a million sessions in turn were determined to be highly satisfying sessions (ie, labeled "No").

Here, in Prop_D, when learning the iterative evaluation model M1, learning data is generated using the method shown in FIG. 17, and as shown in FIG. The features were iteratively learned by the evaluation model M1. Also, in Prop_D, when learning the apology evaluation model M2, learning data was generated using the method shown in FIG. 19, and the apology evaluation model M2 was made to learn the features of the learning data. Also, in Prop_D, the favorable indicator utterance evaluation model M3 and the unfavorable indicator utterance evaluation model M4 were learned using the method shown in FIG. In Prop_D, a model for image classification is adopted as the iterative evaluation model M1, and a model for classifying input information using LSTM is adopted as the models M2 to M4.

Prop_U is a judgment method that uses only the repeat evaluation model M1 and the apology evaluation model M2 among the models M1 to M4 shown in FIG. Such a determination method does not require index utterances for model learning, and thus does not require manual data creation in advance, making it an unsupervised method. For example, in Prop_U, the target session is repeatedly input to the evaluation model M1 and the apology evaluation model M2, and the sum of the degrees of satisfaction output by each model M1 and M2 is calculated as the dissatisfaction score. Also, in Prop_U, the 500,000 sessions with the highest dissatisfaction score calculated are determined to be low-satisfaction sessions (that is, the label is "Yes"), and the dissatisfaction score Half a million sessions were determined to be highly satisfied sessions (ie labeled "No").

Also, BL_S1 used a model that was trained using the above-described training session and adjusted using the adjustment session. Also, in BL_S1, similarly to Prop_D and Prop_U described above, a model for classifying input information using LSTM is adopted.

Also, in BL_S2, an SVM that was trained using a training session and adjusted using an adjustment session was used. Also, in BL_S2, the output of the LSTM layer of the model used in BL_S1 described above is input to the SVM, and the output of the SVM is adopted as the classification result.

In BL_SS, a transductive SVM (TSVM) was used, which was trained using one million unlabeled sessions in addition to training sessions. In addition, the adjustment session and the test session were used for learning as unlabeled sessions by ignoring labels.

Here, in the first experiment, the sessions classified into FL1 to FL4 are classified by the various methods described above, and the accuracy (accuracy), precision (relevance rate), recall (recall rate), and And the F1 value, which is the harmonic mean of precision and recall, was calculated. FIG. 40 shows the results of the first experiment. In addition, in the example shown in FIG. 40, asterisks are attached to methods having a significant difference from Prop_D, and sharp marks are attached to methods having a significant difference from Prop_U. As shown in FIG. 40, when the F1 value is used as a comparison target, among the classification results of the session data classified into FL1, FL2, and FL3, the accuracy of the classification result by Prop_D is the best.

[6-3. Regarding the second experiment]
Subsequently, in order to determine how much each model described above contributes to accuracy, the following second experiment was conducted. In the second experiment, w/o-Rep from which the repeated evaluation model M1 was removed in Prop_D, w/o-Apo from which the apology evaluation model M2 was removed from Prop_D, and w from which the unfavorable index utterance evaluation model M3 was removed from Prop_D. In /o-Unf and Prop_D, a method called w/o-Fav was set by removing the favorable index utterance evaluation model M4.

Also, in the second experiment, a system called w/o-RAUF that does not use the four models M1 to M4 shown in FIG. 1 was set. In w / o-RAUF, 500,000 high-satisfaction sessions and 500,000 low-satisfaction sessions are used as learning data, and the input sessions are high-satisfaction sessions (that is, sessions labeled “No”) or low-satisfaction A model for classifying sessions (that is, sessions with the label “Yes”) was trained, and sessions were classified using the trained model.

Also, in the second experiment, a method called w/o-CandRet was set. In w/o-CandRet, the dissatisfaction score is calculated using each of the models M1 to M4 described above for the history of sessions acquired in advance, and 500,000 sessions are labeled in descending order of the calculated score. Sessions were extracted as "Yes" sessions, and 500,000 sessions with lower calculated scores were extracted as sessions labeled "No". In w/o-CandRet, a model for classifying input sessions is learned using the generated learning data, and sessions are classified using the learned model.

Then, in the second experiment, the sessions classified into FL1 to FL4 were classified by the various methods described above, and the accuracy (accuracy), precision (relevance rate), recall (recall rate), and The F1 value was calculated. FIG. 41 shows the results of the second experiment. As shown in FIG. 41, when the F1 value is used as a comparison target, the accuracy of Prop_D is the highest, and it was found that the accuracy decreases when one of the models M1 to M4 is not used. As a result, it was found that all of the models M1 to M4 contributed to the improvement of session classification accuracy.

[6-4. Regarding the third experiment]
Subsequently, in Prop_D described above, the following third experiment was conducted in order to determine the contribution of the amount of learning data to the classification accuracy. In the following description, the amount of learning data used is indicated by the suffix k. For example, Prop_D_k trains each model M1-M4 using k/2 high-satisfaction sessions and k/2 low-satisfaction sessions, and uses each trained model M1-M4 to classify test sessions. indicates that the Also, in the third experiment, the learning data The contribution of the amount of to the accuracy in each method was investigated.

FIG. 42 is a diagram showing the results of Prop_D in the third experiment. In addition, in FIG. 42, asterisks are given to results having a significant difference from when k is 1M. As shown in FIG. 42, it was found that Prop_D has the highest accuracy when k=2M regardless of the test session classification. This is probably because there is room for improvement in accuracy when the number of learning data is less than 2 million, and over-learning occurs when the number of learning data exceeds 2 million.

[6-5. Regarding the fourth experiment]
Here, Prop_D described above uses a neural network as a model for performing evaluation based on repeated utterances, apologies, and indicator utterances. Now, in order to examine the effect of using neural networks for each recognition, the following fourth experiment was conducted. For example, in the fourth experiment, in the method of Prop_D, instead of the neural network, the session based on repeated utterances is evaluated using edit distance, and the evaluation based on repeated utterances and index utterances is performed by string matching Prop_S. Executed as

FIG. 43 is a diagram showing the results of Prop_S in the fourth experiment. In addition, in FIG. 43, FL2, FL3, and FL4 showed a significant difference from Prop_D. Comparing the results of Prop_D shown in FIGS. 40 and 41 with the results of Prop_S shown in FIG. 43, Prop_S has a higher precision rate and a lower recall rate and F1 value than Prop_D. That is, Prop_S is considered to be less accurate than Prop_D. Therefore, in Prop_D, accuracy can be improved by using a neural network when evaluating a session.

[6-6. About the fifth experiment]
Next, the output of each model used for evaluation in Prop_D was fused, and a model with improved generalization ability (prediction ability for unlearned data) was used to conduct a fifth experiment to evaluate a session. More specifically, in the fifth experiment, the ensemble method was used to further improve the accuracy of Prop_D, which was used as Prop_E.

FIG. 44 shows the results of Prop_E in the fifth experiment. In FIG. 44, the results of Prop_E and the results of Prop_D_2M are shown side by side. As shown in FIG. 44, the result of Prop_E has a higher F1 value than the result of Prop_D_2M. Therefore, by using the ensemble method like Prop_E, it is possible to further improve the accuracy of session evaluation.

[7. About the effect of the embodiment]
An example of the effect of the information providing apparatus 10 that executes each process described above will be described below.

[7-1. Regarding extraction of learning data based on usage mode]
As described above, the information providing apparatus 10 identifies the mode of use of the interactive service by the user. Then, the information providing apparatus 10 uses the session including the user's utterance whose identified usage mode satisfies a predetermined condition as learning data for evaluating the session including the utterance and the response in the dialog service as the history of the dialog service. Extract from

In this way, the information providing apparatus 10 extracts learning data for learning a session evaluation model based on each user's engagement. Therefore, the information providing apparatus 10 can automatically extract learning data for learning a model for determining whether a user is satisfied with a session, for example, without human intervention. As a result of such processing, the information providing apparatus 10 can easily prepare a large amount of learning data, and as a result of improving the accuracy of the model, it is possible to improve the evaluation accuracy of the session.

In addition, the information providing apparatus 10 specifies the usage period of the interactive service by the user as the usage mode, and extracts, as learning data, a session including the user's utterance whose usage period satisfies a predetermined condition. In addition, the information providing apparatus 10 specifies the usage frequency of the interactive service by the user as the usage mode, and extracts sessions including user utterances whose usage frequency satisfies a predetermined condition as learning data. For this reason, for example, when extracting learning data for the purpose of extracting a session with low user satisfaction, the information providing apparatus 10 selects a user with high engagement, that is, a user with high evaluation of the interactive service. By extracting sessions as exceptions and extracting sessions of users with low engagement, i.e., sessions of users with low evaluations of interactive services, as positive examples, it is possible to easily prepare data suitable for learning. can improve the evaluation accuracy of

In addition, the information providing device 10 causes the model to learn the features of the extracted session. Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session.

Further, the information providing apparatus 10 extracts, as learning data having the first attribute, a session including an utterance of a user whose usage mode satisfies a predetermined condition, and also extracts an utterance of a user whose usage mode does not satisfy the predetermined condition as learning data. The sessions containing the training data are extracted as learning data having the second attribute. The information providing apparatus 10 then uses the extracted learning data to learn a model that classifies the input session into either data having the first attribute or data having the second attribute.

For example, the information providing apparatus 10 extracts a session including an utterance of a user whose usage mode satisfies a predetermined condition as learning data that is evaluated favorably by the user, and the usage mode does not satisfy the predetermined condition. A session including user's utterance is extracted as learning data that the user's evaluation is unfavorable. Then, the information providing apparatus 10 uses the extracted learning data to learn a model for determining whether the user's evaluation of the input session is favorable or unfavorable. Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session.

Further, the information providing device 10 evaluates the session including the user's utterance based on the identified usage mode. Then, the information providing device 10 extracts sessions whose evaluations satisfy predetermined conditions as learning data. Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session.

Further, the information providing apparatus 10 extracts an utterance similar to an utterance included in a session whose evaluation satisfies a predetermined condition, or a response similar to a response included in the session from the history of the dialogue service, and extracts the extracted utterance or the extracted utterance or response. A new session with the response is created as a session whose evaluation satisfies a predetermined condition. Then, the information providing apparatus 10 causes the model to learn the characteristics of the new session that has been generated. In this way, the information providing apparatus 10 generates, as learning data, similar sessions from sessions whose usage patterns satisfy predetermined conditions, and performs model learning using the generated learning data. Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session as a result of improving the model determination accuracy.

Further, the information providing apparatus 10 extracts a session whose evaluation satisfies a predetermined condition from the history of the interactive service, and among the utterances included in the extracted session, the utterance whose frequency of appearance satisfies the predetermined condition is included in the utterance. It is extracted as an indicator utterance that serves as an indicator of whether or not the user's evaluation of the session is favorable. For example, the information providing apparatus 10 extracts, as index utterances, utterances in which the ratio of the frequency of appearance in a session in which the evaluation satisfies the predetermined condition to the frequency of appearance in the session in which the evaluation does not satisfy the predetermined condition satisfies a predetermined condition. Therefore, the information providing device 10 can extract index utterances based on the usage mode of the user. Note that such index utterances can be used for session evaluation. Therefore, the information providing device 10 can improve the evaluation accuracy of the session based on the index utterance.

Further, the information providing apparatus 10 evaluates the session based on whether or not the session includes a predetermined response indicating an apology, in addition to the usage mode of the user. In addition to the user's usage mode, the information providing apparatus 10 determines whether the session includes an index utterance that serves as an index of whether the user's evaluation of the session is favorable or not. Evaluate. Therefore, the information providing apparatus 10 can further improve the evaluation accuracy of the session.

In addition, the information providing apparatus 10 identifies the mode of use of the interactive service by the user, and evaluates the session including the user's utterance based on the identified mode of use. Therefore, the information providing apparatus 10 can appropriately evaluate the session without human intervention.

[7-2. Extraction of index utterances based on usage pattern]
In addition, the information providing apparatus 10 acquires a session including an utterance of a user whose usage mode of the interactive service satisfies a predetermined condition and a response to the utterance. Then, the information providing apparatus 10 extracts index utterances, which serve as indices for the user's evaluation of the interactive service, based on the appearance frequency in the acquired session. Such index utterances can be used for session evaluation. Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session.

In addition, the information providing apparatus 10 acquires a session including an utterance of a user who satisfies a predetermined condition for the use period of the interactive service. Further, the information providing apparatus 10 acquires a session including an utterance of a user whose use frequency of the interactive service satisfies a predetermined condition. Then, the information providing apparatus 10 compares the appearance frequency in the session including the user's utterance whose usage mode satisfies the predetermined condition to the appearance frequency in the session including the user's utterance whose usage mode does not satisfy the predetermined condition. extracts an utterance that satisfies a predetermined condition as an index utterance.

In addition, the information providing apparatus 10 indicates that the user has favorably evaluated the dialogue service for an utterance whose appearance frequency exceeds a predetermined threshold value in a session including the user's utterance whose usage mode satisfies a predetermined condition. Extracted as favorable indicator utterances. In addition, the information providing apparatus 10 evaluates the dialogue service unfavorably by the user for an utterance whose appearance frequency exceeds a predetermined threshold in the session including the user's utterance whose usage mode does not satisfy the predetermined condition. It is extracted as an unfavorable index utterance indicating that. As a result of these processes, the information providing apparatus 10 can appropriately extract the index utterance from the user's utterance, thereby improving the evaluation accuracy of the session.

Further, the information providing device 10 evaluates the session based on the appearance frequency of the extracted index utterance. Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session.

In addition, the information providing apparatus 10 also determines, with respect to the session to be evaluated, an utterance whose appearance frequency exceeds a predetermined threshold value in a session including an utterance of a user whose usage mode satisfies a predetermined condition, and an utterance whose usage mode satisfies a predetermined condition. A session to be evaluated is evaluated based on utterances whose appearance frequency exceeds a predetermined threshold in a session including user utterances that do not exist. Therefore, the information providing device 10 can appropriately evaluate the session.

In addition, the information providing apparatus 10 causes the model to learn the characteristics of the session whose evaluation satisfies a predetermined condition. Therefore, the information providing apparatus 10 can implement a model for evaluating sessions.

The information providing device 10 also identifies index utterances extracted from the history of the dialogue service, and extracts a series of dialogues corresponding to the identified index utterances as learning data. Then, the information providing device 10 causes the model to learn the features of the extracted learning data. For example, the information providing device 10 causes the model to learn the features of the unfavorable dialogue example UD and the favorable dialogue example FD. Therefore, the information providing apparatus 10 can implement a model for evaluating sessions.

[7-3. Evaluation of sessions based on index utterances]
In addition, the information providing apparatus 10 acquires a session including user's utterances and responses in the interactive service. Then, the information providing apparatus 10 evaluates the acquired session based on the index utterance, which serves as an index for the user's evaluation of the interactive service. Therefore, the information providing device 10 can automatically learn the session with high accuracy.

In addition, the information providing apparatus 10 outputs a favorable indicator utterance indicating that the user's evaluation of the interactive service is favorable, and a non-favorable indicator utterance indicating that the user's evaluation of the dialogue service is unfavorable. Evaluate sessions based on Then, for example, the information providing apparatus 10 evaluates the session based on the ratio of the appearance frequency of favorable indicator utterances and the appearance frequency of unfavorable indicator utterances. In addition, for example, the information providing apparatus 10 may provide a favorable score indicating how favorable the user's evaluation of the dialogue service is in a session containing favorable indicator utterances, and in a session containing unfavorable indicator utterances The session is rated based on the ratio of the unfavorable score, which indicates how unfavorable the user's evaluation of the interactive service is. Therefore, the information providing device 10 can automatically learn the session with high accuracy.

Further, the information providing apparatus 10 evaluates whether or not the session is favorably evaluated by the user as a highly satisfying session. Then, the information providing device 10 causes the model to learn the characteristics of the highly satisfying session. In addition, the information providing apparatus 10 evaluates whether or not the evaluation by the user is an unfavorable low-satisfaction session. Then, the information providing device 10 causes the model to learn the features of the low-satisfaction session.

Further, the information providing apparatus 10 identifies index utterances from the history of the dialogue service, and extracts a predetermined number of previous responses and utterances from the identified index utterances as learning data. Then, the information providing device 10 causes the model to learn the features of the extracted learning data. For example, the information providing apparatus 10 extracts a predetermined number of previous responses and utterances from the favorable indicator utterances indicating that the user's evaluation of the interactive service is favorable, and extracts the responses and utterances as favorable dialogue examples. An unfavorable index indicating that the user's evaluation is unfavorable A predetermined number of previous responses and utterances are extracted as unfavorable dialogue examples from the utterance, and even if the user's evaluation of the dialogue service is favorable, it is not favorable. A predetermined number of previous responses and utterances from the neutral utterance indicating that it is not a target are extracted as neutral dialogue examples. Then, the information providing apparatus 10 causes the model to learn the features of the extracted favorable dialogue examples, unfavorable dialogue examples, and neutral dialogue examples.

Further, the information providing apparatus 10 evaluates the session based on whether the session includes the index utterance and a predetermined response indicating an apology. As a result of the various types of processing described above, the information providing device 10 can learn a model that appropriately evaluates sessions from various viewpoints. As a result, the information providing apparatus 10 can improve the evaluation accuracy of the session.

In addition, the information providing apparatus 10 estimates the user's future use of the interactive service based on the evaluation result for the session including the user's utterance. For example, the information providing apparatus 10 calculates, as the evaluation, a favorable score indicating how favorable the user's evaluation is, or a non-favorable score indicating how unfavorable the user's evaluation is. Then, based on the ratio of the favorable score and the unfavorable score, the user's future use of the interactive service is estimated. Therefore, the information providing device 10 can appropriately estimate the usage mode of the user.

Further, the information providing apparatus 10 extracts, as index utterances, utterances whose appearance frequency satisfies a predetermined condition from sessions whose evaluation satisfies a predetermined condition. For example, the information providing apparatus 10 evaluates whether the user's evaluation of the session is favorable or unfavorable. An utterance that satisfies a predetermined condition with respect to the frequency of occurrence in a session in which the evaluation by the user is unfavorable is extracted as an index utterance. Therefore, the information providing device 10 can appropriately extract new index utterances.

[7-4. Evaluation of sessions based on distribution images]
In addition, the information providing apparatus 10 acquires a session including user's utterances and responses to the utterances in the interactive service. In addition, the information providing apparatus 10 generates an image indicating whether or not the characters of the utterances included in the session are the same. The information providing device 10 then evaluates the obtained session based on the generated image.

Here, when the same or similar utterances are repeatedly included in the session, the above-described image includes a predetermined pattern. Then, when it is estimated that the image contains a predetermined pattern, it can be estimated that the same or similar utterances are repeatedly contained in the session. In this way, it is estimated that sessions in which the same or similar utterances are repeatedly included are rated poorly by users. Therefore, the information providing apparatus 10 can appropriately evaluate the obtained session based on the generated image.

The information providing apparatus 10 also generates an image showing the distribution of the same characters in the session. For example, when the information providing apparatus 10 associates a character string of an utterance included in a session with each vertical region and each horizontal region of an image, the same character corresponds to the vertical direction and the horizontal direction. An image is generated in which a predetermined color is added to the marked area. Then, based on the image, the information providing apparatus 10 determines whether or not the same or similar character strings appear repeatedly in the session, and evaluates the session based on the determination result. For example, if the same or similar character strings appear repeatedly in a session, the information providing apparatus 10 gives the session a negative evaluation by the user. Therefore, the information providing device 10 can appropriately evaluate the session.

In addition, the information providing apparatus 10 causes the model to learn features of images generated from sessions in which the same or similar character strings appear repeatedly. Then, the information providing apparatus 10 uses the trained model to evaluate the session from which the image is based. Here, it is known that a model that learns an image is more likely to improve the determination accuracy than a model that learns a character. Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session.

Further, the information providing apparatus 10 selects any utterance from the history of the interactive service, and generates an utterance group including a plurality of utterances that are the same as or similar to the selected utterance. Then, the information providing device 10 generates an image from the generated utterance group, and makes the model learn the features of the generated image. Therefore, the information providing device 10 can improve the accuracy of the model as a result of increasing the learning data of the model.

[7-5. Evaluation of sessions based on repeated utterance]
In addition, the information providing apparatus 10 acquires a session including user's utterances and responses to the utterances in the interactive service. Then, the information providing apparatus 10 evaluates the obtained session based on the repetition of the utterance character string included in the session. Therefore, the information providing apparatus 10 can accurately evaluate the session without human intervention.

For example, the information providing apparatus 10 extracts utterance pairs whose character string similarity exceeds a predetermined threshold from the session, and evaluates the session based on the number of extracted pairs. Further, the information providing apparatus 10 extracts utterance pairs whose semantic similarity exceeds a predetermined threshold from the session, and evaluates the session based on the number of extracted pairs. Then, when the number of pairs extracted from the session exceeds a predetermined threshold, the information providing apparatus 10 gives the session an evaluation indicating that the user's evaluation is unfavorable. Therefore, the information providing device 10 can appropriately evaluate the session.

Also, the information providing apparatus 10 evaluates a session using a model that has learned characteristics of sessions in which the same or similar character strings appear repeatedly. For example, the information providing apparatus 10 selects any utterance from the history of the interactive service, and generates an utterance group including multiple utterances that are the same as or similar to the selected utterance. In addition, the information providing device 10 causes the model to learn the features of the generated utterance group. The information providing device 10 then evaluates the session using the trained model.

Further, the information providing apparatus 10 generates a first utterance group including a plurality of utterances that are the same as or similar to the selected utterance, selects a plurality of different utterances from the history of the dialogue service, and generates a second utterance group composed of the selected utterances. A group of utterances is generated, and the model learns the features of the first group of utterances and the features of the second group of utterances. In addition, the information providing apparatus 10 learns a model that determines whether or not the input session is evaluated unfavorably by the user by having the model learn the features of the utterance group. Therefore, the information providing apparatus 10 can accurately evaluate the session.

In addition, the information providing apparatus 10 extracts, from sessions whose evaluation satisfies a predetermined condition, utterances whose appearance frequency satisfies a predetermined condition as index utterances that serve as indices for the user's evaluation of the dialogue service. Therefore, the information providing device 10 can automatically extract new index utterances.

[7-6. Reinforcement learning with utterance as a reward]
In addition, the information providing apparatus 10 acquires an index utterance, which is an evaluation index for the dialogue service, among the utterances for the dialogue model that realizes the dialogue service by generating a response to the input utterance, and obtains the obtained index. Reinforcement learning of dialogue models is performed by setting rewards based on utterances. Therefore, the information providing apparatus 10 can realize learning of a dialogue model that realizes a dialogue service without human intervention, and can improve the accuracy of the dialogue model.

In addition, the information providing apparatus 10 outputs a positive indicator utterance indicating that the user's evaluation of the interactive service is favorable, or a non-favorable indicator that indicates that the user's evaluation of the interactive service is unfavorable. Acquire utterances, set a positive reward for the dialogue model if a favorable indicator utterance is obtained, and set a negative reward for the dialogue model if a non-favorable indicator utterance is obtained do. Therefore, the information providing device 10 can appropriately implement reinforcement learning of the dialogue model.

In addition, the information providing apparatus 10 acquires index utterances output by an utterance model that outputs index utterances in response to responses generated by a dialogue model. For example, the information providing apparatus 10 acquires, as an utterance model, an index utterance output by a model that has learned the characteristics of the history of dialogue between the user and the dialogue model in the dialogue service. For this reason, the information providing apparatus 10 can realize reinforcement learning of a dialogue model even without session history, for example.

[7-7. Extraction of indicator utterances based on co-occurrence]
In addition, the information providing device 10 identifies a preset index utterance, which is an utterance used as an index for the user's evaluation of the dialogue service, from the history of the user's utterances and responses to the utterances in the dialogue service. . Then, the information providing device 10 extracts a new index utterance from the utterances included in the history based on the co-occurrence with the specified index utterance. As described above, such indicator utterances can be used for session evaluation. As a result, the information providing apparatus 10 can improve the evaluation accuracy of the session.

In addition, the information providing apparatus 10 identifies favorable indicator utterances indicating that the user's evaluation of the interactive service is favorable, and newly adds utterances whose co-occurrence with the favorable indicator utterances exceeds a predetermined threshold. Extract as favorable indicator utterances. Further, the information providing apparatus 10 identifies an unfavorable indicator utterance indicating that the user's evaluation of the interactive service is unfavorable, and specifies an utterance whose co-occurrence with the unfavorable indicator utterance exceeds a predetermined threshold. is extracted as a new unfavorable indicator utterance.

In addition, the information providing apparatus 10 determines the number of times the positive indicator utterance indicating that the user's evaluation of the interactive service is favorable is included in the same session, and the number of times the user's evaluation of the interactive service is unfavorable. Count the number of times an unfavorable indicator utterance indicating that something is included in the same session for each utterance of the user, and extract utterances whose ratio of the counted number of times satisfies a predetermined condition as a new indicator utterance. .

Further, the information providing apparatus 10 generates a co-occurrence network that connects utterances or responses having co-occurrence from the history of utterances and responses, identifies index utterances from the co-occurrence network, and generates index utterances on the co-occurrence network. An utterance whose distance from the utterance satisfies a predetermined condition is extracted as a new index utterance. For example, the information providing apparatus 10 identifies, from the co-occurrence network, a favorable indicator utterance indicating that the user's evaluation of the dialogue service is favorable, and the distance from the favorable indicator utterance on the co-occurrence network is a predetermined distance. Utterances below the threshold are extracted as new favorable index utterances. In addition, the information providing apparatus 10 creates a co-occurrence network with a non-favorable indicator utterance indicating that the distance from the favorable indicator utterance is equal to or less than a predetermined threshold and the user's evaluation of the dialogue service is unfavorable. An utterance with a distance above a predetermined threshold or more is extracted as a new favorable index utterance.

Further, the information providing apparatus 10 identifies, from the co-occurrence network, an unfavorable indicator utterance indicating that the user's evaluation of the dialogue service is unfavorable, and determines the distance from the unfavorable indicator utterance on the co-occurrence network. is extracted as a new unfavorable index utterance. For example, the information providing device 10 may be configured such that the distance from the unfavorable index utterance is equal to or less than a predetermined threshold value, and the co-occurrence network with the favorable index utterance indicating that the user's evaluation of the dialogue service is favorable. An utterance whose distance in is equal to or greater than a predetermined threshold is extracted as a new unfavorable index utterance. As a result of these processes, the information providing device 10 can accurately extract index utterances indicating the user's impression.

Further, the information providing device 10 evaluates a session including utterances and responses in the dialogue service based on whether or not the extracted index utterance is included. Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session.

[7-8. Evaluation of sessions using dialogue examples]
In addition, the information providing apparatus 10 extracts a series of dialogues corresponding to index utterances, which serve as indicators of the user's evaluation of the dialogue service, from the history of the dialogue service. That is, the information providing device 10 extracts various dialogue examples. For example, the information providing apparatus 10 extracts an utterance and response a predetermined number before the index utterance as a dialogue example. Then, the information providing device 10 uses the extracted dialogue to learn a model for estimating the user's evaluation of the input dialogue.

In this way, the information providing apparatus 10 extracts various dialogue examples instead of sessions, and makes the model learn the features of the extracted dialogue examples. Extraction of such dialogue examples can be realized more easily than extraction of sessions if there are various indicator utterances. As a result, the information providing apparatus 10 can easily evaluate the session.

Further, the information providing apparatus 10 extracts a predetermined number of previous utterances and responses from the favorable indicator utterances indicating that the user's evaluation of the interactive service is favorable, and uses the extracted utterances and responses. The model is trained using learning data of dialogues that are evaluated favorably by the person. Further, the information providing apparatus 10 extracts a predetermined number of previous utterances and responses from the unfavorable indicator utterances indicating that the user's evaluation of the interactive service is unfavorable, and extracts the extracted utterances and responses. , training of the model is performed using learning data of dialogues that are evaluated unfavorably by users. Further, the information providing apparatus 10 extracts a predetermined number of previous utterances and responses from the neutral index utterances indicating that the user's evaluation of the interactive service is neutral, and transmits the extracted utterances and responses to the user. The model is trained using dialogue learning data that is neutrally evaluated by Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session by the model.

In addition, the information providing apparatus 10 acquires a session including user's utterances and responses in the interactive service. Then, the information providing device 10 evaluates the obtained session using the learned model. Therefore, the information providing apparatus 10 can implement session evaluation.

Further, the information providing apparatus 10 extracts, as index utterances, utterances whose appearance frequency satisfies a predetermined condition from sessions whose evaluation satisfies a predetermined condition. For example, the information providing apparatus 10 evaluates whether the user's evaluation of the session is favorable or unfavorable. An utterance that satisfies a predetermined condition with respect to the frequency of occurrence in a session in which the evaluation by the user is unfavorable is extracted as an index utterance. Therefore, the information providing device 10 can automatically extract new index utterances.

[7-9. Evaluation of sessions using multiple models]
In addition, the information providing apparatus 10 acquires a session including user's utterances and responses in the interactive service. Then, the information providing apparatus 10 evaluates the session using the evaluation results of a plurality of models that evaluate the session based on different types of features among the plurality of features that serve as indicators for evaluating the session by the user. Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session.

Further, the information providing apparatus 10 uses at least the evaluation result of the model for evaluating the session based on whether or not the same or similar speech is repeatedly included in the speech included in the session. Further, the information providing device 10 uses at least the evaluation result of the model for evaluating the session based on whether or not the response included in the session includes a predetermined response indicating an apology. Further, the information providing apparatus 10 decides whether or not the session includes an unfavorable indicator utterance indicating that the user is evaluating the interactive service unfavorably in the utterances included in the session. At least use the evaluation results of the model to be evaluated. Further, the information providing apparatus 10 evaluates the session based on whether or not the utterances included in the session include a favorable indicator utterance indicating that the user is evaluating the interactive service favorably. At least use the evaluation results of the model. Therefore, the information providing apparatus 10 can improve the evaluation accuracy of the session.

In addition, the information providing apparatus 10 apologizes for the evaluation value output by the model that evaluates the session based on whether or not the same or similar utterances are repeatedly included in the utterances included in the session, and the responses included in the session. The session is evaluated based on the value of sum with the evaluation value output by the model that evaluates the session based on whether or not it contains the indicated predetermined response. Therefore, the information providing apparatus 10 can evaluate sessions using a plurality of models that can evaluate sessions without human intervention.

In addition, the information providing apparatus 10 apologizes for the evaluation value output by the model that evaluates the session based on whether or not the same or similar utterances are repeatedly included in the utterances included in the session, and the responses included in the session. The evaluation value output by the model that evaluates the session based on whether or not it contains a predetermined response indicated by the From the sum of the evaluation value output by the model that evaluates the session based on whether or not it contains favorable indicator utterances, it is possible to indicate that the user has favorably evaluated the dialogue service in the utterances included in the session. The session is evaluated based on the value obtained by subtracting the evaluation value output by the model that evaluates the session based on whether or not the indicated favorable indicator utterance is included. For example, the information providing apparatus 10 evaluates sessions depending on whether the value is positive or negative. Therefore, the information providing device 10 can accurately evaluate the session.

As described above, some of the embodiments of the present application have been described in detail based on the drawings. It is possible to carry out the invention in other forms with modifications.

Also, the "section, module, unit" described above can be read as "means" or "circuit". For example, the extraction unit can be read as extraction means or an extraction circuit.

10 information providing device 20 communication unit 30 storage unit 31 session log database 32 session evaluation database 33 index utterance database 34 engagement database 35 model database 40 control unit 41 session acquisition unit 42 usage mode identification unit 50 index utterance extraction processing unit 51 index utterance identification Unit 52 Utterance extraction unit 53 Network generation unit 60 Session evaluation processing unit 61 Learning data extraction unit 62 Learning unit 63 Image generation unit 64 Utterance group generation unit 65 Evaluation unit 70 Engagement prediction processing unit 71 Estimation unit 80 Reinforcement learning processing unit 81 Index utterance Acquisition unit 82 Reward setting unit 83 Reinforcement learning unit 100 Terminal device 200 Dialogue device

Claims (9)

an acquisition unit that acquires a session including a user's utterance and a response to the utterance in the dialogue service;
an evaluation unit that evaluates the session acquired by the acquisition unit based on repetition of character strings of the utterances included in the session; an utterance group generating unit that generates a plurality of utterances as duplicated utterances and sets them as an utterance group of the duplicated utterances;
a learning unit that extracts features of utterances in which the same or similar character strings appear repeatedly in the utterance group generated by the utterance group generation unit, and causes a predetermined utterance evaluation model to learn the extracted features. ,
The evaluation unit uses the predetermined utterance evaluation model to evaluate whether the user is satisfied with the dialogue related to the session based on the repetition mode in the session acquired by the acquisition unit. An evaluation device characterized by: The evaluation unit extracts utterance pairs whose character string similarity exceeds a predetermined threshold from the session, and evaluates the user's satisfaction with the dialogue related to the session based on the number of extracted pairs. The evaluation device according to claim 1, characterized in that: The evaluation unit extracts utterance pairs whose semantic similarity exceeds a predetermined threshold from the session, and evaluates the user's satisfaction with the dialogue related to the session based on the number of extracted pairs. The evaluation device according to claim 1 or 2, characterized in that: wherein, when the number of pairs extracted from the session exceeds a predetermined threshold, the evaluation unit gives an evaluation to the effect that the evaluation by the user is unfavorable to the session. 4. The evaluation device according to item 2 or 3. The utterance group generation unit generates a first utterance group as a repeated session including a plurality of utterances that are the same or similar to the selected utterance, and selects and selects a plurality of different utterances from the history of the dialogue service. generating a second utterance group as a non-repetitive session consisting of utterances,
The learning unit extracts features of utterances of the first utterance group and features of repetitive and non-repetitive utterances of the second utterance group , and applies the extracted features to a predetermined utterance evaluation model. learning a model for estimating an evaluation as to whether or not the user is satisfied with the conversation related to the session obtained by the obtaining unit, using the predetermined utterance evaluation model. The evaluation device according to any one of claims 1 to 4, characterized by: The learning unit includes a seed utterance, a plurality of replicated utterances generated from the seed utterance, and a mixed utterance that is an utterance different from the seed utterance . A plurality of utterances are extracted, and the features related to repetitiveness and non-repetitiveness of an utterance group consisting of non-repetitive sessions in which the extracted utterances are randomly rearranged are applied to a predetermined utterance evaluation model, and the session by the user is applied. For the session acquired by the acquisition unit, learning related to estimation of evaluation is performed, and the predetermined utterance evaluation model is used to determine whether or not the same or similar utterance is repeated in the session. 6. The evaluation device according to any one of claims 1 to 5, characterized in that it learns a model for judging whether or not an evaluation by a person is unfavorable. an utterance extraction unit that extracts utterances whose frequency of appearance satisfies a predetermined condition from sessions whose evaluation by the evaluation unit satisfies a predetermined condition, as index utterances that serve as indices for user evaluation of the dialogue service. The evaluation device according to any one of claims 1 to 6. An evaluation method executed by an evaluation device,
an acquisition step of acquiring a session including a user's utterance and a response to the utterance in the interactive service;
an evaluation step of evaluating the session acquired by the acquisition step based on the repetition of a character string of utterances included in the session; an utterance group generation step of generating a plurality of utterances as replicated utterances to form an utterance group of the replicated utterances;
a learning step of extracting features of utterances in which the same or similar character strings appear repeatedly in the utterance group generated by the utterance group generation step, and allowing a predetermined utterance evaluation model to learn the extracted features ,
The evaluation step uses the predetermined utterance evaluation model to evaluate whether or not the user is satisfied with the dialogue related to the session based on the repetition mode in the session obtained by the obtaining step. An evaluation method characterized by an acquisition procedure for acquiring a session including a user's utterance and a response to the utterance in the interactive service;
an evaluation procedure for evaluating the session acquired by the acquisition procedure based on repetition of a character string of utterances included in the session; an utterance group generation procedure for generating a plurality of utterances as duplicate utterances, and forming an utterance group of the duplicate utterances;
a learning procedure for extracting features of utterances in which the same or similar character strings appear repeatedly in the utterance group generated by the utterance group generation procedure, and allowing a predetermined utterance evaluation model to learn the extracted features; let it run ,
The evaluation procedure uses the predetermined utterance evaluation model to evaluate whether or not the user is satisfied with the dialogue related to the session based on the repetition mode in the session acquired by the acquisition procedure. An evaluation program that makes a computer do things.

Images