JP6704976B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

Conventionally, when the user's conversation content has previously registered words such as listening to or confirming the other party's words, and the conversation content includes these registered words, the conversation is not effectively conducted. There is disclosed a device that judges that the device is a device (see Patent Document 1).

JP, 2007-43356, A

However, in the above-mentioned device, since the judgment of the effectiveness of the conversation depends on the words registered in advance, it may not be possible to make a judgment for other words.

The present invention has been made in consideration of such circumstances, and an information processing apparatus, an information processing method, and an information processing method capable of automatically acquiring a clue for determining whether an unknown conversation is of a predetermined type. One of the purposes is to provide a program.

One aspect of the present invention includes an acquisition unit that acquires a feedback utterance estimated to indicate a predetermined reaction of the second utterance subject with respect to an utterance issued by the first utterance subject, and the acquisition unit obtains the feedback utterance. Further, the information processing apparatus includes a pre-stage generation unit that generates a pre-stage classifier that derives an index estimated that a predetermined type of conversation appears just before the feedback utterance.

According to one aspect of the present invention, it is possible to automatically obtain a clue for determining whether an unknown conversation is a predetermined type.

It is a figure which shows a part of structure of the information processing system 1. It is a figure which shows an example of the content of the label given to a feedback utterance. It is a figure which shows an example of the content of the label given to the conversation just before a feedback utterance. FIG. 6 is a diagram showing an example of feedback utterances input to the utterance classifier 24 and utterance scores output by the utterance classifier 24. It is a figure which shows the other structure of the information processing system 1. It is a figure which shows an example of the feedback utterance biased to the 1st classification side or the 2nd classification side. It is a figure which shows an example of the conversation memorize|stored in the conversation learning data storage device 52. It is a figure which shows the process of learning notionally. FIG. 6 is a diagram showing an example of a conversation input to a conversation classifier 66 and information output by the conversation classifier 66. 6 is a flowchart showing a flow of processing in which a speech classifier 24 is generated by the information processing system 1. 6 is a flowchart showing a flow of processing in which a conversation classifier 66 is generated by the information processing system 1. It is a figure which shows the function structure of the comparative example 1 and the comparative example 2. It is a figure which shows an example of the process result of the information processing system 1, the comparative example 1, and the comparative example 2. It is a figure showing an example of functional composition of information processing system 1A of the modification 1. It is a figure which shows an example of a functional structure of the information processing system 1B of the modification 2. It is a figure showing an example of functional composition of information processing system 1C of the modification 3. It is a figure which shows an example of the processing result of the information processing system 1, the modification 1, the modification 2, and the modification 3.

Hereinafter, embodiments of an information processing apparatus, an information processing method, and a program of the present invention will be described with reference to the drawings. In the following description, the words uttered by the automatic response device or the user are "utterances", the set of utterances is "conversation", and a predetermined reaction of the second utterer to the utterances given by the first utterer is shown. Utterances that are estimated to be present are referred to as “feedback utterances”. An automatic response device is an example of the first utterance subject, and a user (person) is an example of the second utterance subject.

The information processing device is realized by one or more processors. The information processing device derives an index indicating the type of conversation with respect to the conversation conducted between the user and the automatic response device, for example. The type of conversation is, for example, whether the conversation is unnatural (or vice versa). The conversation being unnatural means that the conversation is not established as a result of, for example, an appropriate automatic response not being made. Note that the type of conversation is not limited to whether the conversation is unnatural and may be set arbitrarily.

The information processing device also generates a speech classifier and a conversation classifier in the process of the processing. The utterance classifier derives an index (an utterance score described later) indicating the degree to which an unnatural conversation or a natural conversation is estimated to appear immediately before the feedback utterance, which is an index given to the feedback utterance. It is a thing. In the embodiment described below, an example in which the utterance score is an index indicating the degree to which an unnatural conversation is estimated to appear immediately before the feedback utterance will be described. In addition, hereinafter, the “conversation that appears immediately before” (or “the last conversation”) is a combination of the user's utterance and the utterance of the automatic response device. The conversation classifier derives an index (conversation score described later) indicating the degree of unnaturalness of the conversation, which is an index given to the conversation.

[Constitution]
FIG. 1 is a diagram showing a part of the configuration of the information processing system 1. The information processing system 1 includes, for example, a conversation log storage device 10, a feedback utterance storage device 12, an utterance learning data storage device 14, an acquisition unit 20, an utterance classifier generation unit (previous generation unit) 22, and an utterance classification. And a container 24. The functional configuration described above may be configured as an apparatus.

The acquisition unit 20, the speech classifier generation unit 22, and the speech classifier 24 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Further, some or all of these components are hardware (circuits) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit). Part; including circuitry), or may be realized by cooperation of software and hardware.

Each storage device included in the information processing system 1 is realized by, for example, a ROM (Read Only Memory), a HDD (Hard Disk Drive) flash memory, an SD card, a RAM (Random Access Memory), a register, or the like.

The conversation log storage device 10 stores conversation log information. The log information of the conversation is, for example, text information of the conversation performed by the user and the automatic response device that operates by artificial intelligence (AI). The text information may be converted from a speech utterance by voice recognition.

Feedback utterances are stored in the feedback utterance storage device 12. The feedback utterance stored in the feedback utterance storage device 12 is extracted from the conversation log storage device 10. The feedback utterance is, for example, a preset feedback utterance. For example, a worker extracts from the conversation log information stored in the conversation log storage device 10, or a predetermined device (or other system) stores the conversation log based on a preset wording of the feedback utterance. Feedback utterances are extracted (acquired) by automatically extracting them from the conversation log information stored in the device 10.

As described above, the feedback utterance storage device 12 stores the feedback utterance acquired from the conversation log storage device 10. FIG. 2 is a diagram illustrating an example of feedback utterance. For example, the feedback utterance includes utterances such as (1) "what is different and different" and (2) "thank you". Whether or not the feedback utterance is one that denies or affirms the immediately preceding conversation cannot accurately determine whether or not the immediately previous conversation has been established. Therefore, the information processing system 1 of the present embodiment generates the utterance classifier 24 that outputs only the feedback utterance and outputs the score (probability) indicating whether or not the conversation immediately before the feedback utterance is established.

First, a conversation having a feedback utterance is acquired from the conversation log storage device 10, and as shown in FIG. 3, the conversation immediately before the feedback utterance is natural (label indicating the first type) or unnatural (second type). Label) is added. Then, the learning data in which the label given to the conversation immediately before the feedback utterance is used as the teacher label of the feedback utterance is generated and stored in the utterance learning data storage device 14.

In the example of FIG. 3, since the conversation immediately before “Thank you” is natural, the teacher utterance “natural” is given to the feedback utterance “Thank you”, and the conversation immediately before “What does” is unnatural. Therefore, the learning data with the teacher label "what is meant" is unnatural with respect to the feedback utterance is generated.

The utterance classifier generation unit 22 learns the learning data and generates the utterance classifier 24. The utterance classifier generation unit 22 also performs learning using a deep learning technique using a neural network or a technique such as SVM (Support Vector Machine).

The utterance classifier 24, when given an unknown or known feedback utterance, derives an utterance score that represents the probability that the immediately preceding conversation is unnatural. The feedback utterance given to the utterance classifier 24 is, for example, the feedback utterance acquired by the acquisition unit 20 from the feedback utterance storage device 12. The utterance score is an index indicating the degree to which the utterance made by the automatic response device immediately before the feedback utterance is estimated to be unnatural with respect to the utterance made by a person immediately before the feedback utterance. That is, the higher the utterance score, the higher the probability that the conversation immediately before the feedback utterance between the user and the automatic response device is unnatural.

FIG. 4 is a diagram showing an example of feedback utterances input to the utterance classifier 24 and utterance scores output by the utterance classifier 24. For example, the utterance score derived for the utterance "what is different (FB1 in the figure)" is higher than the utterance score derived for the utterance "It is fairly straightforward and good (FB2 in the figure)". Get higher

The score derived by the utterance classifier 24 makes it possible to judge only a feedback utterance when deciding whether the conversation is natural or unnatural, without scrutinizing the contents of the conversation. Further, in the present embodiment, the assigned label is a binary label such as a natural label indicating that the immediately previous conversation is natural or an unnatural label indicating that the immediately previous conversation is unnatural. The label indicating the type 1 or the label indicating the second type is not limited to whether the immediately preceding conversation is established (or natural) or not established (or unnatural), and is arbitrary. May be added to the feedback utterance. For example, a feedback utterance indicating praise, acceptance, comprehension, appreciation, interest, etc. may be labeled with the first type, or disappointment, poor communication, incomprehension, contempt, boredom, etc. A label indicating the type may be added.

The utterance classifier 24 stores the correspondence information in which the utterance score is associated with the feedback utterance in the scored feedback utterance storage device 50 of the information processing system 1 described later.

FIG. 5 is a diagram showing another configuration of the information processing system 1. In addition to the configuration shown in FIG. 1, the information processing system 1 further includes a conversation log storage device 40, a feedback utterance storage device with score 50, a conversation learning data storage device 52, an extraction unit 62, and a learning data generation unit 63. A conversation classifier generation unit (post-stage generation unit) 64 and a conversation classifier 66. Note that these functional configurations may be configured as a device. Further, of the functional configurations included in the information processing system 1, any functional configuration may be configured as a device.

For example, some or all of the extraction unit 62, the learning data generation unit 63, the conversation classifier generation unit 64, and the conversation classifier 66 are executed by a hardware processor such as a CPU executing a program (software). Will be realized. Further, some or all of these constituent elements may be realized by hardware (including a circuit unit; circuitry) such as LSI, ASIC, FPGA, and GPU, or realized by cooperation of software and hardware. May be done.

The components shown in FIGS. 1 and 5 communicate, for example, by software communication or via a hardware network. The hardware network may include, for example, a WAN (Wide Area Network), a LAN (Local Area Network), the Internet, a dedicated line, a wireless base station, a provider, and the like.

The conversation log storage device 40 stores, for example, conversation log information. The conversation log information may be the same as or different from the information stored in the conversation log storage device 10.

The feedback utterance storage device with score 50 stores the feedback utterance whose utterance score is derived by the utterance classifier 24 and the utterance score for the feedback utterance.

The extraction unit 62 acquires the feedback utterance and the score corresponding to the feedback utterance from the scored feedback utterance storage device 50, and acquires the conversation including the feedback utterance (the feedback utterance and the conversation immediately before it) from the conversation log storage device 40. When the conversation including the feedback utterance is acquired from the conversation log storage device 40, the feedback utterance extracted from the scored feedback utterance storage device 50 is used.

The extraction unit 62 acquires a conversation including the feedback utterance extracted from the conversation log storage device 40, and based on the score attached to the feedback utterance extracted from the scored feedback utterance storage device 50, the conversation immediately before the feedback utterance. Give a score to.

When it is desired to give a score to a conversation including a feedback utterance that is not stored in the scored feedback utterance storage device 50, the feedback utterance of the conversation is given to the utterance classifier 24 to obtain the score.

The learning data generation unit 63 assigns a label representing the type to the conversation immediately before the feedback utterance based on the score given to the conversation immediately before the feedback utterance, and the conversation immediately before the feedback utterance and the type thereof as the teacher label. The learning data is generated and the learning data is stored in the conversation learning data storage device 52.

For example, a conversation immediately before a feedback utterance to which a score equal to or lower than a first threshold value (eg, 0.3) is assigned a label of the first type and a score equal to or higher than a second threshold value (eg, 0.7) is given. The second type of label is given to the conversation immediately before the given feedback utterance. Regarding the labeling, the feedback utterances are rearranged in the descending order of the score, rather than the threshold of the score as described above, and the conversations immediately before the feedback utterances of the upper predetermined ratio (eg, 20%) are classified into the second type. The label may be given, and the first type of label may be given to the conversation immediately before that.

FIG. 6 is a diagram showing an example of feedback utterances to which a score is added. For example, if a label with a score of 0.3 or less is given a label of the first type, and if a score of 0.7 or more is given a label of the second type, ”, and “It’s fairly straightforward and nice” are examples of feedback utterances in which the utterance score is biased toward the first category side, and “What is different and what is different”, This is an example of feedback utterances in which the utterance score is biased toward the second type.

FIG. 7 is a diagram showing an example of a conversation extracted from the conversation log storage device 40. For the conversation immediately before the feedback utterance "what is different and different", a label is given based on the score of the feedback utterance "different and what different". A label immediately before the feedback utterance "cool" is given based on the score of the feedback utterance "cool". For example, for a conversation immediately before a feedback utterance with a score of 0.3 or less, a label of the first type is given, and for a conversation immediately before a feedback utterance with a score of 0.7 or more, When the second type of label is given, the first type of label is given for the conversation of "426 + 129 is" and "The answer is 555", and the first for the conversation of "Show browsing history" and "Fufufu". The learning data to which the label of the type 2 is given is generated as the learning data.

In the conversation learning data storage device 52, a label indicating the type (for example, the first type or the second type) in the conversation immediately before the feedback utterance (not including the feedback utterance) generated by the learning data generating unit 63. The learning data assigned with is stored.

The conversation classifier generation unit 64 generates a conversation classifier 66 that derives a conversation score that is an index indicating the type of unknown conversation, based on the conversation immediately before the feedback utterance extracted by the extraction unit 62. The conversation classifier generation unit 64 includes a conversation immediately before the feedback utterance biased to the first type side, a conversation immediately before the feedback utterance biased to the second type side, and labels given to these immediately previous conversations. Learning is performed based on the (first type or second type) information. Learning is performed by machine learning, for example. The conversation classifier generation unit 64 generates the conversation classifier 66 based on machine learning using the immediately previous conversation extracted by the extraction unit 62 and label information given to the immediately previous conversation as a teacher label. The conversation classifier generation unit 64 may perform the above learning using a deep learning technique using a neural network or the like, or a method such as SVM.

FIG. 8 is a diagram conceptually showing the learning process. For example, if the feedback utterance biased toward the second type is "what is different and different", the conversations a to c that appear immediately before "what is different and different" are extracted. If the feedback utterance biased toward the first type is "cool", the conversations d to f appearing immediately before "cool" are extracted. Thus, in the conversation between the user and the automatic response device, the conversation classifier 66 is learned from the conversation having a high probability of being a natural or unnatural conversation.

When an unknown or known conversation is given, the conversation classifier 66 derives a conversation score (second stage index) indicating the type of the conversation. The conversation score is an index indicating the degree to which the utterance uttered by the automatic response device is estimated to be unnatural with respect to the utterance made by a person immediately before. That is, the higher the conversation score, the higher the probability that the conversation conducted between the user and the automatic response device is unnatural.

FIG. 9 is a diagram showing an example of (unknown) conversation input to the conversation classifier 66 and information output by the conversation classifier 66. For example, when the user's utterance "I can't win a pachinko game" and the utterance "How about a donation?" from the automatic response device, which is a response to the user's utterance, are input to the conversation classifier 66. Outputs, for example, that the probability that the above conversation is unnatural is 95%. In this way, the conversation classifier 66 can determine the naturalness or unnaturalness of a conversation even for an unknown conversation that is not followed by a feedback utterance.

Further, in the above process, it is assumed that it is learned that “show the browsing history” and “fufufu” as conversation A have a high probability of being unnatural. For example, when an unknown conversation “Show History” “Fufufu” is input to the conversation classifier 66, the conversation classifier 66 has the same probability of being unnatural as the conversation A with respect to the unknown conversation. Derive a high conversation score. This is because "browsing history" and "history" are words that are close in meaning.

[Process for generating speech classifier]
FIG. 10 is a flowchart showing the flow of processing in which the utterance classifier 24 is generated by the information processing system 1. First, the utterance classifier generation unit 22 acquires, from the utterance learning data storage device 14, the feedback utterance which is the learning data and the teacher label assigned to the feed utterance (S100).

Next, the utterance classifier generation unit 22 learns the probability that an unnatural conversation or a natural conversation will appear immediately before the feedback utterance, based on the learning data acquired in S100 (S102). Next, the speech classifier generating unit 22 generates the speech classifier 24 based on the learning result of S102 (S104).

Next, the acquisition unit 20 acquires the feedback utterance of the target to which the utterance score is given, and inputs the acquired feedback utterance into the utterance classifier 24. The utterance classifier 24 gives an utterance score to the input feedback utterance, and associates the utterance score of the feedback utterance with the feedback utterance in correspondence information in the scored feedback utterance storage device 50 of the information processing system 1. It is stored (S106). This completes the processing of this flowchart.

Through the above-described processing, the utterance classifier 24 that derives the utterance score indicating the degree to which the immediately preceding conversation is unnatural is generated for the feedback utterance, and the generated utterance classifier 24 responds to the predetermined feedback utterance. A score is given.

[Process for generating conversation classifier]
FIG. 11 is a flowchart showing the flow of processing in which the conversation classifier 66 is generated by the information processing system 1. First, the extraction unit 62 acquires the correspondence information stored in the feedback utterance storage device with score 50 (S200). Next, the extraction unit 62 automatically extracts the feedback utterance to which the utterance score has been added, from the correspondence information acquired in S200 (S202).

Next, the extraction unit 62 extracts the conversation including the feedback utterances extracted in S202 (the feedback utterance and the conversation immediately before the feedback utterance) from the log information stored in the conversation log storage device 40, and the feedback with the score. Based on the score given to the feedback utterance extracted from the utterance storage device 50, a score is given to the conversation immediately before the extracted feedback utterance (S204). Next, the learning data generation unit 63 adds a label indicating the type to the conversation immediately before the feedback utterance based on the score given in step S204, and sets the conversation immediately before the feedback utterance and its type as the teacher label. Learning data including information is generated, and the learning data is stored in the conversation learning data storage device 52 (S206).

Next, the conversation classifier generation unit 64 performs learning based on the learning data generated in S206 and stored in the conversation learning data storage device 52 (S208). Next, the conversation classifier generation unit 64 generates the conversation classifier 66 based on the learning result of S208 (S210). This completes the processing of this flowchart.

Through the above-described processing, the conversation classifier 66 that derives a conversation score indicating the unnaturalness of conversation is performed.

In the above example, the process of generating the utterance classifier 24 and the process of generating the conversation classifier 66 are described as separate processes, but these processes may be a series of processes.

[Summary]
Even if the feedback utterance indicates the first type, the immediately preceding conversation may be unnatural, and even if the feedback utterance indicating the second type may be performed, the immediately previous conversation may be natural. Whether the conversation between the automatic response device and the user is natural or unnatural does not necessarily indicate the type of feedback utterance, and another factor may be involved. For example, when a kind word is uttered by the automatic response device, the user may make a feedback utterance indicating the first type. Further, for example, when an utterance that makes the user angry is made by the automatic response device, the user may make a feedback utterance indicating the second type. Therefore, it is not appropriate to simply judge that the conversation immediately before the feedback utterance indicating the first type is natural and the conversation immediately before the feedback utterance indicating the second type is unnatural.

In addition, since feedback utterances indicating the first type or the second type do not frequently appear in conversations, if the machine learning technique is not applied to the labeled conversations, the feedback utterances that do not follow do not occur. It was sometimes difficult to judge naturalness or unnaturalness in a wide range.

On the other hand, the information processing system 1 of the present embodiment performs machine learning on the conversation immediately before the feedback utterance with the score, which is extracted from the conversation log information, to generate the conversation classifier 66. , The naturalness of the conversation compared to the method of simply recognizing the conversation immediately before the feedback utterance indicating the first type as a natural conversation and the conversation immediately before the feedback utterance indicating the second type as an unnatural conversation. Or, the unnaturalness can be judged in a wide range. For this reason, the conversation classifier 66 can improve the coverage of conversations to be judged, and can judge the naturalness or unnaturalness of conversations even for unknown conversations.

Further, the information processing system 1 according to the present embodiment performs the machine learning on the conversation immediately before the feedback utterance in which the utterance score is biased toward the first type side or the second type side to generate the conversation classifier 66. To do. Therefore, the conversation classifier 66 can more accurately determine whether the conversation is natural or unnatural.

Further, the information processing system 1 of the present embodiment can easily generate the conversation classifier 66 suitable for the task or domain. For example, in the system of the comparative example, when the conversation classifier 66 suitable for a task or domain is generated, log information of the conversation that has appeared in the task or domain is collected and a label is given to the collected conversation. Then, the system of the comparative example performs machine learning on the labeled conversation to generate the conversation classifier 66. In this case, the conversation classifier 66 must be manually created for each task or domain, which increases the cost.

On the other hand, when the information processing system 1 of the present embodiment generates the utterance classifier 24 based on the log information of a certain conversation, the utterance classifier 24 is applied to various tasks and domains. Thus, the conversation classifier 66 can be easily generated. For example, the information processing system 1 extracts the conversation immediately before the feedback utterance with the utterance score from the log information of the conversation that appears in the target task or domain, and performs machine learning on the extracted conversation and utterance score. By performing the process to generate the conversation classifier 66, the conversation classifier 66 suitable for the target task or domain can be generated. As described above, the information processing system 1 applies the utterance classifier 24 to generate the conversation classifier 66 even if no label is attached to the conversation that appears in the target task or domain. You can That is, according to the method of the present embodiment, once a database of scored feedback utterances is created, even if a new task or domain is tackled, a dialogue log of the task or domain and the scored feedback utterance storage device 50 are used. The conversation classifier 66 can be learned automatically and without any trouble, that is, at low cost.

In the above-described embodiment, the conversation classifier 66 is described as deriving the index indicating the unnaturalness of the conversation, but the “unnaturalness” may be replaced with another characteristic. For example, an index indicating the degree to which the conversation immediately before the feedback utterance is of a predetermined type may be derived. For example, an index indicating the degree to which the conversation immediately before the feedback utterance is beneficial to the user, an index indicating the degree to which the conversation improves the mood of the user, and the like may be derived. In these cases, instead of the label indicating the first type or the label indicating the second type for the feedback utterance, a label according to the type of the index is given, and for the conversation immediately before the feedback utterance, Instead of the natural label or the unnatural label, a label according to the type of index is given.

In the above-described embodiment, the conversation classifier 66 has described an example of deriving the probability that the conversation corresponds to one of the two types (for example, the second type). However, instead of this, the conversation is classified. It is also possible to derive a probability that indicates which type among three or more types. In this case, for example, labels indicating three or more types of conversation are prepared. For example, consider a case where, in addition to the labels indicating the first type and the second type, a label indicating the third type indicating neutral conversation is prepared. In this case, the labels immediately before the feedback utterance stored in the utterance learning data storage device 14 are labeled with the first to third types. Then, the information processing system 1 learns the relationship between the first type to the third type and the feedback utterance. In addition, for example, the information processing system 1 automatically extracts, from the correspondence information, a feedback utterance having a score included in a range indicating a natural conversation, an unnatural conversation, and a neutral conversation having a preset utterance score. To do. Then, the information processing system 1 learns the relationship between the extracted conversation immediately before the feedback utterance and the label indicating the type of conversation, thereby generating the conversation classifier 66.

[Comparative Examples 1 and 2]
FIG. 12 is a diagram showing the functional configurations of Comparative Example 1 and Comparative Example 2. Comparative Example 1 shown in the upper diagram of FIG. 12 is a method based on supervised learning that uses data created manually. In Comparative Example 1, the learning unit 100 machine-learns the information stored in the utterance learning data storage device 14, and the conversation classifier 102 is generated based on the learning result. The information stored in the utterance learning data storage device 14 is the conversation immediately before the feedback utterance to which the natural label or the unnatural label is given.

In Comparative Example 2 shown in the lower diagram of FIG. 12, a score is given to a conversation depending on which one of the feedback utterance indicating the first type and the feedback utterance indicating the second type follows most. is there. In Comparative Example 2, the speech classifier 24 and the conversation classifier 66 are not used.

In Comparative Example 2, the score derivation unit 110 performs a conversation based on the information stored in the feedback utterance storage device 12 (feedback utterance without a score) and the log information stored in the conversation log storage device 40. Derive a score for. For example, the score (Score) is derived by the following equation (1). |NEG| is the number of feedback utterances indicating the second type following the conversation in which the log information is focused. |POS| is the number of feedback utterances indicating the first type following the conversation of interest in the log information.
Score=|NEG|-|POS|... (1)

[Comparison with Comparative Examples 1 and 2]
FIG. 13 is a diagram illustrating an example of processing results of the information processing system 1, the comparative example 1, and the comparative example 2. The vertical axis in the figure represents the precision and the horizontal axis represents the recall. The precision is an index indicating how much the correct answer (unnatural conversation) is included in the result of the information processing system determining that the conversation is unnatural. In this case, when the conversation score was equal to or higher than the threshold, it was determined that the conversation was unnatural. The correct answer (unnatural conversation) label is given by a person. The recall rate is an index indicating the degree of determination that the information processing system 1 determines that the conversation is unnatural among the correct answers. AUC (Area Under the Curve) is the area under the curve in the graph.

As illustrated, the information processing system 1 of the present embodiment has the same performance as that of the comparative example 1 or the performance of the comparative example 1 or more. More specifically, the method of Comparative Example 1 is a high-cost method in which the learning data of the conversation classifier must be manually created for each task and each domain. Although it is a low-cost method that does not depend on the domain, it exhibits the same performance as Comparative Example 1. Further, the information processing system 1 of the present embodiment has remarkable performance as compared with Comparative Example 2 in which the feedback utterance is ambiguous and the feedback utterance is infrequent.

Hereinafter, an information processing system 1A of modification 1, an information processing system 1B of modification 2, and an information processing system 1C of modification 3 which are modifications of the information processing system 1 will be described.

[Modification 1]
Modification 1 is an example in which the conversation classifier generation unit 64 is made to further learn the conversation between the user and the automatic response device to which the natural label or the unnatural label is stored, which is stored in the utterance learning data storage device 14. is there. FIG. 14 is a diagram illustrating an example of a functional configuration of the information processing system 1A of the first modification.

[Modification 2]
Modification 2 is an example in which the utterance classifier 24 is omitted. In this case, in the information processing system 1B, the feedback utterance storage device 12 is provided instead of the scored feedback utterance storage device 50. FIG. 15 is a diagram showing an example of the functional configuration of the information processing system 1B of the second modification. The conversation classifier generation unit 64 of the information processing system 1B derives a conversation candidate having a high probability of being a natural conversation and a conversation candidate having a high probability of being an unnatural conversation by using the above-described formula (1).

The information processing system 1B sets, for example, a conversation with a score within a predetermined range as a conversation candidate with a high probability of being a natural conversation, and a conversation with a score within a range different from the predetermined range is an unnatural conversation. Make it a conversation candidate with a high probability.

[Modification 3]
FIG. 16 is a diagram illustrating an example of a functional configuration of the information processing system 1C of Modification 3. In the third modification, the learning data generation unit 63 and the conversation learning data storage device 52 are omitted, and the score derivation unit 120 is provided instead of the conversation classifier 66 of the information processing system 1. Of the feedback utterances stored in the scored feedback utterance storage device 50, the extraction unit 62 has a score in a first range (for example, 20 or 30% from the minimum value) and a second range (for example, 20 or 30 from the maximum value). Extract feedback utterances with a score (in percent). The score derivation unit 120 derives a score using the feedback utterance extracted by the extraction unit 62. Specifically, the score derivation unit derives the score using the above-mentioned formula (1).

[Comparison with the modification]
FIG. 17 is a diagram illustrating an example of processing results of the information processing system 1, the first modification, the second modification, and the third modification. Descriptions similar to those in FIG. 13 are omitted.

As shown in FIG. 17, the information processing system 1, the first modification, and the second modification have higher performance than the third modification that does not have the conversation classifier 66. The information processing system 1 and the modified example 1 have higher performance than the modified example 2 which does not have the utterance classifier 24. That is, it has been clarified from the experiment that the conversation classifier 66 greatly contributes to the performance of the information processing system 1 of this embodiment. The modified example 1 has slightly higher performance than the information processing system 1.

According to the embodiment described above, the information processing system 1 estimates the feedback utterance that is estimated to show the predetermined reaction of the second utterer to the utterance uttered by the first utterer, from the set of conversations. The extraction unit 62 that extracts the immediately preceding conversation based on the utterance score given to the feedback utterance, and the index that indicates the type of the unknown conversation based on the immediately previous conversation extracted by the extraction unit 62. By providing the conversation classifier generation unit 64 that generates the conversation classifier 66, it is possible to automatically obtain a clue to determine whether the unknown conversation is of a predetermined type.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the scope of the present invention. Can be added.

1... Information processing system, 20... Acquisition unit, 22... Utterance classifier generation unit, 24... Utterance classifier, 50... Scored feedback utterance storage device, 62... Extraction unit, 64... Conversation classifier generation unit, 66... Conversation Classifier

Claims (6)

An acquisition unit for acquiring a feedback utterance estimated to exhibit a predetermined reaction of the second utterance subject to the utterance uttered by the first utterer subject;
Immediately before the feedback utterance acquired by the acquisition unit, a pre-stage generation unit that generates a pre-stage classifier that derives an index estimated that a predetermined type of conversation appears.
An information processing apparatus including. Further comprising a pre-stage classifier generated by the pre-stage generation unit,
The information processing apparatus according to claim 1. The acquisition unit acquires the feedback utterance, a conversation immediately before the feedback utterance assigned a first type, and a conversation immediately before the feedback utterance assigned a second type,
The pre-stage generation unit generates a pre-stage classifier based on the information acquired by the acquisition unit,
The information processing apparatus according to claim 2. The pre-stage generation unit performs machine learning to generate the pre-stage classifier by using a label indicating the type given to the feedback utterance acquired by the acquisition unit and the immediately preceding conversation as learning data.
The information processing device according to claim 3. Computer
Acquiring a feedback utterance estimated to show a predetermined reaction of the second utterance subject to the utterance uttered by the first utterer subject,
Immediately before the acquired feedback utterance, a pre-stage classifier that derives an index estimated that a predetermined type of conversation appears is generated.
Information processing method. On the computer,
A feedback utterance presumed to exhibit a predetermined reaction of the second utterer to the utterance uttered by the first utterer is acquired,
Immediately before the obtained feedback utterance, a pre-stage classifier that derives an index estimated that a predetermined type of conversation appears is generated.
program.

Images