KR101131278B1 - Method and Apparatus to Improve Dialog System based on Study - Google Patents

Abstract

The present invention relates to a method and apparatus for improving performance of a learning based conversation system using a conversation log. The method according to the present invention comprises the steps of: collecting a conversation log between the user and the conversation system; generating a speech recognition candidate group for improving a speech recognition model of the conversation system by extracting error utterances from the collected conversation log; Generating a language understanding candidate to improve the language understanding model of the conversation system by extracting error utterances from the log; and extracting a new conversation pattern that does not exist in the conversation system's conversation pattern from the collected conversation logs. Generating a conversation pattern candidate group for improving the speech recognition, verifying the speech recognition candidate group, the language understanding candidate group and the conversation pattern candidate group, and applying the verified candidate group to the improvement of the speech recognition model, the language understanding model, and the conversation model. Include.

Description

Method and Apparatus for Improving Learning-Based Dialog System Performance Using Dialog Log {Method and Apparatus to Improve Dialog System based on Study}

The present invention relates to a method and apparatus for improving performance of a learning-based conversation system, and more particularly, to a method and apparatus for improving performance of a learning-based conversation system using a conversation log.

The conversation system refers to a device that obtains the information that a user needs through a conversation by using voice or text. The conversation system is used as a next-generation intelligent interface for a terminal, a car, a robot, and the use range is gradually expanded. However, it takes a lot of time, effort and cost to build a dialogue system according to each domain that needs it.

1 is a block diagram provided to explain an example of a conventional conversation system.

Referring to FIG. 1, in the conventional conversation system 10, when a voice spoken by a user is input, the final response is determined by the conversation manager 15 through the voice recognition unit 11 and the language understanding unit 13. The system speech is outputted through the generation unit 17 and the speech synthesis unit 19. In order for the parts of the voice recognition unit 11, the language understanding unit 13, and the conversation manager 15 to work properly, a model for processing the parts is required, which is usually before the system 10 is used. It must be made. And if necessary, the model can be modified later, but it requires a lot of effort and help from experts.

In particular, the conventional dialog system is difficult to show excellent performance when the user's utilization is slightly different from the existing model. Experts can intervene to re-analyze user usage and recreate and apply models, but they also require constant management and effort. Workbench tools have been developed to relieve the burden on the managers, but they have not been able to automatically analyze and apply processes that require a lot of human effort and use of the dialogue system.

Therefore, an object of the present invention is to improve the performance of the learning-based conversation system using a conversation log that can improve the performance by modifying the speech recognition model, language understanding model, conversation management model, etc. used in the conversation system with minimal expert intervention. It is to provide a method and an apparatus thereof.

According to an aspect of the present invention, there is provided a method of improving performance of a learning-based conversation system using a conversation log, including: collecting a conversation log between a user and a conversation system; Generating a speech recognition candidate group for extracting and improving a speech recognition model of the conversation system; generating a language understanding candidate group for improving a language understanding model of the conversation system by extracting an error speech from the collected conversation logs; Generating a conversation pattern candidate group for improving a conversation model of the conversation system by extracting a new conversation pattern that does not exist in the conversation pattern of the conversation system from the collected conversation log, the speech recognition candidate group and the language understanding candidate group Verifying the conversation pattern candidate group; And a step of applying the candidates verified in the verification step to improve the speech recognition model, the language understanding model and the dialogue model.

In the verifying step, a candidate group to be applied to the improvement of the speech recognition model, the language understanding model, and the conversation model may be selected by the expert or a predetermined user from among the speech recognition candidate group, the language understanding candidate group, and the conversation pattern candidate group. .

The generating of the speech recognition candidate group may include extracting an error speech from the conversation log, performing speech recognition on the extracted error speech using a speech recognition model having a higher recognition rate than the speech recognition model of the conversation system; And generating a voice recognition candidate group including a voice recognition result and voice recognition reliability according to the voice recognition.

The language understanding candidate group generating step may include extracting an error utterance from the conversation log, and performing a language understanding of the error utterance based on conversation history information included in the conversation log, a language understanding model of the conversation system. Performing a language understanding on the error utterance using the same model as the language understanding, and a language understanding result having a high language understanding reliability among the language understanding results based on the conversation history information and the language understanding results using the language understanding model. And generating the candidate group.

The conversation pattern candidate group generating step may include extracting a conversation pattern that does not exist in the conversation system from the conversation log, removing an error from the conversation pattern, and generating the conversation pattern in which the error is removed as a conversation pattern candidate group. It may include the step.

Error speech extraction for generating the speech recognition candidate group is performed by extracting user speech whose speech recognition reliability is below a predetermined reference value from the conversation log, and error speech extraction for generating the language understanding candidate group is performed in the conversation log. At least one of the understanding reliability may be achieved by extracting user utterance that is less than or equal to a predetermined reference value.

The conversation log may include a user name, speech time, speech speech, speech recognition result, speech recognition reliability, language understanding result, language understanding reliability, and conversation history information.

The conversation log is collected for each user, and generation and verification of the speech recognition candidate group, the language understanding candidate group and the conversation pattern candidate group, and the application of the verified candidate group to the speech recognition model, the language understanding model, and the conversation model are performed. It can be performed per user.

A computer readable medium according to another embodiment of the present invention records a program for causing a computer to execute any one of the above methods.

An apparatus for improving performance of a learning-based conversation system using a conversation log according to another embodiment of the present invention may include a conversation log storage unit for storing a conversation log between a user and a conversation system, and extracting an error utterance from the conversation log. A speech recognition model enhancement unit for generating a speech recognition candidate group for improving a speech recognition model, and a language understanding model enhancement unit for generating a language understanding candidate group for improving a language understanding model of the conversation system by extracting an error speech from the conversation log. A conversation model enhancement unit for generating a conversation pattern candidate group for improving a conversation model of the conversation system by extracting a new conversation pattern that does not exist in the conversation pattern of the conversation system from the conversation log, the speech recognition candidate group, and understanding the language Candidates for verifying the candidate group and the conversation pattern candidate group And a Model and Training of applying a verification unit, and the improvement of speech recognition models, language understanding model and model dialog of the dialog system the verification of candidate.

As described above, according to the present invention, the model can be improved by actual user customization by using the chat log collected in the process of actually using the chat system by the user.

In addition, the optimal performance improvement effect can be expected by reflecting the results of the model improvement candidate group derived through the user dialog log analysis to the model improvement after being verified by experts or experts.

In addition, since the expert only needs to perform the verification of the model candidate group presented by the device, it is economical because the effort and time of the expert may be less than that of the conventional model candidate group.

1 is a block diagram provided to explain an example of a conventional conversation system.
2 is a block diagram provided to explain an apparatus for improving conversational system performance according to an embodiment of the present invention.
FIG. 3 is a block diagram provided to explain the conversation system of FIG. 2.
4 is a block diagram provided to explain the apparatus for improving conversation system performance of FIG. 2.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention.

2 is a block diagram provided to explain an apparatus for improving conversational system performance according to an embodiment of the present invention.

Referring to FIG. 2, the conversation system performance improving apparatus 400 is connected to the conversation system 300 to receive a conversation log. The conversation system performance improving apparatus 400 analyzes the conversation log and performs a function of improving the speech recognition model, the language understanding model, and the conversation management model of the conversation system 300. The conversation system performance improving apparatus 400 and the conversation system 300 may be connected through a wired or wireless communication network or directly through a cable.

The conversation system 300 is a system that provides a service desired by a user by communicating with a user using voice or text. For example, it may be used in a system connected to a broadcast receiving device (not shown) in a home to understand a broadcast program search request through a user's voice, a channel change command, and to operate the broadcast receiving device accordingly.

3, the conversation system 300 of FIG. 2 will be described in detail.

FIG. 3 is a block diagram provided to explain the conversation system of FIG. 2.

Referring to FIG. 3, the conversation system 300 according to the present invention includes a speech recognition model 310, a language understanding model 330, a conversation model 350, a speech recognition unit 320, a language understanding unit 340, The conversation manager 360, a response generator 370, a voice synthesizer 380, and a log manager 390 are included. In addition, according to an embodiment, the conversation system 300 may include a user manager 395.

The speech recognition model 310 is a model used by the speech recognition unit 320 and can be classified into two types. One is a language model that estimates the probability of a sentence, and the other is an acoustic model that estimates the probability based on the input sound for each pronunciation.

When the user speech is input, the voice recognition unit 320 performs a function of finding the most optimal sentence for the user speech using the speech recognition model 310. That is, the voice recognition unit 320 converts the voice signal according to the user's speech into a sentence corresponding thereto. For example, the speech recognition unit 320 may ask the user "What are you doing on the TV now?" Is converted into a text sentence corresponding to the text. The conversion of the user speech into a sentence in the speech recognition unit 320 is called a speech recognition result.

The language understanding model 330 is a model used to determine what purpose / intention a given sentence is uttered, what is a named-entity, and so on. The language understanding model 330 may be trained from sentences tagged with speech intent and named-entity. The language understanding model 330 may adopt a rule-based method, a statistical method, or the like.

The language understanding unit 340 performs a function of outputting, as a language understanding result, a result of identifying a user's speech purpose / intention by using the language understanding model 330. That is, the result of language understanding means a result of analyzing whether the user's speech is caused by any purpose or intention such as a question, a request, a suggestion, or a request. For example, user speech can be analyzed as 'dialogue act', 'Main action', and 'Component Slot'.

Here, 'dialogue act' is an item that is a classification criterion of a conversation sentence. The dialogue act is a user depending on whether the dialogue sentence is a sentence form relating to a user request, a sentence form regarding a WH question, or a sentence form regarding a Yes-No question. Sentences can be classified.

'Main action' is semantic information representing an action desired by a user through a conversation. For example, the user may be classified into one that the user wants to know the program start time (search_time), one that the user wants to know the channel in which the desired program is aired (search_channel).

'Component Slot' is object information about domain that appears in actual user speech. For example, it may be classified into a channel (channel = KBS) in which the broadcast program is broadcast, or a genre (genre = drama) of the broadcast program.

 For example, the result of understanding the language for the case where the user speech is "what are you doing on TV?" Can be analyzed as follows. It can be analyzed as "wh-question" for "dialogue act", "search_time" for "Main action", and [start_time = now] for "Component Slot".

The conversation model 350 is a model used to determine an optimal system response based on conversation situation / discourse information for user speech. The conversation model 350 can be trained from a conversation example. In more detail, the conversation model 350 may automatically extract each conversation example from the conversation example corpus to build a conversation example DB. The dialogue example corpus can be in XML form for easy parsing. With the result of parsing the conversation example corpus, an index table can be created using dialogue act information, driving information, main component information, component slot information, and conversation history information.

The conversation manager 360 determines a system response most suitable for the user's intention of speaking using the conversation model 350. In more detail, the conversation manager 360 extracts the most appropriate conversation example from the conversation model 350 based on the current conversation situation and the conversation history information, and determines the system response based on the conversation example. That is, the conversation manager 360 determines a system response such as an answer, a rejection, or an approval of the conversation system 300 based on the language understanding result and the conversation model.

The response generator 370 generates an output sentence according to the system response determined by the conversation manager 360.

The speech synthesizer 380 converts the output sentence into speech and outputs the speech.

The log manager 390 may store various types of information (hereinafter, referred to as “conversation logs”) that can be obtained while the conversation system 300 is operating in predetermined storage means (not shown). The storage means may be included in the conversation system 300 or may be included in the conversation system performance enhancing apparatus 400. The conversation log collected by the log manager 390 may include a user name, speech time, speech speech, speech recognition result, speech recognition reliability, language understanding result, language understanding reliability, conversation history information, and the like.

Here, the user name may be a name or ID for identifying a user who uses the conversation system 300. The spoken voice refers to a voice input signal received by a user through voice input means such as a microphone (not shown). The speech recognition reliability indicates the degree to which the sentence recognized by the speech recognizer 320 corresponds to user speech, and may be calculated and provided by the speech recognizer 320 in a predetermined manner. The language understanding reliability indicates the degree to which the user's uttering purpose / intention grasped by the language understanding unit 340 corresponds to the actual user speech, and may be calculated and provided by the language understanding unit 340 in a predetermined manner. The conversation history information includes a slot filing vector for the component degree filled up to the current conversation state, and dialog act information and main act information shown in the previous conversation.

The user manager 395 performs a function of separately managing a speech recognition model, a language understanding model, and a conversation model for each user when there are a plurality of users who use the conversation system 300.

Next, referring to FIG. 4, the interactive system performance improving apparatus 400 of FIG. 2 will be described in detail.

4 is a block diagram provided to explain the apparatus for improving conversation system performance of FIG. 2.

Referring to FIG. 4, the apparatus 300 for improving performance of a conversation system includes a conversation log storage unit 410, a speech recognition model enhancement unit 420, a language understanding model enhancement unit 430, a conversation model enhancement unit 440, and a candidate group. The verification unit 450 and the model training unit 460 may be included.

The conversation log storage unit 410 stores a conversation log provided from the log manager 390 of the conversation system 300, and includes a speech recognition model enhancer 420, a language understanding model enhancer 430, and a conversation model enhancer. 440 may be provided.

The speech recognition model enhancer 420 may include an error speech extractor 421, a speech recognition model 423, a speech recognizer 425, and a speech recognition candidate group generator 427.

The error utterance extractor 421 extracts an error utterance from the user utterances stored in the conversation log storage 410. User speech with a speech recognition reliability below a predetermined threshold may be extracted as an error speech.

The speech recognizer 425 generates a speech recognition result for the extracted error speech using the speech recognition model 423. In addition, the speech recognizer 425 may obtain a speech recognition reliability of the generated speech recognition result.

The speech recognition model 423 used in the speech recognition model enhancer 420 may be designed to allow a wider range of recognition than that used in the conversation system 300. The speech recognition model 310 of the conversation system 300 is limited in size and performance due to the constraint of giving a quick response to the user. However, since there is no time constraint in the conversation system performance improving apparatus 400, there is no significant burden on the execution speed of the speech recognizer 425, so that the speech recognition model 423 can be made more freely. In addition to processing modules (not shown), it may be implemented to further identify words not defined in the existing model.

The speech recognition candidate group generator 427 may generate a speech recognition candidate group including the speech recognition result generated by the speech recognizer 425 and the speech recognition reliability and provide the generated speech recognition candidate group to the candidate group verification unit 450. The speech recognition candidate group may be provided as shown in Table 1 as a speech recognition result (sentence) and speech recognition reliability (score) for an error utterance as follows.

Speech recognition results Speech recognition reliability "Then when is the KBS drama?" 0.69

The language understanding model improving unit 430 may include an error speech extraction unit 431, a language understanding result estimating unit 433, a language understanding module 435, a combining unit 437, and a language understanding candidate group generation unit 439. Can be.

The error speech extraction unit 431 extracts a user speech in which the conversation log storage unit 410 has at least one of a speech recognition reliability and a language understanding reliability as an error speech. The reason for considering both the speech recognition reliability as well as the speech recognition reliability is that if the speech recognition is not properly performed, there is a high possibility that the result of the language understanding based on this is also in error.

The language understanding module 435 obtains a language understanding result using the same language understanding model as the language understanding model 330 used in the dialogue system 300 with respect to the extracted error speech.

The language understanding result estimator 433 estimates the language understanding result of the error utterance through the conversation history information in the conversation log storage 410. Compute the N-gram for the Dialog Act / Main Action from the dialogue example. Based on this, estimating the acting / driving information that is most likely to come out in the present according to the acting / driving information of the past.

The combiner 437 compares the language comprehension result obtained by the language comprehension module 435 with the language comprehension result estimated by the language comprehension result estimator 433 to determine which of the two is high.

The language understanding candidate group generation unit 439 generates a language understanding candidate group including the language understanding result and the language understanding reliability determined by the combiner 437 to have high reliability. For example, a language understanding candidate may be provided as shown in Table 2 below.

Language understanding result Language understanding reliability Dialog Act Main action Slots When-question Search_start_time [channel = KBS, genre = drama] 0.87

The conversation model enhancer 440 may include a conversation pattern extractor 441, an error remover 443, and a conversation pattern candidate group generator 445.

The conversation pattern extractor 441 extracts a new conversation pattern that is not in the conversation model of the conversation system 300 from the conversation log storage unit 410. In the conversation log storage unit 410, user speech and system speech are grouped into conversation units as described above. Therefore, the conversation pattern extractor 441 may identify and extract a conversation pattern that does not exist previously for each conversation unit by a predetermined criterion. The criterion used here may be extracted by judging the conversation pattern as a new conversation pattern when the content is below a certain threshold value using a discourse-based n-gram or structured and word interpolated BLUE score (SWB) similar to a language model.

The error remover 443 removes the main error process that was present in the dialogue process with respect to the extracted new dialogue pattern. For example, the main error process could be a case where no conversation example was found, or a speech recognition error or language understanding error was included.

An example of a conversation pattern in which an error is removed from the conversation pattern extracted from the conversation log storage unit 410 is as follows.

* Conversation Pattern Before Eliminating Errors *

================================================== ==========================

User: What are you doing on TV right now?

System: Come to MBC to play, and KBS to broadcast a chat of beauty. Please choose a channel.

User: Then 10 o'clock please <-Voice recognition error / Language understanding error

System: No conversation example found <-No conversation example

User: Then turn on MBC.

System: Move the channel to MBC.

================================================== ==========================

Conversation pattern after error removal

================================================== ==========================

 User: What are you doing on TV right now? // language understanding result + Discourse History [1,0,1,0,0,0,0,0]

 System: Come to MBC to play, and KBS to broadcast the chat of beauty. Please choose a channel.

 User: Then turn on MBC. // language understanding result + Discourse History [1,0,1,1,1,0,0,0]

 System: Move the channel to MBC.

================================================== ===========================

The new conversation pattern in which the error is eliminated may be provided to the conversation candidate group. The conversation candidate group may also be provided by calculating the reliability of the conversation candidate group based on the dialogue act / driving information N-gram.

The candidate group verification unit 450 may cut the candidate system generated by each model from the speech recognition model enhancer 420, the language understanding model enhancer 430, and the conversation model enhancer 440 in the dialogue system 300. Verify it. Preferably, a candidate group for each model may be presented to an expert or an experienced user, and a candidate group to be applied to the performance improvement of the dialogue system 300 may be selected. According to an exemplary embodiment, candidate groups having a reliability higher than or equal to a predetermined criterion may be automatically used to improve the performance of the conversation system 300.

The model training unit 460 performs model enhancement by applying the selected candidate group verified by the candidate group verification unit 450 to each model of the dialogue system 300. More specifically, the model training unit 460 adds a corpus used for training an existing model and a corpus added as a candidate group in the dialogue system performance improving apparatus 400 to add a speech recognition model, a language understanding model, and conversation management of the dialogue system 300. By retraining the model, it replaces the existing model.

Embodiments of the present invention include a computer-readable medium having program instructions for performing various computer-implemented operations. This medium records a program for executing the learning-based conversation system performance improvement method using the conversation log described so far. The medium may include program instructions, data files, data structures, etc., alone or in combination. Examples of such media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CDs and DVDs, floppy disks and program commands such as magnetic-optical media, ROM, RAM and flash memory. Hardware devices configured to store and perform such operations. Alternatively, the medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, Of the right.

Claims (16)

Collecting a chat log between the user and the chat system;
Generating a speech recognition candidate group for improving a speech recognition model of the conversation system by extracting an error utterance from the collected conversation log;
Generating a language understanding candidate group for improving a language understanding model of the conversation system by extracting an error speech from the collected conversation log;
Generating a conversation pattern candidate group for improving a conversation model of the conversation system by extracting a new conversation pattern that does not exist in the conversation pattern of the conversation system from the collected conversation log;
Verifying the speech recognition candidate group, the language understanding candidate group, and the conversation pattern candidate group; and
Applying the candidate group verified in the verification step to the speech recognition model, the language understanding model, and the conversation model;
Learning-based conversation system performance improvement method using a conversation log containing a. The method of claim 1,
The candidate group verification is a conversation log in which an expert or a predetermined user selects a candidate group to be applied to the speech recognition model, the language understanding model, and the conversation model from among the speech recognition candidate group, the language understanding candidate group, and the conversation pattern candidate group. Method for Improving Learning-Based Dialog System Using The method of claim 1,
The voice recognition candidate group generation step,
Extracting an error utterance from the conversation log,
Performing speech recognition on the extracted error speech using a speech recognition model having a recognition rate higher than that of the speech recognition model of the conversation system used to collect the conversation log; and
And generating a speech recognition candidate group including a speech recognition result according to the speech recognition performance and a speech recognition reliability. The method of claim 1,
The language understanding candidate group generation step,
Extracting an error utterance from the conversation log,
Performing a language understanding on the extracted error utterance based on conversation history information included in the conversation log;
Performing language understanding on the extracted error utterance using the same model as the language understanding model of the dialogue system, and
And comparing the reliability value of the language understanding result based on the conversation history information with the reliability value of the language understanding result using the language understanding model to generate a language understanding result having a greater language understanding reliability value as a language understanding candidate group. How to improve the performance of learning-based dialogue system using logs. The method of claim 1,
The conversation pattern candidate group generation step,
Extracting a conversation pattern that does not exist in the conversation system from the conversation log;
Removing the error from the conversation pattern, and
The method for improving performance of a learning-based conversation system using a conversation log comprising generating the conversation pattern from which the error is eliminated as a conversation pattern candidate group. The method according to any one of claims 3 to 5,
The error speech extraction for generating the speech recognition candidate group is performed by extracting a user speech whose reliability of speech recognition is less than or equal to a predetermined reference value from the conversation log.
The error speech extraction for generating the language understanding candidate group is performed by extracting user speech in which at least one of speech recognition reliability and language understanding reliability is less than or equal to a predetermined reference value from the conversation log. The method of claim 1,
The conversation log is a method of improving performance of a learning-based conversation system using a conversation log including a user name, speech time, speech speech, speech recognition result, speech recognition reliability, language understanding result, language understanding reliability, and conversation history information. The method of claim 1,
The chat log is collected for each user,
Generating and verifying the speech recognition candidate group, the language understanding candidate group, and the conversation pattern candidate group, and applying the verified candidate group to the improvement of the speech recognition model, the language understanding model, and the conversation model is learning using a conversation log performed by a user. How to improve performance based conversation system. A chat log storage for storing a chat log between a user and a chat system,
Speech recognition model improving unit for generating a speech recognition candidate group for improving the speech recognition model of the conversation system by extracting an error speech from the conversation log,
A language understanding model improving unit for generating a language understanding candidate group for extracting an error speech from the conversation log to improve a language understanding model of the conversation system;
A conversation model enhancement unit for extracting a new conversation pattern that does not exist in the conversation pattern of the conversation system from the conversation log to generate a conversation pattern candidate group for improving the conversation model of the conversation system;
A candidate group verification unit that verifies the speech recognition candidate group, the language understanding candidate group, and the conversation pattern candidate group, and
And a model training unit for applying the verified candidate group to the improvement of the speech recognition model, the language understanding model, and the conversation model of the conversation system. In claim 9,
The candidate group verification is a conversation in which an expert or a predetermined user selects a candidate group to be applied to the speech recognition model, the language understanding model, and the conversation model of the conversation system from among the speech recognition candidate group, the language understanding candidate group, and the conversation pattern candidate group. Device for improving performance of learning-based dialogue system using logs. In claim 9,
The voice recognition candidate group generation,
Extracting an error utterance from the conversation log, and performing speech recognition on the extracted error utterance using a speech recognition model having a recognition rate higher than that of the speech recognition model of the conversation system used to collect the conversation log, The apparatus for improving performance of a learning-based dialogue system using a dialogue log generating a speech recognition candidate group including a speech recognition result according to the speech recognition and speech recognition reliability. In claim 9,
The language understanding candidate group generation,
Extracting an error utterance from the conversation log, performing a language understanding of the error utterance based on conversation history information included in the conversation log, and using the same model as the language understanding model of the conversation system. Language comprehension results, the language comprehension result having the greater language comprehension reliability value is compared to the language comprehension candidate group by comparing the reliability value of the language comprehension result based on the conversation history information with the reliability value of the language comprehension result using the language comprehension model. An apparatus for improving performance of a learning based conversation system using a conversation log generated. In claim 9,
The conversation pattern candidate group generation is
Improving the performance of a learning-based conversation system using a conversation log extracting a conversation pattern that does not exist in the conversation system from the conversation log, removing an error from the conversation pattern, and generating the conversation pattern from which the error is eliminated as a conversation pattern candidate group. Device. The method according to any one of claims 11 to 13,
The error speech extraction for generating the speech recognition candidate group is performed by extracting a user speech whose reliability of speech recognition is less than or equal to a predetermined reference value from the conversation log.
The error utterance extraction for generating the language understanding candidate group is performed by extracting a user utterance having at least one of a speech recognition reliability and a language understanding reliability from the conversation log using a conversation log. In claim 9,
And the conversation log includes a user name, speech time, speech speech, speech recognition result, speech recognition reliability, language understanding result, language understanding reliability, and conversation history information. In claim 9,
The chat log is collected for each user,
Generating and verifying the speech recognition candidate group, the language understanding candidate group, and the conversation pattern candidate group, and applying the verified candidate group to the improvement of the speech recognition model, the language understanding model, and the conversation model is learning using a conversation log performed by a user. -Based conversation system performance enhancing device.

Images