KR102282695B1 - System and Method for Robust and Scalable Dialogue - Google Patents

Abstract

A dialog system and method used in a task-oriented dialog system are disclosed.
In this embodiment, the scalability of the dialogue system is increased by applying the question intention search, question pattern estimation, and conditional answer process to the user query based on the FAQ KB (Knowledge Base), and weakly labeled ) The purpose is to provide a dialogue system and method with scalability and robustness used in task-oriented dialogue systems by increasing the robustness of the learning model by applying the training process using question pairs.

Description

System and Method for Robust and Scalable Dialogue

The present invention relates to a dialog system and method having scalability and robustness used in a task-oriented dialog system.

The content described below merely provides background information related to the present invention and does not constitute the prior art.

There is a growing interest in task-oriented or target domain-specific dialogue systems that interact with users so that they can achieve goals such as restaurant reservations or flight reservations. is increasing One of the main goals of the task-oriented dialogue system is to provide an answer suitable for the user's query intent.

One way to provide an appropriate answer is to graft Frequently Asked Question (FAQ) answers on a task-oriented dialogue system. A direct method for grafting is to map a user's new request that occurs on the dialog system with an existing question accompanied by a corresponding answer. This type of problem is most closely related to Paraphrase Identification (PI), and the PI model handles the PI problem of identifying whether two sentences are contextually identical.

Recently, by using BERT (Bidirectional Encoder Representations from Transformers, see Non-Patent Document 1), many advances have been made in terms of increasing the accuracy of solving the PI problem.

However, in order to provide an appropriate answer in accordance with the user's intention in the task-oriented dialogue system, two issues of scalability and robustness must be resolved. Here, scalability means that the dialog system accommodates as many users' questions as possible, and robustness means the property of efficiently coping with errors derived from the learning process of the PI model. Therefore, there is a need for a dialogue system and method with scalability and robustness applicable to a task-oriented dialogue system based on an appropriate solution to the two issues.

Non-Patent Document 1: Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. Non-Patent Document 2: Robertson, S.; Zaragoza, H.; et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval.

The present disclosure increases the scalability of the conversation system by applying the question intention search, question pattern estimation, and conditional answer process for user queries based on the FAQ KB (Knowledge Base), and weakly labeled The main purpose is to provide a dialogue system and method having scalability and robustness used in a task-oriented dialogue system by increasing the robustness of the learning model by applying a training process using question pairs.

According to an embodiment of the present invention, a plurality of question pattern groups related to the user's query are selected by searching the knowledge base that stores the answer data to the user's query in the form of data consisting of the question intention, the question pattern, and the conditional answer. and a Question Intent Retriever (QIR) providing the plurality of question pattern groups and a plurality of question intentions connected to the plurality of question pattern groups; a question pattern estimator (Question Pattern Reader: QPR) for selecting a Top-1 question intention using the user query, the plurality of question intentions, and the plurality of question pattern groups; and a Conditional Answerer (CA) that selects an answer related to the top-one question intention by using the conditional answer on the knowledge base.

According to another embodiment of the present invention, in a learning method of a dialog system, Paraphrase Identification (PI) using a dataset including a labeled question pair The process of task fine-tuning the model and refiner; and domain fine-tuning the PI model and the refiner using a domain-specific database including a pair of positive questions with a weak positive label and a pair of negative questions with a negative label. However, the process of fine-tuning the domain provides a learning method implemented on a computer, characterized in that it includes an iterative training process for the PI model and a refinement process for the refiner.

According to another embodiment of the present invention, there is provided a learning apparatus used by a conversation system, comprising: a Paraphrase Identification (PI) model for identifying whether an input question pair is similar; and a refiner for estimating a confidence score corresponding to the input question pair, wherein the PI model uses the confidence score in a training process, and the refiner determines the identification result of the PI model. to use A learning device is provided.

According to another embodiment of the present invention, in a conversation method of a conversation system, a knowledge base storing answer data to a user's query in the form of data consisting of a question intention, a question pattern, and a conditional answer is searched and related to the user's query selecting a plurality of question pattern groups in which there is a question pattern group and providing a plurality of question pattern groups and a plurality of question intentions connected to the plurality of question pattern groups; By inputting the user query and the plurality of question pattern groups into a pre-trained Paraphrase Identification (PI) model, an identification result of whether or not a similar question is asked is calculated, and using the identification result, the A process of selecting a question intention indexing a question pattern group including the highest rate of similar questions among the diagrams as a Top-1 question intention; and selecting an answer related to the top-one question intention by using the conditional answer on the knowledge base.

According to another embodiment of the present invention, there is provided a computer program stored in a computer-readable, non-volatile or non-transitory recording medium in order to execute each step included in the learning method of the conversation system.

According to another embodiment of the present invention, there is provided a computer program stored in a computer-readable, nonvolatile or non-transitory recording medium in order to execute each step included in the conversation method of a conversation system.

As described above, according to this embodiment, based on the FAQ KB (Knowledge Base), the process of searching for a question intention for a user query, estimating a question pattern and a conditional answer is applied to the dialog system, and a weak label is attached (weakly). By providing a dialog system and method used in a task-oriented dialog system to which a training process using labeled) question pairs is applied, there is an effect of increasing the scalability and robustness of the dialog system.

1 is a block diagram of a FAQ answering machine and a learning model according to an embodiment of the present invention.
2 is a conceptual diagram illustrating the operation of the components of the FAQ answering machine according to an embodiment of the present invention.
3 is a conceptual diagram illustrating a training process of a learning model according to an embodiment of the present invention.
4 is a block diagram of a learning model including a PI model and a refiner according to an embodiment of the present invention.
5 is a flowchart illustrating a procedure for answering a user's query by a FAQ answerer according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present embodiments, if it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present embodiments, the detailed description thereof will be omitted.

Also, in describing the components of the present embodiments, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced.

A task-oriented dialog system includes a belief state tracker (also called a dialog state tracker) and an answerer to user queries. can be configured. The present invention particularly relates to the implementation of an answerer in a chat system, and pays attention to a dialog system including a Frequently Asked Question (FAQ) answerer for responding to a user's query.

1 is a block diagram of a FAQ answering machine and a learning model according to an embodiment of the present invention.

1 shows a FAQ answerer 100 used in a task-oriented conversation system, a FAQ knowledge base 110 (Knowledge Base: KB), a question intent retriever (111, Question Intent Retriever, QIR), and a question pattern estimator ( 112, Question Pattern Reader, QPR) and a Conditional Answerer (113, Conditional Answerer: CA). In addition, the QPR 112 includes a Paraphrase Identification (PI) model 114 and a Question Intent Selector 115 .

Components included in the FAQ answering machine 100 according to the present embodiment are not necessarily limited thereto. For example, a learning model and a training device for training the learning model may be additionally provided on the FAQ answering machine, or may be implemented in a form that interworks with an external training device. For example, the learning model 120 shown in FIG. 1 is learned by a training device. The learning model 120 includes a PI model 114 and a refiner 121 . The PI model 114 is provided on the FAQ answerer 100 , but the refiner 121 is the FAQ answerer 100 . exists outside of Accordingly, the refiner 121 is used only for learning, and is not applied when the FAQ answerer 100 operates.

2 is a conceptual diagram illustrating the operation of the components of the FAQ answering machine according to an embodiment of the present invention.

Hereinafter, the form and operation of the components included in the FAQ answering machine 100 will be described with reference to FIGS. 1 and 2 .

The FAQ KB 110 stores data used to answer user queries in the form of triple data. Triple data is in the form of (question diagram, question pattern, conditional answer 'answer condition|corresponding answer').

The question intention is an index for a group of similar question patterns, and serves as an identifier (ID) for the user's question intention, and is used for efficient answer search. If it can be used as an index for a question pattern group, the question intent can be expressed either as a character such as "[phone_belief_description]" or a simple number.

The question pattern is a collection of question sentences actually used by the user or question sentences predicted to be used from conversation records. A question pattern belongs to a question pattern group classified by question intention, and the corresponding question intention is indexed and connected to each other. For example, the natural language sentence "Can you explain about the GG note?" and the keyword combination "HH phone product description" belong to the same group and are linked to the related question intent [phone_belief_description].

The user query and the question pattern in the FAQ KB 110 are compared in context with each other by the PI model 114 . Therefore, the original question "I want to know about GG notes briefly" is changed to the question pattern "I want to know about $phone$" and saved. For example, a plurality of questions and question intentions for various phone product descriptions are integrated into one question intention and question pattern group, and information on specific items such as "GG note" or "HH phone" is After replacing it with the pattern as it is, the handling is delayed by resolving it in the conditional answer.

A conditional answer is a plurality of answering conditions and corresponding answers linked to the intention of a question. For example, to the intent of the question [phone_belief_description], "Answer HH phone is the latest … 'phone_name|HH phone'" or "GG note is the most popular … 'phone_name|GG note'"" Text and 'Answer Conditions | Corresponding Answer'" are linked. Here, the answer condition may be a slot-type that can be delivered from the trust state tracker in the dialog system. When the slot type does not exist, an answer condition may be obtained from an external system using an application specific interface (API) call, or an answer condition may be transmitted from a user using a follow-up question. In case the follow-up question is used, the conditional answer may include a plurality of follow-up questions related to the intent of the question.

The FAQ KB 110 may be generated by using some of the questions in the domain-specific database, such as the conversation record of the communication service system. When generating the FAQ KB 110 , the process of generating a question intention and a question pattern may be performed automatically or manually using a known method.

The reason for using the question pattern instead of the original question and introducing the conditional answer is that it is not easy for the PI model 114 to have enough performance to identify the word corresponding to the answer, and even with the performance, it is difficult to implement complexity. This is because it introduces several problems, such as the required cost, the time required for training, and the difficulty of implementing real-time operations. In addition, the complexity of the PI model is also a constraint in terms of scalability of the FAQ answerer 100 .

Due to the configuration of the FAQ KB 110 according to the present embodiment, it is possible to shorten the time for searching for answers to a user's query and increase the search accuracy. In addition, by introducing components that process each of the question intention, question pattern, and conditional answer constituting the FAQ KB 110 into the FAQ answerer 100, the FAQ answerer 100 can ask as many questions as possible from users at a given time. It has scalability to accommodate

Next, as shown in FIG. 2 , the QIR 111 according to the present embodiment selects a plurality of question intentions related to a user query from the FAQ KB 110 . The screening method is similar to how a search engine searches for documents. When searching, the question intent becomes the document identifier, and the linked question pattern and conditional answer are successively attached to become the text of the document. Using the words constituting the user query as keywords, the QIR 111 searches the FAQ knowledge base. A plurality of question pattern groups are determined based on an order in which many words related to a keyword are included, and a plurality of question intentions for indexing each question pattern group are selected. As shown in FIG. 2 , the output of the QIR 111 according to the present embodiment is a plurality of (represented by Top-X) question intentions and a question pattern group connected to each question intention. Depending on the search capability of the QIR 111 and the size of the FAQ KB 110 , the number of selected question intentions may be increased or decreased.

The QIR 111 filters the vast question patterns belonging to the vast FAQ KB 110 and reduces them to be limited to question patterns connected with a plurality of question intentions closely related to the user's query. The QIR 111 aims to narrow the search range to increase the answer probability, and because of the narrowed search range, it is possible to reduce the computing power required in the reasoning process of the PI model 114 .

Next, as shown in FIG. 2 , the QPR 112 according to this embodiment uses a plurality of question intentions and question pattern groups output by the QIR 111 first to convert a user query and each question pattern into a PI model. Input to (114) to identify whether a duplicate question is present. The QPR 112 uses BERT (Bidirectional Encoder Representations from Transformers), which is one of the neural networks showing performance in natural language processing to identify similar questions. In order to apply to NLU (Natural Language Understanding) such as the operation of the FAQ answerer 100, the PI model is prepared by training the BERT in advance to solve the PI problem (paraphrase identification problem) that identifies whether a similar question exists.

에 속한 질문패턴

에 대하여 QIS(115)가 탑원(Top-1) 질문의도를 선정하는 방법은 수학식 1로 표현한다.Next, an inference method of the QIS 115 based on the output of the PI model 114 will be described. Let the user query be q, and the question pattern group combined with question intent I

question pattern belonging to

A method for the QIS 115 to select a Top-1 question intention is expressed by Equation 1.

는 질문의도 I에 결합된 질문패턴 그룹의 크기를 나타내고,

는 사용자 질의 q와 질문패턴

를 입력받은 PI 모델(114)의 분류 결과로서 유사 질문쌍으로 분류된 경우에는 1, 반대의 경우에는 0이다. here

represents the size of the question pattern group combined in question intention I,

is the user query q and the question pattern

As a result of the classification of the PI model 114 that has received , it is 1 when classified as a similar question pair, and 0 in the opposite case.

In conclusion, using the identification result of the PI model 114, the QPR 112 sets the question intention indexing the question pattern group including the highest rate of similar questions among the plurality of question intentions as the top-one question intention. It is selected and delivered to the CA 113 .

Next, as shown in FIG. 2 , the CA 113 according to the present embodiment selects the correct answer from the list of answers related to the top-one question intention by using the top-one question intention and the connected answer condition selected by the QPR 112 . to provide it to users. As described in the description of the FAQ KB 110, the answer condition may be received from the trust state tracker in the conversation system, obtained from an external system using an API call, or received from the user using a follow-up question.

As described above, the FAQ answerer 100 according to the present embodiment includes a neural network-based learning model, and may perform a training process on the PI model 114 using the provided learning model. This training model is a pre-trained model that makes it possible to handle PI problems based on a labeled dataset and a weakly labeled domain-specific database that can distinguish between similar question pairs. can

3 is a conceptual diagram illustrating a training process of a learning model according to an embodiment of the present invention.

Hereinafter, a training process of the learning model 120 will be described with reference to FIG. 3 . The learning model 120 is composed of a PI model 114 and a refiner 121 in the training process. Both the PI model 114 and the refiner 121 according to the embodiment of the present invention are implemented with BERT, but are not limited thereto, and by any neural network applicable to natural language processing. can also be implemented. Representative examples include a Recurrent Neural Network (RNN) model, and RNN-based neural networks include a Long Short-Term Memory model (LSTM), a Gated Recurrent Unit (GRU), and a Transformer Decoder.

First, the training device (not shown), as shown in FIG. 3, applies unsupervised learning to a large-capacity dialogue corpus that has no correct answer label, PI model 114 and The refiner 121 is pre-trained. As a large-capacity corpus, Wikipedia and Bookscorpus can be used, and hundreds of millions to billions of words extracted from the large-capacity corpus are used for dictionary training.

Next, the training apparatus according to the embodiment of the present invention, as shown in FIG. 3, a dataset including a labeled question pair for discriminating whether the question pair is similar. The PI model 114 and the refiner 121 are task fine-tuned using the PI model 114 . Here, the meaning of the task is the identification of the similarity of the question pair given as input (paraphrase identification). Quora Question Pairs (QQP) is the most widely known dataset and contains about 404K question pairs, and each question pair has a label indicating whether it is similar to each other.

Hereinafter, task fine-tuning for the PI model 114 and the refiner 121 will be described. First, the question pair included in the dataset is denoted by x _i , and the corresponding binary label is denoted _{by y i .} Meanwhile, the question pair included in the dataset may be translated into a target language (eg, Korean) by a natural language translator (NMT) and used.

When the total number of question pairs included in the dataset is N, the parameters of the PI model 114 and the refiner 121 are calculated based on the loss function of the cross-entropy form shown in Equation 2 update

,

)을 모두 출력한다. 따라서 클래스 넘버 C는 2이고, 이진 라벨

및

는 질문쌍의 특성을 반영하여 한쪽은 1, 다른 쪽은 0의 값을 갖는다. 정제기(121)의 최종단은 시그모이드 함수(sigmoid function)로 구현되어 입력된 질문쌍에 부착된 라벨에 대한 확률

을 출력한다. 따라서 클래스 넘버 C는 1이고. 이진 라벨

는 질문쌍의 특성을 반영하여 1 또는 0의 값을 갖는다. Here, C is a class number and depends on the final output of the BERT implementing each of the PI model 114 and the refiner 121 . The final stage of the PI model 114 is implemented as a softmax function, and the probability (

,

) are printed out. So class number C is 2, binary label

and

has a value of 1 on one side and 0 on the other side reflecting the characteristics of the question pair. The final stage of the refiner 121 is implemented as a sigmoid function, so the probability for the label attached to the input question pair

to output So class number C is 1. binary label

has a value of 1 or 0 reflecting the characteristics of the question pair.

In the loss function expressed in Equation 2, cross entropy is used as a metric, but the present invention is not limited thereto, and any metric usable for supervised learning can be used for the loss function.

Next, as shown in FIG. 3, the training apparatus according to an embodiment of the present invention fine-tunes the PI model 114 and the refiner 121 using a domain-specific database having a weakly labeled domain. Domain Fine-tuning). Domain refinement includes iterative training for the PI model 114 and a refinement process for the refiner 121 .

If the domain-specific question pair is applied to the PI model 114 and the refiner 121 after fine-tuning the task using QQP, the accuracy is very low. For example, the question "How can I start the roaming service?" and the question "Do I need to reboot the phone to roam?" do not match in the general sense of the context. However, in domains such as telecommunication services, similar questions can be considered, and the same answer is provided: "Reboot your mobile phone to start roaming service." Accordingly, the PI model 114 and the refiner 121 according to the embodiment of the present invention are domain fine-tuned in order to increase the identification accuracy for the domain-specific query pair.

Hereinafter, a process of generating a question pair from a domain-specific conversation record such as a conversation record of a communication service system will be described. First, a question cluster is generated by applying automatic duplicate question clustering to a domain-specific conversation record. By generating all combinable question pairs in the same question group, a positive question pair is created and a positive label is attached. A positive label may not always be true for a number of reasons, including clustering error, so a positive label can be either a pseudo label or a weak label, and the corresponding question pair is a pseudo positive label. It may be a pseudo-positive question pair. The data representing the positive question pair is in the form of {question 1, question 2, pseudo positive label}. Hereinafter, for convenience, the positive label and the pseudo-positive label are used interchangeably, and the positive question pair and the pseudo positive question pair are also used interchangeably.

A negative question pair expressed as data in the form of {question 1, question 3, negative label} is generated so as to correspond to the data representing the next positive question pair. Here, the corresponding positive and negative question pairs include the same question 1, and question 3 belongs to a different question group. Since two questions belonging to different question groups were created in pairs, there is no error in the negative label.

The training apparatus according to the present invention fine-tunes the PI model 114 using positive and negative question pairs. However, because of the presence of false positive question pairs, the PI model 114 is trained with a weakly supervised method using pseudo-labels. In order to cope with the error contained in the pseudo-label, it is necessary to assign a confidence score in the form of a weight to the positive question pairs used in the training of the PI model. In order to estimate the confidence score, a refiner 121 is added in the embodiment of the present invention.

4 is a block diagram of a learning model including a PI model and a refiner according to an embodiment of the present invention.

Hereinafter, with reference to FIG. 4 , the refinement process of the refiner 121 and iterative training for the PI model 114 will be described. In the iterative training process according to the embodiment of the present invention, the PI model 114 and the refiner 121 use each other's training results. The confidence score of the positive question pair calculated by the refiner 121 is used as a weight in the damage function of the PI model 114 , and the classification result of the PI model 114 is also used in the damage function of the refiner 121 . Also, in the iterative training process, we try to change the label of the positive question pair based on the confidence score of the positive question pair.

를 추정하고, 다른 한번은 부정 질문쌍을 입력하여 부정라벨에 대한 신뢰 점수

를 추정한다. 의사 긍정라벨 및 부정라벨에 대한 신뢰 점수 및 긍정라벨에 대한 PI 모델(114)의 분류 결과가 포함된, 수학식 3에 표현한 손실함수에 기반하여 정제기(121)의 파라미터를 업데이트한다. First, in order to train the refiner 121, a pair of positive questions is input into the refiner 121 once, and the confidence score for the pseudo positive label is

Estimating , and inputting a negative question pair at the other time, the confidence score for the negative label

to estimate The parameters of the refiner 121 are updated based on the loss function expressed in Equation 3, including the confidence scores for the pseudo positive label and the negative label, and the classification result of the PI model 114 for the positive label.

는 긍정 질문쌍 (p_i,q_i)에 대한 PI 모델(114)의 분류 결과로서 유사 질문쌍으로 분류된 경우에는 1, 반대의 경우에는 0이다. 그리고,

는 트레이닝 과정에서 조정되는 하이퍼 파라미터(hyper-parameter)이며, i는 긍정 질문쌍(또는 같은 수의 부정 질문쌍)을 인덱싱한다. 앞에서 설명한 바와 같이 정제기의 최종단은 시그모이드 함수로 구현되므로, 신뢰 점수의 범위는 0에서 1까지 이다. 결과적으로, 본 실시예에서는 긍정 질문쌍에 부착된 의사 긍정라벨이 얼마나 참인지 확신할 수 있는 신뢰 점수를 0과 1사이의 범위 내에서 출력하도록 정제기(121)가 트레이닝된다.here

As the classification result of the PI model 114 for the positive question pair (p _i , q _i ), it is 1 when it is classified as a similar question pair, and 0 in the opposite case. And,

is a hyper-parameter adjusted during the training process, and i indexes positive question pairs (or the same number of negative question pairs). As described above, since the final stage of the refiner is implemented as a sigmoid function, the confidence score ranges from 0 to 1. As a result, in the present embodiment, the refiner 121 is trained to output a confidence score within the range between 0 and 1 with which it can be certain that the pseudo positive label attached to the positive question pair is true.

Next, in order to train the PI model 114, the positive question pair and the negative question pair are alternately input to the PI model 114 to estimate the similarity. By combining the confidence score of the positive question pair provided by the refiner 121 and Equation 2, the parameters of the PI model 114 are updated based on the loss function expressed in Equation 4.

는 수학식 5로 표현된다. function here

is expressed by Equation (5).

는 트레이닝 과정에서 조정되는 하이퍼 파라미터이고, t는 이터레이션 스텝(iteration step)을 나타낸다. 트레이닝 초기에는 정제기(121)가 출력하는 신뢰 점수가 믿을만하지 않으므로, 트레이닝 시 적게 반영되어야 한다. 수학식 5에 나타낸 대로, t가 작은 값이면, 함수

는 지수항에 의존하여 1에 가까운 값이 되므로 신뢰 점수의 영향을 줄일 수 있다. 트레이닝이 진행되어 t가 증가하면, 지수항이 0에 수렴하므로, 함수

는 신뢰 점수에 의존하게 된다.where c and

is a hyperparameter adjusted during the training process, and t represents an iteration step. Since the confidence score output by the refiner 121 is not reliable at the beginning of training, it should be reflected less during training. As shown in Equation 5, if t is a small value, the function

is close to 1 depending on the exponent term, so the influence of the confidence score can be reduced. As the training progresses and t increases, the exponential term converges to 0, so the function

depends on the confidence score.

는 1 이하의 값이므로, 수학식에 4에서

는 긍정라벨에 의한 손실의 영향을 줄이는 가중치 역할을 한다.According to Equation (5), only the confidence score of the positive label is reflected in the loss function of Equation (4). And,

is a value less than or equal to 1, so in Equation 4

serves as a weight to reduce the impact of losses caused by positive labels.

On the other hand, if the confidence score inferred by the refiner 121 with respect to the positive label is less than a preset threshold, the corresponding question pair is converted into a negative question pair, and a negative label is attached. The negative question pair generated by the label change acts as stronger data in the training process than the original negative question pair.

By introducing the label change procedure for the confidence score and the weak label into the training process as described above, the FAQ answerer 100 according to the present embodiment has robustness that can efficiently cope with errors derived from the training process. have

5 is a flowchart illustrating a procedure for answering a user's query by a FAQ answerer according to an embodiment of the present invention.

First, the FAQ answerer 100 obtains a user query (S501), and the QIR 111 searches the FAQ KB 110 to obtain a plurality of question intentions related to the user query and a plurality of questions connected to the plurality of question intentions. A group of question patterns is selected (S502). Here, as described above, the FAQ KB 110 stores answer data to a user's query in the form of data consisting of a question intention, a question pattern, and a conditional answer.

Next, the QPR 112 inputs a user query and a plurality of question pattern groups into the pre-trained PI model 114 to identify whether a similar question exists, and uses the identification result to determine the highest ratio among the plurality of question intentions. A question intention indexing a question pattern group including a similar question of ' is selected as a top-one question intention (S503).

Next, using the answer condition, the CA 113 selects the correct answer from the list of answers related to the top-one question intention (S504). Here, the CA 113 selects an answer related to the top-up question intention by using the answer condition and the answer list constituting the conditional answer on the knowledge base.

Next, the FAQ answering machine 100 provides the selected answer to the user (S505).

Hereinafter, a result of evaluating the performance of the FAQ answering machine 100 according to the present embodiment will be described. The database used for the evaluation is a database that stores conversation records with users at a specific telecommunication company, and consists of millions of question-and-answer pairs. As described above, the learning model 120 was trained using a part of the database of a specific telecommunication company, and the FAQ KB 110 was generated. The accuracy of the answer provided by the FAQ answering machine 100 was measured using the generated FAQ KB 110 . The measured accuracy was 84%, which showed better performance compared to the 70% shown by the conventional IR (Information Retrieval) model (BM25, see Non-Patent Document 2).

As described above, according to this embodiment, based on the FAQ KB, the process of searching for the intention of a user query, estimating a question pattern, and a conditional answering process is applied to the conversation system, and a weakly labeled question pair is applied. By providing a dialog system and method used in a task-oriented dialog system to which a training process using

Although it is described that each process is sequentially executed in each flowchart according to the present embodiment, the present invention is not limited thereto. In other words, since it may be applicable to change and execute the processes described in the flowchart or to execute one or more processes in parallel, the flowchart is not limited to a time-series order.

Various implementations of the systems and techniques described herein include digital electronic circuitry, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combination can be realized. These various implementations may include being implemented in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. or may be a general-purpose processor). Computer programs (also known as programs, software, software applications or code) contain instructions for a programmable processor and are stored on a “computer-readable medium”.

A computer-readable medium includes any computer program product, apparatus, and/or device (eg, a CD-ROM, ROM, memory card, a non-volatile or non-transitory recording medium such as a hard disk, a magneto-optical disk, and a storage device).

Various implementations of the systems and techniques described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or other types of storage systems or combinations thereof), and at least one communication interface. For example, a programmable computer may be one of a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a Personal Data Assistant (PDA), a cloud computing system, or a mobile device.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are for explanation rather than limiting the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

100: FAQ Answerer 110: FAQ Knowledge Base
111: question intention searcher 112: question pattern estimator
113: conditional answerer 114: PI model
120: learning model 121: refiner

Claims (22)

Searching a knowledge base that stores answer data to a user's query in the form of data consisting of a question intention, a question pattern, and a conditional answer, selecting a plurality of question pattern groups related to the user query, and selecting the plurality of question pattern groups and a Question Intent Retriever (QIR) providing a plurality of question intentions connected to the plurality of question pattern groups;
a Question Pattern Reader (QPR) for selecting a Top-1 question intention using the user query, the plurality of question intentions, and the plurality of question pattern groups; and
Conditional Answerer (CA) that selects an answer related to the top-one question intention by using the conditional answer on the knowledge base
including,
The question pattern estimator,
a paraphrase identification (PI) model for generating an identification result as to whether a similar question exists between the user query and the question pattern; and
A question intention selector that selects a question intention indexing a question pattern group including a similar question of the highest rate among the plurality of question intentions as the top-one question intention by using the identification result (Question Intent Selector, QIS)
Conversation system comprising a. According to claim 1,
The question intent functions as an identifier for the question pattern, the question pattern is a question in which an item related to the user query is replaced with a pattern, and the conditional answer is a plurality of answers connected to the question intent A dialogue system, characterized in that it consists of a condition and a corresponding answer. According to claim 1,
The question pattern group is
The dialogue system according to claim 1, wherein the question intention is a group of question patterns having the same question intention, and the question intention indexes the question pattern group. According to claim 1,
The inquiry intent searcher,
Using a word constituting the user query as a keyword, searching the knowledge base to select the plurality of question pattern groups based on relevance to the keyword, the plurality of question pattern groups and the plurality of question pattern groups and providing the plurality of question intentions for indexing to the question pattern estimator. delete The method of claim 1
The paraphrase identification model is,
A dialogue system characterized in that it has been pre-trained using a database containing domain-specific, weakly labeled question pairs. delete 3. The method of claim 2,
The conditional answerer,
The answer condition is obtained using an estimated belief state for the user query, obtained from an external system using an API (Application Specific Interface) call, or delivered from the user using a follow-up question. Conversation system, characterized in that. delete delete delete delete delete delete delete delete delete delete delete In the dialogue method of the dialogue system,
Searching a knowledge base that stores answer data to a user's query in the form of data consisting of question intent, question pattern, and conditional answer to select a plurality of question pattern groups related to the user query and providing a plurality of question intentions connected to the plurality of question pattern groups.
By inputting the user query and the plurality of question pattern groups into a pre-trained Paraphrase Identification (PI) model, an identification result for whether or not a similar question exists is calculated, and using the identification result, the A process of selecting a question intention indexing a question pattern group including the highest rate of similar questions among the diagrams as a Top-1 question intention; and
The process of selecting an answer related to the top-one question intention by using the conditional answer on the knowledge base
Conversation method implemented on a computer, characterized in that it comprises a. delete A computer program stored in a computer-readable, non-volatile or non-transitory recording medium to execute each step included in the conversation method of the conversation system according to claim 20.

Images