KR102401333B1 - System and Method for Robust and Scalable Dialogue - Google Patents

Abstract

A dialog system and method used in a task-oriented dialog system are disclosed.
In this embodiment, the scalability of the dialogue system is increased by applying the question intention search, question pattern estimation, and conditional answer process to the user query based on the FAQ KB (Knowledge Base), and weakly labeled ) The purpose of this is to provide a dialog system and method with scalability and robustness used in task-oriented dialog systems by increasing the robustness of the learning model by applying the training process using question pairs.

Description

System and Method for Robust and Scalable Dialogue

The present invention relates to a dialog system and method having scalability and robustness used in a task-oriented dialog system.

The content described below merely provides background information related to the present invention and does not constitute the prior art.

There is an interest in a task-oriented or target domain-specific dialogue system that interacts with a user so that the user can achieve a goal, such as a restaurant reservation or a flight reservation. is increasing One of the main goals of the task-oriented dialog system is to provide an answer suitable for the user's query intent.

One way to provide an appropriate answer is to graft Frequently Asked Question (FAQ) answers on a task-oriented dialogue system. A direct method for grafting is to map a user's new request that occurs on the dialog system with an existing question accompanied by a corresponding answer. This type of problem is most closely related to Paraphrase Identification (PI), and the PI model handles the PI problem of identifying whether two sentences are contextually identical.

Recently, by using BERT (Bidirectional Encoder Representations from Transformers, see Non-Patent Document 1), many advances have been made in terms of increasing the accuracy of solving the PI problem.

However, in order to provide an appropriate answer that meets the user's intention in the task-oriented dialogue system, two issues of scalability and robustness must be resolved. Here, scalability means that the dialog system accommodates as many questions as possible from users, and robustness means the property of efficiently coping with errors derived from the learning process of the PI model. Therefore, there is a need for a dialogue system and method with scalability and robustness applicable to a task-oriented dialogue system based on an appropriate solution to the two issues.

Non-Patent Document 1: Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. Non-Patent Document 2: Robertson, S.; Zaragoza, H.; et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval.

The present disclosure increases the scalability of a conversational system by applying a question intention search, question pattern estimation, and conditional answer process to a user query based on the FAQ KB (Knowledge Base), and weakly labeled The main purpose is to provide a dialog system and method having scalability and robustness used in a task-oriented dialog system by increasing the robustness of the learning model by applying a training process using question pairs.

According to an embodiment of the present invention, a plurality of question pattern groups related to the user's query are selected by searching the knowledge base that stores the answer data to the user's query in the form of data consisting of the question intention, the question pattern, and the conditional answer. and a Question Intent Retriever (QIR) providing the plurality of question pattern groups and a plurality of question intentions connected to the plurality of question pattern groups; a question pattern estimator (Question Pattern Reader: QPR) for selecting a Top-1 question intention using the user query, the plurality of question intentions, and the plurality of question pattern groups; and a Conditional Answerer (CA) that selects an answer related to the top-one question intention by using the conditional answer on the knowledge base.

According to another embodiment of the present invention, in a learning method performed by a computing device, Paraphrase Identification using a dataset including a labeled question pair PI) a process of task fine-tuning a model and a refiner, wherein the task indicates identification of whether an input question pair is similar; and domain fine tuning the PI model and the refiner using a domain-specific database, comprising a positive question pair with a weak positive label, and a negative query pair with a negative label. -tuning), wherein the process of fine-tuning the domain provides a learning method, characterized in that it includes an iterative training process for the PI model, and a refinement process for the refiner .

According to another embodiment of the present invention, there is provided a learning apparatus used by a conversation system, comprising: a Paraphrase Identification (PI) model for generating an identification result for whether an input question pair is similar; and a refiner for estimating a confidence score corresponding to the input question pair, and using a dataset including a labeled question pair for discriminating similarity. After task fine-tuning the PI model and the refiner, a domain comprising a positive question pair with a weak positive label, and a negative question pair with a negative label Domain fine-tuning the PI model and the refiner using a specialized database, but the domain fine-tuning, iterative training for the PI model using the confidence score , by using the identification result to provide a learning apparatus characterized in that the refiner is refined.

According to another embodiment of the present invention, in a conversation method of a conversation system, a knowledge base storing answer data to a user's query in the form of data consisting of a question intention, a question pattern, and a conditional answer is searched and related to the user's query selecting a plurality of question pattern groups having By inputting the user query and the plurality of question pattern groups into a pre-trained Paraphrase Identification (PI) model, an identification result of whether or not a similar question is asked is calculated, and using the identification result, the A process of selecting a question intention indexing a question pattern group including the highest rate of similar questions among the diagrams as a Top-1 question intention; and selecting an answer related to the top-one question intention by using the conditional answer on the knowledge base.

According to another embodiment of the present invention, there is provided a computer program stored in a computer-readable recording medium to execute each step included in a learning method performed by a computing device.

According to another embodiment of the present invention, there is provided a computer program stored in a computer-readable recording medium to execute each step included in the conversation method of the conversation system.

As described above, according to the present embodiment, based on the FAQ KB (Knowledge Base), the process of searching for a question intention for a user's query, estimating a question pattern, and a conditional answer process is applied to the dialog system, and a weak label is attached (weakly). By providing a dialog system and method used in a task-oriented dialog system to which a training process using labeled) question pairs is applied, it is effective to increase the scalability and robustness of the dialog system.

1 is a block diagram of a FAQ answering machine and a learning model according to an embodiment of the present invention.
2 is a conceptual diagram for explaining the operation of the components of the FAQ answering machine according to an embodiment of the present invention.
3 is a conceptual diagram illustrating a training process of a learning model according to an embodiment of the present invention.
4 is a block diagram of a learning model including a PI model and a refiner according to an embodiment of the present invention.
5 is a flowchart illustrating a procedure for answering a user's query by a FAQ answerer according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in the description of the present embodiments, if it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present embodiments, the detailed description thereof will be omitted.

Also, in describing the components of the present embodiments, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced.

A task-oriented dialog system includes a belief state tracker (also called a dialog state tracker) and an answerer to a user query. can be configured. The present invention particularly relates to the implementation of an answerer in a chat system, and pays attention to a dialog system including a Frequently Asked Question (FAQ) answerer for responding to a user's query.

1 is a block diagram of a FAQ answering machine and a learning model according to an embodiment of the present invention.

1 shows a FAQ answerer 100 used in a task-oriented conversation system, a FAQ knowledge base 110 (Knowledge Base: KB), a question intent retriever (111, Question Intent Retriever, QIR), and a question pattern estimator ( 112, Question Pattern Reader, QPR) and a Conditional Answerer (113, Conditional Answerer: CA). In addition, the QPR 112 includes a Paraphrase Identification (PI) model 114 and a Question Intent Selector 115 .

Components included in the FAQ answering machine 100 according to the present embodiment are not necessarily limited thereto. For example, a learning model and a training device for training the learning model may be additionally provided on the FAQ answering machine, or may be implemented in a form that interworks with an external training device. For example, the learning model 120 shown in FIG. 1 is learned by a training device. The learning model 120 includes a PI model 114 and a refiner 121 . The PI model 114 is provided on the FAQ answerer 100 , but the refiner 121 is the FAQ answerer 100 . exists outside of Accordingly, the refiner 121 is used only for learning, and is not applied to the operation of the FAQ answerer 100 .

2 is a conceptual diagram for explaining the operation of the components of the FAQ answering machine according to an embodiment of the present invention.

Hereinafter, the form and operation of the components included in the FAQ answering machine 100 will be described with reference to FIGS. 1 and 2 .

The FAQ KB 110 stores data used to answer user queries in the form of triple data. Triple data is in the form of (question diagram, question pattern, conditional answer 'answer condition|corresponding answer').

The question intention is an index to a group of similar question patterns, and serves as an identifier (ID) for the user's question intention, and is used for efficient answer search. If it can be used as an index for a question pattern group, the question intent can be expressed either as a character such as "[phone_belief_description]" or a simple number.

The question pattern is a collection of question sentences actually used by the user or question sentences predicted to be used from conversation records. A question pattern belongs to a question pattern group classified by question intention, and the corresponding question intention becomes an index and is connected to each other. For example, the natural language sentence "Can you explain the GG note?" and the keyword combination "HH phone product description" belong to the same group and are linked to [phone_belief_description], which is the intent of the related question.

The user query and the question pattern in the FAQ KB 110 are compared in context with each other by the PI model 114 . Therefore, the original question "I want to know about GG notes" is changed to the question pattern "I want to know about $phone$ simply" and saved. For example, a plurality of questions and question intentions for various phone product descriptions are integrated into one question intention and question pattern group, and information on specific items such as "GG note" or "HH phone" is After replacing the pattern with the correct one, the handling is delayed by resolving it in the conditional answer.

A conditional answer is a plurality of answering conditions and corresponding answers linked to the intention of a question. For example, to the intent of the question [phone_belief_description], "Answer HH phone is the latest … 'phone_name|HH phone'" or "GG note is the most popular … 'phone_name|GG note'"" Text and 'Answer Conditions | Corresponding Answer'" are linked. Here, the answer condition may be a slot-type that can be delivered from the trust state tracker in the dialog system. When the slot type does not exist, an answer condition may be obtained from an external system using an application specific interface (API) call, or an answer condition may be transmitted from a user using a follow-up question. In case the follow-up question is used, the conditional answer may include a plurality of follow-up questions related to the intent of the question.

The FAQ KB 110 may be generated using some of the questions in the domain-specific database, such as the conversation record of the communication service system. When generating the FAQ KB 110 , the process of generating a question intention and a question pattern may be performed automatically or manually using a known method.

The reason for using the question pattern instead of the original question and introducing the conditional answer is that it is not easy for the PI model 114 to have enough performance to identify the word corresponding to the answer, and even if it has the performance, it is difficult to implement complexity. This is because it introduces several problems, such as required cost, time required for training, and difficulty in implementing real-time operation. In addition, the complexity of the PI model is also a constraint in terms of scalability of the FAQ answerer 100 .

Due to the configuration of the FAQ KB 110 according to the present embodiment, it is possible to shorten the time for searching for answers to a user's query and increase the search accuracy. In addition, by introducing components that process each of the question intention, question pattern, and conditional answer constituting the FAQ KB 110 in the FAQ answerer 100, the FAQ answerer 100 can ask as many questions as possible from users at a given time. It has scalability to accommodate

Next, as shown in FIG. 2 , the QIR 111 according to the present embodiment selects a plurality of question intentions related to a user query from the FAQ KB 110 . The screening method is similar to how a search engine searches for documents. When searching, the question intent becomes the document identifier, and the linked question pattern and conditional answer are successively attached to become the text of the document. Using the words constituting the user query as keywords, the QIR 111 searches the FAQ knowledge base. A plurality of question pattern groups are determined based on an order in which many words related to a keyword are included, and a plurality of question intentions for indexing each question pattern group are selected. As shown in FIG. 2 , the output of the QIR 111 according to the present embodiment is a plurality of (represented by Top-X) question intentions and a question pattern group connected to each question intention. Depending on the search capability of the QIR 111 and the size of the FAQ KB 110 , the number of selected question intentions may be increased or decreased.

The QIR 111 filters the vast question patterns belonging to the vast FAQ KB 110 and reduces it to be limited to the question patterns connected with a plurality of question intentions closely related to the user's query. The QIR 111 aims to narrow the search range to increase the answer probability, and because of the narrowed search range, it is possible to reduce the computing power required in the reasoning process of the PI model 114 .

Next, as shown in FIG. 2 , the QPR 112 according to this embodiment uses a plurality of question intentions and question pattern groups output by the QIR 111 first to convert a user query and each question pattern into a PI model. Input to (114) to identify whether a duplicate question is present. The QPR 112 uses BERT (Bidirectional Encoder Representations from Transformers), which is one of the neural networks showing performance in natural language processing to identify similar questions. In order to apply to NLU (Natural Language Understanding) such as the operation of the FAQ answerer 100, the BERT is trained in advance to solve the PI problem (paraphrase identification problem) that identifies whether a similar question exists to prepare a PI model.

에 속한 질문패턴

에 대하여 QIS(115)가 탑원(Top-1) 질문의도를 선정하는 방법은 수학식 1로 표현한다.Next, an inference method of the QIS 115 based on the output of the PI model 114 will be described. Let the user query be q, and the question pattern group combined with question intent I

question pattern belonging to

A method for the QIS 115 to select a Top-1 question intention is expressed by Equation 1.

는 질문의도 I에 결합된 질문패턴 그룹의 크기를 나타내고,

는 사용자 질의 q와 질문패턴

를 입력받은 PI 모델(114)의 분류 결과로서 유사 질문쌍으로 분류된 경우에는 1, 반대의 경우에는 0이다. here

represents the size of the question pattern group combined in question intention I,

is the user query q and the question pattern

As a result of the classification of the PI model 114 receiving , it is 1 when classified as a similar question pair, and 0 in the opposite case.

In conclusion, using the identification result of the PI model 114, the QPR 112 sets the question intention indexing the question pattern group including the highest rate of similar questions among the plurality of question intentions as the top-one question intention. It is selected and delivered to CA (113).

Next, as shown in FIG. 2 , the CA 113 according to the present embodiment selects the correct answer from the list of answers related to the top-one question intention by using the top-one question intention and the connected answer condition selected by the QPR 112 . to provide it to users. As described in the description of the FAQ KB 110, the answer condition may be received from the trust state tracker in the conversation system, obtained from an external system using an API call, or received from the user using a follow-up question.

As described above, the FAQ answerer 100 according to the present embodiment has a neural network-based learning model, and may perform a training process for the PI model 114 using the provided learning model. This training model is a pre-trained model that makes it possible to handle PI problems based on a labeled dataset and a weakly labeled domain-specific database that can distinguish between similar question pairs. can

3 is a conceptual diagram illustrating a training process of a learning model according to an embodiment of the present invention.

Hereinafter, a training process of the learning model 120 will be described with reference to FIG. 3 . The learning model 120 is composed of a PI model 114 and a refiner 121 in the training process. Both the PI model 114 and the refiner 121 according to the embodiment of the present invention are implemented with BERT, but are not limited thereto, and by any neural network applicable to natural language processing. can also be implemented. Representative examples include a Recurrent Neural Network (RNN) model, and RNN-based neural networks include a Long Short-Term Memory model (LSTM), a Gated Recurrent Unit (GRU), and a Transformer Decoder.

First, the training device (not shown), as shown in FIG. 3, applies unsupervised learning to a large-capacity dialogue corpus without an answer label to PI model 114 and The refiner 121 is pre-trained. Wikipedia and Bookscorpus can be used as the large-capacity corpus, and hundreds of millions to billions of words extracted from the large-capacity corpus are used for dictionary training.

Next, the training apparatus according to the embodiment of the present invention, as shown in FIG. 3, a dataset including a labeled question pair for discriminating whether the question pair is similar. The PI model 114 and the refiner 121 are task fine-tuned using the PI model 114 . Here, the meaning of the task is the identification of the similarity of the question pair given as input (paraphrase identification). Quora Question Pairs (QQP) is the most widely known dataset and includes about 404K question pairs, and each question pair has a label indicating whether they are similar to each other.

Hereinafter, task fine-tuning for the PI model 114 and the refiner 121 will be described. First, the question pair included in the dataset is denoted by x _i , and the corresponding binary label is denoted by y _i . Meanwhile, the question pair included in the dataset may be translated into a target language (eg, Korean) by a natural language translator (NMT) and used.

When the total number of question pairs included in the dataset is N, the parameters of the PI model 114 and the refiner 121 are calculated based on the loss function of the cross-entropy form shown in Equation 2 update

,

)을 모두 출력한다. 따라서 클래스 넘버 C는 2이고, 이진 라벨

및

는 질문쌍의 특성을 반영하여 한쪽은 1, 다른 쪽은 0의 값을 갖는다. 정제기(121)의 최종단은 시그모이드 함수(sigmoid function)로 구현되어 입력된 질문쌍에 부착된 라벨에 대한 확률

을 출력한다. 따라서 클래스 넘버 C는 1이고. 이진 라벨

는 질문쌍의 특성을 반영하여 1 또는 0의 값을 갖는다. Here, C is a class number and depends on the final output of the BERT implementing each of the PI model 114 and the refiner 121 . The final stage of the PI model 114 is implemented as a softmax function, and the probability (

,

) are printed out. So class number C is 2, binary label

and

has a value of 1 on one side and 0 on the other side reflecting the characteristics of the question pair. The final stage of the refiner 121 is implemented as a sigmoid function, so the probability for the label attached to the input question pair

to output So class number C is 1. binary label

has a value of 1 or 0 reflecting the characteristics of the question pair.

In the loss function expressed in Equation 2, cross entropy is used as a metric, but the present invention is not limited thereto, and any metric usable for supervised learning can be used for the loss function.

Next, as shown in FIG. 3, the training apparatus according to the embodiment of the present invention fine-tunes the PI model 114 and the refiner 121 using a domain-specific database having a weakly labeled domain ( Domain Fine-tuning). The domain refinement includes iterative training for the PI model 114 and a refinement process for the refiner 121 .

If a domain-specific question pair is applied to the PI model 114 and the refiner 121 after fine-tuning the task using QQP, the accuracy is very low. For example, the question "How can I start the roaming service?" and the question "Do I need to reboot my phone to roam?" do not match in the general sense. However, in domains such as telecommunication services, similar questions can be considered, and the same answer is provided: "Reboot your mobile phone to start roaming service." Accordingly, the PI model 114 and the refiner 121 according to the embodiment of the present invention are domain fine-tuned in order to increase the identification accuracy for the domain-specific query pair.

Hereinafter, a process of generating a question pair from a domain-specific conversation record such as a conversation record of a communication service system will be described. First, a question cluster is generated by applying automatic duplicate question clustering to a domain-specific conversation record. By generating all combinable question pairs in the same question group, a positive question pair is created and a positive label is attached. Because positive labels may not always be true for a number of reasons, including clustering error, positive labels can be pseudo-labels or weak labels, and the corresponding question pair is pseudo-positive. It may be a pseudo-positive question pair. The data representing the positive question pair is in the form of {question 1, question 2, pseudo positive label}. Hereinafter, for convenience, the positive label and the pseudo-positive label are used interchangeably, and the positive question pair and the pseudo positive question pair are also used interchangeably.

Next, a negative question pair expressed as data in the form of {question 1, question 3, negative label} is generated to correspond to the data representing one positive question pair. Here, the corresponding positive and negative question pairs include the same question 1, and question 3 belongs to a different question group. Since two questions belonging to different question groups were created in pairs, there is no error in the negative label.

The training apparatus according to the present invention fine-tunes the PI model 114 by using positive and negative question pairs. However, because of the presence of false positive question pairs, the PI model 114 is trained with a weakly supervised method using pseudo labels. In order to cope with the error contained in the pseudo-label, it is necessary to assign a confidence score in the form of a weight to the positive question pairs used for training of the PI model. In order to estimate the confidence score, a refiner 121 is added in the embodiment of the present invention.

4 is a block diagram of a learning model including a PI model and a refiner according to an embodiment of the present invention.

Hereinafter, with reference to FIG. 4 , the refinement process of the refiner 121 and iterative training for the PI model 114 will be described. In the iterative training process according to the embodiment of the present invention, the PI model 114 and the refiner 121 use each other's training results. The confidence score of the positive question pair calculated by the refiner 121 is used as a weight in the damage function of the PI model 114 , and the classification result of the PI model 114 is also used in the damage function of the refiner 121 . Also, in the iterative training process, we try to change the label of the positive question pair based on the confidence score of the positive question pair.

를 추정하고, 다른 한번은 부정 질문쌍을 입력하여 부정라벨에 대한 신뢰 점수

를 추정한다. 의사 긍정라벨 및 부정라벨에 대한 신뢰 점수 및 긍정라벨에 대한 PI 모델(114)의 분류 결과가 포함된, 수학식 3에 표현한 손실함수에 기반하여 정제기(121)의 파라미터를 업데이트한다. First, in order to train the refiner 121, an affirmative question pair is inputted into the refiner 121 once, and the confidence score for the pseudo positive label is

Estimating , and inputting a negative question pair at the other time, the confidence score for the negative label

to estimate The parameters of the refiner 121 are updated based on the loss function expressed in Equation 3, including the confidence scores for the pseudo positive label and the negative label, and the classification result of the PI model 114 for the positive label.

는 긍정 질문쌍 (p_i,q_i)에 대한 PI 모델(114)의 분류 결과로서 유사 질문쌍으로 분류된 경우에는 1, 반대의 경우에는 0이다. 그리고,

는 트레이닝 과정에서 조정되는 하이퍼 파라미터(hyper-parameter)이며, i는 긍정 질문쌍(또는 같은 수의 부정 질문쌍)을 인덱싱한다. 앞에서 설명한 바와 같이 정제기의 최종단은 시그모이드 함수로 구현되므로, 신뢰 점수의 범위는 0에서 1까지 이다. 결과적으로, 본 실시예에서는 긍정 질문쌍에 부착된 의사 긍정라벨이 얼마나 참인지 확신할 수 있는 신뢰 점수를 0과 1사이의 범위 내에서 출력하도록 정제기(121)가 트레이닝된다.here

is the classification result of the PI model 114 for the positive question pair (p _i , q _i ). And,

is a hyper-parameter adjusted during the training process, and i indexes positive question pairs (or the same number of negative question pairs). As described above, since the final stage of the refiner is implemented as a sigmoid function, the confidence score ranges from 0 to 1. As a result, in the present embodiment, the refiner 121 is trained to output a confidence score within the range between 0 and 1 with which it can be sure how true the pseudo positive label attached to the positive question pair is.

Next, in order to train the PI model 114, the positive question pair and the negative question pair are alternately input to the PI model 114 to estimate the similarity. By combining the confidence score of the positive question pair provided by the refiner 121 and Equation 2, the parameters of the PI model 114 are updated based on the loss function expressed in Equation 4.

는 수학식 5로 표현된다. function here

is expressed by Equation (5).

는 트레이닝 과정에서 조정되는 하이퍼 파라미터이고, t는 이터레이션 스텝(iteration step)을 나타낸다. 트레이닝 초기에는 정제기(121)가 출력하는 신뢰 점수가 믿을만하지 않으므로, 트레이닝 시 적게 반영되어야 한다. 수학식 5에 나타낸 대로, t가 작은 값이면, 함수

는 지수항에 의존하여 1에 가까운 값이 되므로 신뢰 점수의 영향을 줄일 수 있다. 트레이닝이 진행되어 t가 증가하면, 지수항이 0에 수렴하므로, 함수

는 신뢰 점수에 의존하게 된다.where c and

is a hyperparameter adjusted during the training process, and t represents an iteration step. Since the confidence score output by the refiner 121 is not reliable at the beginning of training, it should be reflected less during training. As shown in Equation 5, if t is a small value, the function

is close to 1 depending on the exponent term, so the influence of the confidence score can be reduced. As the training progresses and t increases, the exponential term converges to 0, so the function

will depend on the confidence score.

는 1 이하의 값이므로, 수학식에 4에서

는 긍정라벨에 의한 손실의 영향을 줄이는 가중치 역할을 한다.According to Equation (5), only the confidence score of the positive label is reflected in the loss function of Equation (4). And,

is a value less than or equal to 1, so in Equation 4

is a weighting factor that reduces the impact of losses caused by positive labels.

On the other hand, if the confidence score inferred by the refiner 121 for the positive label is less than a preset threshold, the corresponding question pair is converted into a negative question pair, and a negative label is attached. The negative question pair generated by the label change acts as stronger data in the training process than the original negative question pair.

By introducing the label change procedure for the confidence score and the weak label into the training process as described above, the FAQ answerer 100 according to the present embodiment provides robustness that can efficiently cope with errors derived from the training process. have

5 is a flowchart illustrating a procedure for answering a user's query by a FAQ answerer according to an embodiment of the present invention.

First, the FAQ answerer 100 obtains a user query (S501), and the QIR 111 searches the FAQ KB 110 to obtain a plurality of question intentions related to the user query and a plurality of questions connected to the plurality of question intentions. of the question pattern group is selected (S502). Here, as described above, the FAQ KB 110 stores answer data to a user's query in the form of data composed of a question intention, a question pattern, and a conditional answer.

Next, the QPR 112 inputs a user query and a plurality of question pattern groups into the pre-trained PI model 114 to identify whether a similar question exists, and uses the identification result to determine the highest ratio among the plurality of question intentions. A question intention indexing a question pattern group including similar questions of

Next, using the answer condition, the CA 113 selects an accurate answer from the list of answers related to the top-one question intention (S504). Here, the CA 113 selects an answer related to the top-one question intention by using the answer condition and the answer list constituting the conditional answer on the knowledge base.

Next, the FAQ answering machine 100 provides the selected answer to the user (S505).

Hereinafter, a result of evaluating the performance of the FAQ answering machine 100 according to the present embodiment will be described. The database used for evaluation is a database that stores conversation records with users at a specific telecommunication company, and consists of millions of question-answer pairs. As described above, the learning model 120 was trained using a part of the database of a specific telecommunication company, and the FAQ KB 110 was generated. The accuracy of the answer provided by the FAQ answerer 100 was measured using the generated FAQ KB 110 . The measured accuracy was 84%, which showed better performance compared to 70% shown by the conventional IR (Information Retrieval) model (BM25, see Non-Patent Document 2).

As described above, according to this embodiment, based on the FAQ KB, the process of searching for the intention of a user query, estimating a question pattern, and a conditional answering process is applied to the conversation system, and a weakly labeled question pair is applied. By providing a dialog system and method used in a task-oriented dialog system to which a training process using

Although it is described that each process is sequentially executed in each flowchart according to the present embodiment, the present invention is not limited thereto. In other words, since it may be applicable to change and execute the processes described in the flowchart or to execute one or more processes in parallel, the flowchart is not limited to a time-series order.

Various implementations of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuitry, field programmable gate array (FPGA), application specific integrated circuit (ASIC), computer hardware, firmware, software, and/or combination can be realized. These various implementations may include being implemented in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. or may be a general-purpose processor). Computer programs (also known as programs, software, software applications, or code) contain instructions for a programmable processor and are stored on a “computer-readable medium”.

A computer-readable medium includes any computer program product, apparatus, and/or device (eg, a CD-ROM, ROM, memory card, a non-volatile or non-transitory recording medium such as a hard disk, a magneto-optical disk, and a storage device).

Various implementations of the systems and techniques described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or other types of storage systems or combinations thereof), and at least one communication interface. For example, a programmable computer may be one of a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a Personal Data Assistant (PDA), a cloud computing system, or a mobile device.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible without departing from the essential characteristics of the present embodiment by those skilled in the art to which this embodiment belongs. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

100: FAQ answerer 110: FAQ knowledge base
111: question intent searcher 112: question pattern estimator
113: conditional answerer 114: PI model
120: learning model 121: refiner

Claims (12)

In the learning method performed by the computing device,
Task fine-tuning of a Paraphrase Identification (PI) model and refiner using a dataset containing a labeled question pair that distinguishes whether it is similar or not. process, wherein the task indicates identification of whether an input question pair is similar; and
The PI model and the refiner are domain fine-tuned using a domain-specific database, including positive question pairs with weak positive labels, and negative query pairs with negative labels. tuning)
including,
The process of fine-tuning the domain is
an iterative training process for the PI model, and a purification process for the refiner, wherein the refiner estimates a confidence score for the weak positive label and the negative label to determine the positive question A learning method characterized in that it copes with errors in the pair creation process. 2. The method of claim 1
The process of fine-tuning the task is
A learning method, characterized in that the PI model and the refiner are pre-trained using a large-capacity dialogue corpus without an answer label (unlabeled). 2. The method of claim 1
The process of fine-tuning the above task is
A learning method, characterized in that the parameter of the PI model is updated based on a distance metric between the label for distinguishing the similarity and the output of the PI model. 2. The method of claim 1
The process of fine-tuning the above task is
The learning method according to claim 1, wherein the parameter of the refiner is updated based on a distance metric between the label for discriminating whether the similarity is present and the output of the refiner. 2. The method of claim 1
The process of fine-tuning the domain is
A learning method, characterized in that by training the PI model based on the weak positive label and the negative label, identifying whether the PI model is similar to a question pair included in the domain-specific database. delete According to claim 1,
The purification process is
The learning method according to claim 1, wherein the parameter of the refiner is updated based on a loss function combining the confidence scores for the weak positive label and the negative label, and the identification result of the PI model for the positive question pair. According to claim 1,
The iterative training process is
A learning method, characterized in that the weight for the positive question pair is calculated by using the confidence score for the weak positive label. 9. The method of claim 8,
The iterative training process is
and updating the parameter of the PI model based on a distance metric between the weak positive label and the output of the PI model, a distance metric between the negative label and the output of the PI model, and the weight. According to claim 1,
The iterative training process is
When the confidence score for the weak positive label is less than a preset threshold, the corresponding question pair is corrected as the negative question pair, and the negative label is attached. In the learning device used by the dialogue system,
Paraphrase Identification (PI) model for generating identification results for similarity of input question pairs; and
A refiner for estimating a confidence score corresponding to the input question pair
including,
After task fine-tuning the PI model and the refiner using a dataset including a labeled question pair to distinguish whether or not they are similar, a weak positive label domain fine-tuning the PI model and the refiner using a domain-specific database, comprising a positive query pair with a label, and a negative query pair with a negative label,
The domain precision adjustment is
performing iterative training on the PI model using the confidence score, and performing purification on the refiner using the identification result;
The purifier,
and estimating the confidence score for the weak positive label and the negative label to cope with an error in the generation of the positive question pair. A computer program stored in a computer-readable recording medium to execute the learning method performed by the computing device according to any one of claims 1 to 5, and any one of claims 7 to 10.

Images