TW202234285A - Dialogue data processing system and method thereof and computer readable medium - Google Patents

Abstract

The present invention is a dialogue data processing system and method thereof. The system is used to collect user dialogue records, filter out intent sentences that can be marked from the dialogue records according to the filter index, select representative sentences by using the original data with tags in each intent category, analyze the average semantic distance between the intent sentence and each representative sentence, and then mark the intent sentence as the intent category of the corresponding representative sentence, or mark it as a new intent category. In addition, the system can enhance the intent category with insufficient sample number by checking the intent category itself and the balance of the number of samples between each intent category. According to this, the present invention can automatically mark the user’s dialogue record, and check and enhance the balance of the intent category. The present invention further provides a computer-readable medium for performing a dialogue data processing method.

Description

Conversational data processing system, method therefor, and computer-readable medium

The present invention relates to data processing technology, and more particularly, to a dialog data processing system, method and computer-readable medium.

In the past, the system for tagging the user's dialogue data could only mark the data intent category label according to whether the user's feedback is satisfied or not after receiving a large amount of user's dialogue records. For example, in a certain dialogue data, the user's feedback result In order to be satisfied, the dialog data does not need to change the intent category label. On the contrary, when the user feedback in the dialog data is unsatisfactory, the artificial intelligence trainer (AI trainer) needs to modify the dialog data to another appropriate intent category , to tag it, or add a new intent category to the conversation data to mark the conversation data with a new intent class label.

However, most users do not actively report whether they are satisfied or not, so the actual amount of conversation data that can be collected is relatively sparse. In addition, users usually reveal their satisfaction or dissatisfaction in the sentences of the conversation, so the user's satisfaction can be known by retrieving the user's conversation record and analyzing the sentences in the conversation record. However, due to the user's The data of the dialogue records are often complex. If all the dialogue data are directly fed back to the AI trainer for marking, the required markings are required. The amount of data recorded is too large, and it may be difficult to implement. In addition, AI trainers often do not check the balance of the amount of data in the labeled taxonomic groups, that is, there may be large differences in the amount of data among the taxonomic groups, which will also affect the subsequent data training results.

To sum up, if we can find a data processing technology that can effectively classify and mark dialogue records, and can check the balance of the amount of data in each classification group, it will be beneficial to the subsequent model training, which will become the technology of the present invention. People in the field are eager to pursue the goal of a solution.

In view of the above problems, the present invention proposes a dialogue data processing system, which includes: a dialogue data acquisition module, which is used for collecting dialogue records with sentences, so as to filter the dialogue records according to the filtering index to obtain intent sentences; and dialogue data The grouping labeling module is used to select representative sentences from original data with labels in a plurality of intention categories, and then compare the intention sentences with the representative sentences, so as to obtain and compare the intention sentences with each representative sentence. If the average semantic distance between the representative sentences is the largest, so that when the average semantic distance is smaller than a predetermined threshold value, the intention sentence and the original data hiding the label are grouped and marked with each representative sentence as the center, Make the intent sentence and the original data in the same group have the same label, or when the largest average semantic distance exceeds the predetermined threshold, the intent sentence with the largest average semantic distance is a new intent category. A new representative sentence, and the intention sentence and the original data with the label hidden are grouped and labeled with all representative sentences and new representative sentences as the center, so that the intention sentence and the original data in the same group have the same label .

In one embodiment, the dialog data grouping and marking module takes a cluster center of a plurality of semantic vectors corresponding to a plurality of sentences of each of the plurality of intention categories, so as to obtain and The one with the smallest distance between the group centers is used as the representative sentence.

In another embodiment, the average semantic distance is calculated by first calculating the distance between the intent sentence and each of the representative sentences, and then averaging the distances.

In another embodiment, the dialogue data processing system further includes a dialogue data enhancement module for data augmentation, after the number of samples of each of the intent categories is insufficient or the difference in the number of samples among all the intent categories is compared. , and data enhancement is performed for the small sample size.

In another embodiment, the data augmentation uses synonym replacement, random insertion, random exchange, random deletion, data augmentation methods based on machine learning and deep learning, or any combination thereof.

In another embodiment, the filter index includes a satisfaction feedback value, positive and negative polarity of sentence sentiment, confidence in the intention of the dialogue text, or whether there is a transfer request.

In another embodiment, the dialogue data capturing module further includes an emotion recognition unit, which uses an emotion recognition model to recognize the emotion polarity of the intent sentence, so as to generate the positive and negative emotion polarity of the sentence.

In another embodiment, the dialogue data capturing module further includes an intention recognition unit, which uses an intention recognition model to calculate the intention recognition confidence level of the intention sentence to generate the dialogue text intention confidence level.

The present invention further proposes a dialogue data processing method, which includes: collecting dialogue records with sentences; filtering the dialogue records according to filtering indicators to obtain intent sentences; selecting representative sentences from original data with tags in multiple intent categories; and compare the intent sentence with each of the representative sentences respectively, so as to obtain the largest average semantic distance between the intent sentence and each of the representative sentences When the maximum average semantic distance is less than a predetermined threshold value, the intention sentence and the original data hiding the tag are grouped and marked with each representative sentence as the center, so that the intention sentence in the same group is grouped and marked. and the original data have the same label, or when the largest average semantic distance exceeds the predetermined threshold, the intent sentence with the largest average semantic distance is a new representative sentence of the new intent category, and all representative sentences are and the new representative sentence as the center to group and mark the intent sentence and the original data with the label hidden, so that the intent sentence and the original data in the same group have the same label.

In another embodiment, the step of selecting a representative sentence is to obtain a cluster center of a plurality of semantic vectors corresponding to a plurality of sentences of each of the plurality of intent categories to obtain from the plurality of semantic vectors The one with the smallest distance from the group center is used as the representative sentence.

In another embodiment, the average semantic distance is calculated by first calculating the distance between the intent sentence and each of the representative sentences, and then averaging the distances.

In another embodiment, the step of filtering the dialogue record to obtain the intent sentence according to the filtering index further includes that after the number of samples of each of the intent categories is insufficient or the difference of the number of samples among all the intent categories is compared, the number of samples is compared. Data augmentation with fewer intent categories.

In another embodiment, the data augmentation uses synonym replacement, random insertion, random exchange, random deletion, data augmentation methods based on machine learning and deep learning, or any combination thereof.

In another embodiment, the filter index includes a satisfaction feedback value, positive and negative polarity of sentence sentiment, confidence in the intention of the dialogue text, or whether there is a transfer request.

In another embodiment, the positive and negative polarity of the sentence emotion is generated by using an emotion recognition model to identify the emotional polarity of the intent sentence.

In yet another embodiment, the dialog text intention confidence level is generated by calculating the intention recognition confidence level of the intention sentence by using an intention recognition model.

The present invention further provides a computer-readable medium, which is applied to a computing device or a computer and stores an instruction to execute the above-mentioned method for processing dialogue data.

To sum up, the dialogue data processing system, method and computer-readable medium of the present invention are capable of filtering out the intent sentences that can be marked from the dialogue records according to the set filtering index after collecting the user's dialogue records, The representative sentence is selected from the intent category, and semantic analysis of the intent sentence and the representative sentence is performed to determine whether the intent sentence should be marked as the same intent category as the representative sentence, or should be marked as a new intent category. Therefore, the present invention can To achieve the purpose of automatically classifying users' conversation records and marking them. In addition, after the intent sentence is classified, the present invention can also check the number of samples in each intent category and the difference in the number of samples in all intent categories, so as to enhance the intent categories with insufficient samples, so as to avoid the occurrence of intent categories. The purpose of the imbalance problem caused by the difference in the amount of data.

10, 10': Dialogue data processing system

11: Dialogue data capture module

111: Emotion Recognition Unit

112: Intent recognition unit

12: Dialogue data group marking module

13: Dialogue data enhancement module

S401~S405: Steps

601~608: Process

FIG. 1 is a schematic structural diagram of the dialogue data processing system of the present invention.

FIG. 2 is a schematic diagram of the structure of the dialogue data capturing module of the present invention.

FIG. 3 is a schematic structural diagram of another embodiment of the dialogue data processing system of the present invention.

FIG. 4 is a flow chart of the steps of the dialogue data processing method of the present invention.

FIG. 5 is a flow chart of steps of another embodiment of the dialogue data processing method of the present invention.

FIG. 6 is a flow chart of the method for processing dialogue data according to the present invention.

The following describes the technical content of the present invention through specific embodiments, and those skilled in the art can easily understand the advantages and effects of the present invention from the content disclosed in this specification. However, the present invention can also be implemented or applied by other different specific embodiments.

FIG. 1 is a schematic structural diagram of the dialogue data processing system of the present invention. As shown in the figure, the dialogue data processing system 10 of the present invention includes a dialogue data capture module 11 and a dialogue data grouping and marking module 12. The dialogue data capture module 11 obtains the user's dialogue records and then filters them. The analysis and calculation are performed through the dialogue data grouping and marking module 12, so as to mark the sentences in the filtered dialogue records after judging the type. The description of the present invention is as follows.

The dialogue data capture module 11 is used for receiving or collecting dialogue records with sentences, wherein the dialogue records refer to the sentences in which the dialogue data capture module 11 captures the sentences (such as explanation, narration or inquiry) stated by the user on the system. Formed dialogue text data. Conversation records may include a single statement stated by the user or multiple statements narrated by the user during an event or over a period of time. In addition, the dialogue data processing system 10 of the present invention can further be provided with a database for storing data, and the dialogue data capturing module 11 can store the dialogue records in the database after collecting the dialogue records.

After collecting the user's dialog records, the dialog data capturing module 11 can filter the user's dialog records according to a preset filter index, so as to obtain the intent sentence. Specifically, the filtering indicators include satisfaction feedback value, positive and negative polarity of sentence emotion, confidence level of dialogue text intention, or whether there is a transfer request or not. The positive and negative polarity of the dialogue text sentiment, the model confidence of the system's prediction of the dialogue text intention, whether the user transfers to a special person, etc. are used to filter and filter the sentences suitable for marking in the dialogue record to form the intention sentence.

The dialog data grouping and labeling module 12 selects representative sentences from original data with labels in multiple intent categories. Specifically, each intent category may include multiple sentences. And each sentence has a corresponding semantic vector, and the method of selecting the representative sentence by the dialogue data grouping and marking module 12 is to average all the semantic vectors in each intention category to obtain the corresponding group center, and then calculate each semantic vector and The distance between the group hearts is used to select the sentence corresponding to the semantic vector of the smallest distance as the representative sentence. The dialogue data grouping and marking module 12 calculates the dialogue data acquisition module 11 according to the selected representative sentence. The distance between the intended sentence and each representative sentence is the one with the largest average semantic distance between the intended sentence and each representative sentence. The dialogue data grouping and marking module 12 first calculates the distance between the intended sentence and each representative sentence. distance, and then average the distances to obtain the average semantic distance.

Furthermore, the dialogue data processing system 10 of the present invention can preset the threshold value of the average semantic distance. According to this, after obtaining the largest average semantic distance, the average semantic distance is compared with the predetermined threshold to determine the difference between the intended sentence and the predetermined threshold. The semantic similarity between the representative sentences is used to classify the intent sentences. That is, when the largest average semantic distance in the intent sentence is less than or equal to a predetermined threshold value, the intent sentence and the original data hiding the label are grouped and marked with each representative sentence as the center, that is, the The intent sentence is classified into the intent category marked by the representative sentence with similar semantics and marked; or when the intent sentence with the largest average semantic distance exceeds or equals to a predetermined threshold, the intent sentence with the largest average semantic distance It is the representative sentence of the new intention category, and the intention sentence and the original data with the label hidden are grouped and marked with all representative sentences as the center, so that the intention sentence and the original data in the same group have the same Labeling means that the intent sentence is classified into a new intent category and then marked, and finally, the marked intent sentence and the original data become model training data for training the intent recognition model. Accordingly, the present invention can filter and filter after collecting a large number of conversation records of users, so as to obtain conversation records suitable for marking, so as to reduce the number of conversation records that need to be marked, and then perform filtering on the filtered conversation records. Marking, to achieve the purpose of automatically marking the dialogue records, according to this, it can avoid the problem of poor quality of marked data that may be caused by conventional AI trainers when faced with a large number of dialogue records that need to be marked.

FIG. 2 is a schematic structural diagram of the dialogue data capturing module of the present invention. As shown in the figure, the dialogue data capturing module 11 includes an emotion recognition unit 111 and an intention recognition unit 112, wherein the emotion recognition unit 111 uses the emotion recognition model to recognize the emotion polarity of the intention sentence, so as to generate the positive and negative polarity of the sentence emotion, for example The emotion recognition unit 111 can use the deep learning long short-term memory network to train the emotion recognition model to recognize the positive and negative polarity of the sentence emotion of the sentence. In addition, the intention recognition unit 112 uses the intention recognition model to calculate the intention recognition confidence of the intention sentence, so as to To generate the confidence level of the intention of the dialogue text, for example, the intention recognition unit 112 can use the keyword comparison score, the intention model classification probability value of machine learning or deep learning to calculate and predict the intention confidence level of the sentence.

FIG. 3 is a schematic structural diagram of another embodiment of the dialogue data processing system of the present invention. As shown in the figure, the dialogue data processing system 10' of this embodiment is substantially the same as the dialogue data processing system 10 of the first embodiment, the difference is that in this embodiment, the dialogue data processing system 10' further includes a dialogue data enhancement module Group 13, wherein the dialogue data capturing module 11 and the dialogue data group marking module 12 are the same as those described above and will not be described in detail.

The dialogue data enhancement module 13 is to perform data enhancement for the small number of samples after the number of samples in each intent category is insufficient or after comparing the difference in the number of samples among all intent categories, that is, in addition to checking whether the number of samples in a single intent category is In addition to being sufficient, the difference in the number of samples of each sentence between each original intent category or between the original intent category and the new intent category is also compared, so as to perform data enhancement for the intent category with insufficient samples. In addition, the dialogue data enhancement module 13 can also set a threshold value of the number of samples, so that when the difference in the number of samples exceeds the threshold value of the number of samples, data enhancement is performed on the intent category with less sentence data, wherein the dialogue data enhancement module 13 Can use synonym replacement, random insertion, random exchange, random Machine deletion, data enhancement methods based on machine learning and deep learning, or a combination of the aforementioned methods, are used to enhance sentence data, so as to achieve the purpose of balancing the number of samples in each group and to provide better training data.

In other words, after the sentence is marked in the present invention, the dialogue data enhancement module 13 performs data augmentation according to the number of training samples contained in each intent category, so as to maintain that the difference in the proportion of the data samples of each intent category is not greater than the threshold value. If the difference in the proportion of the number of data samples of each category is greater than the threshold value, the sentence data in the intent category with fewer samples will be expanded to maintain an appropriate proportion. In this way, the present invention can maintain the balance of the number of samples of each intent category, and automatically expand the number of samples, so that the AI trainer only needs to label effectively and a small amount of data to achieve the effect of completing a large amount of high-quality labeling data, This enables the final model training data to provide better training effects.

FIG. 4 is a flow chart of the steps of the dialog data processing method of the present invention.

In step S401, a conversation record with sentences is collected. In the present invention, a user interface can be set for the user to communicate with the system, so as to collect the user's dialog record, wherein the dialog record includes the sentences stated by the user. Specifically, the present invention collects the dialogue records of the dialogue between the user and the system by collecting the user's sentences in the online dialogue of the user, which can be set within a period of time (for example, every or several hours, every day, every week , monthly) to collect user sentences, to accumulate all sentences of users within the time interval, and form conversation records.

In step S402, the dialog record is filtered according to the filter index to obtain the intent sentence. Among them, the present invention uses filtering indicators such as the positive and negative polarity of the sentence emotion, the confidence level of the dialogue text intention, the satisfaction feedback value, whether the user makes a transfer request, or a combination indicator of the above-mentioned filtering indicators to perform the collected dialogue records. Filter to filter out intent sentences that can be used for marking from all user sentences in the conversation record.

The positive and negative polarity of sentence emotion in the filter index is to use the deep learning long short-term memory network to train an emotion recognition model to identify the emotional polarity of the collected sentences. In detail, the present invention trains an emotion recognition model through a deep learning long short-term memory network including formulating emotional word scores and comparing emotional words, machine learning or deep learning emotional models, etc., to calculate the positive and negative emotions of all the sentences collected. Polarity, and then identify the emotional polarity of the sentences in the dialogue records.

The dialog text intent confidence in the filtering index is calculated by using the keyword comparison score, the intent model classification probability value of machine learning or deep learning to calculate the intent confidence of the intent sentence. That is, the present invention calculates the confidence level of intent recognition of user sentences by using methods such as formulating keyword comparison scores, machine learning or deep learning intent model classification probability values, or a combination thereof. Intent recognition confidence of the sentence.

In step S403, representative sentences are selected from original data with tags in a plurality of intent categories. Among them, the selection of the representative sentence is to average multiple semantic vectors corresponding to multiple sentences in each original intent category to obtain the cluster center of the semantic vector, and then select the cluster center from the multiple semantic vectors. The one with the smallest distance is used as the representative sentence.

In step S404, the intent sentence is compared with each of the representative sentences, so as to obtain the one with the largest average semantic distance from the intended sentence and each of the representative sentences, so that the one with the largest average semantic distance is smaller than a predetermined threshold When the value is set, the intent statement and the original data that hide the label are grouped and labeled with each representative statement as the center, so that the intent statement and the original data in the same group have the same label, or the original data in the same group have the same label. When the one with the largest average semantic distance exceeds the predetermined threshold value, the intent sentence with the largest average semantic distance is set as the representative sentence of the new intent category, and the intent sentence and the original one that hides the label are centered on all representative sentences. The data is grouped and labeled so that the intent statement and the original data in the same group have the same label. This step is to calculate the intent statement and each representative The average semantic distance between sentences, and find out the one with the largest average semantic distance between the intent sentence and each representative sentence. The distance is averaged. In short, when the distance of the largest average semantic distance is less than the predetermined threshold, the intent sentence is classified into the original intent category corresponding to the semantically similar intent sentence in each representative sentence, and marked. In addition, if the distance of the one with the largest average semantic distance exceeds a predetermined threshold value, the intent sentence is classified into a new intent category and subsequent marking is performed. Furthermore, the marked intent sentence and the original data are used for training the intent recognition model. In one embodiment, the new intent category and its representative sentence are subsequently processed again for dialogue data, and in step S403, the new intent category can become one of multiple intent categories, and the representative sentence is selected according to its steps.

For example, when the present invention distinguishes the intent category to which the intent sentence belongs, first, a representative sentence R is selected from each intent category in the original marked data A that has been marked before, and if there are N categories in all intent categories, the N sentences are selected to represent the representative sentences R _1~N of their respective intention categories. Next, semantically compare the selected intent sentences in the unmarked new data B with the representative sentences R _1~N to calculate the average semantic distance, and when the average semantic distance exceeds the threshold value of the semantic distance , to add a new intent category to the intent sentence. In addition, if multiple intent sentences are filtered out, after calculating the average semantic distance of each intent sentence corresponding to the representative sentence, find the largest average semantic distance from the average semantic distances. If the average semantic distance of O exceeds the preset threshold value, select O as another representative sentence R _N+1 of the new intent. Next, hide the data mark of the original marked data A, and add the intent sentence of the new data B, and perform unsupervised or semi-supervised grouping calculation centered on the representative sentences R _1~N+1 , including the original marked data A And all sentence data of new data B are divided into N+1 groups, among which, the first N groups are the number of original intent categories, the N+1th group is the possible new intent categories, and then the sentences with similar semantics are divided into The same intent category. Finally, display all the original data labels in the original marked data A, and then automatically mark all the unmarked intent sentences in the new data B by using the similar and known data labels in the same group to form N Or N+1 groups of intent categories, to achieve the purpose of automatically marking the dialogue sentences in the dialogue record.

FIG. 5 is a flow chart of steps of another embodiment of the dialogue data processing method of the present invention. As shown in the figure, steps S401 to S404 in this embodiment are the same as those in the previous embodiment, and the difference is that this embodiment further includes step S405. In step S405, after the number of samples of each of the intent categories is insufficient or the difference in the number of samples among all the intent categories is compared, data enhancement is performed on the intent categories with a small number of samples. This step compares the difference between the respective sample numbers of all intent categories (including the original intent category or adding a new intent category), and performs data enhancement for the intent category with a small number of samples. The methods of data enhancement include synonym replacement, random insertion, random exchange, random deletion, data enhancement methods based on machine learning and deep learning, or any combination of the above.

Accordingly, the present invention augments the sentence data in each intent category, wherein it is first checked whether the number of samples of sentences in each intent category is sufficient, and then the difference in the number of samples between each intent category is compared, and the Data enhancement is performed on the intent categories with a small number of samples, so as to continue to expand until the number of samples in each intent category reaches a balance, that is, the difference in the number of samples is less than a preset threshold value α of the number of samples.

FIG. 6 is a flowchart of a method for processing dialogue data according to the present invention. As shown in the figure, the flow of a specific embodiment is described as follows.

In the process 601, first, the dialogue data capture module in the dialogue data processing system of the present invention collects the dialogue records. The dialogue data capture module collects user sentences when the user asks for relevant information online, and accumulates a paragraph All sentences in the time can be used to form a conversation record. In addition, it can also receive the user's conversation data obtained from an external database or other conversation data retrieval devices. Specifically, the present invention can design an input interface on the front end, such as web or communication software, etc., for the user to input natural language sentences, and the system of the present invention transmits the conversation record with the user's sentence to the back-end data library for recording and storage.

For example, for example, the user enters "I have a problem with my bill" in the front-end input interface; the system responds: "What's the problem?"; The user asks: "The money is wrongly calculated"; meaning"; the user asked: "It sucks". In the above dialogue, the present invention records the sentences inquired by the user to the system, such as T=["There is a problem with my bill"; "The money is miscalculated"; "It's too bad"]. Then, after word segmentation, each sentence is converted into sentence vector by using the method of text turning vector (for example, using the word2vec model). Take "I have a problem with my bill" as an example. After the word segmentation, it can be divided into 5 words such as "I", "of", "billing", "yes", "problem"", and then enter each word in In the word2vec model, the corresponding sentence vector can be obtained. According to this, the sentence can be converted into a two-dimensional vector ""I": [0.123, 0.456], "De": [0.233, 0.536], "Bill": [0.322, 0.689], "has": [0.111, 0.422], "problem": [0.777, 0.543]", to express.

In the process 602, the emotion recognition module recognizes the emotional polarity of the dialogue text, wherein the emotion recognition module can use the deep learning Long Short-Term Memory (LSTM) network to train the emotion recognition model to recognize the dialogue text emotion Polarity, calculate the positive and negative polarity of emotion of each sentence through the emotion recognition model of all the user dialogue sentences collected above, so as to generate a probability distribution of emotional polarity between 0 and 1, where the probability value is closer to 1, the Indicates that the more positive their emotions are, on the contrary, the closer to 0, the more negative their emotions are. By analyzing the sentences collected by the dialogue data acquisition module T=["There is a problem with my bill"; "The money is wrongly calculated. ";"It sucks"], to get the emotional probability E=[0.4,0.5,0.1].

In the process 603, the intention recognition module identifies the confidence level of the dialogue text intention, and the intention recognition module uses a supervised machine learning neural network (Neural Network) to train the intention recognition model to identify the dialogue text intention confidence level. The intention recognition The module is used to identify all the dialogue sentences of the user collected by the dialogue data acquisition module, and calculate the intention recognition confidence of each sentence through the intention recognition model, so as to generate a confidence probability between 0 and 1 corresponding to each intention type distribution, and the probability sum of all intent categories is 1. Specifically, the present invention can use the softmax function to perform analysis operations. For example, assuming that the marked data has three intent categories, namely [“Billing”, “Mobile Phone”, “Internet”], its output value is [1, 2, 3], then it is calculated by the following function, and the softmax function is as follows:

上述計算得到[0.09,0.245,0.665]之機率分布，其中，意圖類別“網路”之機率為三者中最高，則將語句之意圖分類為“網路”，是以，所得到之機率值即為意圖分類之信心度。舉例而言，以對話資料擷取模組所蒐集之T=[“我的帳單有問題”；“錢算錯”；“太爛了吧”]對話紀錄為例，對話紀錄T中之每一個語句將各自產生三維向量，以分別代表[“帳單”,“手機”,“網路”]三種意圖類別，且其總和為1，最後集合為I=[[0.8,0.1,0.1],[0.4,0.3,0.3],[0.3,0.4,0.3]]，據此，每個語句之意圖信心度為各個意圖類別中最大之機率值α=[0.8,0.4,0.4]。

The above calculation obtains the probability distribution of [0.09, 0.245, 0.665], in which, the probability of the intent category "Internet" is the highest among the three, then the intent of the sentence is classified as "Internet". Therefore, the obtained probability value It is the confidence of intention classification. For example, take T=["There is a problem with my bill";"The money is wrongly calculated";"Itsucks"] dialogue records collected by the dialogue data acquisition module as an example. A statement will each generate three-dimensional vectors to represent the three intent categories ["Billing", "Mobile", "Internet"], and their sum is 1, and the final set is I=[[0.8,0.1,0.1], [0.4, 0.3, 0.3], [0.3, 0.4, 0.3]], according to which, the intention confidence of each sentence is the maximum probability value α=[0.8, 0.4, 0.4] in each intention category.

In the process 604, the dialogue data capture module filters the dialogue records, that is, the dialogue data capture module filters the dialogue records according to the filter index, wherein the filter index refers to the user feedback, whether the user transfers to a special person, and the emotion recognition method. The emotional polarity of the user's dialogue sentence identified by the module is positive and negative, and the intentional confidence of the dialogue sentence identified by the intention recognition module is based on this, the dialogue data extraction module uses the filter index as a reference for selecting the user's dialogue sentence.

By way of example, the selection criteria of the present invention are exemplified as follows. In example 1, the user has a dissatisfied sentence. For example, the user responded: "The phone cannot be answered", and the system replied: "Your network speed is normal". Therefore, the system's response is related to the "mobile phone" that the user wants to ask. "The problem is different, so the user gives unsatisfactory feedback. In example 2, when the user transfers the special person, the sentence that the user talks to the special person, that is, the user puts forward the requirement of "transferring the special person" during the dialogue with the system. At this time, the selection of the dialogue record is the dialogue sentence between the user and the special person. . In Example 3, the probability of positive and negative emotions generated by the emotion recognition module is lower than 0.3, and the above-mentioned dialogue record T=[“There is a problem with my bill”; “The money is wrong”; “Too bad. The semantic emotional probability E=[0.4, 0.5, 0.1], so the emotional probability value of the sentence “too bad” is 0.1, which is lower than the threshold value of 0.3, so this sentence is selected. In Example 4, the confidence level of the dialogue sentence of the intention recognition module is lower than 0.7, so the dialogue record T=["There is a problem with my bill"; "The money is wrong"; "It's too bad"] If the semantic intent confidence is α=[0.8, 0.4, 0.4], the two sentences [“money count is wrong”; “too bad”] will be selected. Accordingly, the sentences selected by the dialogue data capturing module according to the aforementioned four criteria are used to carry out the subsequent steps.

進一步求出各對話語句與 群心μ之距離，再自各意圖類別中選出離群心最近的對話語句作為代表語句R。 In the process 605, the dialogue data grouping and marking module identifies the group to which the dialogue sentence belongs. First, a representative sentence R is selected from each intent category in the original marked data A previously marked by the system. The representative sentence can be selected by averaging the semantic vectors of the dialogue sentences in each intent category to obtain a cluster. center μ , through the distance formula

The distance between each dialogue sentence and the group center μ is further obtained, and the dialogue sentence closest to the group center is selected as the representative sentence R from each intention category.

，且依次求出d(r₂ ,μ)=0，d(r₃ ,μ)=

，藉以選出距離群心μ最小之對應語句為代表語句R，因而R=“我想查帳 單資訊”。另外，原有標記資料A中之意圖類別共有三類，即N=3類，故代表語句R_1~3分為[“我想查帳單資訊”,“有新的手機嗎”,“網路速度太慢”]，而各自代表之語意向量為[[0.3,0.4],[0.7,0.7],[0.9,0.9]]。 Specifically, as shown in Table 1 below, taking the "billing" intent category as an example, it is assumed that a two-dimensional vector represents the position of each sentence in the semantic space, that is, the vector "[0.1, 0.2], [ 0.3, 0.4], [0.5, 0.6]” respectively represent the semantic vector of the sentences in the intent category “Billing”, ““The bill is wrong”, “I want to check the billing information”, and “See the payment information””, First, calculate the average of each dimension to get the group center, the group center is [(0.1+0.3+0.5)/3=0.3, (0.2+0.4+0.6)/3=0.4], so μ =[0.3 , 0.4] , according to the distance d(r ₁ ,μ ) =

, and in turn obtain d(r ₂ , μ )=0, d(r ₃ , μ )=

, so that the corresponding sentence with the smallest distance from the cluster center μ is selected as the representative sentence R, so R=“I want to check the billing information”. In addition, there are three categories of intent categories in the original marked data A, namely N=3 categories, so the representative sentences R _1~3 are divided into [“I want to check the billing information”, “Do you have a new mobile phone”, “Internet The speed of the road is too slow”], and the semantic vectors represented by each are [[0.3, 0.4], [0.7, 0.7], [0.9, 0.9]].

，再將同一語句T_i對代表語句R所求得之各 距離取平均，以T₁=“我的帳單有問題”之語句而言，假設其語句向量為[0.3, 0.3]，可得到T₁語句與代表語句R_1~3之間的距離為d(t_1,r_1)=√0.01，d(t_1,r_2)=√0.32，d(t_1,r_3)=√0.72，如此T₁語句之平均語意距離為d(t_1,r)=√(0.01+0.32+0.72)÷3=0.34，另外，T₂=“錢算錯”之語句設其語句向量為[0.2,0.2]，則其平均語意距離為d(t_2,r)=√(0.05+0.5+0.99)÷3=0.41，以及T₃=“太爛了吧”之語句若其語句向量為[0.1,0.1]，則其平均語意距離為d(t_3,r)=√(0.13+0.72+1.28)÷3=0.49。據此，可得到對話資料T之語句中與R距離最大之語句O即為“太爛了吧”，其中，可設定預設之門檻值=0.3，因而，語句O的平均語意距離超過預設之門檻值=0.3，則挑選O為另一新意圖類別的代表語句R_N+1，亦即，新的意圖類別之代表語句為R_N+1=“太爛了吧”。 Further, the sentences in the new data T=[“There is a problem with my bill”; “The money is wrongly calculated”; “It’s too bad”] selected by the dialogue data capture module are respectively and the representative sentence R _1~3 = ["I want to check the billing information", "Do you have a new mobile phone", "The network speed is too slow"] to perform semantic comparison to find the sentence O with the largest average semantic distance. Specifically, the calculation method of the average semantic distance is to calculate the distance between each sentence in T and R

, and then average the distances obtained by the same sentence T _i on the representative sentence R. For the sentence T ₁ = "I have a problem with my bill", assuming that the sentence vector is [0.3, 0.3], we can get The distance between the T ₁ statement and the representative statement R _1~3 is d(t_1,r_1)=√0.01, d(t_1,r_2)=√0.32, d(t_1,r_3) ₌ √0.72, so the T1 statement The average semantic distance is d(t_1,r)=√(0.01+0.32+0.72)÷3=0.34. In addition, the sentence vector of T ₂ = "miscalculation of money" is set to be [0.2, 0.2], then the average If the semantic distance is d(t_2,r)=√(0.05+0.5+0.99)÷3=0.41, and T ₃ = "too bad", if its sentence vector is [0.1, 0.1], then its average semantic The distance is d(t_3,r)=√(0.13+0.72+1.28)÷3=0.49. According to this, it can be obtained that the sentence O with the largest distance from R among the sentences in the dialogue data T is “too bad”, and the preset threshold value can be set to be 0.3. Therefore, the average semantic distance of the sentence O exceeds the preset value. The threshold value = 0.3, then O is selected as the representative sentence R _N+1 of another new intention category, that is, the representative sentence of the new intention category is R _N+1 = "too bad".

Then, the original marked data A and the new data T are used as training data, and the representative sentence R _1~N+1 = [“I want to check the billing information”, “Do you have a new mobile phone”, “Internet speed” "Too slow", "too bad"] for the preset grouping center to perform grouping calculation, and divide all data into N+1 groups, where N=3 is the number of original intent categories, and the N+1th group is For the newly added intent category. Accordingly, the present invention uses the k-means grouping method, sets k as 4 groups, and uses the group heart of the first round of k-means to represent the sentence R _1~N+1 = [“I want to check the billing information”, "Do you have a new phone?", "The internet is too slow", "It's too bad"] instead, using an algorithm to classify sentences with similar semantics into the same group. For example: a group with R ₁ = "I want to check billing information" U ₁ = ["I want to check billing information", "The bill is wrong", "See billing information", "My There is a problem with the bill"].

In the process 606, the dialog data grouping labeling module labels the intent category for each group data. For all unmarked new data T _k , it is automatically marked with _Ak of the closest and known intent category in the same group. For example, the statement T ₁ = ["I have a problem with my bill"] can use the nearest neighbor method to find the same group U ₁ = ["I want to check the billing information", "The bill is wrong", "It depends on the payment information", "I have a problem with my bill"], the nearest neighbor to T1 Nb = "I want to check the billing information", and then mark the intent category of T ₁ = ["I have a problem with my bill"] as the same as Nb="I want to check billing information" same intent category "Billing". In this way, all the sentences in the new data T can be marked with the corresponding intent category, so, I _bill = ["I want to check the billing information", "The bill is wrong", "See the payment information" , "I have a problem with my bill"], I _{mobile phone} = ["I want to check the new model of the mobile phone", "Do you have a new mobile phone", "Do you have a new phone?"], I _network = ["The network speed is too slow ”, “internet disconnected”, “cannot connect to internet”], and I _other = [“money miscalculation”; “too bad”].

In the processes 607-608, the dialog data enhancement module augments the data of each intent category. Among them, the dialogue data enhancement module will first check whether the number of samples of each group of intent categories is sufficient, that is, whether the sentences in the intent categories are sufficient, the dialogue data enhancement module Data enhancement is performed on the intent category with a small number of samples. The data enhancement method includes synonym replacement (randomly select N non-stop words and replace them with their synonyms), random insertion (randomly select 1 non-stop word and replace its synonyms with its synonyms). Insert any position in the sentence N times), random exchange (optionally exchange two words in the sentence, repeat the above N times), random deletion (randomly select N words to delete), data enhancement methods based on machine learning and deep learning, etc. , so that the sentence data of each group of intent categories continues to expand until the number of samples in each intent category reaches a balance (the difference ratio is less than a set threshold value α), so as to achieve the purpose of balancing the number of samples of the intent categories. In addition, a threshold value of the number of samples can be set in the dialogue data enhancement module, so that after calculating the number Q of each intent category, it can be determined whether the number Q of each intent category reaches a predetermined threshold value (assuming the threshold value is 2), that is, Q (I _bill ) = 4, Q (I _{mobile phone} ) = 3, Q (I _network ) = 3 and Q (I _other ) = 2, all of which are greater than the predetermined threshold value, so the evaluation of whether the data is balanced, that is , calculate the proportion of the difference in the number of each category, the formula is E = x _i /max( x ), according to this, the proportion of the difference in the number of sentences in each intention category E(I _bill ) = 1, E(I _{mobile phone} ) = 0.75, E(I _network ) = 0.75, E(I _other ) = 0.5, further, according to the quantity difference threshold α = 0.6, it is determined that the amount of data of intention category I _other is obviously too low, so it is necessary to carry out data enhanced.

In the data enhancement of the present invention, the common synonyms in natural language processing can be used to replace the data to enhance the data. Through the predefined synonym table [“too bad”, “too bad”, “too poor”], the intent category is The sentence "too bad" in I _Others can be replaced by synonyms, which can be added "Too bad", and the intention category calculation E(I _Others )=0.75 After exceeding the threshold value of quantity difference α=0.6, stop the data enhanced. The above process can obtain the best training data set, and can provide continuous learning and updating of the intention recognition model of the system, wherein the updating method can use two methods of retraining and fine-tuning training (Fine tune) or a combination thereof, wherein the retraining is Add the new data to the old data to retrain the model together. In addition, fine-tuning training keeps the previous model weights and only updates the upper layer weights in the network model.

Each module and unit in the present invention can be software, hardware or firmware; if it is hardware, it can be a processing unit, processor, computer or server with data processing and computing capabilities; if it is software or firmware, which may include instructions executable by a processing unit, processor, computer, or server.

In addition, the present invention also discloses a computer-readable medium, which is applied to a computing device or computer having a processor (eg, CPU, GPU, etc.) and/or memory, and stores instructions, and can utilize the computing device or computer. The computer executes the computer-readable medium through a processor and/or a memory, so as to execute the above-mentioned methods and steps when executing the computer-readable medium.

To sum up, the present invention provides a dialogue data processing system with automatic data capture, group marking and enhancement, a method and a computer-readable medium thereof, which utilizes a dialogue data capture module to collect user feedback dialogue data and based on The user's emotional positive and negative, the confidence of the dialogue text intention, whether to transfer to a special person and other indicators are used to filter the user's dialogue sentence data to filter the user's dialogue sentences. In addition, the dialogue data grouping and labeling module of the present invention performs unsupervised grouping calculation on the existing dialogue data and automatically marks it in advance, so as to assist the AI trainer to quickly and effectively classify the intention categories of the dialogue data, which can improve the learning AI training. There are so many categories of intentions that teachers face when marking, it is difficult to intuitively establish The problem of classifying a new sentence data. In addition, the present invention adjusts the number of samples included in each intent category, and enhances data for categories with insufficient samples, so as to achieve the purpose of maintaining the balance of the number of samples in the training data set.

Therefore, the present invention can achieve the following effects.

First, improve the AI trainer's marking of a large number of dialogue records, which may cause the shortcomings of poor quality of the marked data.

Second, the present invention automatically calculates and selects the best training data by considering methods such as data filtering and retrieval, aggregation and expansion of similar meanings.

Third, the dialogue data retrieval not only considers whether the user's feedback is satisfied, whether the user transfers to a special person, etc., but also adds two indicators, such as the system's detection of the positive and negative polarity of the user's dialogue text, and the model's confidence in the system's prediction of the intention of the question. Use the filter to filter the conversation transcripts that are suitable for the AI trainer to mark.

Fourth, through group calculation and automatic tagging of dialogue data, AI trainers can quickly process a large amount of tagged data and reduce data tagging manpower.

Fifth, the augmentation of dialogue data is used to enhance the richness and diversity of the data of each intent category, and through a check mechanism that combines the balance of each category of data, it is ensured that the quality of subsequent intent detection model predictions is not biased by data imbalance.

The above-mentioned implementation cases are only illustrative and specific descriptions, rather than limiting the scope of the present invention. Any equivalent modifications or changes made to them do not depart from the spirit and scope of the present invention, and should be included in the patent of this case. in the range.

10: Dialogue data processing system

11: Dialogue data capture module

12: Dialogue data group marking module

Claims (15)

A dialogue data processing system, comprising: The dialogue data acquisition module is used to collect dialogue records with sentences, so as to filter the dialogue records according to the filter index to obtain the intention sentences; and The dialog data grouping and tagging module is used to select representative sentences from the original data with labels in a plurality of intention categories, and then compare the intention sentences with the representative sentences respectively, so as to obtain from the intention sentences The one with the largest average semantic distance from each of the representative sentences, so that when the one with the largest average semantic distance is less than a predetermined threshold value, grouping the intended sentence and the original data hiding the label with each representative sentence as the center. mark, so that the intent sentence and the original data in the same group have the same label, or when the average semantic distance exceeds the predetermined threshold, the intent sentence with the largest average semantic distance is the new intent The new representative sentence of the category, and the intention sentence and the original data that hide the label are grouped and labeled with all the representative sentence and the new representative sentence as the center, so that the intention sentence and the original data in the same group are grouped and labeled. Profiles have the same label. The dialog data processing system of claim 1, wherein the dialog data grouping and marking module takes a group center of a plurality of semantic vectors corresponding to a plurality of sentences of each of the plurality of intent categories, so as to obtain the The one with the smallest distance from the cluster center among the multiple semantic vectors is taken as the representative sentence. The dialogue data processing system as claimed in claim 1, further comprising a dialogue data enhancement module for data augmentation, when the number of samples for each of the intent categories is insufficient or after comparing the differences in the number of samples among all the intent categories , and data enhancement is performed for the small sample size. The dialogue data processing system of claim 3, wherein the data augmentation uses synonym replacement, random insertion, random exchange, random deletion, data augmentation methods based on machine learning and deep learning, or any combination thereof. The dialogue data processing system according to claim 1, wherein the filter index includes satisfaction feedback value, positive and negative polarity of sentence emotion, confidence level of dialogue text intention, or whether there is a transfer request. The dialogue data processing system of claim 5, wherein the dialogue data acquisition module further comprises an emotion recognition unit, which uses an emotion recognition model to recognize the emotion polarity of the intent sentence to generate the positive and negative emotion polarity of the sentence. The dialogue data processing system of claim 5, wherein the dialogue data acquisition module further comprises an intention recognition unit, which uses an intention recognition model to calculate the intention recognition confidence level of the intention sentence to generate the dialogue text intention confidence level . A dialogue data processing method, comprising: Collect conversation records with sentences; Filter the conversation record according to the filter index to obtain the intent sentence; Select representative sentences from each of the tagged pre-existing data within multiple intent categories; and The intent sentence is compared with each of the representative sentences, so as to obtain the one with the largest average semantic distance from the intended sentence and each of the representative sentences, so that when the largest average semantic distance is less than a predetermined threshold, each The representative sentence is centered on grouping and labeling the intent sentence and the original data that hides the label, so that the intent sentence and the original data in the same group have the same label, or the one with the largest average semantic distance When the predetermined threshold value is exceeded, the intent sentence with the largest average semantic distance is the new representative sentence of the new intent category, and all the representative sentences and the new representative sentence are centered on the intent sentence and the one that hides the label. The original data is grouped and labeled, so that the intent statement and the original data in the same group have the same label. The dialogue data processing method according to claim 8, wherein the step of selecting a representative sentence is to take a group center of a plurality of semantic vectors corresponding to a plurality of sentences of each of the plurality of intention categories, so as to obtain the The one with the smallest distance from the cluster center among the plurality of semantic vectors is taken as the representative sentence. The dialogue data processing method as claimed in claim 9, wherein the step of filtering the dialogue record according to the filtering index to obtain the intent sentence further comprises that the number of samples of each of the intent categories is insufficient or that the samples among all the intent categories are compared. After the difference in the number of samples, data enhancement is performed on the intent categories with a small number of samples. The dialogue data processing method as claimed in claim 10, wherein the data augmentation uses synonym replacement, random insertion, random exchange, random deletion, data augmentation methods based on machine learning and deep learning, or any combination thereof. The dialogue data processing method according to claim 8, wherein the filter index includes satisfaction feedback value, positive and negative polarity of sentence emotion, confidence level of dialogue text intention or whether there is a transfer request. The dialogue data processing method according to claim 12, wherein the positive and negative polarity of the sentence emotion is generated by identifying the emotional polarity of the intention sentence by using an emotion recognition model. The dialogue data processing method according to claim 12, wherein the intention confidence level of the dialogue text is generated by calculating the intention recognition confidence level of the intention sentence by using an intention recognition model. A computer-readable medium used in a computing device or a computer and storing instructions for executing the method for processing session data as described in any one of claims 8 to 14.

Images