TWI737101B - Question-answering learning method and question-answering learning system using the same and computer program product thereof - Google Patents

Abstract

An embodiment of question-answering learning method includes the following steps. A plurality of classifiers are created according to N1 labeled sentences of N sentences. Each of the classifiers determines sentence type of each of unlabeled N2 sentences in the N sentences. According to the degree of inconsistency of the determined sentence types of the classifiers, N3 sentences are selected from the unlabeled N2 sentences, wherein the determined sentence types of the N3 sentences of the classifiers are inconsistent. N4 identified sentences to be labeled that are complementary to each other are selected from the N3 sentences. After the N4 identified sentences are labeled, a plurality of updated classifiers are re-created, according to the labeled N1 sentences and the N4 identified sentences. At least one of the classifiers before re-created is added to the updated classifiers as member(s) of the updated classifiers.

Description

Question-and-answer learning method, question-and-answer learning system and computer using it Program product

This disclosure relates to a learning method and a learning system using it, and in particular, it relates to a question-and-answer learning method and a question-and-answer learning system using it.

In the learning method of conventional question answering, a large number of sentences are usually classified manually, and then a question answering classification model is established accordingly. Question answering systems are generally called automatic question answering systems, dialogue systems, conversation systems, automatic customer service systems, customer service robots, or text interactive assistants, instant messaging robots, etc. In the follow-up Q&A learning, the Q&A classification model will usually hand over all new sentences that cannot be classified to manually mark the sentence types (give corresponding answers to the new sentences). However, such a method requires a lot of manual processing time and the accuracy of the question and answer obtained may not steadily increase.

Therefore, how to reduce manual processing man-hours and stably improve the accuracy of question and answer has become one of the goals of the industry in this technical field.

An embodiment of the present disclosure provides a question-and-answer learning method. The Q&A learning method includes the following steps. A classifier generation module creates a classifier module based on the marked N1 sentences in the N sentences. The classifier module includes several classifiers, each of which represents a different question and answer classification model, where N and N1 are positive Integer; each of these classifiers judges the sentence type of each of the N2 unmarked sentences in the N sentences, where N2 is a positive integer; in the consistency evaluation step, the consistency evaluation module is based on The degree of consistency of the judgment results of these classifiers is that N3 sentences are selected from the N2 sentences that are not marked, and the judgment results of these classifiers for each N3 sentence are inconsistent, where N3 is a positive integer; in complementary In the degree evaluation step, a complementary degree evaluation module selects N4 sentences complementary to each other from the N3 sentences as several sentences to be marked for selection, where N4 is a positive integer; after marking the N4 sentences to be marked for selection, The classifier generation module re-creates these classifiers of the classifier module based on the marked N1 sentences and N4 sentences to be marked; and, a classifier evaluation module of the classifiers before reconstruction At least one of them is added to the classifier module as a member of the classifier module.

Another embodiment of the present disclosure provides a question-and-answer learning system. The question and answer learning system includes a classifier generation module, a consistency degree evaluation module, a complementary degree evaluation module, and a classifier evaluation module. The classifier generation module is used to: create a classifier module based on the marked N1 sentences in the N sentences. The classifier module includes several classifiers, each of which represents a different question and answer classification model, where N and N1 Is a positive integer. Each of these classifiers determines the sentence type of each of the N2 unmarked sentences in the N sentences, where N2 is a positive integer. Consistency Evaluation Module Used for: In the consistency evaluation step, according to the consistency of the judgment results of these classifiers, N3 sentences are selected from the N2 sentences that are not marked, and the judgments of these classifiers on each N3 sentence The results are inconsistent, where N3 is a positive integer. The degree of complementarity evaluation module is used for: in a step of evaluating the degree of complementarity, N4 sentences that are complementary to each other are selected from N3 sentences as a number of sentences to be marked for selection, where N4 is a positive integer. The classifier generation module is further used to: after marking the N4 sentences to be marked for selection, re-build these classifiers of the classifier module based on the marked N1 sentences and the N4 sentences to be marked for selection. The classifier evaluation module is used for adding at least one of the classifiers before reconstruction to the classifier module as a member of the classifier module.

Another embodiment of the present disclosure provides a computer program product. The computer program product is used to be loaded into a question-and-answer learning system to execute a question-and-answer learning method. The Q&A learning method includes the following steps: a classifier generation module builds a classifier module based on the labeled N1 sentences in the N sentences, the classifier module includes several classifiers, and each classifier represents a different question and answer classification model , Where N and N1 are positive integers; each of these classifiers judges the sentence type of each of the N2 unmarked sentences in the N sentences, where N2 is a positive integer; in the consistency evaluation step, The consistency evaluation module selects N3 sentences from the N2 sentences that are not marked according to the consistency of the judgment results of these classifiers, and the judgment results of these classifiers for each N3 sentence are inconsistent. N3 is a positive integer; in the complementary degree evaluation step, a complementary degree evaluation module selects N4 sentences that are complementary to each other from the N3 sentences as a number of sentences to be marked, where N4 is a positive integer; in marking N4 sentences After the sentences to be marked are selected, the classifier generation module is based on the marked N1 sentences and N4 A sentence to be marked is selected, and the classifiers of the classifier module are re-created; and, a classifier evaluation module adds at least one of the classifiers before reconstruction to the classifier module for classification A member of the device module.

In order to have a better understanding of the above and other aspects of the present disclosure, the following examples are specially cited, and the accompanying drawings are described in detail as follows:

100: Question and Answer Learning System

110: Classifier generation module

120: classifier module

130: Consistency Evaluation Module

140: Complementarity Evaluation Module

150: database

160: classifier evaluation module

C1, C2: classifier

S110~S160, S141~S149: steps

N, N1, N2, N3, N4: Quantity

q _i , q ₁ ~q ₁₀₀ : statement

FIG. 1 is a functional block diagram of a question and answer learning system according to an embodiment of the disclosure.

Figure 2 shows a flowchart of an embodiment of the question and answer learning method of the question and answer learning system shown in Figure 1.

FIG. 3 is a schematic diagram of the judgment of the degree of complementarity of sentences according to an embodiment of the present disclosure.

FIG. 4 is a flowchart of judgment of sentence complementarity according to an embodiment of the present disclosure.

FIG. 5 is a schematic diagram of an embodiment of the classifier reconstruction of the question answering learning system 100 in FIG. 1.

Please refer to FIG. 1, which shows a functional block diagram of the question and answer learning system 100 according to an embodiment of the present disclosure. The question and answer learning system 100 includes a classifier generation module 110, a classifier module 120, a consistency evaluation module 130, a complementarity evaluation module 140, a database 150, and a classifier evaluation module 160.

The classifier generation module 110, the classifier module 120, the consistency evaluation module 130, the complementarity evaluation module 140, and the classifier evaluation module 160 include At least one of them can be a combination of one or more of a chip, a circuit, a circuit board, and a recording medium storing an array of program codes formed by a semiconductor process. At least two of the classifier generation module 110, the classifier module 120, the consistency evaluation module 130, the complementarity evaluation module 140, and the classifier evaluation module 160 can be integrated into a single module, or the classifier generation module At least one of the group 110, the classifier module 120, the consistency evaluation module 130, the complementarity evaluation module 140, and the classifier evaluation module 160 can be integrated into a processor or a controller middle. In addition, the database 150 can be stored in a storage module, such as a memory.

The classifier generation module 110 _{creates a classifier module 120 based on the marked N1 sentences in the N sentences q i} . The classifier module 120 includes several classifiers C1, and each classifier C1 represents a different question and answer. Classification model, where N and N1 are positive integers. Each classifier is used to determine the sentence type of each of the N2 unmarked sentences in the N sentences, where N2 is a positive integer. The consistency evaluation module 130 is used in the consistency evaluation step to select N3 sentences from the N2 unmarked sentences according to the consistency of the judgment results of these classifiers C1, among which these classifiers C1's judgment results for each N3 sentence are inconsistent, and N3 is a positive integer. The complementarity evaluation module 140 is used to select N4 complementary sentences from the N3 sentences as a number of sentences to be marked for selection in the complementary degree evaluation step, where N4 is a positive integer. The content words and/or context of the N4 sentences are, for example, non-repetitive, dissimilar to each other, not implying each other, and/or not generating each other. After marking N4 sentences to be marked for selection, the classifier generation module 110 re-creates several classifiers C2 (classifier C2) of the classifier module 120 according to the marked N1 sentences and N4 sentences to be marked for selection. An embodiment is shown in Figure 5). The classifier evaluation module 160 adds at least one of the classifiers C1 before reconstruction to the classifier module 120 as a member of the classifier module 120. In one embodiment, N1 is less than N, N2 is less than N, N3 is not greater than N2, and N4 is not greater than N3, but the embodiment of the present disclosure is not limited thereto. In addition, the "sentences" in this article are, for example, natural language descriptive sentences, natural language question sentences or oral sentences, question sentences, straightforward sentences, or any other grammatical/sentence sentences.

In addition, the classifier generation module 110 can use Neural Networks (NN) technology, Deep Neural Networks (DNN) technology, or Support Vector Machine (Support Vector Machine) technology for training and learning. The question and answer learning system 100 can actively select a small number of sentences from unmarked sentences to be manually labeled, and then feed them to the classifier generation module 110 for relearning. The classifier generation module 110 builds at least one classifier C2 based on the manual marking results. . In other words, the question answering learning method of the question answering learning system 100 is a machine active learning method.

Compared with marking a large number of N sentences, in the question-and-answer learning system of the disclosed embodiment, the manual only needs to mark (for example, provide a response to the sentence, such as answer sentence, answer, function menu, pattern and/or picture) by The sentence type of the N4 sentences selected by the question and answer learning system 100, wherein the value of N4 is less than the value of N, so it can save a lot of time for manual marking. In addition, since the N4 sentences selected by the question and answer learning system 100 have a low degree of consistency and a high degree of complementarity with previously marked sentences, the probability of manually repetitively marking sentences of the same sentence type can be reduced, and the performance of the question and answer learning system 100 can be improved. Overall Q&A accuracy rate. Furthermore, since at least one of the classifiers before reconstruction is added to the members of the classifier module after reconstruction, in the iterative update process, the question and answer The accuracy rate can maintain a stable improvement, reducing the chance of system performance degradation due to updates, thus improving the management convenience of the question and answer system.

For example, in an example, the value of N is for example 10000, N1 is for example 100, N2 is for example 100 of 9900 (N-N1=9900), N3 is for example 8, and N4 is for example 3. indivual. In this example, the manual only needs to mark 3 sentences (that is, the value of N4), and compared with the previously marked sentences, the consistency of these 3 sentences is low and the degree of complementarity is high. They are meaningful markings and can be effective Improve the overall Q&A accuracy rate of the Q&A learning system 100. In addition, the embodiments of the present disclosure do not limit the aforementioned values of N, N1, N2, N3, and N4, which may be larger or smaller values.

The process of selecting N4 sentences by the question answering learning system 100 will be further explained below. Please refer to FIG. 2, which shows a flowchart of an embodiment of the question and answer learning method of the question and answer learning system 100 in FIG. 1.

In step S110, the classifier generation module 110 creates a classifier module 120 based on the marked N1 sentences in the N sentences. The classifier module 120 includes C1 classifiers, and each C1 classifier represents a different question and answer. Classification model, where N and N1 are positive integers and N1 is less than N. In other words, the active learning question answering method of the disclosed embodiment may not mark all N sentences one by one, and only select a smaller number of meaningful (that can improve the accuracy of question answering) sentences (ie N4) from a relatively small number of N1 sentences. Just mark it.

In step S120, each classifier C1 determines the sentence type of each of the N2 unmarked sentences in the N sentences, where N2 is a positive integer and N2 is less than N.

Taking N2 as 100 for example, each classifier C1 determines the sentence type of each of _{the N2 unlabeled sentences q 1} ~q _100. Taking one sentence q ₁ as an example, each classifier C1 judges the sentence _{type of the sentence q 1.} When all the classifiers C1 consistently judge that the sentence q ₁ belongs to the same sentence type, the sentence q _{1 is} defined as having consistency (Or completely consistent); when not all classifiers C1 consistently judge that the sentence q ₁ belongs to the same sentence type, the sentence q _{1 is} defined as inconsistent (that is, inconsistent).

In step S130, in the consistency evaluation step, the consistency evaluation module 130 selects N3 sentences from the N2 unmarked sentences according to the consistency of the judgment results of these classifiers C1, where this The judgment results of some classifiers C1 for each N3 sentence are inconsistent, where N3 is a positive integer and not greater than N2. For a sentence, the more sentence types judged by these classifiers C1, the higher the inconsistency of the sentence (or the lower the degree of consistency); on the contrary, for a sentence, when these classifiers The fewer sentence types judged by C1, the lower the inconsistency of the sentence (or the higher the degree of consistency).

This embodiment takes as an example that eight sentences q ₁ to q _{8 among the N3 unmarked sentences q 1} to q ₁₀₀ are inconsistent for subsequent description.

In step S140, in the complementary degree evaluation step, the complementary degree evaluation module 140 _{selects N4 sentences complementary to each other from the N3 sentences q 1} to q ₈ as several sentences to be marked for selection, where N4 is a positive integer, And N4 is not greater than N3. Depending on how _{complementary the N3 sentences q 1} to q ₈ are to each other, the number of N4 confirmed sentences to be marked may be equal to or less than N3.

For example, please refer to FIGS. 3 and 4. FIG. 3 shows a schematic diagram of sentence complementarity judgment according to an embodiment of the present disclosure, and FIG. 4 shows a flow of judgment of sentence complementarity according to an embodiment of the present disclosure. picture.

In step S141, the complementarity evaluation module 140 can sort the N3 sentences according to the degree of consistency. For example, as shown in Figure 3, the complementarity evaluation module 140 arranges sentences into q ₄ , q ₆ , q ₃ , q ₈ , q ₅ , q ₂ , q in the order of the degree of consistency from low to high. ₇ and q ₁ .

Then, in step S142, as shown in FIG. 3, in the first batch selection, the complementarity evaluation module 140 selects the sentence q _{4 with the} highest inconsistency as the sentence to be marked for selection.

Then, in step S143, the complementarity evaluation module 140 sets the initial value of i to be equal to 2.

Then, in step S144, in the second batch of selection, the complementarity evaluation module 140 compares the complementarity of the sentence with the second highest inconsistency among the N3 sentences and the sentences to be marked for selection. For example, the degree of complementarity evaluation module 140 compares the degree of complementarity between the i-th (i-th highest inconsistency) sentence q _{6 and the sentence q 4} to be marked for selection.

Then, in step S145, the complementarity evaluation module 140 determines whether the sentence with the second highest inconsistency (such as the i-th highest) is complementary to each sentence to be marked. For example, the complementarity evaluation module 140 determines whether the sentence q ₆ is complementary _{to the sentence q 4} to be marked for selection. When the sentence q _{6 is complementary to the sentence q 4} to be marked for selection, the flow proceeds to step S146. In addition, when the sentence with the second highest inconsistency (such as the i-th highest) is not complementary to the sentences to be marked for selection, the complementarity evaluation module 140 does not consider the sentence with the second highest inconsistency as the sentence to be marked for selection. member. For example, when the sentence q _{6 is} not complementary to the sentence to be marked for selection q ₄ , the complementarity evaluation module 140 does not use the sentence q ₆ as a member of the sentence to be marked for selection, for example, the sentence q ₆ is ignored, and then the flow enters Step S147.

In step S146, when the sentence with the second highest inconsistency (such as the i-th highest) is complementary to each sentence to be marked for selection, the complementarity evaluation module 140 uses the sentence with the second highest inconsistency as the sentence to be marked for selection. The members of the statement. For example, as shown in FIG. 3, when the sentence q _{6 is} complementary to the sentence to be marked for selection q ₄ , the complementarity evaluation module 140 selects the sentence q ₆ as the sentence to be marked for selection.

Then, in step S147, the complementarity evaluation module 140 determines whether the number of sentences to be marked for selection has reached N4 or there are no sentences to be selected (for example, i is equal to N3). When the number of sentences to be marked for selection has reached N4 or there are no sentences to choose from, the process goes to step S149; when the number of sentences for which to be marked for selection does not reach N4 or i is not equal to N3, the process goes to In step S148, the value of i is accumulated and the flow returns to step S144 to continue the judgment of the complementarity of the next sentence.

As shown in Figure 3, since the number of sentences to be marked is only 2 selected in the second batch selection, which does not reach N4 (for example, 3), the judgment of the complementarity of the next sentence is continued. For example, after accumulating the value of i (i=3), the complementarity evaluation module 140 compares the i-th sentence (the third highest inconsistency) sentence q ₃ with the sentence q ₄ and q ₆ to be marked. Complementary degree. As shown in the figure, although the sentence q _{3 is} complementary to the sentence to be marked for selection q ₄ , when the sentence q _{3 is} not complementary to another sentence to be marked for selection q ₆ , the complementarity evaluation module 140 ignores (or discards) The sentence q ₃ does not use the sentence q ₃ as a member of the sentence to be marked.

As shown in Figure 3, since the number of sentences to be marked in the third batch of selection is still only 2, which does not reach N4 (N4 is 3, for example), the judgment of the complementarity of the next sentence is continued. For example, after accumulating the value of i (i=4), the complementarity evaluation module 140 compares the i-th (4th highest inconsistency) sentence q ₈ with the sentences q ₄ and q ₆ to be marked. Complementary degree. As shown in the figure, since the sentence q _{8 is complementary to all the sentences q 4} and q ₆ to be marked for selection, the complementarity evaluation module 140 selects the sentence q ₈ as the sentence to be marked for selection.

As shown in FIG. 3, the number of sentences to be marked in the fourth batch selection has reached N4 (N4 is 3, for example) or there are no sentences to be selected, so the process goes to step S149 to stop the complementation degree judgment step. Then, the manual only needs to mark the sentences to be marked. Compared with the larger number of N, N1, N2, and N3, the manual only needs to mark a relatively small number of N4 sentences to be marked, which saves a lot of processing man-hours. In addition, since these selected sentences to be marked are highly complementary (or complementary) and low in consistency (or inconsistent) compared with other marked sentences, these sentences to be marked for selection are added to the question-and-answer learning system 100. , Which can effectively and steadily improve the accuracy of question and answer.

The following explains the method of judging whether the two sentences are complementary.

The complementarity of the two sentences indicates the amount of information that the two sentences do not repeat each other, that is, the content words and/or the context are not repeated, dissimilar to each other, do not imply each other, and/or do not generate each other. The repetition degree value of the amount of information is based on different text analysis methods and adopts a binary value (0 means no implication, and 1 means implication), a proportional value, or a probability value. For example, using literal implication to identify the logical inference relationship between the two sentences q1 and q2, if the sentence q1 can infer the complete meaning of the sentence q2, (that is, the sentence q1 semantic implication sentence q2), It means that the information amount of the sentence q1 already includes the information amount of the sentence q2, so the two sentences q1 and q2 are not complementary, and the opposite is complementary. In addition, the amount of repetitive information of the two sentences can be measured by repeated words. If the words of sentence q1 and the words of sentence q2 are repeated more (for example, 60%, but the embodiment of the present invention is not limited by this), It means that the higher the amount of repetitive information of the two sentences q1 and q2, the lower the degree of complementarity between the two sentences q1 and q2 (for example, 40%, but the embodiment of the present invention is not limited by this). In addition, word repetition analysis can be expanded by similar words, antonyms, related words, similar words, semantic word nets, ontology word nets, proper noun recognition, word embedding, etc., that is, if the semantics of sentence q1 and sentence q2 are similar The higher the degree, the lower the complementarity of the two sentences q1 and q2. Alternatively, the amount of repetitive information of the two sentences can be measured by a language model. If the language model of sentence q1 can generate sentence q2, the higher the probability value (for example, 60%, but the embodiment of the present invention is not limited by this), it means two sentences The higher the amount of repeated information of q1 and q2, the lower the degree of complementarity between the two sentences q1 and q2 (for example, 40%, but the embodiment of the present invention is not limited by this).

Taking the sentence q1 as "Taipei 101 has a bus stop" and sentence q2 as "You can take public transportation to Taipei 101", the sentence q1 implies sentence q2, and the recognition result of the two can be 1 to represent the implication (binary value) ( Information repetition means non-complementarity), or a probability value of 90%, for example, indicates a high possibility of implication (a high degree of information repetition means a low degree of complementarity).

In summary, the degree of complementarity of two sentences can be judged according to one of the following or a combination: (1). The degree of repetition of words in the second sentence, the degree of repetition of synonyms or synonyms or antonyms or related words of the words, and the related word network of words The degree of repetition, the similarity of the meaning of the word, the similarity of the upper and lower ontology concepts of the word, the degree of the word The degree of similarity of related words, the degree of similarity of the related word network of words; (2). The degree of repetition of phrases, clauses or proper nouns in the second sentence; (3). The phrase, clause or proprietary of the second sentence The degree of similarity of nouns; (4). The degree of similarity of the word embedding (word vector) of the two sentences; (5). The degree of similarity of the sentence patterns of the two sentences; (6). The degree of semantic similarity of the two sentences; (7). The implication relation of sentences; (8). The implied probability of the two sentences; (9). The similarity of the language model of the two sentences.

In one embodiment, the complementarity evaluation module 140 can determine the complementarity of two sentences based on the same concept (or synonymous) word and sentence pattern of the sentence. For example, when the two sentences are "How to go to Taipei" and "How to go to Tianlongguo", because "Taipei" and "Tianlongguo" belong to the same conceptual (or synonymous) word and the sentence patterns of the two sentences are the same, the degree of complementarity is assessed The module 140 determines that the two sentences are not complementary (the degree of complementarity is low). In another example, when the two sentences are "How to go to Taipei" and "What activities are there in Taipei", the two sentences have different sentence patterns, so the complementarity assessment module 140 judges the two sentences as complementary (degree of complementarity). high). In summary, the complementarity evaluation module 140 uses sentence text analysis technology to determine the complementarity of two sentences.

Next, in step S150 in FIG. 2, please also refer to FIG. 5, which shows a schematic diagram of the update of the classifier of the question answering learning system 100 in FIG. 1. After manually marking the selected N4 sentences to be marked, the classifier generation module 110 re-creates several classifiers C2 of the classifier module 120 according to the marked N1 sentences and N4 sentences to be marked. . In addition, the classifier generation module 110 may pre-store the classifier C1 of the previous iteration (each process of rebuilding a batch of classifiers is called an iteration) in the database 150.

Then, in step S160, as shown in FIG. 5, the classifier evaluation module 160 adds at least one of the classifiers C1 before reconstruction to the classifier module 120 to be used as the classifier module 120 member. In other words, the classifier evaluation module 160 can add at least one of the plurality of classifiers C1 generated in the previous iteration to the plurality of classifiers C2 generated in the current iteration as a classifier model. Member of several classifiers in group 120. Since different classifiers represent different question and answer classification models, adding several classifiers C1 generated by the previous iteration can reduce the impact of the newly selected sentence to be marked on the accuracy of the question and answer to the range not covered by the classifier C1. The possibility that the Q&A accuracy performance will be unstable or decreased after the classifier is updated is reduced, and the Q&A accuracy rate of the Q&A learning system 100 can be stably improved.

The following describes multiple embodiments in which the classifier evaluation module 160 adds at least one classifier C1 of the previous iteration. The classifier evaluation module 160 can be based on the classification accuracy rate of the classifier C1, the number of iterations, the attenuation rate of the number of iterations, the upper or lower limit of the members of the classifier module, all the classifiers of each iteration, or the above conditions的组合。 The combination.

For example, in one embodiment, the classifier evaluation module 160 determines the composition of the classifier module members 120 as follows: retain all the classifiers C1 of all previous iterations. For example, if the classifier module members of the first iteration have 4 classifiers, then the second iteration has 8 classifiers, 4 of which are all 4 classifiers of the first iteration, and the first iteration has 8 classifiers. The three-generation classifier module has 12 classifiers, 4 of which are the 4 classifiers of the first iteration and the other 4 are the 4 classifiers of the second iteration.

In another embodiment, the classifier evaluation module 160 determines the composition of the classifier module members 120 as follows: only retain all the classifiers C1 of the previous iteration. For example, if the classifier module members of the first iteration have 4 classifiers, then the second iteration has 8 classifiers, 4 of which are the 4 classifiers of the first iteration, and the third The members of the classifier module of the iteration level are still 8 classifiers, of which 4 are the 4 classifiers of the second iteration.

In other embodiments, the classifier evaluation module 160 determines the composition of the classifier module members 120 as follows: only keep the top n ranked by the classification accuracy of the classifier C1 in the previous iteration, where n is the middle class. The number between 1% and 50% of the number of classifiers in the previous iteration; or, n is any positive integer between 1 and the number of classifiers in the previous iteration. For example, if the classifier module members of the first iteration have 4 classifiers, then there are 6 classifiers in the second iteration, and 2 of them are classified in the 4 classifiers of the first iteration. The top 2 ranked by correct rate.

To sum up, in the question and answer learning method of the embodiment of the present disclosure, the manual only needs to mark a relatively small number of selected sentences to be marked, so a large amount of manual processing man-hours can be saved. In addition, since the previous classifier can be retained during the learning iterative process, the sentence to be marked is highly complementary (or complementary) and low in consistency (or inconsistent) compared with other marked sentences, so it can ensure that every After adding the selected sentences to be marked for a generation to the question and answer learning system, they can steadily and continuously improve the accuracy of the question and answer.

To sum up, although the present disclosure has been disclosed as above through the embodiments, it is not intended to limit the present disclosure. Those who have general knowledge in the technical field to which this disclosure pertains will not deviate from Within the spirit and scope of this disclosure, various changes and modifications can be made. Therefore, the scope of protection of this disclosure shall be subject to the scope of the attached patent application.

100: Question and Answer Learning System

110: Classifier generation module

120: classifier module

130: Consistency Evaluation Module

140: Complementarity Evaluation Module

150: database

160: classifier evaluation module

C1: classifier

N1: Quantity

q _i , q ₁ ~q ₁₀₀ : statement

Claims (17)

A question-and-answer learning method, including: a classifier generation module creates a classifier module based on N1 of the N sentences that have been marked. The classifier module includes a plurality of classifiers, each of which represents a different Question and answer classification model, where N and N1 are positive integers; each of these classifiers judges the sentence type of each of the N2 unmarked sentences in the N sentences, where N2 is a positive integer; in agreement In the sexual degree evaluation step, the consistency degree evaluation module selects N3 sentences from the unmarked N2 sentences according to the consistency degree of the judgment results of the classifiers, and the classifiers perform the same for each of the N3 sentences. The judgment results of the sentences are inconsistent, and N3 is a positive integer; in the complementary degree evaluation step, a complementary degree evaluation module selects N4 sentences complementary to each other from the N3 sentences as plural sentences to be marked for selection , Where N4 is a positive integer; after marking the selected sentences to be marked, the classifier generation module re-creates the classifier module based on the marked N1 sentences and the selected sentences to be marked Some classifiers; and a classifier evaluation module adds at least one of the classifiers before the re-creation to the classifier module as a member of the classifier module; wherein the two complementary sentences are each other There is no repetitive amount of information. For example, the question-and-answer learning method described in item 1 of the scope of patent application includes: The N3 sentences are sorted in the order of the degree of consistency from high to low. For example, the question and answer learning method described in item 1 of the scope of patent application, wherein the complementary degree evaluation step includes: the complementary degree evaluation module selects the sentence with the highest inconsistency among the N3 sentences as the selected sentence to be marked member. For example, the question and answer learning method described in item 3 of the scope of patent application, wherein the complementary degree evaluation step includes: the complementary degree evaluation module compares the sentence with the second highest inconsistency among the N3 sentences and the sentence to be marked. The degree of complementarity of each; when the sentence with the second highest inconsistency is complementary to each of the sentences to be marked for selection, the complementary degree evaluation module regards the sentence with the second highest inconsistency as the sentence These confirm the members of the sentence to be marked. For example, the question-and-answer learning method described in item 4 of the scope of patent application, wherein the complementary degree evaluation step includes: when the sentence with the second highest inconsistency is not complementary to any one of the selected sentences to be marked, the complementary The degree evaluation module does not use the sentence with the second highest inconsistency as a member of the sentences to be marked. For example, the question and answer learning method described in item 1 of the scope of patent application, wherein the complementary degree evaluation step includes: (a) the complementary degree evaluation module selects the sentence with the highest inconsistency among the N3 sentences as the candidates for confirmation Mark the members of the sentence; (b) The complementarity evaluation module compares the complementarity of the i-th highest inconsistency among the N3 sentences with each of the sentences to be marked, where the initial value of i is equal to 2; (c) When the sentences with the i-th highest inconsistency are complementary to each of the sentences to be marked for selection, the complementary degree evaluation module uses the sentences with the i-th highest inconsistency as the sentences to be marked for selection. The members of the statement; and (d) when the value of i is not equal to N4, add up the value of i, and then repeat steps (b) ~ (d). For example, the question-and-answer learning method described in item 6 of the scope of patent application, wherein the complementary degree evaluation step includes: the complementary degree evaluation module judges whether the number of selected sentences to be marked has reached N4; If the number of marked sentences reaches N4 or there are no sentences to choose from, the complementary degree evaluation module stops executing the complementary degree evaluation step. The question-and-answer learning method described in item 6 of the scope of patent application, wherein step (c) includes: analyzing the meaning of the sentence with the highest inconsistency and the meaning of each of the sentences to be marked; and When the context of the sentence with the i-th highest inconsistency is complementary to the context of any one of the selected sentences to be marked, the complementarity evaluation module uses the sentence with the i-th highest inconsistency as the sentence These confirm the members of the sentence to be marked. A question-and-answer learning system, including: A classifier generation module is used to create a classifier module based on the marked N1 of the N sentences. The classifier module includes a plurality of classifiers, each of which represents a different question and answer classification model, Where N and N1 are positive integers; each of the classifiers judges the sentence type of each of the N2 unmarked sentences in the N sentences, where N2 is a positive integer; the consistency evaluation module Used for: In the consistency evaluation step, according to the consistency of the judgment results of the classifiers, N3 sentences are selected from the N2 unlabeled sentences, wherein the classifiers perform each of the N3 sentences The judgment results of sentences are inconsistent, where N3 is a positive integer; a complementary degree evaluation module is used to: in a complementary degree evaluation step, select N4 sentences that are complementary to each other from the N3 sentences as plural confirmations The sentence to be marked, where N4 is a positive integer; wherein, the classifier generation module is further used to: after marking the sentences to be marked for selection, according to the marked N1 sentences and the sentences to be marked for selection, Re-establish the classifiers of the classifier module; and wherein, the question and answer learning system further includes: a classifier evaluation module for adding at least one of the classifiers before the re-establishment to the classifier module In the group, as a member of the classifier module; among them, the two complementary sentences have no overlapping information. For example, in the question-and-answer learning system described in item 9 of the scope of patent application, the complementary degree evaluation module is further used to sort the N3 sentences in the order of inconsistency from high to low. For example, in the question and answer learning system described in item 9 of the scope of patent application, the complementary degree evaluation module is further used to select the sentence with the highest inconsistency among the N3 sentences as the members of the sentences to be marked. For example, the question-and-answer learning system described in item 11 of the scope of patent application, wherein the degree of complementarity assessment module is further used to: compare the sentence with the second highest inconsistency among the N3 sentences with each of the sentences to be marked. Complementary degree; when the sentences with the second highest inconsistency are complementary to each of the sentences to be marked for selection, the sentences with the second highest inconsistency are used as members of the sentences to be marked for selection. For example, the question and answer learning system described in item 12 of the scope of patent application, wherein the degree of complementarity assessment module is further used: when the sentence with the second highest inconsistency is not complementary to any one of the selected sentences to be marked, The sentence with the second highest inconsistency is not used as a member of the sentences to be marked. For example, the question-and-answer learning system described in item 9 of the scope of patent application, wherein the degree of complementarity assessment module is further used to: (a) select the sentence with the highest inconsistency among the N3 sentences as the members of the selected sentences to be marked (B) Compare the complementarity of the i-th highest inconsistency among the N3 sentences with each of the sentences to be marked, where the initial value of i is equal to 2; (c) When the i-th sentence with the highest inconsistency is complementary to each of the sentences to be marked for selection, use the sentence with the highest inconsistency as the member of the sentences to be marked for selection ; And (d) When the value of i is not equal to N4, accumulate the value of i, and then repeat steps (b) ~ (d). For example, in the question-and-answer learning system described in item 14 of the scope of patent application, the complementary degree evaluation module is further used to determine whether the number of sentences to be marked has reached N4 or there are no sentences to be selected; and when the sentences are confirmed The number of selected sentences to be marked reaches N4, and the execution of the complementary degree evaluation step is stopped. For example, the question-and-answer learning system described in item 14 of the scope of patent application, wherein the degree of complementarity evaluation module is further used in step (c) to analyze the meaning of the sentence with the highest inconsistency and the sentences to be marked for selection The context of each of the sentences; and when the context of the sentence with the highest inconsistency is complementary to the context of any of the sentences to be marked, the complementarity evaluation module will be inconsistent The sentence with the highest sex is the member of the sentence to be marked. A computer program product for loading into a question-and-answer learning system to implement a question-and-answer learning method. The question-and-answer learning method includes: a classifier generation module creates a classifier based on N1 of the N sentences that have been marked Module, the classifier module includes a plurality of classifiers, each of which represents a different question and answer classification model, where N and N1 are positive integers; Each of the classifiers judges the sentence type of each of the N2 unmarked sentences in the N sentences, where N2 is a positive integer; in the consistency evaluation step, the consistency evaluation module According to the consistency of the judgment results of the classifiers, N3 sentences are selected from the unlabeled N2 sentences, wherein the judgment results of the N3 sentences by the classifiers are inconsistent, and N3 is positive Integer; in the complementary degree evaluation step, a complementary degree evaluation module selects N4 sentences that are complementary to each other from the N3 sentences as plural sentences to be marked for confirmation, where N4 is a positive integer; in marking the confirmation After the sentences to be marked are selected, the classifier generation module re-creates the classifiers of the classifier module based on the marked N1 sentences and the sentences to be marked; and a classifier evaluation module will At least one of the classifiers before the re-creation is added to the classifier module as a member of the classifier module; wherein the two complementary sentences do not have overlapping information with each other.

Images