CN114417887A

CN114417887A - Natural language inference method and device fusing semantic parsing

Info

Publication number: CN114417887A
Application number: CN202210281854.6A
Authority: CN
Inventors: 杜振东
Original assignee: Nanjing Yunwen Network Technology Co ltd
Current assignee: Nanjing Yunwen Network Technology Co ltd
Priority date: 2022-03-22
Filing date: 2022-03-22
Publication date: 2022-04-29
Anticipated expiration: 2042-03-22
Also published as: CN114417887B

Abstract

The invention provides a natural language inference method and a device fusing semantic analysis, wherein the method comprises the following steps: obtaining a precondition T and a corresponding hypothesis Q thereof; semantic analysis is respectively carried out on the premises T and the hypothesis Q, and a plurality of features are obtained; unifying and fusing the plurality of features to obtain fused feature input _ i; and inputting the fusion characteristic input _ i into a JudgeNLIModel model, and outputting an answer of a corresponding hypothesis. The invention combines natural language understanding to extract specific characteristics in the premises and the hypothesis, thereby helping to infer the relationship between the premises and the hypothesis, improving the accuracy of natural language inference and providing an improved reference direction for the subsequent natural language inference method.

Description

Natural language inference method and device fusing semantic parsing

Technical Field

The invention relates to the technical field of language identification, in particular to a natural language inference method and a natural language inference device integrating semantic parsing.

Background

Natural Language Inference (NLI), also known as text implication recognition, is one of the most important issues in Natural Language processing, which requires the Inference of a logical relationship between two given sentences. This task presents two sentences, called preconditions and hypotheses, respectively, with the goal of determining the logical relationship between them as necessary, neutral or contradictory.

The traditional natural language inference method comprises text similarity, text alignment, logic calculation and text conversion. The text similarity is to calculate the similarity between the premise and the hypothesis to judge whether the premise and the hypothesis form an implication relation; the text alignment is to align the parts with similar premises and assumptions, and then to use logistic regression to analyze implications and perform conflict detection; the logic calculation is to express the text into a mathematical logic expression to form a fact set, and then a logic inference rule judges whether to infer a hypothesis according to the premise; the text conversion is to rewrite the premises and the hypotheses into a syntax tree, then design an inference rule by using background knowledge, rewrite the premises, and if the premises can be rewritten into a assumed syntax tree form, the implication relationship is considered to exist.

However, the above method relies heavily on transformation rules, some from the knowledge base and some from the corpus. In addition, when a piece of premise and related assumptions are given, the traditional natural language inference method, such as text similarity, cannot find differences in numerical comparison of two pieces of text. For example: the premise is that the full price ticket needs to be purchased when the height is larger than 1.3 meters, and the problem is that the full price ticket needs not to be purchased when the height is not larger than 1.2 meters, the text is not obviously different, only one figure is different, and the model can be judged to be consistent by mistake under many conditions.

Disclosure of Invention

In view of the above problems, the present invention provides a natural language inference method and apparatus fusing semantic parsing.

In order to solve the technical problems, the invention adopts the technical scheme that: a natural language inference method fusing semantic parsing comprises the following steps: obtaining a precondition T and a corresponding hypothesis Q thereof; semantic analysis is respectively carried out on the premises T and the hypothesis Q, and a plurality of features are obtained; unifying and fusing the plurality of features to obtain fused feature input _ i; and inputting the fusion characteristic input _ i into a JudgeNLIModel model, and outputting an answer of a corresponding hypothesis.

As a preferred scheme, semantic parsing is performed on the precondition T and the hypothesis Q respectively, specifically: and respectively identifying characters and units in the premises T and the premises Q by using a regular expression rule to obtain a plurality of characteristics.

Preferably, assuming that the precondition T performs semantic analysis to obtain a plurality of features T1, T2 … tn, and assuming that Q performs semantic analysis to obtain a plurality of features Q1, Q2 … qn, and the Q is represented by serialization, the fusion feature input _ i is [ CLS ] T [ SEP ] [ nlu _ T ] T1 [ SEP ] [ nlu _ T ] T2 [ SEP ] … [ nlu _ T ] tn [ SEP ] Q [ SEP ] [ nlu _ Q ] Q1 [ SEP ] [ nlu _ Q ] Q2 [ SEP ] … [ nlu _ Q ] qn [ SEP ], wherein each feature includes a start value num _ begin, a start value close condition num _ begin _ interval, an end value num _ end, an end value close condition num _ end _ interval, and a unit feature unit, and the content of the end is replaced by no.

Preferably, the unification is to unify units in the premise T and the hypothesis Q.

As a preferred scheme, the construction of the Judge NLIModel model comprises the following steps: analyzing the document, collecting a plurality of pairs of preconditions T and hypotheses Q, and generating a data set D; dividing the data set D into a training set, a test set and a verification set according to a preset proportion; respectively extracting the characteristics of the premises T and the hypothesis Q in the training set, the test set and the verification set to obtain the characteristics of the training set, the test set and the verification set; after the characteristics of the training set are unified and fused, inputting the training set into an initial model for training; setting an epoch turn of the initial model, outputting the model once every epoch turn, and storing the model as a JudgeNLIModels model; inputting the verification set characteristics into Judge NLIModels models, evaluating all Judge NLIModels models by adopting an f1 index, and selecting the model with the best effect, namely the Judge NLIModel model; and testing the JudgeNLIModel model by using the test set characteristics to obtain the effect of the JudgeNLIModel model.

Preferably, the calculation formula of the f1 index is as follows:

wherein, P is accuracy and R is recall.

Preferably, the initial model is a BERT model, which includes an input layer, an embedding layer, a bidirectional layer and an output layer, wherein the embedding layer is obtained by summing word embedding, segment embedding and position embedding; in the position embedding, the position coding of the features in the preconditions T and hypotheses Q is kept consistent with the position of their values.

The invention also provides a natural language inference device fused with semantic parsing, which comprises: the acquisition module is used for acquiring the preconditions T and the corresponding hypotheses Q; the analysis module is used for performing semantic analysis on the premises T and the hypothesis Q respectively to obtain a plurality of characteristics; the fusion module is used for unifying and fusing the plurality of characteristics to obtain fused characteristics input _ i; and the input and output module is used for inputting the fusion characteristic input _ i into the JudgeNLIModel model and outputting answers corresponding to the hypotheses.

As a preferred scheme, the system further comprises a model building module, wherein the model building module comprises: the data set generating unit is used for analyzing the document, acquiring a plurality of pairs of preconditions T and hypotheses Q and generating a data set D; the data set dividing unit is used for dividing the data set D into a training set, a test set and a verification set according to a preset proportion; the characteristic extraction unit is used for respectively extracting the characteristics of the preconditions T and the hypothesis Q in the training set, the test set and the verification set to obtain the characteristics of the training set, the test set and the verification set; the model training unit is used for inputting the training set characteristics into an initial model for training after the characteristics of the training set are unified and fused; the model output unit is used for setting an epoch turn of the initial model, outputting the model once every epoch turn, and storing the model as a Judge NLIModels model; the model selection unit is used for inputting the verification set characteristics into Judge NLIModels models, evaluating all Judge NLIModels models by adopting an f1 index, and selecting the model with the best effect, namely the Judge NLIModel model; and the model testing unit is used for testing the JudgeNLIModel model by utilizing the test set characteristics to obtain the effect of the JudgeNLIModel model.

Compared with the prior art, the invention has the beneficial effects that: by adopting the natural language inference method fusing semantic analysis, the features are extracted from a plurality of preconditions and hypotheses and feature fusion is carried out, so that the relationship between the preconditions and the hypotheses is deduced, the accuracy of natural language inference is improved, and the problem that the relationship between the preconditions and the two sections of assumed texts is judged by using a method for calculating text similarity in the traditional natural language inference, and the wrong inference is generated when the two sections of texts have partial numerical difference is solved. Meanwhile, the method can support natural language inference of data in a specific field, and can realize the natural language inference in the specific field as long as the labeled data in the specific field is subjected to model training, so that the applicability of the method is further improved.

Drawings

The disclosure of the present invention is illustrated with reference to the accompanying drawings. It is to be understood that the drawings are designed solely for the purposes of illustration and not as a definition of the limits of the invention. In the drawings, like reference numerals are used to refer to like parts. Wherein:

FIG. 1 is a flow chart of a natural language inference method incorporating semantic parsing according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of a JudgenNLIModel model constructed according to an embodiment of the present invention;

FIG. 3 is an architecture diagram of the JudgeNLIModel model according to an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of an embedding layer in the JudgeNLIModel model according to the embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a natural language inference apparatus incorporating semantic parsing according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a model building module according to an embodiment of the present invention.

Detailed Description

It is easily understood that according to the technical solution of the present invention, a person skilled in the art can propose various alternative structures and implementation ways without changing the spirit of the present invention. Therefore, the following detailed description and the accompanying drawings are merely illustrative of the technical aspects of the present invention, and should not be construed as all of the present invention or as limitations or limitations on the technical aspects of the present invention.

An embodiment according to the present invention is shown in connection with fig. 1. A natural language inference method fusing semantic parsing comprises the following steps:

s101, obtaining a precondition T and a corresponding hypothesis Q.

For example: the premise T is that the online monitoring device for the dissolved gas in the transformer oil needs routine verification at regular intervals, and the verification period is 1 year-2 years. Regarding the assumption Q of the premise T, the periodic routine verification period required for the online monitoring device of the dissolved gas in the transformer oil is 3 years to 4 years.

S102, semantic analysis is respectively carried out on the premises T and the hypothesis Q, and a plurality of features are obtained. The method specifically comprises the following steps: and respectively identifying characters and units in the premises T and the premises Q by using a regular expression rule to obtain a plurality of characteristics. Regular expression rules are set by the regular expression rules according to needs.

For example: the premise T is that the online monitoring device for the dissolved gas in the transformer oil needs routine verification at regular intervals, the verification period is 1 year-2 years, and the premise T is subjected to semantic analysis to obtain a numerical characteristic as follows: [ nlu _ t ]1, include, 2, include, year [ SEP ].

S103, unifying and fusing the plurality of features to obtain fused feature input _ i.

The unification is to unify unit features in the preconditions T and the assumptions Q. For example: the premise T is "height 1.7 m", and the premise Q is "height 170 cm", and the features in the premise Q need to be converted into "1.7 m".

In the embodiment of the invention, a precondition T is assumed to carry out semantic analysis to obtain a plurality of characteristics T1 and T2 … tn, a Q is assumed to carry out semantic analysis to obtain a plurality of characteristics Q1 and Q2 … qn, and after serialization representation, fusion characteristics input _ i are [ CLS ] T [ SEP ] [ nlu _ T ] T1 [ SEP ] [ nlu _ T ] T2 [ SEP ] … [ nlu _ T ] tn [ SEP ] Q [ SEP ] [ nlu _ Q ] Q1 [ SEP ] [ nlu _ Q ] Q2 [ SEP ] … [ nlu _ Q ] qn [ SEP ], wherein each characteristic comprises a starting value num _ begin, a starting value close condition num _ begin, an ending value num _ end, and a unit characteristic unit, and the content of the ending value close condition num _ end _ interval is replaced by None.

For example: let premise T be "the on-line monitoring device for dissolved gas in transformer oil needs routine calibration at regular intervals, and the calibration period is 1 year-2 years", and let Q be "the on-line monitoring device for dissolved gas in transformer oil needs routine calibration period at regular intervals is 3 years-4 years". The fusion feature input _ i is: the dissolved gas on-line monitoring device in the [ CLS ] transformer oil needs routine verification at regular intervals, and the verification period is 1 year-2 years. The on-line monitoring device for the dissolved gas in the [ SEP ] [ nlu _ t ]1, include, 2, include and annual [ SEP ] transformer oil needs a regular routine verification period of 3-4 years. [ SEP ] [ nlu-q ]3, include, 4, include, annual [ SEP ] ". Where the [ CLS ] flag is placed at the head of the first sentence and the [ SEP ] flag is used to separate the two input sentences.

S104, inputting the fusion characteristics input _ i into the JudgeNLIModel model, and outputting answers corresponding to the hypotheses. I.e. outputs the result if the assumption is correct.

Referring to fig. 2, the construction of the judgeinimodel model includes the following steps:

s201, analyzing the document, collecting a plurality of pairs of preconditions T and hypotheses Q, and generating a data set D.

S202, dividing the data set D into a training set D _ train, a test set D _ test and a verification set D _ dev according to a preset proportion. Specifically, the preset ratio is set as 8: 1: 1.

s203, respectively extracting the preconditions and the assumed features in the training set D _ train, the test set D _ test and the verification set D _ dev to obtain the training set features, the test set features and the verification set features. The feature extraction method refers to step S102 described above.

And S204, after the extracted features are unified and fused, inputting the features into an initial model for training. Referring to the step S103, the preconditions and assumed features in the training set D _ train, the test set D _ test, and the verification set D _ dev are unified and fused to obtain a fused feature.

S205, setting an epoch of the initial model, outputting the model once every epoch, and storing the model as a JudgeNLIModels model. Specifically, the pattern epoch (period) is set to 100, and the pattern is output every 100 rounds. 1 epoch represents all samples in the 1-pass training set.

S206, inputting the verification set characteristics into JudgeNLIModels models, evaluating all JudgeNLIModels models by adopting an f1 index, and selecting the model with the best effect, namely the JudgeNLIModel model.

The formula for the calculation of the f1 index is as follows:

wherein, P is accuracy and R is recall; the larger the f1 numerical value is, the better the model effect is, the correct inference precondition and the assumed logarithm number refer to the number of the model inference calculation result consistent with the result in the verification set; the number of the model-inferred preconditions and hypothesis pairs refers to the number of the results calculated by the model, and if the result is 'yes', the model-inferred preconditions and hypothesis are the correct results of the model-inferred preconditions and hypothesis pairs; the number of preconditions and hypothesis pairs that should be inferred refers to the number of preconditions and hypothesis pairs in the validation set that are the result.

S206, testing the JudgeNLIModel model by using the test set characteristics to obtain the effect of the JudgeNLIModel model. The test effect was also evaluated using the f1 index.

It is to be appreciated that domain-specific annotation data can be employed for directional training to accommodate natural language inferences in various domains.

The architecture of the JudgeNLIModel model is detailed below:

referring to fig. 3 and 4, the judgeinimedel model includes an Input _ i (Input) layer, an Embedding layer, a transform (bidirectional) layer and an output layer, wherein the Embedding layer is formed by summing three Embedding layers, which are Token Embedding, Segment Embedding and Position Embedding,

token Embedding is a word vector, the first word of which is a CLS flag, used for later classification needs.

Segment Embedding is used for distinguishing sentences, and the invention needs to distinguish 4 sentences in total, namely, the preconditions, the feature extracted from the preconditions, the hypothesis and the feature extracted from the hypothesis.

Position Embedding, namely Position coding, the premise and the hypothesis part in the input of the invention carry out Position coding in sequence, the Position coding of the premise characteristic is coded according to the Position of the numerical value in the premise, and the Position coding of the hypothesis characteristic is also consistent with the Position of the numerical value in the hypothesis, so that the model can focus attention on the numerical value and better judge the relationship between the premise and the hypothesis.

In the embodiment of the invention, the JudgeNLIModel model uses a BERT (pre-training) algorithm, wherein a Self Attention mechanism can capture semantic features between different words in the same sentence to acquire important information. The formula for Self Attention is as follows:

in the above formula, Q, K, and V are vectors initialized by each word, and these three vectors are length 64 vectors, and the formula is as follows:

where X is the word embedding of the word, the form length is 512. W is a 512 x 64 weight matrix, and applying the above formula, the length of Q, K, V is initialized to 64. In the formula, SoftMax () function acts on the matrix according to rows and factors

The adjustment function is achieved, so that the inner volume is not too large, a more stable gradient is achieved in the training process, and if the inner volume is too large, softmax can enter a gradient-free area. W is the corresponding weight.

The Q, K and V are all obtained by X in the original word embedding form, which is equivalent to the information of the bottom layer, and the attention required to be enhanced can be further increased by the multiplication of the matrix.

Fig. 3 shows the model structure, input preconditions and assumptions and corresponding features of the present invention, where the model is encoded according to the foregoing Embedding method, and then subjected to two-layer transform processing after Embedding.

The model used by the invention directly takes the first [ CLS ]]token's final hidden state

Adding a layer of weight

And predicting the label probability P through a softmax function:

the following are the parameters and values of the model adjustment:

batch size 64

Learning ratio Learning rate:3e-5

Number of cycles of epochs:4

Referring to fig. 5 and 6, the present invention further provides a natural language inference apparatus fusing semantic parsing, including:

an obtaining module 101, configured to obtain a precondition T and a corresponding hypothesis Q thereof.

And the analysis module 102 is configured to perform semantic analysis on the preconditions T and the hypothesis Q, respectively, to obtain a plurality of features.

And the fusion module 103 is configured to unify and fuse the plurality of features to obtain a fusion feature input _ i.

And the input and output module 104 is used for inputting the fusion characteristic input _ i into the JudgeNLIModel model and outputting answers of corresponding hypotheses.

Further comprising a model building module 200, the model building module 200 comprising:

and the data set generating unit 201 is used for analyzing the document, acquiring a plurality of pairs of preconditions T and assumptions Q, and generating a data set D.

The data set dividing unit 202 is configured to divide the data set D into a training set, a test set, and a verification set according to a preset ratio.

And the feature extraction unit 203 is configured to extract features of the preconditions T and the hypotheses Q in the training set, the test set, and the verification set, respectively, to obtain training set features, test set features, and verification set features.

And the model training unit 204 is configured to input the training set features into an initial model for training after unification and feature fusion of the training set features.

And the model output unit 205 is used for setting the epoch of the initial model, outputting the model once every epoch, and storing the model as a JudgeNLIModels model.

And the model selection unit 206 is configured to input the verification set features into the judgeinimodel models, evaluate all the judgeinimodel models by using an f1 index, and select a model with the best effect, namely the judgeinimodel model.

And the model testing unit 207 is used for testing the JudgeNLIModel model by using the test set characteristics to obtain the effect of the JudgeNLIModel model.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In summary, the beneficial effects of the invention include: the natural language inference method by fusing semantic analysis is to give a plurality of preconditions and hypothesis pairs, and the model of the invention can extract the characteristics in the preconditions and the hypotheses, perform fusion in an embedding layer and improve the effect of NLI (natural language inference method) by combining NLU (natural language understanding). Aiming at the problem that NLI cannot solve the judgment of the preconditions and the hypotheses, the NLU + NLI method is used for fusing semantic analysis, so that the given preconditions are solved, and the task of judging the wrong hypotheses is performed. The invention combines natural language understanding and extracts specific characteristics in the premises and the hypothesis, thereby helping to infer the relationship between the premises and the hypothesis, improving the accuracy of natural language inference and providing an improved reference direction for the subsequent natural language inference method. Meanwhile, the method can support natural language inference of data in a specific field, and can realize the natural language inference in the specific field as long as the labeled data in the specific field is subjected to model training, so that the applicability of the method is further improved. The method solves the problem that the traditional semantic inference method cannot correctly infer numerical judgment when calculating the text similarity, and improves the accuracy of natural language inference.

It should be understood that the integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The technical scope of the present invention is not limited to the above description, and those skilled in the art can make various changes and modifications to the above-described embodiments without departing from the technical spirit of the present invention, and such changes and modifications should fall within the protective scope of the present invention.

Claims

1. A natural language inference method fused with semantic parsing is characterized by comprising the following steps:

obtaining a precondition T and a corresponding hypothesis Q thereof;

semantic analysis is respectively carried out on the premises T and the hypothesis Q, and a plurality of features are obtained;

unifying and fusing the plurality of features to obtain fused feature input _ i;

and inputting the fusion characteristic input _ i into a JudgeNLIModel model, and outputting an answer of a corresponding hypothesis.

2. The natural language inference method fused with semantic parsing according to claim 1, wherein the preconditions T and the hypothesis Q are semantically parsed, respectively, specifically: and respectively identifying characters and units in the premises T and the premises Q by using a regular expression rule to obtain a plurality of characteristics.

3. The natural language inference method fusing semantic parsing according to claim 1, wherein feature fusion is performed on the plurality of features to obtain a fused feature input _ i, specifically:

assuming that a precondition T carries out semantic analysis to obtain a plurality of characteristics T1 and T2 … tn, and a Q carries out semantic analysis to obtain a plurality of characteristics Q1 and Q2 … qn, after serialization representation, the fusion characteristics input _ i is [ CLS ] T [ SEP ] [ nlu _ T ] T1 [ SEP ] [ nlu _ T ] T2 [ SEP ] … [ nlu _ T ] tn [ SEP ] Q [ SEP ] [ nlu _ Q ] Q1 [ SEP ] [ nlu _ Q ] Q2 [ SEP ] … [ nlu _ Q ] qn [ SEP ], wherein each characteristic comprises a starting value num _ begin, a starting value close condition num _ begin, an ending value num _ end, an ending value close condition num _ end _ unit, and missing content, and None is used for replacing the unit characteristic unit.

4. The natural language inference method fused with semantic parsing according to claim 1, wherein the unification is to unify units in a precondition T and a hypothesis Q.

5. The natural language inference method fused with semantic parsing according to claim 1, wherein the construction of the JudgeNLIModel model comprises the following steps:

analyzing the document, collecting a plurality of pairs of preconditions T and hypotheses Q, and generating a data set D;

dividing the data set D into a training set, a test set and a verification set according to a preset proportion;

respectively extracting the characteristics of the premises T and the hypothesis Q in the training set, the test set and the verification set to obtain the characteristics of the training set, the test set and the verification set;

after the characteristics of the training set are unified and fused, inputting the training set into an initial model for training;

setting an epoch turn of the initial model, outputting the model once every epoch turn, and storing the model as a JudgeNLIModels model;

inputting the verification set characteristics into Judge NLIModels models, evaluating all Judge NLIModels models by adopting an f1 index, and selecting the model with the best effect, namely the Judge NLIModel model;

and testing the JudgeNLIModel model by using the test set characteristics to obtain the effect of the JudgeNLIModel model.

6. The natural language inference method fused with semantic parsing according to claim 5, wherein the f1 index is calculated as follows:

wherein, P is accuracy and R is recall.

7. The natural language inference method fused with semantic parsing of claim 5, wherein the initial model is a BERT model comprising an input layer, an embedding layer, a bi-directional layer and an output layer, the embedding layer being obtained by summing word embedding, segment embedding and position embedding;

in the position embedding, the position coding of the features in the preconditions T and hypotheses Q is kept consistent with the position of their values.

8. A natural language inference apparatus fused with semantic parsing, comprising:

the acquisition module is used for acquiring the preconditions T and the corresponding hypotheses Q;

the analysis module is used for performing semantic analysis on the premises T and the hypothesis Q respectively to obtain a plurality of characteristics;

the fusion module is used for unifying and fusing the plurality of characteristics to obtain fused characteristics input _ i;

and the input and output module is used for inputting the fusion characteristic input _ i into the JudgeNLIModel model and outputting answers corresponding to the hypotheses.

9. The natural language inference device fused with semantic parsing of claim 8, further comprising a model building module, the model building module comprising:

the data set generating unit is used for analyzing the document, acquiring a plurality of pairs of preconditions T and hypotheses Q and generating a data set D;

the data set dividing unit is used for dividing the data set D into a training set, a test set and a verification set according to a preset proportion;

the characteristic extraction unit is used for respectively extracting the characteristics of the preconditions T and the hypothesis Q in the training set, the test set and the verification set to obtain the characteristics of the training set, the test set and the verification set;

the model training unit is used for inputting the training set characteristics into an initial model for training after the characteristics of the training set are unified and fused;

the model output unit is used for setting an epoch turn of the initial model, outputting the model once every epoch turn, and storing the model as a Judge NLIModels model;

the model selection unit is used for inputting the verification set characteristics into Judge NLIModels models, evaluating all Judge NLIModels models by adopting an f1 index, and selecting the model with the best effect, namely the Judge NLIModel model;

and the model testing unit is used for testing the JudgeNLIModel model by utilizing the test set characteristics to obtain the effect of the JudgeNLIModel model.