CN115270713A - A human-machine collaborative corpus construction method and system - Google Patents

Abstract

The invention relates to a method and a system for constructing a man-machine collaborative corpus, and belongs to the field of corpus construction. The method comprises the steps of constructing an automatic labeling model of a corpus and a data set, carrying out manual labeling on the data set and constructing a basic corpus, training the automatic labeling model of the corpus by using the basic corpus, carrying out machine labeling by using the trained model, calculating the labeling consistency of two labeling modes, and completing the construction of a man-machine collaborative corpus if the labeling consistency meets the requirement. The invention utilizes the man-machine collaborative corpus constructed by the method of combining the manual marking and the machine marking, can shorten the marking period and improve the marking quality.

Description

A method and system for building a human-computer collaborative corpus

technical field

The invention relates to the field of corpus construction, in particular to a method and system for man-machine collaborative corpus construction.

Background technique

For the construction of the corpus, the main work is to label the data according to the manually formulated entity relationship types and labeling specifications. At present, there are mainly three annotation modes for corpus. The first is domain expert annotation. This annotation mode is suitable for corpus annotation in professional fields and can ensure the quality of annotation, but the annotation cost is high and the cycle is long; the second is crowdsourcing annotation. , this annotation mode uses online users to annotate the same data and vote to obtain annotations. This method has a low annotation cost, but it is only suitable for simple annotation tasks that require little domain knowledge; the third is group Annotation, this annotation mode does not rely on experts, and the quality of the final corpus constructed is high, but it has high requirements for the annotation community. These three corpus annotation modes are all manual annotation methods. The manual annotation method can be completed with relatively little corpus preparation, and can be completed with high quality on quite complex phenomena. However, the manual labeling cycle is long and difficult to implement. In the current era of big data, it is a major task to carry out large-scale labeling work covering tens of thousands of pieces of data. The machine annotation speed is faster, the cycle is shorter, and the construction of the corpus can be completed in a short period of time, but the quality of pure machine annotation is poor.

Contents of the invention

The purpose of the present invention is to provide a method and system for constructing a human-machine collaborative corpus, so as to solve the problems in the prior art that the cycle of manual labeling is long and the quality of machine labeling is poor.

To achieve the above object, the present invention provides the following scheme:

A method for constructing a human-machine collaborative corpus, comprising:

Build corpus automatic labeling models and datasets; the texts in the datasets include subject textbooks, teaching designs and tutorial plans;

Carrying out manual labeling on the data set, obtaining manual labeling results and constructing a basic corpus; manual labeling includes manual pre-labeling and manual formal labeling; the manual labeling results include manual formal labeling results and manual pre-labeling results;

Using the basic corpus to train the corpus automatic labeling model to obtain the trained corpus automatic labeling model;

Using the trained corpus automatic labeling model to perform machine labeling on the text to be labeled to obtain a machine labeling result; the text to be labelled is the text selected when the manual formal labeling is performed;

Performing labeling consistency calculation on the machine labeling result and the manual official labeling result to obtain the first labeling consistency;

Judging whether the consistency of the first annotation reaches the threshold of consistency of the first annotation, and obtaining a judgment result;

If the judgment result is that the first labeling consistency reaches the first labeling consistency threshold, outputting the trained corpus automatic labeling model, and using the trained corpus automatic labeling model as a corpus;

If the judgment result is that the first annotation consistency does not reach the first annotation consistency threshold, then expand the basic corpus, and return "use the basic corpus to train the automatic annotation model of the corpus, and obtain The trained corpus automatically annotates the model" step.

Optionally, said constructing the corpus to automatically label the model and the data set further includes:

The data set is preprocessed to obtain a processed data set.

Optionally, the preprocessing of the data set to obtain the processed data set specifically includes:

performing format conversion on the text to obtain a text in a preset format;

Irregular text in the pre-formatted text is eliminated to obtain a processed data set; the irregular text includes tables, pictures, school names and teacher names.

Optionally, the manual labeling of the data set is performed to obtain a basic corpus, which specifically includes:

According to the dimensions of knowledge and the classification methods of other fields, combined with the characteristics of junior high school mathematics, formulate labeling specifications; the other fields mentioned include the medical field and the military field;

Carrying out lexical and syntactic analysis to the processed data set, constructing event triples;

Perform manual pre-labeling on the first preset quantity of text twice according to the event triplet to obtain a pre-labeling result, and update the labeling specification according to the pre-labeling result; the pre-labeling result is the first pre-labeling result and the second pre-labeled result;

According to the event triplet, the second preset number of texts is manually annotated twice to obtain an artificial official annotating result and construct a basic corpus; the artificial official annotating result includes a first official annotating result and a second official annotating result As a result; the second preset amount of text is different from the first preset amount of text.

Optionally, performing manual pre-labeling on the first preset quantity of text twice according to the event triplet to obtain a pre-labeling result, and updating the labeling specification according to the pre-labeling result, specifically includes:

Carrying out manual pre-marking on the first preset quantity of text twice according to the event triplet to obtain a pre-marking result;

calculating a second sign consistency between the first pre-labeled result and the second pre-labeled result;

judging whether to iterate the preset number of times to obtain a second judging result;

If the second judgment result is that the preset number of times has not been iterated, then comparing the first pre-marking result and the second pre-marking result to obtain a comparison result;

Updating the labeling specification according to the comparison result to obtain a new labeling specification, and returning "carry out manual pre-labeling on the first preset number of texts twice according to the event triplet to obtain a pre-labeling result";

If the second judgment result is the preset number of iterations, the manual pre-labeling is ended, and the new labeling specification is output.

Optionally, the second preset number of texts is manually formally marked twice according to the event triplet, to obtain a manual formal markup result, and to construct a basic corpus, specifically including:

Carrying out manual formal labeling twice on the text of the second preset quantity according to the event triplet, and obtaining manual formal labeling results;

calculating a third label consistency between the first formal labeling result and the second formal labeling result;

Judging whether the consistency of the third label reaches the second label consistency threshold, and obtaining a third judgment result;

If the third judgment result is that the consistency of the third label reaches the second label consistency threshold, the manual formal labeling is completed, and the manual formal labeling result is output;

Constructing the basic corpus according to the manual formal labeling results;

If the third judgment result is that the third labeling consistency does not reach the second labeling consistency threshold, return "according to the event triplet, perform manual formal labeling on the second preset number of texts twice." , to obtain the manual formal labeling results”.

A human-computer collaborative corpus construction system, comprising:

Model and data set building blocks are used to construct corpus automatic labeling models and data sets; the texts in the data set include subject textbooks, teaching designs and teaching plans;

The manual labeling module is used to manually label the data set, obtain manual labeling results and build a basic corpus; manual labeling includes manual pre-labeling and manual formal labeling; the manual labeling results include manual manual labeling results and manual pre-labeling results ;

A training module, configured to use the basic corpus to train the automatic labeling model of the corpus to obtain the automatic labeling model of the corpus after training;

The machine labeling module is used to use the trained corpus automatic labeling model to carry out machine labeling on the text to be labeled to obtain a machine labeling result; the text to be labelled is the text selected when the manual formal labeling;

A calculation module, configured to perform labeling consistency calculation on the machine labeling result and the manual formal labeling result to obtain the first labeling consistency;

A judging module, configured to judge whether the first label consistency reaches the first label consistency threshold, and obtain a judgment result;

The first execution module is configured to output the trained corpus automatic labeling model if the judgment result is that the first labeling consistency reaches the first labeling consistency threshold, and automatically label the trained corpus Label the model as a corpus;

The second execution module is configured to expand the basic corpus if the judgment result is that the consistency of the first annotations does not reach the threshold of consistency of the first annotations, and return "Use the basic corpus to analyze the corpus Automatic labeling model is trained, and the step of obtaining the trained corpus automatic labeling model".

Optionally, also include:

The data processing module is used to preprocess the data set to obtain the processed data set.

Optionally, the data processing module includes:

a format conversion unit, configured to perform format conversion on the text to obtain a text in a preset format;

The data elimination unit is used to eliminate irregular text in the text in the preset format to obtain a processed data set; the irregular text includes tables, pictures, school names and teacher names.

Optionally, the manual labeling module includes:

An annotation specification formulation unit is used to formulate an annotation specification according to the dimension of knowledge and the classification methods of other fields, combined with the characteristics of junior high school mathematics; the other fields include the medical field and the military field;

A data analysis unit, configured to perform lexical and syntactic analysis on the processed data set to construct event triples;

A pre-labeling unit, configured to manually pre-label the first preset quantity of text twice according to the event triplet, obtain a pre-labeled result, and update the labeling specification according to the pre-labeled result; the pre-labeled The results are the first pre-labeled result and the second pre-labeled result;

The formal labeling unit is used to perform manual formal labeling on the second preset quantity of text twice according to the event triplet, obtain the manual formal labeling result, and construct a basic corpus; the manual manual labeling result includes the first formal labeling A result and a second official labeling result; the second preset number of texts is different from the first preset number of texts.

According to the specific embodiments provided by the invention, the invention discloses the following technical effects:

The human-machine collaborative corpus construction method and system of the present invention, by constructing the corpus automatic labeling model and data set, manually labeling the data set and constructing the basic corpus, using the basic corpus to train the corpus automatic labeling model, using the trained model Machine labeling, calculate the labeling consistency of the two labeling methods, if the labeling consistency meets the requirements, the construction of the human-machine collaborative corpus will be completed. The invention utilizes the man-machine collaborative corpus constructed by the method of combining manual labeling and machine labeling, which can shorten the labeling cycle and improve the labeling quality.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without paying creative labor.

Fig. 1 is the flow chart of a kind of man-machine cooperative corpus construction method provided by the present invention;

Fig. 2 is the structural diagram of the named entity recognition model based on pre-training language model of the present invention;

Fig. 3 is the structural diagram of the relationship extraction model based on graph convolutional neural network of the present invention;

Fig. 4 is the manual pre-labeling flowchart of the present invention;

Fig. 5 is the flow chart of manual formal labeling of the present invention;

Fig. 6 is a diagram of the automatic labeling and recognition results of the model of the present invention;

Fig. 7 is a flow chart of the human-machine collaborative labeling of the present invention;

FIG. 8 is an overall framework diagram of the human-computer collaborative corpus construction method of the present invention;

FIG. 9 is a structural diagram of a human-machine collaborative corpus construction system provided by the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The purpose of the present invention is to provide a method and system for constructing a human-machine collaborative corpus, so as to solve the problems in the prior art that the cycle of manual labeling is long and the quality of machine labeling is poor. The human-computer collaborative corpus construction method mainly includes two stages: manual processing and machine processing. The manual processing stage includes data collection and processing, annotation specification formulation and manual annotation; the machine processing stage includes lexical and syntactic analysis and machine annotation.

The invention adds machine processing operation on the basis of manual labeling, and studies the method of man-machine cooperative construction of corpus. Taking junior high school mathematics as an example, referring to the classification dimensions of knowledge in Bloom's "Taxonomy of Educational Objectives" and the classification methods of other fields (such as medical field, military field, etc.), combined with the characteristics of junior high school mathematics knowledge, the formula Entity and relationship categories have constructed a junior high school mathematics corpus, which has a practical supporting role in the construction and application of knowledge maps for follow-up education.

In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Fig. 1 is the flow chart of a kind of man-machine cooperative corpus construction method provided by the present invention, as shown in Fig. 1, the method comprises:

Step 101: Construct a corpus automatic labeling model and a data set. The texts in the data set include subject textbooks, instructional design and tutorial plans.

As an optional implementation manner, the step 101 further includes:

The data set is preprocessed to obtain a processed data set.

As an optional implementation manner, the preprocessing of the data set to obtain the processed data set specifically includes:

Perform format conversion on the text to obtain text in a preset format.

Irregular text in the pre-formatted text is eliminated to obtain a processed data set; the irregular text includes tables, pictures, school names and teacher names.

The human-machine collaborative corpus annotation mainly includes two parts: manual annotation and model automatic annotation. The manual annotation is performed strictly according to the annotation process, and the automatic annotation adopts the corpus automatic annotation model to realize automatic annotation. In the human-computer collaborative construction method of the subject corpus, manual annotation and machine annotation are interspersed and coordinated, and the overall framework is shown in Figure 8.

In practical applications, the automatic corpus tagging process is mainly the process of automatically identifying entities and entity relationships in the corpus data through a deep learning-based model (corpus automatic tagging model). The corpus automatic tagging model is divided into pre-trained language model-based Named entity recognition model (BERT-wwm-BiLSTM-CRF) and graph convolutional neural network-based relation extraction model (BERT-GCN).

On the basis of the model frame construction of the corpus automatic labeling model, the present invention takes the mathematics corpus data of the basic education subject as an example, combines multi-source educational teaching texts such as teaching design, teaching guides and test booklets, etc., and prepares the corpus automatic Label the training corpus of the model and carry out model training, which is divided into the following steps: data collection and processing, labeling standard formulation, lexical and syntactic analysis, and model training.

The present invention is mainly oriented to the basic education subject mathematics, the school period is junior high school, and the textbook version is the People's Education Publishing House. A total of 688 teaching designs and electronic textbooks have been collected, including each of the 29 chapters and 91 subsections of the junior high school mathematics textbook in the People's Education Edition. Each section of the chapter ensures the full coverage of the junior middle school mathematics entity relationship. The invention collects data such as junior high school mathematics textbooks, teaching design, and teaching plans, and performs text standardization processing on them.

The data collected on the website is mainly stored in pdf, doc, and docx formats. In the teaching design, there are some private information such as school names and teacher names, and some content formats are presented in the form of tables. There are some pictures and exercises. The above are not conducive to the subsequent labeling work, so the original data needs to be processed. Firstly, format conversion is performed on the collected data, and the texts in different formats are uniformly encoded in UTF-8 and stored in txt format. Secondly, the tables, pictures, exercises, school names, and teacher names in the text are removed to facilitate subsequent work.

Named Entity Recognition (NER) refers to the recognition of entities with specific meaning in text, mainly including names of people, places, institutions, proper nouns, etc. The named entity recognition model BERT-wwm-BiLSTM-CRF based on the pre-trained language model used in the present invention mainly includes three modules, and the model diagram is shown in FIG. 2 .

First, the junior high school mathematics text is extracted from rich text features through the BERT-wwm pre-training language model to obtain the word vector, and then the obtained junior high school mathematics word vector is input into the BiLSTM module to fully extract the context information in the mathematics sentence, and finally through the CRF module. The output results of the sentence context semantic features are decoded to obtain the globally optimal label sequence, thereby completing the named entity recognition of junior high school mathematics texts.

The input of the BERT-wwm model is junior high school mathematics text, including token embedding word vector, segment embedding sentence vector and position embedding position vector, using the Transformer Encoder structure, which can express the specific semantics of words in the context according to the semantic relationship of the context. Transformer is based on the attention mechanism to model junior high school mathematics sentences. The Attention method is Self-Attention. By learning three parameters, the feature after the same embedding is converted to calculate the Attention score of this sentence. The calculation is as follows: formula (1 ) shown.

Among them, Q, K and V are the input word vector matrix, d _k is the dimension, and QK ^T is used to calculate the relationship between the input word vectors. After d _k is reduced, the weight representation is obtained through softmax normalization. Finally, the output is a weighted sum of all word vectors in the sentence, so that the representation of each word contains information about other words in the sentence, which is context-dependent and more global than traditional word embedding models.

BiLSTM receives the character embedding generated by the BERT-wwm pre-training model and predicts the probability of each character label. The bidirectional LSTM structure can use contextual information more effectively. Each junior high school mathematics sentence is calculated in order and in reverse order, and then the hidden layer representation is obtained by vector splicing. Suppose the input at the last moment is h _t-1 , and the input at this moment is x _t , then the formula is as follows.

f _t ＝σ(w _f ·[h _t-1 ,x _t ]+b _f )

i _t =σ(w _i ·[h _t-1 ,x _t ]+b _i )

O _t ＝σ(w _o ·[h _t-1 ,x _t ]+b _o )

h _t ＝O _t ·tanh·(S _t )

表示当前时刻的状态，是由tanh层创建的新候选值向量，S_t表示由状态S_t-1更新的新单元状态，h_t表示t时刻最终的输出。Among them, σ is the sigmoid activation function, f _t is the forget gate, it is the input gate, O _t is the output gate, _{x t is} the current input, and tanh is the hyperbolic tangent function. b _f is the offset vector of the forget gate, b _i is the offset vector of the input gate, b _o is the offset vector of the output gate, b _c is the offset vector of the memory unit, w _f is the weight matrix of the forget gate, w _i is the weight matrix of the input gate, w _o is the weight matrix of the output gate, w _c represents the weight matrix of the memory unit,

Represents the state at the current moment, which is a new candidate value vector created by the tanh layer, S _t represents the new unit state updated by state S _t-1 , and h _t represents the final output at time t.

The input of the CRF layer is the word vector obtained from the training of the BiLSTM layer. The input of the CRF layer is denoted as X=(x ₁ ,x ₂ ,…,x _n ), where x _n represents the nth word; the prediction sequence Y=(y ₁ ,y ₂ ,…,y _n ), where y _n denotes the predicted label for the nth word. Its score function expression is shown in formula (3).

Use the softmax function to calculate the conditional probability, such as formula (4).

表示除当前计算的yi之外，全集Y中剩下的其他y。Among them, A _yi,yi+1 represents the transition probability of the label from yi to yi+1, which is obtained from the transition matrix learned by CRF. P _i,yi represents the unnormalized probability of the word Xi in the label yi, also known as the state score, which is obtained from the output of BiLSTM.

Indicates other y left in the corpus Y other than the currently computed yi.

Relation Extraction (Relation Extraction, RE) is to find the relationship between the subject and the object in unstructured or semi-structured data (if there are two entities with a relationship, the two entities can be called the subject and the object respectively) , and represent it as an entity-relationship triple, namely (subject, relation, object). The graph convolutional neural network-based relation extraction model BERT-GCN model used in the present invention is mainly divided into three parts, and the model diagram is shown in FIG. 3 .

First, the text is input into the BERT pre-training model to extract context-rich semantic features, and the corresponding word vector representation is obtained. At the same time, the dependency syntax analysis is performed on the text to build a dependency syntax graph, and then the adjacency matrix of the graph is obtained, which is used to store sentences. Grammatical structure information; secondly, the obtained word vector and adjacency matrix are sent to GCN for further feature extraction, and finally the feature vector is classified through the softmax layer to predict the probability of the relationship.

First, the text is input into the BERT pre-training model to extract rich text features to obtain corresponding word vectors. Secondly, by analyzing the syntactic dependence of the input junior high school mathematics sentences and constructing a syntactic dependency graph, the graph neural network can learn the grammatical information of junior high school mathematics sentences from the dependency structure. In this paper, G=(V, E) is used to represent the dependency syntax graph, where V=v1,v2...,vn represents the set of points in the graph, E=el,e2,...em represents the set of edges in the graph, and the dependency The adjacency matrix A corresponding to the syntax graph is shown in formula (5). When A _ij is 1, it means that there is an edge between vertices vi and vj, and when A _ij is 0, it means that there is no edge.

According to the adjacency matrix storage method of an undirected graph, the adjacency matrix can be obtained through dependency syntax analysis, and the adjacency matrix can be sent to the graph neural network for further feature extraction. The calculation process of the hidden state of node i is shown in formula (6) shown.

where A _ij represents the elements corresponding to nodes i and j in the adjacency matrix, W ^(j) represents the weight matrix of node j, h _j represents the hidden state of node j, b ^(j) represents the bias of node j, f represents Non-linear function, what the present invention uses is RELU non-linear function.

The input of the graph neural network of layer l is composed of the output of layer l-1 and the adjacency matrix. After the operation of layer 1, the hidden state representation of node i after the graph neural network of layer l is shown in formula (7).

Where W ^(l) represents the weight matrix of layer l, h _j ^(l-1) represents the hidden state of node j after passing through layer l-1 graph neural network, and b ^(l) represents the bias of layer l.

After the l-layer graph convolution operation, if the graph convolution operation is applied one-sidedly, the result of the node representation may be different, because the degree of the node varies greatly. This may affect the sentence representation of high-level nodes, no matter what information the nodes contain, the node information in the l-1th layer graph neural network is transferred to the lth layer. Therefore, the data is normalized before being sent to the nonlinear layer, and self-loops are added on each graph node. As shown in formula (8).

I是单位矩阵，

是节点i的度。in

I is the identity matrix,

is the degree of node i.

The relationship extraction task of the present invention can be regarded as a multi-classification task. Finally, the probability distribution of the relationship is obtained through the softmax function classifier, the relationship label is predicted, and the cross-entropy function is used as the loss function of the model during the training process.

Step 102: Carry out manual labeling on the data set, obtain manual labeling results and build a basic corpus. Manual labeling includes manual pre-labeling and manual formal labeling; the manual labeling results include manual formal labeling results and manual pre-labeling results.

As an optional implementation manner, the step 102 specifically includes:

Step 1021: According to the dimensions of knowledge and the classification methods of other fields, combined with the characteristics of junior high school mathematics, formulate labeling specifications. Said other fields include the medical field and the military field.

In practical application, the present invention refers to the classification dimension of knowledge and the classification methods of other fields (such as medical field, military field, etc.) There are two types of entities, knowledge points, knowledge units, and knowledge clusters. Among them, the knowledge point is the smallest knowledge that cannot be further divided; there are multiple knowledge points that cannot be further divided under the knowledge unit; the knowledge cluster is the general term for this type of knowledge. As shown in Table 1.

Table 1 Entity category table

Five types of relationships, forerunner and successor, containment, parallel, father and son, brother. The predecessor-successor relationship represents that there is a certain dependence between entities A and B, that is, a predecessor-successor relationship. The containment relationship represents the whole and part relationship between entities A and B. A parallel relationship means that there is a parallel relationship between entities A and B. A parent-child relationship means that entities A and B have some common attributes. A sibling relationship means that entities A and B have the same parent class. As shown in table 2.

Table 2 Relationship type table

relationship type relationship example predecessor successor relationship (polynomial, equation, predecessor-sequence relationship) containment relationship (triangle, edge, containment relationship) parallel relationship (independent variable, dependent variable, parallel relationship) father-son relationship (function, one-time function, parent-child relationship) Brotherhood (primary function, quadratic function, sibling relationship)

Before the official labeling work starts, first understand the overall teaching system and knowledge system of junior high school mathematics, and then complete the formulation of labeling specifications. First of all, there is a nesting situation for the entities of junior high school mathematics, which may cause the boundary of entities to be unable to be confirmed during the labeling and recognition process. For example, the entity "system of binary linear equations" exists in the text, but this word also includes two entities of "binary linear equations" and "equation". Therefore, during labeling and recognition, entity boundaries should be confirmed according to the context.

Secondly, entities in junior high school mathematics are usually composed of specific ending words, such as "real numbers", "integer numbers", "complex numbers" and "rational numbers", all of which end with the word "number". Therefore, in the process of entity recognition, the accuracy of labeling can be enhanced according to this feature. When labeling entities, the principle of marking the maximum range of entities should be followed. For the relationship between mathematical entities in junior high school, if there is a predecessor-successor relationship between two entities during the labeling process, it is necessary to ensure that the predecessor entity is in front and the successor entity is in the back. If there is a predecessor-successor relationship between "equation" and "function", then the marked entity 1 is "equation" and entity 2 is "function". Finally, when marking, it is necessary to ensure that the marked entities do not overlap or nest, and the marked content cannot contain punctuation marks.

Step 1022: Perform lexical and syntactic analysis on the processed data set to construct event triples. In practical applications, lexical and syntactic analysis is performed, firstly Chinese word segmentation is performed on the text, and a mathematical thesaurus is introduced; then syntactic analysis is performed, and event triples are constructed in combination with semantic role analysis.

In practical application, Chinese word segmentation refers to splitting a sentence into individual words. In cross-domain word segmentation, there are often a large number of professional terminology vocabulary. However, professional terms in the field of education have obvious disciplinary tendencies and are highly specialized. Existing word segmentation tools cannot correctly segment professional words in basic subjects in the field of education, which has a great impact on subsequent experiments. For example, the word segmentation tool jieba word segmentation used in the present invention will segment "unary linear equation" into two words "unary" and "linear equation", and "combination method" will be segmented into two words "combination" and "method".

Aiming at the poor performance of Chinese word segmentation in the field of education, the invention adds a mathematical thesaurus to improve the accuracy of word segmentation. Download the Mathematics Vocabulary Encyclopedia from Sogou Input Method Thesaurus Encyclopedia, but the file format is scel, which cannot be directly used in jieba word segmentation. Convert the file format of the downloaded Mathematics Vocabulary Encyclopedia, from the original scel format to UTF-8 encoded txt format. The converted mathematical thesaurus has a total of 135,593 words, including mathematical words at various stages. Each word in the thesaurus forms its own line. Introduce this mathematical thesaurus into the jieba word segmentation, and perform Chinese word segmentation again. The word segmentation results of the above-mentioned words such as "matching", "method", "unary" and "linear equation" are accurate. It proves that when performing Chinese word segmentation in a limited field, the introduction of a professional thesaurus in this field will improve the accuracy of the word segmentation results and be more helpful for subsequent work.

In the field of education, due to the existence of some special syntactic structures in instructional design texts, such as "on the basis of ..., learn ...", "understand (knowledge, master) ...", these syntactic structures can be clearly seen Outline the predecessor-successor relationship between entities. Therefore, on the basis of the original treebank, some special syntactic structures unique to the teaching design are added to complete the Chinese syntactic analysis of the junior high school mathematics corpus, mainly using the Chinese dependency treebank of Harbin Institute of Technology for annotation analysis. Harbin Institute of Technology Chinese Dependency Treebank data comes from the corpus of "People's Daily". The treebank includes a total of 8,000 sentences, and there are 14 types of dependency relationships.

Taking the sentence "Students have learned the concept of square root, which is the basis for learning cube roots and real numbers" as an example, the specific meaning of each column in the dependency category is shown in Table 3 below, and the labeling results are shown in Table 4, mainly Pay attention to the core relationship in this sentence, as well as the subject-predicate relationship and the verb-object relationship.

Table 3 Symbol meaning table

symbol meaning ID serial number FORM word or punctuation LEMMA In Chinese, same as FORM CPOSTAG Part of speech (coarse-grained) POSTAG Part of speech (fine-grained) FEATS — HEAD head word subscript DEPREL Dependency with the head word

Table 4 Annotation result table

The tree bank based on the special syntactic structure added to the teaching design is sent to the CRF model for training, and the trained model is imported into the pyLTP syntactic analyzer for syntactic analysis of junior high school mathematics corpus. In addition, semantic role analysis is added on the basis of dependency syntax analysis, and finally a simple event triple is constructed to help manual relationship labeling. First check whether there is a subject-verb-object structure, if it exists, extract it in the form of subject-verb-object, otherwise change to the method of dependent syntax. For each word, generate a child node that depends on the syntax; and generate a parent-child array dependency structure for the word, mainly recording the part of speech, the part of speech of the parent node and their relationship; loop each word, find the verb-object, attributive suffix The subject-verb-verb-complement relationship of verb-object and pre-object is extracted. Based on the above extraction process, the construction of a simple event triplet is completed.

When performing manual labeling operations, two labelers perform manual pre-labeling and manual formal labeling based on word segmentation results and syntactic analysis results. In the pre-labeling stage, two labelers label a part of the same text. These same documents are mainly It is used to calculate the consistency of labeling each time, and it will go through three rounds to gradually improve the labeling specifications. Complete formal annotation based on the latest annotation specifications, and build a basic corpus.

Step 1023: Carry out manual pre-labeling on the first preset amount of text twice according to the event triplet to obtain a pre-labeling result, and update the labeling specification according to the pre-labeling result. The pre-labeled results are a first pre-labeled result and a second pre-labeled result.

As an optional implementation manner, the step 1023 specifically includes:

The first preset amount of text is manually pre-labeled twice according to the event triplet to obtain a pre-labeled result.

Computing a second flag consistency between the first pre-labeled result and the second pre-labeled result.

It is judged whether to iterate a preset number of times to obtain a second judgment result.

If the second judgment result is that the preset times have not been iterated, then comparing the first pre-marking result with the second pre-marking result to obtain a comparison result.

The labeling specification is updated according to the comparison result to obtain a new labeling specification, and "the first preset number of texts are manually pre-labeled twice according to the event triplet to obtain a pre-labeling result" is returned.

If the second judgment result is the preset number of iterations, the manual pre-labeling is ended, and the new labeling specification is output.

In practical applications, in order to perfect the established labeling specifications and improve the labeling quality in the labeling stage, the corpus pre-labeling work is carried out before the official labeling starts. Among them, the manual pre-labeling process is shown in Figure 4. 50 documents were randomly selected, and two annotators independently annotated and analyzed different annotations. Under the guidance of professional junior high school mathematics teachers, they revised and improved the annotation specifications. Rounds, calculate the annotation consistency of each round.

Step 1024: According to the event triples, the second preset number of texts are formally annotated manually twice to obtain the formal annotated results and construct a basic corpus. The artificial formal marking result includes a first formal marking result and a second formal marking result; the second preset number of texts is different from the first preset number of texts.

As an optional implementation manner, the step 1024 specifically includes:

The second preset number of texts are manually formally marked twice according to the event triples to obtain a manual formally marked result.

Calculate a third annotation consistency between the first formal annotation result and the second official annotation result.

It is judged whether the consistency of the third annotation reaches the second threshold of consistency of annotation, and a third judgment result is obtained.

If the third judgment result is that the consistency of the third labeling reaches the second labeling consistency threshold, the manual formal labeling is completed, and the manual formal labeling result is output.

The basic corpus is constructed according to the manual formal annotation results.

If the third judgment result is that the third labeling consistency does not reach the second labeling consistency threshold, return "according to the event triplet, perform manual formal labeling on the second preset number of texts twice." , to obtain the manual formal labeling results”.

In practical applications, after the pre-labeling work is over, it enters the formal labeling stage. Two labelers respectively label 200 documents other than the pre-labeled ones, and at the same time, 50 documents are randomly selected and labeled by the two labelers for After the annotation is completed, the annotation consistency is calculated. The process of the formal labeling stage is shown in Figure 5.

Complete the manual labeling of entities, and export the data into ann format. Then select the BIO mode in the sequence labeling mode to perform sequence labeling on the exported data, that is, label each word in the sentence as "B-X", "I-X" or "O", where B is the beginning of the entity, and I is the entity In the middle or end of , O is not an entity. Accordingly, combined with the defined three types of entity types, a total of seven types of labels can be obtained. The knowledge point symbol is "KNO", the knowledge unit symbol is "KUN", and the knowledge cluster symbol is "KNG". As shown in Table 5, the final base corpus with BIO tags is obtained.

Table 5 Manual labeling result table of entity relationship

serial number label type label definition 1 B-KNO The first word of knowledge point entity 2 I-KNO Knowledge point entity middle or end 3 B-KUN The first word of the knowledge unit entity 4 I-KUN Knowledge unit entity middle or end 5 B-KNG The first word of the knowledge cluster entity 6 I-KNG Knowledge cluster entity middle or end 7 o not an entity

Complete the manual labeling of the relationship, export the data into ann format and process it, and finally form the format required for training the relationship extraction model (relation type serial number sentence entity 1 start position entity 1 end position entity 2 start position entity 2 end position), where Entity 1 is separated by "#", and entity 2 is separated by "$". The labels corresponding to the relationship types are shown in Table 6.

Table 6 Annotation result table of relation type

Step 103: Using the basic corpus to train the corpus automatic tagging model to obtain a trained corpus automatic tagging model.

In practical applications, for the training of the named entity recognition model (BERT-wwm-BiLSTM-CRF) based on the pre-trained language model, the data used for the training data of the model comes from the junior high school mathematics corpus constructed by human-computer cooperation, in which the entity corpus There are 22803 sentences in total, and the training set and test set are randomly divided according to the ratio of 8:2. The number of entities in the junior high school mathematics corpus is 2756 knowledge points KNO, 808 knowledge units KUN, and 185 knowledge clusters KNG. The model training parameters mainly include using the Adam optimizer, the batch processing parameter batch_size is set to 64, the number of iterations epoch is set to 100, the learning rate is set to 5e-5, the inactivation rate drop_out_rate is set to 0.5, and the maximum sequence length max_seq_length is set to 128. The loss function of the named entity recognition model training based on the pre-trained language model is as follows:

Among them, Z(x) represents the CRF score accumulation function, where score( ) score function is the sum of emission probability and transition probability in CRF.

For the training of the relational extraction model based on graph convolutional neural network (BERT-GCN), the triplet data used in the training data of this model comes from the junior high school mathematics corpus in Chapter 3. The relational corpus has a total of 26,764 sentences. In the experiment, according to The ratio of 8:2 is randomly divided into training set and test set. The relationships in the junior middle school mathematics corpus include 6351 predecessor-successor relationships, 2044 inclusion relationships, 554 parallel relationships, 460 parent-child relationships and 209 brother relationships. The model training parameters mainly include using the Adam optimizer, the batch processing parameter batch_size is set to 32, the number of iterations epoch is set to 100, the learning rate is set to 1.7e-5, the inactivation rate drop_out_rate is set to 0.4, and the maximum sequence length max_seq_length is set to 128. Weight decay is set to 1e-3. The loss function for training uses cross entropy as follows:

Among them, D represents the number of relation types, N represents the number of triples, Y represents the real relation label, and Z represents the predicted relation probability.

Step 104: Use the trained corpus automatic labeling model to perform machine labeling on the text to be labelled, and obtain a machine labeling result. The text to be marked is the text selected during the manual formal marking.

In practical applications, machine annotation is mainly through the trained model (named entity recognition model based on pre-trained language model (BERT-wwm-BiLSTM-CRF) and relationship extraction model based on graph convolutional neural network (BERT-GCN) ) to automatically identify entity relationships. After obtaining the named entity recognition model and relation extraction model (trained corpus automatic labeling model) trained with the manually labeled basic corpus, the trained corpus automatic labeling model is embedded into the labeling tool developed by the team for machine Automatic labeling. Import the text to be labeled into the labeling tool, and after the automatic labeling mode is turned on, the labeling tool will visually label the recognition results of the model, as shown in Figure 6.

Step 105: Carry out labeling consistency calculation on the machine labeling result and the manual official labeling result to obtain a first labeling consistency.

Step 106: Judging whether the consistency of the first annotation reaches the first threshold of consistency of annotation, and obtaining a judgment result. If yes, go to step 107; if not, go to step 108.

Step 107: Output the trained corpus automatic tagging model, and use the trained corpus automatic tagging model as a corpus.

Step 108: expand the basic corpus, and return to "step 103".

In practical applications, 50 documents randomly selected during manual formal labeling are also used for machine labeling, the purpose is to calculate labeling consistency. Taking the 50 documents marked manually as the benchmark, compare the marking consistency of the part of the document marked by the machine with the same manual marking, and judge whether the marking consistency with the 50 documents marked by the machine has reached 80%. When it reaches 80%, it is necessary to expand the basic corpus and continue to train the model until the final corpus of junior high school mathematics is formed. If it reaches 80%, machine annotation is completed to form the final junior high school mathematics corpus.

Specifically, after the manual and machine annotations are completed, the consistency of annotations in the corpus is analyzed and compared. The present invention uses F value or Kappa value to evaluate the labeling consistency of the corpus. Among them, the Kappa value is usually used to evaluate the positive and negative examples of labeling. For the corpus of the present invention, only unlabeled corpus can be used as negative examples, which is difficult to count. In this case, the F value is generally used to evaluate, and the two labeling parties (manual and machine) are respectively A and B, and A is used as a reference group (the selection of the reference group is random) to calculate the accuracy P of B, The calculation formula is:

Then the recall rate R of B is:

Based on the accuracy P and recall rate R of B, the F value is calculated:

The higher the F value, the higher the accuracy of B labeling, that is, the higher the labeling consistency. When calculating the consistency of entity labels, when the entity strings and types are the same, the labels are considered to be consistent. The above indicators are also used in the calculation of relationship consistency, when the entity pairs and the relationship between entities are the same, they are labeled consistently.

Table 7 Entity annotation comparison result table

Table 8 Relational annotation comparison result table

It can be seen from Table 7 and Table 8 that during the pre-labeling process, the value of labeling consistency is gradually increasing in each round. This shows that after each round of pre-labeling, the labeling specification will be further revised according to the labeling results, and the gradual improvement of the labeling specification will also gradually improve the labeling consistency.

If the consistency of the machine-labeled corpus does not reach 80%, then the basic corpus needs to be expanded, and another 1,000 pieces of data are manually labeled to expand the basic corpus. After the expansion is completed, iterate from the first step until the labeling consistency is greater than 80%.

In the human-computer collaborative corpus construction method in the present invention, the manual processing stage includes data collection and processing, label specification formulation and manual labeling; the machine processing stage includes lexical syntax analysis and machine labeling, as shown in FIG. 7 . This method, when labeling massive data, can effectively reduce the time and cost of manual labeling, has applicability, and can effectively solve the problem of large-scale data labeling.

Fig. 9 is a structural diagram of a human-machine collaborative corpus construction system provided by the present invention. As shown in Fig. 9, the system includes:

The model and data set construction module 901 is used to construct corpus automatic labeling models and data sets; the texts in the data set include subject textbooks, teaching designs and tutorial plans.

The manual labeling module 902 is configured to perform manual labeling on the dataset, obtain manual labeling results and construct a basic corpus. Manual labeling includes manual pre-labeling and manual formal labeling; the manual labeling results include manual formal labeling results and manual pre-labeling results.

The training module 903 is configured to use the basic corpus to train the corpus automatic tagging model to obtain a trained corpus automatic tagging model.

The machine labeling module 904 is configured to use the trained corpus automatic labeling model to perform machine labeling on the text to be labelled, and obtain a machine labeling result. The text to be marked is the text selected during the manual formal marking.

The calculation module 905 is configured to perform labeling consistency calculation on the machine labeling result and the manual formal labeling result to obtain a first labeling consistency.

A judging module 906, configured to judge whether the consistency of the first annotation reaches the first threshold of consistency of annotation, and obtain a judgment result.

The first execution module 907 is configured to output the trained corpus automatic labeling model if the judgment result is that the first labeling consistency reaches the first labeling consistency threshold, and the trained corpus Automatically annotate the model as a corpus.

The second execution module 908 is configured to expand the basic corpus and return "Use the basic corpus for the The corpus automatic labeling model is trained to obtain the trained corpus automatic labeling model".

As an optional implementation, it also includes:

The data processing module is used to preprocess the data set to obtain the processed data set.

As an optional implementation, the data processing module includes:

The format conversion unit is configured to perform format conversion on the text to obtain a text in a preset format.

The data elimination unit is used to eliminate irregular text in the text in the preset format to obtain a processed data set; the irregular text includes tables, pictures, school names and teacher names.

As an optional implementation manner, the manual labeling module 902 includes:

The labeling specification formulation unit is used to formulate labeling specifications based on the dimensions of knowledge and classification methods in other fields, combined with the characteristics of junior high school mathematics. Said other fields include the medical field and the military field.

The data analysis unit is configured to perform lexical and syntactic analysis on the processed data set to construct event triples.

The pre-labeling unit is configured to manually pre-label the first preset quantity of text twice according to the event triplet to obtain a pre-labeling result, and update the labeling specification according to the pre-labeling result. The pre-labeled results are a first pre-labeled result and a second pre-labeled result.

The formal labeling unit is configured to perform manual formal labeling twice on a second preset number of texts according to the event triples, obtain manual formal labeling results, and construct a basic corpus. The artificial formal marking result includes a first formal marking result and a second formal marking result; the second preset number of texts is different from the first preset number of texts.

As an optional implementation manner, the pre-marking unit includes:

The pre-labeling subunit is configured to manually pre-label the first preset number of texts twice according to the event triples to obtain a pre-labeling result.

A first calculating subunit, configured to calculate a second flag consistency between the first pre-labeled result and the second pre-labeled result.

The first judging subunit is used for judging whether to iterate a preset number of times to obtain a second judging result.

The first execution subunit is configured to compare the first pre-marking result with the second pre-marking result to obtain a comparison result if the second judgment result is that the preset number of times has not been iterated.

The update subunit is used to update the labeling specification according to the comparison result to obtain a new labeling specification, and return "according to the event triplet, perform manual pre-labeling on the first preset number of texts twice, and obtain the pre-labeled Annotate the result".

The second executing subunit is configured to end the manual pre-labeling and output the new labeling specification if the second judgment result is the preset number of iterations.

As an optional implementation manner, the formal labeling unit includes:

The formal labeling subunit is configured to perform manual formal labeling twice on a second preset number of texts according to the event triples to obtain manual formal labeling results.

The second calculation subunit is configured to calculate a third label consistency between the first formal labeling result and the second formal labeling result.

The second judging subunit is configured to judge whether the consistency of the third annotation reaches the second threshold of consistency of annotation, and obtain a third judgment result.

A third executing subunit, configured to complete the manual formal labeling and output the manual formal labeling result if the third judgment result is that the third labeling consistency reaches the second labeling consistency threshold.

The basic corpus construction subunit is configured to construct the basic corpus according to the manual formal annotation result.

The fourth execution subunit is configured to return "According to the event triplet for the second preset number of The text of the text is manually annotated twice, and the result of manual official annotation is obtained.

The human-computer collaborative corpus construction system in the present invention can complete the human-computer collaborative corpus construction, which has reference value for the research on data-driven corpus construction methods, and the formed junior high school mathematics corpus has practical support for the construction and application of follow-up education knowledge maps .

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part.

In this paper, specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; meanwhile, for those of ordinary skill in the art, according to the present invention Thoughts, there will be changes in specific implementation methods and application ranges. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. A method for constructing a human-computer collaborative corpus is characterized by comprising the following steps:

constructing an automatic labeling model and a data set of a corpus; the text in the data set comprises subject textbooks, teaching designs and teaching guide cases;

carrying out manual annotation on the data set to obtain a manual annotation result and construct a basic corpus; the manual marking comprises manual pre-marking and manual formal marking; the manual annotation result comprises a manual formal annotation result and a manual pre-annotation result;

training the corpus automatic annotation model by using the basic corpus to obtain a trained corpus automatic annotation model;

performing machine labeling on a text to be labeled by using the trained corpus automatic labeling model to obtain a machine labeling result; the text to be marked is the text selected during the manual formal marking;

carrying out labeling consistency calculation on the machine labeling result and the manual formal labeling result to obtain a first labeling consistency;

judging whether the first labeling consistency reaches a first labeling consistency threshold value or not to obtain a judgment result;

if the judgment result shows that the first labeling consistency reaches the first labeling consistency threshold value, outputting the trained corpus automatic labeling model, and taking the trained corpus automatic labeling model as a corpus;

and if the judgment result is that the first annotation consistency does not reach the first annotation consistency threshold, expanding the basic corpus, and returning to the step of training the automatic annotation model of the corpus by using the basic corpus to obtain the trained automatic annotation model of the corpus.

2. The method for constructing a human-computer collaborative corpus according to claim 1, wherein the constructing corpus automatically labels a model and a data set, and then further comprises:

and preprocessing the data set to obtain a processed data set.

3. The method for constructing the human-computer collaborative corpus of claim 2, wherein the preprocessing the data set to obtain a processed data set specifically comprises:

carrying out format conversion on the text to obtain a text with a preset format;

eliminating irregular texts in the preset format to obtain a processed data set; the irregular text comprises a table, a picture, a school name and a teacher name.

4. The method for constructing a human-computer collaborative corpus according to claim 3, wherein the manually labeling the data set to obtain a basic corpus specifically comprises:

according to knowledge dimensions and classification methods in other fields, and by combining the characteristics of junior middle school mathematics disciplines, marking specifications are formulated; such other fields include the medical field and the military field;

performing lexical and syntactic analysis on the processed data set to construct event triples;

manually pre-labeling the texts with a first preset number twice according to the event triad to obtain a pre-labeling result, and updating the labeling specification according to the pre-labeling result; the pre-labeling results comprise a first pre-labeling result and a second pre-labeling result;

carrying out manual formal annotation twice on a second preset number of texts according to the event triplet to obtain a manual formal annotation result, and constructing a basic corpus; the manual formal annotation result comprises a first formal annotation result and a second formal annotation result; the second preset number of texts is different from the first preset number of texts.

5. The method for constructing the human-computer collaborative corpus of claim 4, wherein the manually pre-labeling a first preset number of texts twice according to the event triplet to obtain pre-labeling results, and updating the labeling specifications according to the pre-labeling results specifically comprises:

manually pre-labeling the texts with the first preset number twice according to the event triad to obtain a pre-labeling result;

calculating the second mark consistency of the first pre-labeling result and the second pre-labeling result;

judging whether iteration is carried out for a preset number of times to obtain a second judgment result;

if the second judgment result is that the preset times are not iterated, comparing the first pre-labeling result with the second pre-labeling result to obtain a comparison result;

updating the marking specification according to the comparison result to obtain a new marking specification, and returning to the step of performing manual pre-marking twice on the texts with the first preset number according to the event triplet to obtain a pre-marking result;

and if the second judgment result is that the preset times are iterated, ending the manual pre-labeling and outputting the new labeling specification.

6. The method for constructing a human-computer collaborative corpus of claim 5, wherein the performing two times of artificial formal annotation on a second preset number of texts according to the event triplet to obtain an artificial formal annotation result, and constructing a basic corpus specifically comprises:

carrying out manual formal annotation twice on a second preset number of texts according to the event triad to obtain a manual formal annotation result;

calculating third labeling consistency of the first formal labeling result and the second formal labeling result;

judging whether the third labeling consistency reaches a second labeling consistency threshold value or not to obtain a third judgment result;

if the third judgment result is that the third labeling consistency reaches the second labeling consistency threshold, completing the manual formal labeling and outputting a manual formal labeling result;

constructing the basic corpus according to the artificial formal annotation result;

and if the third judgment result shows that the third annotation consistency does not reach the second annotation consistency threshold, returning to the step of carrying out twice manual formal annotation on the texts with the second preset number according to the event triad to obtain a manual formal annotation result.

7. A human-computer collaborative corpus construction system, comprising:

the model and data set construction module is used for constructing an automatic annotation model and a data set of a corpus; the texts in the data set comprise subject textbooks, teaching designs and teaching guide cases;

the manual labeling module is used for performing manual labeling on the data set to obtain a manual labeling result and construct a basic corpus; the manual marking comprises manual pre-marking and manual formal marking; the manual annotation result comprises a manual formal annotation result and a manual pre-annotation result;

the training module is used for training the corpus automatic annotation model by utilizing the basic corpus to obtain a trained corpus automatic annotation model;

the machine labeling module is used for performing machine labeling on a text to be labeled by using the trained corpus automatic labeling model to obtain a machine labeling result; the text to be marked is the text selected during the manual formal marking;

the calculation module is used for carrying out marking consistency calculation on the machine marking result and the manual formal marking result to obtain first marking consistency;

the judging module is used for judging whether the first marking consistency reaches a first marking consistency threshold value to obtain a judging result;

the first execution module is used for outputting the trained corpus automatic labeling model if the judgment result shows that the first labeling consistency reaches the first labeling consistency threshold value, and taking the trained corpus automatic labeling model as a corpus;

and the second execution module is used for expanding the basic corpus and returning to the step of training the corpus automatic labeling model by using the basic corpus to obtain the trained corpus automatic labeling model if the judgment result shows that the first labeling consistency does not reach the first labeling consistency threshold value.

8. The system for constructing a human-computer collaborative corpus of claim 7, further comprising:

and the data processing module is used for preprocessing the data set to obtain a processed data set.

9. The system for constructing a human-computer collaborative corpus of claim 8, wherein the data processing module comprises:

the format conversion unit is used for carrying out format conversion on the text to obtain the text with a preset format;

the data removing unit is used for removing the irregular texts in the texts with the preset formats to obtain a processed data set; the irregular text comprises a table, a picture, a school name and a teacher name.

10. The system for constructing a human-computer collaborative corpus of claim 9, wherein the manual labeling module comprises:

the marking specification making unit is used for making a marking specification according to the dimensionality of knowledge and classification methods in other fields by combining the characteristics of the junior high school mathematics subject; such other fields include the medical field and the military field;

the data analysis unit is used for carrying out lexical and syntactic analysis on the processed data set and constructing event triples;

the pre-labeling unit is used for manually pre-labeling the texts with the first preset number twice according to the event triad to obtain a pre-labeling result and updating the labeling specification according to the pre-labeling result; the pre-labeling result comprises a first pre-labeling result and a second pre-labeling result;

the manual labeling unit is used for carrying out two times of manual formal labeling on the second preset number of texts according to the event triplet, obtaining manual formal labeling results and constructing a basic corpus; the manual formal annotation result comprises a first formal annotation result and a second formal annotation result; the second preset number of texts is different from the first preset number of texts.

Images