CN115878777A - Judicial document index extraction method based on few-sample comparative learning - Google Patents

Abstract

A method for extracting element indexes aiming at judicial texts comprises the following steps: 1) Acquiring judicial writing data, cleaning the judicial writing data, and constructing training data based on the referee writing. 2) A structured language (JDISL) of the judicial literature indexes is provided, the JDISL is adopted to conduct prompt guidance, a structural form of the indexes based on the prompt is obtained, and the training corpus is further processed into the prompt training corpus. 3) And carrying out few-sample negative sampling amplification on the prompt training corpus. 4) On the basis of an open-source Unilm basic pre-training model, a prompt training corpus is used as input. 5) And (4) taking a Unilm model hidden layer vector, and processing the hidden layer vector into Gaussian embedding. 6) And calculating contrast learning loss of Gaussian embedding, and performing iterative updating to obtain a final model. 7) And inputting the judicial documents of the indexes to be extracted into the trained model, and extracting information by the index extraction model to obtain the information extraction result of each label type.

Description

Judicial document index extraction method based on few-sample comparative learning

technical field

The invention belongs to the technical field of judgment document data processing, and specifically designs a judicial document index extraction method based on few-sample comparative learning (that is, based on knowledge graph).

Background technique

In the information age, the amount of Internet data is growing exponentially. With the rapid development of Internet technology, the information on the network is growing explosively. Not only the scale of information is expanding, but also the types of information are increasing. At the same time, the successful application of a large amount of data in various fields has announced the arrival of the era of big data. Big data plays an increasingly important role in the development of society, and its value has been generally recognized by the society. In recent years, with the continuous deepening of legalization in our country, the trial of judicial cases has become more transparent. The disclosure of judgment documents on the Internet is a typical example. Judgment documents, as "judicial products" that carry the trial process and trial results of court cases, contain rich judicial information, including the court of judgment, case number, litigation claims of the parties, case name, judgment result, applicable law, etc., which precisely gather The core elements of court "big data". Through in-depth mining of this information, it is possible to summarize the law of trial trials, predict trial trends, enhance judicial credibility, and provide technical support for the realization of judicial justice and the construction of a legal society. However, adjudication documents are semi-structured field texts, which include both stylized legal language and everyday common language. At the same time, the writing of adjudication documents is largely determined by judges, which makes adjudication documents polymorphic. A series of characteristics such as sex, heterogeneity and randomness. Therefore, how to extract valuable information from such special texts is a subject of great value and significance.

A large number of judicial texts in the judicial field have a wide range of index extraction needs, which are used to evaluate the effectiveness of judicial procedures or judicial trial reforms. With the construction of a society ruled by law and the development of information technology in our country, the number of judicial documents in the judicial field has grown exponentially. It is difficult to meet the needs of case handling when there are too few cases, and the differences in the professionalism of judicial personnel will also have a certain degree of impact on case judgments. Therefore, the intelligent extraction of key elements of judicial documents can provide reference for judicial personnel, assist in handling cases, improve work efficiency, and is also the key to subsequent in-depth analysis and efficient judgment.

At present, the available annotation data in the field of judicial documents is scarce, the annotation is difficult, and there is a problem of scarcity of available data resources. It belongs to the low-resource and few-sample application scenario; the judicial document index extraction task includes multiple subtasks such as entity extraction, relationship extraction, and event extraction. Different subtasks require different labeled data and different algorithm models, which leads to the application requirements of training multiple subtask models in low-resource and few-sample scenarios. In summary, the extraction of judicial document indicators faces the problems of multi-task algorithm model training in low-resource and few-sample application scenarios, which is difficult to converge, low extraction accuracy, and poor generalization. Such a judicial document indicator extraction model cannot satisfy the needs of judicial personnel. The goal of handling cases and improving work efficiency. Therefore, the present invention proposes a judicial document index extraction method based on comparative learning with few samples, which effectively solves the above-mentioned problems.

Contents of the invention

Aiming at the deficiencies of the prior art, the present invention provides a judicial document index extraction method based on few-sample comparative learning (that is, based on knowledge graph). Firstly, obtain judicial document data, perform data cleaning on judicial document data, and construct training data based on judgment documents. Then use the designed Judicial Document Index Structured Language (JDISL) to conduct prompt guidance on the judicial document indicators, and further process the training corpus into prompt training corpus. Then the prompt training corpus is amplified by few-sample negative sampling. On the open-source UniLM basic pre-training model, the prompt training corpus is used as input, the hidden layer vector of the UniLM model is taken, processed into a Gaussian embedding, the contrastive learning loss of the Gaussian embedding is calculated, and the final model is iteratively updated. Input the judicial documents of the indicators to be extracted into the trained model, and obtain the information extraction results of each label type.

Aiming at the deficiencies of the prior art, the present invention provides a judicial document index extraction method based on comparative learning with few samples.

According to one embodiment of the present invention, there is provided a method for extracting judicial document indicators based on few-sample contrastive learning, the method comprising the following steps:

1) Obtain judicial document data (such as judgment documents or judgment document data published on the Internet), perform data cleaning on judicial document data, and construct training data containing training corpus based on judgment documents (that is, judicial documents);

2) A Judicial Document Indicator Structured Language (JDISL) is proposed for judicial document indicators. JDISL is used for prompt guidance, and the prompt-based construction form of indicators is obtained, and the training corpus is further processed into prompt training corpus;

3) Amplify the prompt training corpus by negative sampling with few samples to obtain the amplified prompt training corpus;

4) On the basis of the open source Unilm basic pre-training model, the amplified prompt training corpus is used as input;

5) Take the hidden layer vector of the Unilm model and process it as a Gaussian embedding;

6) Calculate the contrastive learning loss of Gaussian embedding, and iteratively update to obtain the final model;

7) Input the judicial documents of the indicators to be extracted into the trained final model, and the indicator extraction model performs information extraction to obtain the information extraction results of each label type.

Preferably, step 2) adopts prompt guidance to the judicial document index, obtains the construction form based on prompt of the index, and further processes the training corpus into the prompt training corpus and includes the following two substeps of 2.1) and 2.2):

Step 2.1) Prompt construction of court trial element indicators in judicial documents by using information extraction structured generation means (or information extraction means) including entity extraction, relationship extraction, event extraction and response extraction. In order to create a framework from text to prompt structure for unified modeling, unified modeling is realized for entity indicators, relationship indicators, event indicators, and response indicators in judicial documents.

Information extraction is affected by different goals, heterogeneous structures, and specific demand patterns, and is divided into entity extraction, relationship extraction, event extraction, and response extraction. Prompt construction is carried out on the index of court trial elements. Unified modeling is a generation framework from text to prompt structure, and realizes unified modeling of entity indicators, relationship indicators, event indicators, and response indicators in judicial documents.

The structured generation of information extraction can be decomposed into two atomic operations.

1) Locating: locating target information fragments from sentences, such information fragments include, for example, entities and events in judicial documents.

2) Association: Connecting different pieces of information according to the required association, such as a connection between (one) entity and (another) entity, between an entity and an event, or between (one) event and (another) event the connection between.

Different information extraction task structures can be generated by combining atomic operations.

According to the specific structure of judicial document indicators, a judicial document indicator structured language (JDISL) is designed, in which the symbol ":" of JDISL represents the mapping from information to its location or associated fragments, and the symbols "[], ()" are used It is used to express the hierarchical structure between extracted index information.

JDISL consists of three parts:

1) info_name: Indicates a specific piece of information that exists in the source text

2) relation_name: Indicates a specific piece of information that exists in the source text, and this piece of information is associated with the upper-level info information in the structure.

3) info_span: Indicates the specific information of the source text or the text segment span corresponding to the associated segment.

Through JDISL, different information extraction structures can be uniformly encoded, and different judicial document index information extraction tasks can be uniformly modeled as the same generation process from text to structure.

For entity recognition and event detection tasks can be modeled as:

(info_name:info_span),

For relation extraction and event extraction tasks, it can be modeled as:

(info_name:info_span(relation_name:info_span),...),

Taking court trial elements as an example, the indicators to be extracted include witnesses, plaintiffs, defendants, evidence, courts, and judges, and the relationship between indicators includes (plaintiff) suing (defendant). It is prompted to guide it, and for the entity, it is expressed in parallel as:

[witness, plaintiff, defendant, judge]

For attribute and relationship dependencies, express it as:

[court: evidence, witness: (testimony), [plaintiff: (suit)], defendant, judge]

Step 2.2) After the prompt construction is performed on the court trial element indicators in the judicial documents, the training data containing the training corpus is processed based on the prompt construction, so as to obtain the prompt training corpus.

After the prompt construction of the court trial element indicators, the training corpus needs to be processed based on the prompt construction to obtain the prompt training corpus.

For example, for the original corpus:

Content: "In the labor contract dispute case between the plaintiff Cui and the defendant Zhang, the testimony of the witness Liu showed that Cui was at fault."

Label: [2,3, "Plaintiff"], [7,8, "Defendant"], [20,21, "Witness"], [27,32, "Testimony"], [(2,3),( 7,8), "Sue"]

Based on the prompt structure, it is processed into a prompt structured corpus:

Content: "A labor contract dispute between the plaintiff Cui and the defendant Zhang"

Result: [(2,3, "Cui"): ("Sued"), (7,8, "Zhang"): ("Sued"), (20,21, "Liu"): ( 27,32,"Cui was at fault")]

Prompt is constructed as a predefined pattern, and the training text is processed into a prompt structured corpus.

Preferably, in step 3), the process of carrying out the few-sample negative sampling amplification of the prompt training corpus more specifically includes the following steps:

Step 3.1) For prompt construction corpus:

Content: text

Result: [[info_type_1],[info_type_2],…,[info_type_n]]

Among them, [info_type_n] means that the extracted information set whose label type is n has the structure form [[info_name], [relation_name], [info_span]].

Process Result into multiple result subsets based on promt:

Result_1:[info_type_1][prompt:label_1]

Result_2:[info_type_2][prompt:label_2]

…

Result_n:[info_type_n][prompt:label_n]

Where label_n represents the label type n corresponding to the [info_type_n] set;

Then perform negative sampling based on the prompt type, and expand the amount of data from n to n^2, such as expanding Result_1 to n times:

Result_1:[info_type_1][prompt:label_1]

Result_1:[info_type_1][prompt:label_2]

…

Result_1:[info_type_1][prompt:label_n].

Thus, the amplified prompt training corpus is obtained.

Preferably, in step 4), on the basis of the Unilm basic pre-training model of open source, the process of using the prompt training corpus as input more specifically includes:

Step 4.1) Process the prompt training corpus into the input form required by the Unilm model. The input forms accepted by the Unilm model are:

{x ₁ , x ₂ , x ₃ ,..., x _m-1 , x _m }

The prompt structured information obtained by JDISL is [[info_name], [relation_name], [text]], where text is the information referred to within the info_span span in the source text.

Then the prompt structured information is used as a prefix to associate with the segment information to obtain the model input corpus: {info_name_1,...info_name_n,...,relation_name_1,...,relation_name_n,...,text_1,...,text_n, x ₁ , x ₂ , x ₃ , ..., x _m-1 , x _m };

Above, the value of n refers to the number of entity/relationship fragment information. The value of m refers to the sequence length of the input document.

Unilm can be used for both generation and classification tasks. After the input, the tokenizer pre-trained by Unilm is used to segment the input, and then the vector embedding is performed. The vector embedding includes token embedding, segment embedding, and position embedding.

Token embedding refers to inserting the [CLS] tag at the beginning of each sentence and the [SEP] tag at the end of the sentence, where the [CLS] tag represents the vector of the current sentence, and the [SEP] tag represents the sentence used to segment the sentence in the text .

Segment embeddings are used to distinguish two sentences. Different sentences are marked with A and B respectively, so the input sentence is expressed as (E _A , E _B , E _A , E _B ,…).

Position embedding is to add position information according to the relative position of the subscript index of the sentence, that is, or such as [0,1,2,3...].

After embedding, it will go through 24 serial transformer structure layers, and calculate the hidden layer output vector through the multi-head attention mechanism.

h={H; h ₁ , h ₂ , h ₃ , . . . , h _m-1 , h _m }

Among them, h _m represents the hidden layer state of the m-th judicial document sequence, and H represents the state set of prompt structured information.

As preferably, in step 5), get Unilm model hidden layer vector, the process that is processed as Gaussian embedding comprises more specifically:

According to step 5.1), use the projection network f _u and f _∑ consisting of fully connected layers and nonlinear activation functions to generate Gaussian distribution parameters:

u _i =f _u (h _i );

Σ _i ＝ELu(f _∑ (h _i ))+(1+∈)

where u _i , Σ _i represent the average covariance and diagonal covariance of the Gaussian embedding respectively (non-zero elements are only along the diagonal of the matrix), both f _u and f _∑ are composed of a single-layer linear layer, with Relu as the activation function, For stability ∈ take e^(-14).

Process the hidden layer output vector into a Gaussian embedding (u _i , Σ _i ).

Preferably, in step 6), the contrastive learning loss of Gaussian embedding is calculated, and the process of iteratively updating to obtain the final model more specifically includes the following two substeps of 6.1) and 6.2):

Step 6.1) To compute the contrast loss, consider the KL divergence between all valid token pairs in the sampled corpus, two tokens x _q and x _p are considered positive examples if they have the same label y _q = y _p , for which The Gaussian embedding N(u _p , Σ _p ) and N(u _q , ∑ _q ), calculate its KL divergence:

Both directions of KL divergence are calculated since it is not symmetrical.

d(p,q)＝1/2(D _KL |N _p ||N _q |+D _KL |N _q ||N _p |)

For a positive sample x _p , compute the Gaussian embedding loss of x _p with respect to other valid tokens in the batch:

χ _p ＝{(χ _q ，y _q )∈χ|y _p ＝y _q ·p≠q}

Finally, the contrastive learning loss is calculated using KL divergence and Gaussian embedding loss:

Step 6.2) Backpropagation is performed based on the contrastive learning loss function, and the parameters of the UniLM model are iteratively updated to obtain an information extraction model that can finally be used for index extraction of judicial documents, that is, the final model.

At present, the available annotation data in the field of judicial documents is scarce, the annotation is difficult, and there is a problem of scarcity of available data resources. It belongs to the low-resource and few-sample application scenario; the judicial document index extraction task includes multiple subtasks such as entity extraction, relationship extraction, and event extraction. Different subtasks require different labeled data and different algorithm models, which leads to the application requirements of training multiple subtask models in low-resource and few-sample scenarios. Therefore, there is an urgent need for a judicial document index extraction method to solve the problems of difficult convergence, low extraction accuracy, and poor generalization of multi-task algorithm models in low-resource and few-sample application scenarios.

In the present invention, prompt guidance is adopted for the indicators of judicial documents, the prompt-based construction form of the indicators is obtained, and the training corpus is further processed into prompt training corpus.

Information extraction is affected by different goals, heterogeneous structures, and specific demand patterns, and is divided into entity extraction, relationship extraction, event extraction, and response extraction. Prompt construction is carried out on the index of court trial elements. Unified modeling is a generation framework from text to prompt structure, and realizes unified modeling of entity indicators, relationship indicators, event indicators, and response indicators in judicial documents.

The structured generation of information extraction can be decomposed into two atomic operations.

1) Locating: Locate target information fragments from sentences, such as entities and events in documents.

2) Association: Connect different pieces of information according to the required association, such as the connection between entities and entities and the connections between entities and events.

Different information extraction task structures can be generated by combining atomic operations.

According to the specific structure of judicial document indicators, a judicial document indicator structured language (JDISL) is designed. JDISL: represents the mapping from information to its location or associated fragments, [], () are used to represent the extracted index information hierarchy.

JDISL consists of three parts:

1) info_name: Indicates a specific piece of information that exists in the source text

2) relation_name: Indicates a specific piece of information that exists in the source text, and this piece of information is associated with the upper-level info information in the structure.

3) info_span: Indicates the specific information of the source text or the text segment span corresponding to the associated segment.

Through JDISL, different information extraction structures can be uniformly encoded, and different judicial document index information extraction tasks can be uniformly modeled as the same generation process from text to structure.

For entity recognition and event detection tasks can be modeled as:

(info_name:info_span),

For relation extraction and event extraction tasks, it can be modeled as:

(info_name:info_span(relation_name:info_span),...),

Taking court trial elements as an example, the indicators to be extracted include witnesses, plaintiffs, defendants, evidence, courts, and judges, and the relationship between indicators includes (plaintiff) suing (defendant). It is prompted to guide it, and for the entity, it is expressed in parallel as:

[witness, plaintiff, defendant, judge]

For attribute and relationship dependencies, express it as:

[court: evidence, witness: (testimony), [plaintiff: (suit)], defendant, judge]

After the prompt construction of the court trial element indicators, the training corpus needs to be processed based on the prompt construction to obtain the prompt training corpus.

For example, for the original corpus:

Content: "In the labor contract dispute case between the plaintiff Cui and the defendant Zhang, the testimony of the witness Liu showed that Cui was at fault."

Label: [2,3, "Plaintiff"], [7,8, "Defendant"], [20,21, "Witness"], [27,32, "Testimony"], [(2,3),( 7,8), "Sue"]

Based on the prompt structure, it is processed into a prompt structured corpus:

Content: "A labor contract dispute between the plaintiff Cui and the defendant Zhang"

Result: [(2,3,"Cui"):("Sued"), (7,8,"Zhang"):("Sued"), (20,21,"Liu"):( 27,32,"Cui was at fault")]

Prompt is constructed as a predefined pattern, and the training text is processed into a prompt structured corpus.

In the present invention, the prompt training corpus is subjected to few-sample negative sampling amplification.

For prompt construction corpus:

Content: text

Result: [[info_type_1],[info_type_2],…,[info_type_n]]

Among them, [info_type_n] means that the extracted information set whose label type is n has the structure form [[info_name], [relation_name], [info_span]].

Process Result into multiple result subsets based on promt:

Result_1:[info_type_1][prompt:label_1]

Result_2:[info_type_2][prompt:label_2]

…

Result_n:[info_type_n][prompt:label_n]

Where label_n represents the label type n corresponding to the [info_type_n] set;

Then perform negative sampling based on the prompt type, and expand the amount of data from n to n^2, such as expanding Result_1 to n times:

Result_1:[info_type_1][prompt:label_1]

Result_1:[info_type_1][prompt:label_2]

…

Result_1:[info_type_1][prompt:label_n].

The amplified prompt training corpus is obtained above.

In the present invention, on the basis of the open-source Unilm basic pre-training model, the prompt training corpus is used as input.

Process the prompt training corpus into the input form required by the Unilm model. The input forms accepted by the Unilm model are:

{x ₁ , x ₂ , x ₃ ,..., x _m-1 , x _m }

The prompt structured information obtained by JDISL is [[info_name], [relation_name], [text]], where text is the information referred to within the info_span span in the source text.

Then associate the prompt structured information as a prefix with the fragment information to obtain the model input corpus:

{info_name_1,...info_name_n,...,relation_name_1,...,relation_name_n,...,text_1,...,text_n,x ₁ ,x ₂ ,x ₃ ,...,x _m-1 ,x _m }

Unilm can be used for both generation tasks and classification tasks. After the input, the tokenizer pre-trained by Unilm is used to segment the input, and then vector embedding is performed. The vector embedding includes token embedding, segment embedding, and position embedding.

Token embedding refers to inserting the [CLS] tag at the beginning of each sentence and the [SEP] tag at the end of the sentence, where the [CLS] tag represents the vector of the current sentence, and the [SEP] tag represents the sentence used to segment the sentence in the text .

Segment embeddings are used to distinguish two sentences. Different sentences are marked with A and B respectively, so the input sentence is expressed as (E _A , E _B , E _A , E _B ,…).

Position embedding is to add position information according to the relative position of the subscript index of the sentence, such as [0,1,2,3...].

After embedding, it will go through 24 serial transformer structure layers, and calculate the hidden layer output vector through the multi-head attention mechanism.

h={H; h ₁ , h ₂ , h ₃ , . . . , h _m-1 , h _m }

Among them, h _m represents the hidden layer state of the m-th judicial document sequence, and H represents the state set of prompt structured information.

In the present invention, the hidden layer vector of the Unilm model is taken and processed as a Gaussian embedding.

Use a projection network f _u and f _∑ consisting of fully connected layers and nonlinear activation functions to generate Gaussian distribution parameters:

u _i =f _u (h _i );

∑ _i ＝ ELU(f _∑ (h _i ))+(1+∈)

where u _i , ∑ _i represent the average covariance and diagonal covariance of the Gaussian embedding respectively (non-zero elements are only along the diagonal of the matrix), both f _u and f _∑ are composed of a single linear layer, with Relu as the activation function, For stability ∈ take e^(-14).

Process the hidden layer output vector into a Gaussian embedding (u _i , Σ _i ).

In the present invention, the contrastive learning loss of the Gaussian embedding is calculated, and iteratively updated to obtain the final model.

To compute the contrast loss, consider the KL divergence between all valid token pairs in the sampled corpus, two tokens x _q and x _p are considered positive examples if they have the same label y _q = y _p , for their Gaussian embeddings N(u _p , ∑ _p ) and N(u _q , Σ _q ), calculate its KL divergence:

Both directions of KL divergence are calculated since it is not symmetrical.

d(p,q)＝1/2(D _KL |N _p ||N _q |+D _KL |N _q ||N _P |)

For a positive sample x _p , compute the Gaussian embedding loss of x _p with respect to other valid tokens in the batch:

χ _p ={(χ _q , y _g )∈χ[y _p =y _q , p≠q)

Finally, the contrastive learning loss is calculated using KL divergence and Gaussian embedding loss:

Based on the backpropagation of the contrastive learning loss function, the parameters of the UniLM model are iteratively updated to obtain an information extraction model that can finally be used for the extraction of judicial document indicators.

Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

1, in the present invention, by proposing a kind of judicial document index structured language (JDISL), adopt JDISL to carry out prompt guidance, obtain the index based on the structural form of prompt, further train corpus to be processed as prompt training corpus, effectively to different The information extraction structure is uniformly encoded, so that effective joint extraction can be carried out, which effectively solves the difficulty of unified modeling of multiple extraction subtasks of the judicial document index extraction multi-task algorithm model, and the problem of excessive training costs.

2. In the present invention, the prompt training corpus is amplified by negative sampling with few samples, which effectively solves the problem of difficult learning with few samples caused by too little available labeling data in the low-resource field of judicial documents.

3. In the present invention, the hidden layer vector of the Unilm model is taken and processed as Gaussian embedding, and the information class distribution is clearly modeled by Gaussian embedding, which promotes the generalized feature representation of various indicators and contributes to the development of a few-sample target domain. Adaptive, which effectively solves the problems of poor generalization performance and low extraction accuracy of the few-shot learning model.

4. In the present invention, the comparative learning loss function is used to calculate the loss training model, and iteratively updated to obtain the final model, which effectively solves the problems of difficult model training convergence and low extraction accuracy.

Description of drawings

Fig. 1 is a schematic diagram of the structure of the method for extracting judicial document indicators based on few-sample comparative learning in the present invention.

Fig. 2 is a structure diagram of a few-sample data processing module of the judicial document index extraction method based on few-sample contrastive learning in the present invention.

Fig. 3 is a network structure diagram of a few-sample comparative learning based on a judicial document index extraction method based on a few-sample comparative learning of the present invention.

Detailed ways

The technical solutions of the present invention are illustrated below, and the protection scope of the present invention includes but is not limited to the embodiments.

A method for extracting judicial document indicators based on comparative learning with few samples, the method includes the following steps:

1) Obtain judicial document data, perform data cleaning on judicial document data, and construct training data based on judgment documents.

2) A Judicial Document Indicator Structured Language (JDISL) is proposed for judicial document indicators. JDISL is used for prompt guidance, and the prompt-based construction form of indicators is obtained, and the training corpus is further processed into prompt training corpus.

3) Amplify the prompt training corpus by negative sampling with few samples.

4) Based on the open source Unilm basic pre-training model, the prompt training corpus is used as input.

5) Take the hidden layer vector of the Unilm model and process it as a Gaussian embedding.

6) Calculate the contrastive learning loss of the Gaussian embedding, and update iteratively to obtain the final model.

7) Input the judicial documents of the indicators to be extracted into the trained model, and the indicator extraction model performs information extraction to obtain the information extraction results of each label type.

As preferably, step 2) adopts prompt guidance to judicial document index, obtains index based on the structural form of prompt, and further training corpus is processed as prompt training corpus and includes:

Step 2.1) Information extraction is affected by different objectives, heterogeneous structures, and specific demand patterns, and is divided into entity extraction, relation extraction, event extraction, response extraction, etc. Prompt construction is carried out on the index of court trial elements. Unified modeling is a generation framework from text to prompt structure, and realizes unified modeling of entity indicators, relationship indicators, event indicators, and response indicators in judicial documents.

The structured generation of information extraction can be decomposed into two atomic operations.

1) Locating: Locate target information fragments from sentences, such as entities and events in documents.

2) Association: Connect different pieces of information according to the required association, such as the connection between entities and entities and the connections between entities and events.

Different information extraction task structures can be generated by combining atomic operations.

According to the specific structure of judicial document indicators, a judicial document indicator structured language (JDISL) is designed. JDISL: represents the mapping from information to its location or associated fragments, [], () are used to represent the extracted index information hierarchy.

JDISL consists of three parts:

1) info_name: Indicates a specific piece of information that exists in the source text

2) relation_name: Indicates a specific piece of information that exists in the source text, and this piece of information is associated with the upper-level info information in the structure.

3) info_span: Indicates the specific information of the source text or the text segment span corresponding to the associated segment.

Through JDISL, different information extraction structures can be uniformly encoded, and different judicial document index information extraction tasks can be uniformly modeled as the same generation process from text to structure.

For entity recognition and event detection tasks can be modeled as:

(info_name:info_span)

For relation extraction and event extraction tasks, it can be modeled as:

(info_name:info_span(relation_name:info_span),...)

Taking court trial elements as an example, the indicators to be extracted include witnesses, plaintiffs, defendants, evidence, courts, and judges, and the relationship between indicators includes (plaintiff) suing (defendant). It is prompted to guide it, and for the entity, it is expressed in parallel as:

[witness, plaintiff, defendant, judge]

For attribute and relationship dependencies, express it as:

[court: evidence, witness: (testimony), [plaintiff: (suit)], defendant, judge]

Step 2.2) After the prompt construction of the court trial element indicators, the training corpus needs to be processed based on the prompt construction, so as to obtain the prompt training corpus.

For example, for the original corpus:

Content: "In the labor contract dispute case between the plaintiff Cui and the defendant Zhang, the testimony of the witness Liu showed that Cui was at fault."

Label: [2,3, "Plaintiff"], [7,8, "Defendant"], [20,21, "Witness"], [27,32, "Testimony"], [(2,3),( 7,8), "Sue"]

Based on the prompt structure, it is processed into a prompt structured corpus:

Content: "A labor contract dispute between the plaintiff Cui and the defendant Zhang"

Result: [(2,3,"Cui"):("Sued"), (7,8,"Zhang"):("Sued"), (20,21,"Liu"):( 27,32,"Cui was at fault")]

Prompt is constructed as a predefined pattern, and the training text is processed into a prompt structured corpus.

As a preference, step 3) carries out the few-sample negative sampling amplification of the prompt training corpus:

Step 3.1) For prompt construction corpus:

Content: text

Result: [[info_type_1],[info_type_2],…,[info_type_n]]

Among them, [info_type_n] means that the extracted information set whose label type is n has the structure form [[info_name], [relation_name], [info_span]].

Process Result into multiple result subsets based on promt:

Result_1:[info_type_1][prompt:label_1]

Result_2:[info_type_2][prompt:label_2]

…

Result_n:[info_type_n][prompt:label_n]

Where label_n represents the label type n corresponding to the [info_type_n] collection

Then perform negative sampling based on the prompt type, and expand the amount of data from n to n^2, such as expanding Result_1 to n times:

Result_1:[info_type_1][prompt:label_1]

Result_1:[info_type_1][prompt:label_2]

…

Result_1:[info_type_1][prompt:label_n]

As a preference, step 4) uses the prompt training corpus as input on the basis of the open source Unilm basic pre-training model:

Step 4.1) Process the prompt training corpus into the input form required by the Unilm model. The input forms accepted by the Unilm model are:

{x ₁ , x ₂ , x ₃ ,..., x _m-1 , x _m }

The prompt structured information obtained by JDISL is [[info_name], [relation_name], [text]], where text is the information referred to within the info_span span in the source text.

Then associate the prompt structured information as a prefix with the fragment information to obtain the model input corpus:

{info_name_1,...info_name_n,...,relation_name_1,...,relation_name_n,...,text_1,...,text_n,x ₁ ,x ₂ ,x ₃ ,...,x _m-1 ,x _m }

Unilm can be used for both generation tasks and classification tasks. After the input, first use Unilm pre-trained tokenizer to segment the input, and then perform vector embedding, including token embedding, segment embedding, and position embedding.

Token embedding refers to inserting the [CLS] tag at the beginning of each sentence and the [SEP] tag at the end of the sentence, where the [CLS] tag represents the vector of the current sentence, and the [SEP] tag represents the sentence used to segment the sentence in the text .

Segment embeddings are used to distinguish two sentences. Different sentences are marked with A and B respectively, so the input sentence is expressed as (E _A , E _B , E _A , E _B ,…).

Position embedding is to add position information according to the relative position of the subscript index of the sentence, such as [0,1,2,3...].

After embedding, it will go through 24 serial transformer structure layers, and calculate the hidden layer output vector through the multi-head attention mechanism.

h={H; h ₁ , h ₂ , h ₃ , . . . , h _m-1 , h _m }

Among them, h _m represents the hidden layer state of the m-th judicial document sequence, and H represents the state set of prompt structured information.

Preferably, step 5) takes the hidden layer vector of the Unilm model and processes it as a Gaussian embedding:

According to step 5.1), use the projection network f _u and f _∑ consisting of fully connected layers and nonlinear activation functions to generate Gaussian distribution parameters:

u _i =f _u (h _i );

∑ _i ＝ ELU(f _∑ (h _i ))+(1+∈)

where u _i , ∑ _i represent the average covariance and diagonal covariance of the Gaussian embedding respectively (non-zero elements are only along the diagonal of the matrix), both f _u and f _Σ are composed of a single-layer linear layer, with Relu as the activation function, For stability ∈ take e^(-14).

Then the hidden layer output vector is processed into a Gaussian embedding (u _i , Σ _i ).

As a preference, step 6) calculates the contrastive learning loss of Gaussian embedding, and iteratively updates to obtain the final model:

Step 6.1) To compute the contrast loss, consider the KL divergence between all valid _token pairs in the sampled corpus, two tokens x _q and x _p are considered as positive examples if they have the same label y _q = y _p , For their Gaussian embeddings N(u _p , Σ _p ) and N(u _q , ∑ _q ), calculate its KL divergence:

Both directions of KL divergence are calculated since it is not symmetrical.

d(p,q)＝1/2(D _KL |N _p ||N _q |+D _KL |N _q ||N _p |)

For a positive sample x _p , compute the Gaussian embedding loss of x _p with respect to other valid tokens in the batch:

χ _p ＝((χ _q ，y _q )∈χ|y _p ＝y _q ，p≠q}

Finally, the contrastive learning loss is calculated using KL divergence and Gaussian embedding loss:

Step 6.2) Backpropagation is performed based on the contrastive learning loss function, and the parameters of the UniLM model are iteratively updated to obtain an information extraction model that can finally be used for the extraction of judicial document indicators.

Example 1

As shown in Figure 1, a judicial document index extraction method based on few-sample comparative learning, the method includes the following steps:

1) Obtain judicial document data, perform data cleaning on judicial document data, and construct training data based on judgment documents.

2) A Judicial Document Indicator Structured Language (JDISL) is proposed for judicial document indicators. JDISL is used for prompt guidance, and the prompt-based construction form of indicators is obtained, and the training corpus is further processed into prompt training corpus.

3) Amplify the prompt training corpus by negative sampling with few samples.

4) Based on the open source Unilm basic pre-training model, the prompt training corpus is used as input.

5) Take the hidden layer vector of the Unilm model and process it as a Gaussian embedding.

6) Calculate the contrastive learning loss of the Gaussian embedding, and update iteratively to obtain the final model.

7) Input the judicial documents of the indicators to be extracted into the trained model, and the indicator extraction model performs information extraction to obtain the information extraction results of each label type.

Example 2

Repeat embodiment 1, just step 2) adopts prompt guidance to the judicial document index, obtains the index based on the construction form of prompt, further training corpus is processed as prompt training corpus and includes, as shown in Figure 2:

Step 2.1) Information extraction is affected by different objectives, heterogeneous structures, and specific demand patterns, and is divided into entity extraction, relation extraction, event extraction, response extraction, etc. Prompt construction is carried out on the index of court trial elements. Unified modeling is a generation framework from text to prompt structure, and realizes unified modeling of entity indicators, relationship indicators, event indicators, and response indicators in judicial documents.

The structured generation of information extraction can be decomposed into two atomic operations.

1) Locating: Locate target information fragments from sentences, such as entities and events in documents.

2) Association: Connect different pieces of information according to the required association, such as the connection between entities and entities and the connections between entities and events.

Different information extraction task structures can be generated by combining atomic operations.

According to the specific structure of judicial document indicators, a judicial document indicator structured language (JDISL) is designed. JDISL: represents the mapping from information to its location or associated fragments, [], () are used to represent the extracted index information hierarchy.

JDISL consists of three parts:

1) info_name: Indicates a specific piece of information that exists in the source text

2) relation_name: Indicates a specific piece of information that exists in the source text, and this piece of information is associated with the upper-level info information in the structure.

3) info_span: Indicates the specific information of the source text or the text segment span corresponding to the associated segment.

Through JDISL, different information extraction structures can be uniformly encoded, and different judicial document index information extraction tasks can be uniformly modeled as the same generation process from text to structure.

For entity recognition and event detection tasks can be modeled as:

(info_name:info_span)

For relation extraction and event extraction tasks, it can be modeled as:

(info_name:info_span(relation_name:info_span),...)

Taking court trial elements as an example, the indicators to be extracted include witnesses, plaintiffs, defendants, evidence, courts, and judges, and the relationship between indicators includes (plaintiff) suing (defendant). It is prompted to guide it, and for the entity, it is expressed in parallel as:

[witness, plaintiff, defendant, judge]

For attribute and relationship dependencies, express it as:

[court: evidence, witness: (testimony), [plaintiff: (suit)], defendant, judge]

Step 2.2) After the prompt construction of the court trial element indicators, the training corpus needs to be processed based on the prompt construction, so as to obtain the prompt training corpus.

For example, for the original corpus:

Content: "In the labor contract dispute case between the plaintiff Cui and the defendant Zhang, the testimony of the witness Liu showed that Cui was at fault."

Label: [2,3, "Plaintiff"], [7,8, "Defendant"], [20,21, "Witness"], [27,32, "Testimony"], [(2,3),( 7,8), "Sue"]

Based on the prompt structure, it is processed into a prompt structured corpus:

Content: "A labor contract dispute between the plaintiff Cui and the defendant Zhang"

Result: [(2,3,"Cui"):("Sued"), (7,8,"Zhang"):("Sued"), (20,21,"Liu"):( 27,32,"Cui was at fault")]

Prompt is constructed as a predefined pattern, and the training text is processed into a prompt structured corpus.

Example 3

Repeat embodiment 2, just step 3) the prompt training corpus is carried out few sample negative sampling amplification, as shown in Figure 2:

Step 3.1) For prompt construction corpus:

Content: text

Result: [[info_type_1],[info_type_2],…,[info_type_n]]

Among them, [info_type_n] indicates that the extracted information set whose label type is n has the structure form [[info_name], [relation_name], [info_span]].

Process Result into multiple result subsets based on promt:

Result_1:[info_type_1][prompt:label_1]

Result_2:[info_type_2][prompt:label_2]

…

Result_n:[info_type_n][prompt:label_n]

Where label_n represents the label type n corresponding to the [info_type_n] collection

Then perform negative sampling based on the prompt type, and expand the amount of data from n to n^2, such as expanding Result_1 to n times:

Result_1:[info_type_1][prompt:label_1]

Result_1:[info_type_1][prompt:label_2]

…

Result_1:[info_type_1][prompt:label_n]

Example 4

Repeat embodiment 3, as shown in Figure 2, just step 4) on the basis of the Unilm basic pre-training model of open source, prompt training corpus is used as input, as shown in Figure 3:

Step 4.1) Process the prompt training corpus into the input form required by the Unilm model. The input forms accepted by the Unilm model are:

{x ₁ , x ₂ , x ₃ ,..., x _m-1 , x _m }

The prompt structured information obtained by JDISL is [[info_name], [relation_name], [text]], where text is the information referred to within the info_span span in the source text.

Then associate the prompt structured information as a prefix with the fragment information to obtain the model input corpus:

{info_name_1,...info_name_n,...,relation_name_1,...,relation_name_n,...,text_1,...,text_n,x ₁ ,x ₂ ,x ₃ ,...,x _m-1 ,x _m }

Unilm can be used for both generation tasks and classification tasks. After the input, first use Unilm pre-trained tokenizer to segment the input, and then perform vector embedding, including token embedding, segment embedding, and position embedding.

Token embedding refers to inserting the [CLS] tag at the beginning of each sentence and the [SEP] tag at the end of the sentence, where the [CLS] tag represents the vector of the current sentence, and the [SEP] tag represents the sentence used to segment the sentence in the text .

Segment embeddings are used to distinguish two sentences. Different sentences are marked with A and B respectively, so the input sentence is expressed as (E _A , E _B , E _A , E _B ,…).

Position embedding is to add position information according to the relative position of the subscript index of the sentence, ie or for example [0,1,2,3...].

After embedding, it will go through 24 serial transformer structure layers, and calculate the hidden layer output vector through the multi-head attention mechanism.

h={H; h ₁ , h ₂ , h ₃ , . . . , h _m-1 , h _m }

Among them, h _m represents the hidden layer state of the m-th judicial document sequence, and H represents the state set of prompt structured information.

Example 5

Repeat embodiment 4, as shown in Figure 3, just get Unilm model hidden layer vector in step 5), be processed as Gaussian embedding:

According to step 5.1), use the projection network f _u and f _∑ composed of fully connected layers and nonlinear activation functions to generate Gaussian distribution parameters, as shown in Figure 3:

u _i =f _u (h _i );

∑ _i ＝ ELU(f _∑ (h ₁ )+(1+∈)

where u _i , ∑ _i represent the average covariance and diagonal covariance of the Gaussian embedding respectively (non-zero elements are only along the diagonal of the matrix), both f _u and f _∑ are composed of a single linear layer, with Relu as the activation function, For stability ∈ take e^(-14).

Then the hidden layer output vector is processed into a Gaussian embedding (u _i , ∑ _i ).

Example 6

Repeat embodiment 5, as shown in Figure 3, only step 6) calculates the comparative learning loss of Gaussian embedding, and iteratively updates to obtain the final model:

Step 6.1) To calculate the contrast loss, consider the KL divergence between all valid token pairs in the sampled corpus, two tokens x _q and x _p are considered as positive examples if they have the same label y _q = y _q , for which Gaussian embedding N(u _p , ∑ _p ) and N(u _q , ∑ _q ), calculate its KL divergence:

Both directions of KL divergence are calculated since it is not symmetrical.

d(p,q)＝1/2(D _KL |N _p ||N _q |+D _KL |N _q ||N _p |)

For a positive sample x _p , compute the Gaussian embedding loss of x _p with respect to other valid tokens in the batch:

x _p ＝{(x _q ，y _q )∈χ|yp=y _q ，p≠q}

Finally, the contrastive learning loss is calculated using KL divergence and Gaussian embedding loss:

Step 6.2), as shown in Figure 3, performs backpropagation based on the comparative learning loss function, and iteratively updates the parameters of the UniLM model to obtain an information extraction model that can finally be used for the extraction of judicial document indicators.

Claims (6)

1. A method for extracting judicial document indicators based on few-sample contrastive learning, the method comprising the following steps: 1) Obtain judicial document data, perform data cleaning on judicial document data, and construct training data containing training corpus based on judicial documents; 2) A Judicial Document Indicator Structured Language (JDISL) is proposed for judicial document indicators. JDISL is used for prompt guidance, and the prompt-based construction form of indicators is obtained, and the training corpus is further processed into prompt training corpus; 3) Amplify the prompt training corpus by negative sampling with few samples to obtain the amplified prompt training corpus; 4) On the basis of the open source Unilm basic pre-training model, the amplified prompt training corpus is used as input; 5) Take the hidden layer vector of the Unilm model and process it as a Gaussian embedding; 6) Calculate the contrastive learning loss of Gaussian embedding, and iteratively update to obtain the final model; 7) Input the judicial documents of the indicators to be extracted into the trained final model, and the indicator extraction model performs information extraction to obtain the information extraction results of each label type. 2. the judicial document index extraction method according to claim 1, it is characterized in that, step 2) adopts prompt guidance to judicial document index, obtains index based on the construction form of prompt, further training corpus is processed as the process of prompt training corpus include: Step 2.1) Use the information extraction structured generation means or information extraction means including entity extraction, relationship extraction, event extraction and response extraction to carry out prompt construction on the court trial element indicators in judicial documents, so as to uniformly model from text to prompt structure The generation framework realizes unified modeling of entity indicators, relationship indicators, event indicators, and response indicators in judicial documents; Among them, the structured generation of information extraction is decomposed into two atomic operations: 1) Locate: Locate the target information fragment from the sentence, the information fragment includes entities and events in judicial documents; 2) Association: connect different pieces of information according to the required association, that is, the connection between entities, the connection between entities and events, or the connection between events and events; According to the specific structure of judicial document indicators, a judicial document indicator structured language (JDISL) is designed. The symbol ":" of JDISL indicates the mapping from information to its location or associated fragments, and the symbols "[], ()" are used for Represents the hierarchical structure between the extracted indicator information; Among them, JDISL consists of three parts: 1) info_name: represents a specific piece of information that exists in the source text; 2) relation_name: represents a specific information fragment that exists in the source text, and this information fragment is associated with the upper-level info information in the structure; 3) info_span: represents the specific information of the source text or the corresponding text fragment span of the associated fragment; Through JDISL, different information extraction structures are uniformly encoded, and different judicial document index information extraction tasks are uniformly modeled as the same generation process from text to structure; For entity recognition and event detection tasks modeled as: (info_name:info_span) For relation extraction and event extraction tasks, the unified modeling is: (info_name:info_span(relation_name:info_span),...); Step 2.2) After carrying out the prompt construction to the court trial element index in the judicial document, the training data containing the training corpus is processed based on the prompt construction, thereby obtaining the prompt training corpus; preferably, the Prompt construction is used as a predefined pattern, and the training text is processed as prompt Structured corpus form. 3. according to claim 1 or 2 described judicial document index extracting methods, it is characterized in that, step 3) in the process that prompt training corpus is carried out few sample negative sampling amplification comprises: Step 3.1) For prompt construction corpus: Content: text Result: [[info_type_1],[info_type_2],…,[info_type_n]] Among them, [info_type_n] indicates that the extracted label type is n and the structure form is [[info_name], [relation_name], [info_span]] information set; Process Result into multiple result subsets based on promt: Result_1:[info_type_1][prompt:label_1] Result_2:[info_type_2][prompt:label_2] … Result_n:[info_type_n][prompt:label_n] Where label_n represents the label type n corresponding to the [info_type_n] set; Then perform negative sampling based on the prompt type, and expand the amount of data from n to n^2, such as expanding Result_1 to n times: Result_1:[info_type_1][prompt:label_1] Result_1:[info_type_1][prompt:label_2] … Result_1:[info_type_1][prompt:label_n], Thus, the amplified prompt training corpus is obtained. 4. according to the judicial document index extracting method of any one in claim 1-3, it is characterized in that, step 4) on the basis of the Unilm basic pre-training model of open source, the process with prompt training corpus as input comprises: Step 4.1) Process the prompt training corpus into the input form required by the Unilm model. The input forms accepted by the Unilm model are: {x ₁ , x ₂ , x ₃ ,..., x _m-1 , x _m } The prompt structured information obtained by JDISL is [[info_name], [relation_name], [text]], where text is the information referred to within the info_span span in the source text; Then associate the prompt structured information as a prefix with the fragment information to obtain the model input corpus: {info_name_1,...info_name_n,...,relation_name_1,...,relation_name_n,...,text_1,...,text_n, _x1 , _x2 , _x3 ,...,xm _-1 , _xm }; Use Unilm for both generation tasks and classification tasks; after the input, first use the Unilm pre-trained tokenizer to segment the input, and then perform vector embedding, which includes token embedding, segment embedding, and position embedding; Among them, Token embedding refers to inserting the [CLS] tag at the beginning of each sentence, and inserting the [SEP] tag at the end of each sentence, where the [CLS] tag represents the vector of the current sentence, and the [SEP] tag represents the sentence used to segment the text. sentence; Among them, the segment embedding is used to distinguish two sentences, and the different sentences are marked by A and B respectively, so the input sentence is expressed as (E _A , E _B , E _A , E _B ,…); Among them, position embedding is to add position information according to the relative position of the subscript index of the sentence (ie or for example, [0,1,2,3...]); After embedding, it will go through 24 serial transformer structure layers, and calculate the hidden layer output vector through the multi-head attention mechanism. h={H; h ₁ , h ₂ , h ₃ , . . . , h _m-1 , h _m } Among them, h _m represents the hidden layer state of the m-th judicial document sequence, and H represents the state set of prompt structured information. 5. according to the judicial document index extracting method described in any one in claim 1-4, it is characterized in that, in step 5), get Unilm model hidden layer vector, be processed as the process of Gaussian embedding and comprise: Step 5.1) Use a projection network f _u and f _∑ consisting of fully connected layers and nonlinear activation functions to generate Gaussian distribution parameters: u _i =f _u (h _i ); ∑ _i = ELU(f _∑ (h _i ))+(1+∈); where u _i , ∑ _i represent the average covariance and diagonal covariance of the Gaussian embedding, respectively, that is, the non-zero elements are only along the diagonal of the matrix, both f _u and f _∑ are composed of a single-layer linear layer, with Relu as the activation function, For stability ∈ take e^(-14); thus the hidden layer output vector is processed into a Gaussian embedding (u _i , ∑ _i ). 6. According to the judicial document index extraction method described in any one of claims 1-5, it is characterized in that, in step 6), the contrastive learning loss of Gaussian embedding is calculated, and the process of iteratively updating to obtain the final model includes the following 6.1) and 6.2 ) two sub-steps: Step 6.1) To compute the contrast loss, considering the KL divergence between all valid token pairs in the sampled corpus, two tokens x _q and x _p are considered positive examples if they have the same label y _q = _y p , for which Gaussian embedding N(u _p , ∑ _p ) and N(u _q , ∑ _q ), calculate its KL divergence:

Both directions of KL divergence are calculated since it is not symmetric: d(p,q)＝1/2(D _KL |N _p ||N _q |+D _KL |N _q ||N _p |) For a positive sample x _p , compute the Gaussian embedding loss of x _p with respect to other valid tokens in the batch: χ _p ＝{(χ _q ，y _q )∈χ|y _p ＝y _q ，p≠q} Finally, the contrastive learning loss is calculated using KL divergence and Gaussian embedding loss:

Step 6.2) Backpropagation is performed based on the contrastive learning loss function, and the parameters of the UniLM model are iteratively updated to obtain an information extraction model that can finally be used for the extraction of judicial document indicators.

Images