CN115878777A - Judicial writing index extraction method based on few-sample contrast learning - Google Patents

Abstract

A method for extracting element indexes aiming at judicial texts comprises the following steps: 1) Acquiring judicial writing data, cleaning the judicial writing data, and constructing training data based on the referee writing. 2) A structured language (JDISL) of the judicial literature indexes is provided, the JDISL is adopted to conduct prompt guidance, a structural form of the indexes based on the prompt is obtained, and the training corpus is further processed into the prompt training corpus. 3) And carrying out few-sample negative sampling amplification on the prompt training corpus. 4) On the basis of an open-source Unilm basic pre-training model, a prompt training corpus is used as input. 5) And (4) taking a Unilm model hidden layer vector, and processing the hidden layer vector into Gaussian embedding. 6) And calculating contrast learning loss of Gaussian embedding, and performing iterative updating to obtain a final model. 7) And inputting the judicial documents of the indexes to be extracted into the trained model, and extracting information by the index extraction model to obtain the information extraction result of each label type.

Description

Judicial literature index extraction method based on few-sample comparison learning

Technical Field

The invention belongs to the technical field of referee document data processing, and particularly relates to a judicial document index extraction method based on less sample comparison learning (namely based on a knowledge map).

Background

The data volume of the internet in the information age increases exponentially, and with the rapid development of internet technologies, the information on the network increases explosively, so that not only is the information scale continuously enlarged, but also the information types are continuously increased. Meanwhile, the successful application of a large amount of data in various fields announces the arrival of the big data era, the big data plays an increasingly important role in the development of the society, and the value of the big data is generally accepted by the society. In recent years, with the deepening of the law-making construction of China, the trial and study of judicial cases become more transparent, and the disclosure of official documents on the internet is a typical example. The referee document is used as a 'judicial product' bearing the case trial process and the trial result of the court, and contains rich judicial information including a judge court, a case number, a party litigation request, a case name, a judgment result, applicable laws and the like, which just gather the core elements of 'big data' of the court. By deeply mining the information, case trial rules can be summarized, trial trends can be predicted, judicial credibility can be improved, and technical support is provided for judicial justice and construction law society. However, a referee document is a semi-structured domain text which has both stylized grammatical language and daily common language, and the writing of the referee document is largely determined by the judge, which makes the referee document have a series of characteristics such as polymorphism, isomerism and randomness. Therefore, how to extract valuable information from such special texts is a topic with important value and significance.

A large amount of judicial texts in the judicial field have wide index extraction requirements, and are used for evaluating the effect of judicial programs or judicial judgment and reform and the like. With the construction of the law-oriented society of China and the development of information technology, the number of judicial documents in the judicial field increases exponentially. The case processing requirement is difficult to meet under the condition of few cases, and the case judgment is influenced to a certain degree by the difference of professional literacy of judicial personnel. Therefore, the intelligent extraction of the key elements of the judicial documents can provide reference for judicial personnel, assist in case handling and improve the working efficiency, and is also the key for subsequent deep analysis and efficient case judgment.

Available marking data in the field of the current judicial documents are rare, marking difficulty is high, and the problem of scarce available data resources exists, and the method belongs to a low-resource and few-sample application scene; the judicial literature index extraction tasks comprise a plurality of subtasks such as entity extraction, relation extraction and event extraction, and different subtasks require different marking data and different algorithm models, so that the application requirements for training a plurality of subtask models are met under the low-resource and few-sample scene. In conclusion, the judicial literature index extraction faces the problems that the multi-task algorithm model is difficult to be trained under the low-resource and few-sample application scene, the extraction precision is low, and the generalization performance is poor, and the judicial literature index extraction model cannot meet the goals of assisting the judicial staff to handle cases and improving the working efficiency. Therefore, the judicial literature index extraction method based on the few-sample comparison learning effectively solves the problems.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides a judicial literature index extraction method based on less sample comparison learning (namely, based on a knowledge graph). Firstly, acquiring judicial literature data, performing data cleaning on the judicial literature data, and constructing training data based on referee texts. And then, carrying out prompt guidance on the judicial literature indexes by using a designed structured language of the judicial literature indexes (JDISL), and further processing the training corpus into a prompt training corpus. And then performing few-sample negative sampling amplification on the prompt training corpus. On an open-source UniLM (unified modeling language) basic pre-training model, a prompt training corpus is used as input, a UniLM model hidden layer vector is taken and processed into Gaussian embedding, the contrast learning loss of the Gaussian embedding is calculated, and a final model is obtained through iterative updating. And inputting the judicial documents of the indexes to be extracted into the trained model to obtain the information extraction result of each label type.

Aiming at the defects of the prior art, the invention provides a judicial literature index extraction method based on less sample comparison learning.

According to an embodiment of the present invention, there is provided a judicial literature index extraction method based on few-sample contrast learning, the method including the steps of:

1) Acquiring judicial literature data (such as official documents or official document data disclosed on the Internet), performing data cleaning on the judicial literature data, and constructing training data containing training corpuses based on the official documents (namely the judicial literature);

2) Providing a structured language (JDISL) of the judicial literature indexes, adopting the JDISL to conduct prompt guidance to obtain a structural form of the indexes based on the prompt, and further processing the training corpus into the prompt training corpus;

3) Carrying out few-sample negative sampling amplification on the prompt training corpus to obtain an amplified prompt training corpus;

4) Taking the amplified prompt training corpus as input on the basis of an open-source Unilm base pre-training model;

5) Taking a Unilm model hidden layer vector, and processing the Unilm model hidden layer vector into Gaussian embedding;

6) Calculating contrast learning loss of Gaussian embedding, and performing iterative updating to obtain a final model;

7) And inputting the judicial documents of the indexes to be extracted into the trained final model, and extracting information by the index extraction model to obtain the information extraction result of each label type.

Preferably, the step 2) is to adopt prompt guidance for the judicial literature index to obtain a construction form of the index based on prompt, and the process of further processing the corpus into the prompt corpus comprises the following two substeps of 2.1) and 2.2):

and 2.1) utilizing an information extraction structured generation means (or an information extraction means) comprising entity extraction, relation extraction, event extraction and response extraction to carry out prompt construction on the court trial element indexes in the judicial literature. So as to uniformly model a generation frame from text to prompt structure, and realize uniform modeling on entity indexes, relation indexes, event indexes and response indexes in the judicial literature.

The information extraction is affected by different targets, heterogeneous structures and specific demand patterns and is divided into entity extraction, relationship extraction, event extraction, response extraction and the like. And performing a prompt structure on the court trial element indexes. The unified modeling is a generation frame from text to prompt structure, and realizes unified modeling on entity indexes, relation indexes, event indexes and response indexes in the judicial literature.

The information extraction structured generation may be decomposed into two atomic operations.

1) Positioning: target pieces of information are located from sentences, including, for example, entities and events in a judicial paper.

2) And (3) association: different pieces of information are connected according to the required associations, such as a connection between an entity(s) and an entity(s), a connection between an entity and an event, or a connection between an event(s) and an event(s).

Different information extraction task structures can be generated by combining atomic operations.

According to the structure specificity of the judicial writing index, a structured language of the judicial writing index (JDISL) is designed, wherein the symbol of JDISL ": "denotes the mapping from information to its positioning or associated segments, and the symbols" [ ], () "are used to denote the hierarchy between the extracted index information.

JDISL consists of three parts:

1) info _ name-a specific piece of information that indicates the existence of the source text

2) A relation _ name, a specific information segment which indicates the existence of the source text and is associated with the upper-layer info information in the structure.

3) info _ span-a span of text segments corresponding to a specific information or associated segment representing the source text type.

Different information extraction structures can be uniformly coded through JDISL, and different judicial literature index information extraction tasks are uniformly modeled into the same generation process from text to structuring.

The task for entity recognition and event detection can be modeled as:

(info_name:info_span)，

the relationship extraction and event extraction tasks can be modeled uniformly as:

(info_name:info_span(relation_name:info_span),…)，

taking court trial elements as an example, the indexes to be extracted comprise witnesses, original reports, defendants, evidences, courts and judges, and the relationship among the indexes comprises the onset (defendant) and the prosecution (defendant). The entity is subjected to prompt guidance, and is represented in parallel as follows:

[ witnesses, plains, defendents, judges ]

For attribute and relationship dependencies, it is expressed as:

[ court: evidence, witness: (testimony), [ original: (prosecution) ], quilt, official ]

And 2.2) after the prompt construction is carried out on the court trial element indexes in the judicial literature, processing the training data containing the training corpus based on the prompt construction so as to obtain the prompt training corpus.

After the prompt construction is carried out on the court trial element indexes, the corpus needs to be processed based on the prompt construction, so that the prompt corpus is obtained.

As for the raw corpus:

content shows the dispute between original and reported labor contracts, and the testimony show that the rule is wrong. "

Label: [2,3, "original" ], [7,8, "defended" ], [20,21, "witness" ], [27,32, "witness" ], [ (2, 3), (7, 8), "prosecution" ]

And constructing and processing a prompt structured corpus based on the prompt:

content, a case of dispute between original and reported labor contracts "

Result: [ (2, 3, "from") to ("from"), (7, 8, "from"): ("filed"), (20, 21, "Liu-Gong"): (27, 32, "true for error") ]

And the Prompt construction is used as a predefined mode, and the training text is processed into a Prompt structured corpus form.

Preferably, the process of performing sample-less negative sampling amplification on the prompt training corpus in the step 3) more specifically comprises the following steps:

step 3.1) for the prompt construction corpus:

Content:text

Result:[[info_type_1],[info_type_2],…,[info_type_n]]

wherein [ info _ type _ n ] represents an information set in which the extracted structure of tag type n is in the form of [ [ info _ name ], [ relationship _ name ], [ info _ span ] ].

Processing results into a plurality of Result subsets based on the proxy:

Result_1:[info_type_1][prompt:label_1]

Result_2:[info_type_2][prompt:label_2]

…

Result_n:[info_type_n][prompt:label_n]

wherein label _ n represents a label type n corresponding to the [ info _ type _ n ] set;

then, based on the prompt type, negative sampling is carried out, and the data size is expanded from n to n ^2, such as expanding Result _1 to n times:

Result_1:[info_type_1][prompt:label_1]

Result_1:[info_type_1][prompt:label_2]

…

Result_1:[info_type_1][prompt:label_n]。

thereby obtaining the amplified prompt training corpus.

Preferably, the process of using the prompt corpus as an input in step 4) on the basis of the open-source Unilm base pre-training model more specifically includes:

and 4.1) processing the prompt training corpus into an input form required by the Unilm model. The Unilm model can accept the input form:

{x ₁ ，x ₂ ，x ₃ ，...，x _m-1 ，x _m }

the template structured information obtained by JDISL is [ [ info _ name ], [ relationship _ name ], [ text ] ], where text is the information referred to within the info _ span in the source text.

And then associating the prompt structural information serving as a prefix with fragment information to obtain a model input corpus: { info _ name _1, \8230; info _ name _ n, \8230;, translation _ name _1, \8230;, translation _ name _ n, \8230;, text _1, \8230;, text _ n, x ₁ ，x ₂ ，x ₃ ，...，x _m-1 ，x _m }；

Above, the value of n refers to the number of entity/relationship fragment information. The value of m refers to the sequence length of the input document.

Unilm can be used for both generating and classifying tasks. After the input, the input is first tokenizer pre-trained using Unilm followed by vector embedding, which includes token embedding, segment embedding, and location embedding.

The Token embedding means that [ CLS ] marks are inserted into the beginning of each sentence, and [ SEP ] marks are inserted into the end of each sentence, wherein the [ CLS ] marks represent vectors of the current sentence, and the [ SEP ] marks represent clauses used for segmenting the sentences in the text.

Segment embedding is used to distinguish two sentences, the different sentences being preceded by a and B labels, respectively, so that the input sentence is represented as (E) _A ，E _B ，E _A ，E _B ,……)。

Position embedding is to add position information according to the subscript index of sentence, i.e. or as [0,1,2,3 \8230 ].

After embedding, the hidden layer output vector is obtained through calculation of a multi-head attention mechanism through 24 series-connected transform structure layers

h＝{H；h ₁ ，h ₂ ，h ₃ ，...，h _m-1 ，h _m }

Wherein h is _m Represents the hidden layer state of the mth judicial writing sequence, and H represents the state set of the prompt structured information.

Preferably, the process of taking the hidden layer vector of the Unilm model in the step 5) and processing the hidden layer vector into Gaussian embedding more specifically comprises the following steps:

using a projection network f consisting of a fully connected layer and a nonlinear activation function according to step 5.1) _u And f _∑ Generating a gaussian distribution parameter:

u _i ＝f _u (h _i )；

Σ _i ＝ELu(f _∑ (h _i ))+(1+∈)

wherein u is _i ，Σ _i Mean covariance and diagonal covariance representing gaussian embedding, respectively (non-zero elements only along the diagonal of the matrix), f _u And f _∑ Both consist of a single linear layer with Relu as the activation function, taking e ^ (14) for stability.

Processing hidden layer output vectors for gaussian embedding (u) _i ，Σ _i )。

Preferably, the step 6) of calculating the contrast learning loss of the gaussian embedding, and the process of iteratively updating to obtain the final model more specifically comprises the following two substeps of 6.1) and 6.2):

step 6.1) to calculate the contrast loss, consider the KL divergence between all valid token pairs in the sampled corpus if two labels x _q And x _p Having the same label y _q ＝y _p They are considered as positive examples for their gaussian embedding of N (u) _p ，Σ _p ) And N (u) _q ，∑ _q ) Its KL divergence was calculated:

both directions of KL divergence are calculated because it is not symmetric.

d(p，q)＝1/2(D _KL |N _p ||N _q |+D _KL |N _q ||N _p |)

For positive samples x _p Calculating x _p Gaussian embedding loss relative to other valid tokens in the batch:

χ _p ＝{(χ _q ，y _q )∈χ|y _p ＝y _q ·p≠q}

and finally, calculating the comparative learning loss by utilizing KL divergence and Gaussian embedding loss:

and 6.2) performing back propagation based on the comparison learning loss function, and performing iterative update on the UniLM model parameters to obtain an information extraction model which can be finally used for judicial literature index extraction, namely a final model.

The method comprises the following steps that available marking data in the field of judicial documents at present are rare, marking difficulty is high, and the problem of scarce available data resources exists, so that the method belongs to a low-resource and few-sample application scene; the judicial literature index extraction tasks comprise a plurality of subtasks such as entity extraction, relation extraction and event extraction, and different subtasks require different marking data and different algorithm models, so that the application requirements for training a plurality of subtask models are met under the low-resource and few-sample scene. Therefore, a judicial literature index extraction method is needed to solve the problems of difficult convergence, low extraction precision and poor generalization of the multi-task algorithm model in the low-resource and few-sample application scene.

In the invention, the judicial literature index is guided by the prompt to obtain a structural form of the index based on the prompt, and the training corpus is further processed into the prompt training corpus.

The information extraction is influenced by different targets, heterogeneous structures and specific demand patterns and is divided into entity extraction, relation extraction, event extraction, response extraction and the like. And performing a prompt structure on the court trial element indexes. The unified modeling is a generation frame from text to prompt structure, and realizes unified modeling on entity indexes, relation indexes, event indexes and response indexes in the judicial literature.

The information extraction structured generation may be decomposed into two atomic operations.

1) Positioning: target pieces of information, such as entities and events in the document, are located from the sentences.

2) And (3) association: different information fragments are connected according to the required association, such as connection between entities and connection between events.

Different information extraction task structures can be generated by combining atomic operations.

According to the structure specificity of the judicial literature indexes, a structured language of the judicial literature indexes (JDISL) is designed, wherein the JDISL comprises the following steps: representing the mapping of information to its location or associated segment, [ ], () is used to represent the hierarchy between the extracted metric information.

JDISL consists of three parts:

1) info _ name-a specific piece of information that indicates the existence of the source text

2) A relation _ name, a specific piece of information indicating the existence of the source text, the piece of information having an association with the upper info information in the structure.

3) info _ span-a span of text segments corresponding to a particular piece of information or associated segment representing the source text species.

Different information extraction structures can be uniformly coded through JDISL, and different judicial literature index information extraction tasks are uniformly modeled into the same generation process from text to structuring.

The task for entity recognition and event detection can be modeled as:

(info_name:info_span)，

the relationship extraction and event extraction tasks can be modeled uniformly as:

(info_name:info_span(relation_name:info_span),…)，

taking court trial elements as an example, the indexes to be extracted comprise witnesses, original reports, evidences, courts and judges, and the relationship among the indexes comprises (original reports) prosecution (reports). The entity is subjected to prompt guidance, and is represented in parallel as follows:

[ witnesses, plains, defendents, judges ]

For attribute and relationship dependencies, it is expressed as:

[ court: evidence, witness: (testimony), [ original: (prosecution) ], quilt, official ]

After the prompt construction is carried out on the court trial element indexes, the corpus needs to be processed based on the prompt construction, so that the prompt corpus is obtained.

As for the raw corpus:

content shows the dispute between original and reported labor contracts, and the testimony show that the rule is wrong. "

Label: [2,3, "original" ], [7,8, "defended" ], [20,21, "witness" ], [27,32, "witness" ], [ (2, 3), (7, 8), "prosecution" ]

And constructing and processing a prompt structured corpus based on the prompt:

content, a case of dispute between original and reported labor contracts "

Result: [ (2, 3, "from") to ("from"), (7, 8, "from"): ("filed"), (20, 21, "Liu-Gong"): (27, 32, "true for error") ]

The Prompt structure serves as a predefined mode, and the training text is processed into a Prompt structured corpus form.

In the invention, the prompt training corpus is subjected to the negative sampling amplification of few samples.

For prompt construction corpora:

Content:text

Result:[[info_type_1],[info_type_2],…,[info_type_n]]

where [ info _ type _ n ] represents an information set in which the extracted structure of tag type n is in the form of [ [ info _ name ], [ relation _ name ], [ info _ span ] ].

Processing results into a plurality of Result subsets based on the proxy:

Result_1:[info_type_1][prompt:label_1]

Result_2:[info_type_2][prompt:label_2]

…

Result_n:[info_type_n][prompt:label_n]

wherein label _ n represents a label type n corresponding to the [ info _ type _ n ] set;

then, based on the prompt type, negative sampling is carried out, and the data volume is expanded from n to n ^2, such as expanding Result _1 to n times:

Result_1:[info_type_1][prompt:label_1]

Result_1:[info_type_1][prompt:label_2]

…

Result_1:[info_type_1][prompt:label_n]。

thus obtaining the amplified prompt training corpus.

In the invention, a prompt training corpus is used as input on the basis of an open-source Unilm basic pre-training model.

And processing the prompt training corpus into an input form required by the Unilm model. The Unilm model can accept the input form:

{x ₁ ，x ₂ ，x ₃ ，...，x _m-1 ，x _m }

the template structured information obtained by JDISL is [ [ info _ name ], [ relationship _ name ], [ text ] ], where text is the information referred to within the info _ span in the source text.

And then associating the prompt structural information serving as a prefix with fragment information to obtain a model input corpus:

{info_name_1,…info_name_n,…,relation_name_1,…,relation_name_n,…,text_1,…,text_n,x ₁ ，x ₂ ，x ₃ ，...，x _m-1 ，x _m }

unilm can be used both for generating tasks and for classifying tasks, after input, first tokenizer pre-trained using Unilm performs tokenizer on the input, followed by vector embedding, including token embedding, segment embedding, and position embedding.

Token embedding refers to inserting [ CLS ] marks at the beginning and [ SEP ] marks at the end of each sentence, wherein the [ CLS ] marks represent vectors of current sentences, and the [ SEP ] marks represent clauses used for segmenting sentences in the text.

Segment embedding is used to distinguish two sentences, the different sentences being preceded by a and B labels, respectively, so that the input sentence is represented as (E) _A ，E _B ，E _A ，E _B ,……)。

Position embedding is to add position information such as 0,1,2,3 8230, relative to the position according to the subscript index of the sentence.

After embedding, the hidden layer output vector is obtained through calculation of a multi-head attention mechanism through 24 series-connected transform structure layers

h＝{H；h ₁ ，h ₂ ，h ₃ ，...，h _m-1 ，h _m }

Wherein h is _m Represents the hidden layer state of the mth judicial writing sequence, and H represents the state set of the prompt structured information.

In the invention, a Unilm model hidden layer vector is taken and processed as Gaussian embedding.

Using a projection network f consisting of a fully connected layer and a non-linear activation function _u And f _∑ Generating a gaussian distribution parameter:

u _i ＝f _u (h _i )；

∑ _i ＝ELU(f _∑ (h _i ))+(1+∈)

wherein u is _i ，∑ _i Mean covariance and diagonal covariance representing gaussian embedding, respectively (non-zero elements only along the diagonal of the matrix), f _u And f _∑ Both consist of a single linear layer with Relu as the activation function, taking e ^ (14) for stability.

Processing hidden layer output vectors for gaussian embedding (u) _i ，Σ _i )。

In the invention, the contrast learning loss of Gaussian embedding is calculated, and the final model is obtained by iterative updating.

To calculate the contrast loss, consider the KL divergence between all valid token pairs in the sample corpus if two tokens x _q And x _p Having the same label y _q ＝y _p They are considered as positive examples, with N (u) embedded for their gaussian _p ，∑ _p ) And N (u) _q ，Σ _q ) Its KL divergence was calculated:

both directions of KL divergence are calculated because it is not symmetric.

d(p,q)＝1/2(D _KL |N _p ||N _q |+D _KL |N _q ||N _P |)

For positive samples x _p Calculating x _p Gaussian embedding loss relative to other valid tokens in the batch:

χ _p ＝{(χ _q ，y _g )∈χ[y _p ＝y _q ，p≠q)

and finally, calculating the comparative learning loss by utilizing KL divergence and Gaussian embedding loss:

and performing back propagation based on the comparison learning loss function, and performing iterative update on the parameters of the UniLM model to obtain an information extraction model which can be finally used for extracting judicial literature indexes.

Compared with the prior art, the technical scheme of the invention has the following beneficial technical effects:

1. in the invention, a structured language of a judicial literature index (JDISL) is provided, the JDISL is adopted for prompt guidance to obtain a structural form of the index based on prompt, and the training corpus is further processed into the prompt training corpus, so that different information extraction structures are effectively and uniformly coded, and therefore, effective combined extraction can be performed, and the problems of difficult uniform modeling and overlarge training cost of extracting a plurality of extraction subtasks of a multi-task algorithm model by the judicial literature index are effectively solved.

2. In the invention, the prompt training corpus is subjected to the negative sampling amplification of few samples, so that the problem of difficulty in learning of few samples caused by too few available labeled data in the field of low resources of judicial documents is effectively solved.

3. In the invention, a Unilm model hidden layer vector is taken and processed as Gaussian embedding, information distribution is definitely modeled through the Gaussian embedding, generalized characteristic representation of various indexes is promoted, self-adaptation of a less-sample target domain is facilitated, and the problems of poor generalization performance and low extraction precision of a less-sample learning model are effectively solved.

4. In the invention, the comparative learning loss function is used for calculating the loss training model, and the final model is obtained by iterative updating, thereby effectively solving the problems of difficult convergence and low extraction precision of model training.

Drawings

FIG. 1 is a schematic structural diagram of a judicial literature index extraction method based on few-sample comparison learning.

Fig. 2 is a structural diagram of a few-sample data processing module of the judicial literature index extraction method based on few-sample contrast learning according to the present invention.

Fig. 3 is a network structure diagram of the few-sample contrast learning based on the judicial literature index extraction method of the few-sample contrast learning of the present invention.

Detailed Description

The technical solutions of the present invention are illustrated below, and the scope of the present invention includes, but is not limited to, examples.

A judicial writing index extraction method based on few-sample comparison learning comprises the following steps:

1) Acquiring judicial writing data, cleaning the judicial writing data, and constructing training data based on the referee writing.

2) A structured language (JDISL) of the judicial literature indexes is provided, the JDISL is adopted to conduct prompt guidance, a structural form of the indexes based on the prompt is obtained, and the training corpus is further processed into the prompt training corpus.

3) And carrying out few-sample negative sampling amplification on the prompt training corpus.

4) On the basis of an open-source Unilm basic pre-training model, a prompt training corpus is used as input.

5) And (4) taking a Unilm model hidden layer vector, and processing the hidden layer vector into Gaussian embedding.

6) And calculating contrast learning loss of Gaussian embedding, and performing iterative updating to obtain a final model.

7) And inputting the judicial writing of the index to be extracted into the trained model, and extracting information by using the index extraction model to obtain the information extraction result of each label type.

Preferably, the step 2) of guiding the judicial literature indexes by the prompt to obtain a structural form of the indexes based on the prompt, and further processing the corpus into the prompt corpus comprises:

and 2.1) information extraction is influenced by different targets, heterogeneous structures and specific requirement modes and is divided into entity extraction, relation extraction, event extraction, response extraction and the like. And performing a prompt structure on the court trial element indexes. The unified modeling is a generation frame from text to prompt structure, and realizes unified modeling on entity indexes, relation indexes, event indexes and response indexes in the judicial literature.

The information extraction structured generation can be decomposed into two atomic operations.

1) Positioning: target pieces of information, such as entities and events in the document, are located from the sentences.

2) And (3) association: different information fragments are connected according to the required association, such as connection between entities and connection between events.

Different information extraction task structures can be generated by combining atomic operations.

According to the structure specificity of the judicial literature indexes, a structured language of the judicial literature indexes (JDISL) is designed, wherein the JDISL comprises the following steps: representing the mapping from information to its location or associated segment, [ ], () is used to represent the hierarchy between the extracted metric information.

JDISL consists of three parts:

1) info _ name-a specific piece of information that indicates the existence of the source text

2) A relation _ name, a specific piece of information indicating the existence of the source text, the piece of information having an association with the upper info information in the structure.

3) info _ span-a span of text segments corresponding to a particular piece of information or associated segment representing the source text species.

Different information extraction structures can be uniformly coded through JDISL, and different judicial literature index information extraction tasks are uniformly modeled into the same generation process from text to structuring.

The task for entity recognition and event detection can be modeled as:

(info_name:info_span)

the relationship extraction and event extraction tasks can be modeled uniformly as:

(info_name:info_span(relation_name:info_span),…)

taking court trial elements as an example, the indexes to be extracted comprise witnesses, original reports, evidences, courts and judges, and the relationship among the indexes comprises (original reports) prosecution (reports). It is subjected to prompt guidance, and for an entity, it is expressed in parallel as:

[ witness, original notice, quilt notice, judge ]

For attribute and relationship dependencies, it is expressed as:

[ court: evidence, witness: (testimony), [ original: (prosecution) ], quilt, official ]

And 2.2) after the prompt construction is carried out on the court trial element indexes, processing the corpus based on the prompt construction to obtain the prompt corpus.

As for the raw corpus:

content shows the dispute between original and reported labor contracts, and the testimony show that the rule is wrong. "

Label: [2,3, "original" ], [7,8, "defended" ], [20,21, "witness" ], [27,32, "witness" ], [ (2, 3), (7, 8), "appellation" ]

Processing into a prompt structured corpus based on a prompt structure:

content: "a case of dispute between original true contract and reported work contract"

Result: [ (2, 3, "from") to ("from"), (7, 8, "from"): ("filed"), (20, 21, "Liu-Gong"): (27, 32, "true for error") ]

The Prompt structure serves as a predefined mode, and the training text is processed into a Prompt structured corpus form.

Preferably, step 3) performs sample-less negative sampling amplification on the prompt training corpus:

step 3.1) for the prompt construction corpus:

Content:text

Result:[[info_type_1],[info_type_2],…,[info_type_n]]

where [ info _ type _ n ] represents an information set in which the extracted structure of tag type n is in the form of [ [ info _ name ], [ relation _ name ], [ info _ span ] ].

Processing results into a plurality of Result subsets based on promt:

Result_1:[info_type_1][prompt:label_1]

Result_2:[info_type_2][prompt:label_2]

…

Result_n:[info_type_n][prompt:label_n]

wherein label _ n represents the label type n corresponding to [ info _ type _ n ] set

Then, based on the prompt type, negative sampling is carried out, and the data volume is expanded from n to n ^2, such as expanding Result _1 to n times:

Result_1:[info_type_1][prompt:label_1]

Result_1:[info_type_1][prompt:label_2]

…

Result_1:[info_type_1][prompt:label_n]

preferably, step 4) takes the prompt corpus as input on the basis of the open-source Unilm base pre-training model:

and 4.1) processing the prompt training corpus into an input form required by the Unilm model. The Unilm model can accept the input form:

{x ₁ ，x ₂ ，x ₃ ，...，x _m-1 ，x _m }

the template structured information obtained by JDISL is [ [ info _ name ], [ relationship _ name ], [ text ] ], where text is the information referred to within the info _ span in the source text.

And then associating the prompt structural information serving as a prefix with fragment information to obtain a model input corpus:

{info_name_1,…info_name_n,…,relation_name_1,…,relation_name_n,…,text_1,…,text_n,x ₁ ，x ₂ ，x ₃ ，...，x _m-1 ，x _m }

unilm can be used for both generating and classifying tasks, and after input, the input is firstly participled by using a token pre-trained by Unilm, and then vector embedding is carried out, wherein the token embedding, the segment embedding and the position embedding are included.

Token embedding refers to inserting [ CLS ] marks at the beginning and [ SEP ] marks at the end of each sentence, wherein the [ CLS ] marks represent vectors of current sentences, and the [ SEP ] marks represent clauses used for segmenting sentences in the text.

Segment embedding is used to distinguish two sentences, the different sentences being preceded by a and B labels, respectively, so that the input sentence is represented as (E) _A ，E _B ，E _A ，E _B ,……)。

Position embedding is to add position information such as 0,1,2,3 8230, relative to the position according to the subscript index of the sentence.

After embedding, the hidden layer output vector is obtained through calculation of a multi-head attention mechanism through 24 series-connected transform structure layers

h＝{H；h ₁ ，h ₂ ，h ₃ ，...，h _m-1 ，h _m }

Wherein h is _m Represents the hidden layer state of the mth judicial writing sequence, and H represents the state set of the prompt structural information.

Preferably, step 5) takes a Unilm model hidden layer vector, and processes the hidden layer vector into Gaussian embedding:

using a projection network f consisting of fully connected layers and nonlinear activation functions according to step 5.1) _u And f _∑ Generate a heightThe parameters of the Si distribution:

u _i ＝f _u (h _i )；

∑ _i ＝ELU(f _∑ (h _i ))+(1+∈)

wherein u _i ，∑ _i Mean covariance and diagonal covariance of the Gaussian embedding (non-zero elements only along the diagonal of the matrix), respectively, f _u And f _Σ Both consist of a single linear layer with Relu as the activation function, taking e ^ (14) for stability.

The hidden layer output vector is processed for gaussian embedding (u) _i ，Σ _i )。

Preferably, step 6) calculates the contrast learning loss of gaussian embedding, and iteratively updates to obtain a final model:

step 6.1) to calculate the contrast loss, consider all valid t's in the sample corpus _o KL divergence between ken pairs if two markers x _q And x _p Having the same label y _q ＝y _p They are considered as positive examples for their gaussian embedding of N (u) _p ，Σ _p ) And N (u) _q ，∑ _q ) Its KL divergence was calculated:

both directions of KL divergence are calculated because it is not symmetric.

d(p，q)＝1/2(D _KL |N _p ||N _q |+D _KL |N _q ||N _p |)

For positive samples x _p Calculating x _p Gaussian embedding loss with respect to other valid tokens in the batch:

χ _p ＝((χ _q ，y _q )∈χ|y _p ＝y _q ，p≠q}

and finally, calculating the comparative learning loss by utilizing KL divergence and Gaussian embedding loss:

and 6.2) performing back propagation based on the comparison learning loss function, and performing iterative updating on the UniLM model parameters to obtain an information extraction model which can be finally used for extracting judicial literature indexes.

Example 1

As shown in fig. 1, a judicial literature index extraction method based on few-sample comparison learning includes the following steps:

1) Acquiring judicial writing data, cleaning the judicial writing data, and constructing training data based on the referee writing.

2) A structured language of a judicial literature index (JDISL) is provided for the judicial literature index, the JDISL is adopted for prompt guidance, a structural form of the index based on prompt is obtained, and the training corpus is further processed into the prompt training corpus.

3) And carrying out negative sampling amplification on the prompt training corpus with few samples.

4) On the basis of an open-source Unilm basic pre-training model, a prompt training corpus is used as input.

5) And (4) taking a Unilm model hidden layer vector, and processing the hidden layer vector into Gaussian embedding.

6) And calculating contrast learning loss of Gaussian embedding, and iteratively updating to obtain a final model.

7) And inputting the judicial writing of the index to be extracted into the trained model, and extracting information by using the index extraction model to obtain the information extraction result of each label type.

Example 2

The embodiment 1 is repeated, except that the prompt guidance is adopted for the judicial literature indexes in the step 2), a structural form of the indexes based on the prompt is obtained, and the corpus is further processed into the prompt corpus as shown in fig. 2:

and 2.1) information extraction is influenced by different targets, heterogeneous structures and specific requirement modes and is divided into entity extraction, relation extraction, event extraction, response extraction and the like. And performing a prompt structure on the court trial element indexes. The unified modeling is a generation frame from text to prompt structure, and realizes unified modeling on entity indexes, relation indexes, event indexes and response indexes in the judicial documents.

The information extraction structured generation may be decomposed into two atomic operations.

1) Positioning: target pieces of information, such as entities and events in the document, are located from the sentences.

2) And (3) association: different information fragments are connected according to the required association, such as connection between entities and events.

Different information extraction task structures can be generated by atomic operation combination.

According to the structure specificity of the judicial literature indexes, a structured language of the judicial literature indexes (JDISL) is designed, wherein the JDISL comprises the following steps: representing the mapping from information to its location or associated segment, [ ], () is used to represent the hierarchy between the extracted metric information.

JDISL consists of three parts:

1) info _ name-a specific piece of information that indicates the existence of the source text

2) A relation _ name, a specific information segment which indicates the existence of the source text and is associated with the upper-layer info information in the structure.

3) info _ span-a span of text segments corresponding to a particular piece of information or associated segment representing the source text species.

Different information extraction structures can be uniformly coded through JDISL, and different judicial writing index information extraction tasks are uniformly modeled into the same generation process from text to structuring.

The task for entity recognition and event detection can be modeled as:

(info_name:info_span)

the relationship extraction and event extraction tasks can be modeled uniformly as:

(info_name:info_span(relation_name:info_span),…)

taking court trial elements as an example, the indexes to be extracted comprise witnesses, original reports, defendants, evidences, courts and judges, and the relationship among the indexes comprises the onset (defendant) and the prosecution (defendant). The entity is subjected to prompt guidance, and is represented in parallel as follows:

[ witnesses, plains, defendents, judges ]

For attribute and relationship dependencies, it is expressed as:

[ court: evidence, witness: (testimony), [ original: (prosecution) ], quilt, official ]

And 2.2) after the prompt construction is carried out on the court trial element indexes, processing the corpus based on the prompt construction to obtain the prompt corpus.

As for the raw corpus:

content shows the dispute between original and reported labor contracts, and the testimony show that the rule is wrong. "

Label: [2,3, "original" ], [7,8, "defended" ], [20,21, "witness" ], [27,32, "witness" ], [ (2, 3), (7, 8), "prosecution" ]

And constructing and processing a prompt structured corpus based on the prompt:

content, a case of dispute between original and reported labor contracts "

Result: [ (2,3, "trette"): ("filed"), (20, 21, "Liu-Gong"): (27, 32, "true for error") ]

The Prompt structure serves as a predefined mode, and the training text is processed into a Prompt structured corpus form.

Example 3

Example 2 is repeated except that step 3) is performed to amplify the prompt corpus by sample-less negative sampling, as shown in fig. 2:

step 3.1) constructing corpus for prompt:

Content:text

Result:[[info_type_1],[info_type_2],…,[info_type_n]]

wherein [ info _ type _ n ] represents an information set in which the extracted structure of tag type n is in the form of [ [ info _ name ], [ relationship _ name ], [ info _ span ] ].

Processing results into a plurality of Result subsets based on the proxy:

Result_1:[info_type_1][prompt:label_1]

Result_2:[info_type_2][prompt:label_2]

…

Result_n:[info_type_n][prompt:label_n]

wherein label _ n represents the label type n corresponding to [ info _ type _ n ] set

Then, based on the prompt type, negative sampling is carried out, and the data volume is expanded from n to n ^2, such as expanding Result _1 to n times:

Result_1:[info_type_1][prompt:label_1]

Result_1:[info_type_1][prompt:label_2]

…

Result_1:[info_type_1][prompt:label_n]

example 4

Example 3 is repeated, as shown in fig. 2, except that step 4) takes a prompt corpus as an input on the basis of the open-source Unilm base pre-training model, as shown in fig. 3:

and 4.1) processing the prompt training corpus into an input form required by the Unilm model. The Unilm model can accept the input form:

{x ₁ ，x ₂ ，x ₃ ，...，x _m-1 ，x _m }

the template structured information obtained by JDISL is [ [ info _ name ], [ relationship _ name ], [ text ] ], where text is the information referred to within the info _ span in the source text.

And then associating the prompt structural information serving as a prefix with the fragment information to obtain a model input corpus:

{info_name_1,…info_name_n,…,relation_name_1,…,relation_name_n,…,text_1,…,text_n,x ₁ ，x ₂ ，x ₃ ，...，x _m-1 ，x _m }

the Unilm can be used for generating tasks and classifying tasks, after input, firstly, the Unilm pre-trained token is used for word segmentation of the input, and then vector embedding including token embedding, segment embedding and position embedding is carried out.

Token embedding refers to inserting [ CLS ] marks at the beginning and [ SEP ] marks at the end of each sentence, wherein the [ CLS ] marks represent vectors of current sentences, and the [ SEP ] marks represent clauses used for segmenting sentences in the text.

Segment embedding is used to distinguish two sentences, the different sentences being preceded by a and B labels, respectively, so that the input sentence is represented as (E) _A ，E _B ，E _A ，E _B ,……)。

Position embedding is the addition of position information, i.e., or for example [0,1,2, 3' \ 8230 ], relative to the position according to the subscript index of the sentence.

After embedding, the hidden layer output vector is obtained through calculation of a multi-head attention mechanism through 24 series-connected transform structure layers

h＝{H；h ₁ ，h ₂ ，h ₃ ，...，h _m-1 ，h _m }

Wherein h is _m Represents the hidden layer state of the mth judicial writing sequence, and H represents the state set of the prompt structural information.

Example 5

Example 4 was repeated, as shown in fig. 3, except that Unilm model hidden layer vectors were taken in step 5), and processed as gaussian embedding:

using a projection network f consisting of fully connected layers and nonlinear activation functions according to step 5.1) _u And f _∑ Generating gaussian distribution parameters, as shown in fig. 3:

u _i ＝f _u (h _i )；

∑ _i ＝ELU(f _∑ (h ₁ )+(1+∈)

wherein u is _i ，∑ _i Mean covariance and diagonal covariance representing gaussian embedding, respectively (non-zero elements only along the diagonal of the matrix), f _u And f _∑ Both consist of a single linear layer with Relu as the activation function, taking e ^ (14) for stability.

The hidden layer output vector is processed intoGaussian embedding (u) _i ，∑ _i )。

Example 6

Example 5 was repeated as shown in fig. 3, except that the gaussian-embedded contrast learning loss was calculated in step 6), and the final model was obtained by iterative updating:

step 6.1) in order to calculate the contrast loss, consider the KL divergence between all valid token pairs in the sampled corpus, if two tokens x _q And x _p Having the same label y _q ＝y _q They are considered as positive examples for their gaussian embedding of N (u) _p ，∑ _p ) And N (u) _q ，∑ _q ) Its KL divergence was calculated:

both directions of KL divergence are calculated because it is not symmetric.

d(p,q)＝1/2(D _KL |N _p ||N _q |+D _KL |N _q ||N _p |)

For positive samples x _p Calculating x _p Gaussian embedding loss with respect to other valid tokens in the batch:

x _p ＝{(x _q ，y _q )∈χ|yp=y _q ，p≠q}

and finally, calculating the comparative learning loss by utilizing KL divergence and Gaussian embedding loss:

and 6.2) performing back propagation based on the comparison learning loss function, and performing iterative updating on the UniLM model parameters to obtain an information extraction model which can be finally used for extracting judicial literature indexes, as shown in FIG. 3.

Claims (6)

1. A judicial literature index extraction method based on few-sample comparison learning comprises the following steps:

1) Acquiring judicial literature data, performing data cleaning on the judicial literature data, and constructing training data containing training corpora based on the judicial literature;

2) Providing a structured language of a judicial literature index (JDISL) aiming at the judicial literature index, adopting the JDISL to conduct prompt guidance to obtain a structural form of the index based on the prompt, and further processing the training corpus into the prompt training corpus;

3) Carrying out few-sample negative sampling amplification on the prompt training corpus to obtain an amplified prompt training corpus;

4) Taking the amplified prompt training corpus as input on the basis of an open-source Unilm base pre-training model;

5) Taking a hidden layer vector of a Unilm model, and processing the hidden layer vector into Gaussian embedding;

6) Calculating contrast learning loss of Gaussian embedding, and performing iterative updating to obtain a final model;

7) And inputting the judicial documents of the indexes to be extracted into the trained final model, and extracting information by the index extraction model to obtain the information extraction result of each label type.

2. The method for extracting judicial literature indexes according to claim 1, wherein the step 2) of using prompt guidance for the judicial literature indexes to obtain a structural form of the indexes based on prompt, and the process of further processing the training corpus into the prompt training corpus comprises:

step 2.1) performing a prompt construction on court trial element indexes in the judicial writing by using an information extraction structured generation means or an information extraction means comprising entity extraction, relationship extraction, event extraction and response extraction so as to uniformly model a generation frame from a text to the prompt structure and realize uniform modeling on the entity indexes, the relationship indexes, the event indexes and the response indexes in the judicial writing;

the information extraction structured generation is decomposed into two atomic operations:

1) Positioning: locating a target information fragment from the sentence, wherein the information fragment comprises an entity and an event in the judicial literature;

2) And (3) association: connecting different information fragments according to the required association, namely the connection between the entities, the connection between the entities or the connection between the events;

according to the structure specificity of the judicial writing index, a structured language of the judicial writing index (JDISL) is designed, and the symbol of the JDISL is': "represents the mapping from information to its positioning or associated segments, the symbols" [ ], () "are used to represent the hierarchy between the extracted index information;

the JDISL consists of three parts:

1) info _ name, a specific piece of information indicating the existence of the source text;

2) A relation _ name, which represents a specific information segment existing in the source text, wherein the information segment is associated with the upper-layer info information in the structure;

3) info _ span represents the text fragment span corresponding to the specific information or associated fragment of the source text;

uniformly coding different information extraction structures through JDISL, and uniformly modeling different judicial literature index information extraction tasks into the same generation process from text to structuring;

the task of entity identification and event detection is modeled as:

(info_name:info_span)

the relationship extraction and event extraction tasks are uniformly modeled as follows:

(info_name:info_span(relation_name:info_span),…)；

step 2.2) after a prompt construction is carried out on the court trial element indexes in the judicial writing, processing training data containing the training corpus on the basis of the prompt construction so as to obtain the prompt training corpus; preferably, the Prompt structure is used as a predefined mode, and the training text is processed into a Prompt structured corpus form.

3. The method for extracting judicial writing index according to claim 1 or 2, wherein the process of performing the sample-less negative sampling amplification on the prompt training corpus in the step 3) comprises:

step 3.1) constructing corpus for prompt:

Content:text

Result:[[info_type_1],[info_type_2],…,[info_type_n]]

wherein [ info _ type _ n ] represents an information set in which the extracted structure having a tag type of n is in the form of [ [ info _ name ], [ relation _ name ], [ info _ span ] ];

processing results into a plurality of Result subsets based on promt:

Result_1:[info_type_1][prompt:label_1]

Result_2:[info_type_2][prompt:label_2]

…

Result_n:[info_type_n][prompt:label_n]

wherein label _ n represents a label type n corresponding to the [ info _ type _ n ] set;

then, based on the prompt type, negative sampling is carried out, and the data volume is expanded from n to n ^2, such as expanding Result _1 to n times:

Result_1:[info_type_1][prompt:label_1]

Result_1:[info_type_1][prompt:label_2]

…

Result_1:[info_type_1][prompt:label_n]，

thereby obtaining the amplified prompt training corpus.

4. The method for extracting judicial writing index according to any one of claims 1 to 3, wherein the process of taking the prompt corpus as input in step 4) on the basis of the open-source Unilm base pre-training model comprises:

and 4.1) processing the prompt training corpus into an input form required by the Unilm model. The input form accepted by the Unilm model is:

{x ₁ ，x ₂ ，x ₃ ，...，x _m-1 ，x _m }

the template structured information obtained by JDISL is [ [ info _ name ], [ relationship _ name ], [ text ] ], where text is the information referred to within the info _ span in the source text;

and then associating the prompt structural information serving as a prefix with the fragment information to obtain a model input corpus:

{info_name_1,…info_name_n,…,relation_name_1,…,relation_name_n,…,text_1,…,text_n,x ₁ ，x ₂ ，x ₃ ，...，x _m-1 ，x _m }；

using Unilm for both generating tasks and classifying tasks; after input, the input is first tokenizer pre-trained using Unilm followed by vector embedding including token embedding, segment embedding, and position embedding;

wherein, token embedding means that [ CLS ] marks are inserted into the beginning of each sentence, and [ SEP ] marks are inserted into the end of each sentence, wherein the [ CLS ] marks represent the vector of the current sentence, and the [ SEP ] marks represent the sentences used for segmenting the sentences in the text;

wherein segment embedding is used to distinguish two sentences, the different sentences being preceded by a and B labels, respectively, so that the input sentence is represented as (E) _A ，E _B ，E _A ，E _B ,……)；

Wherein, the position embedding is to add position information (i.e. or e.g. [0,1,2,3 \8230;) to the relative position according to the subscript index of the sentence;

after embedding, the hidden layer output vector is obtained through calculation of a multi-head attention mechanism through 24 series-connected transform structure layers

h＝{H；h ₁ ，h ₂ ，h ₃ ，...，h _m-1 ，h _m }

Wherein h is _m Represents the hidden layer state of the mth judicial writing sequence, and H represents the state set of the prompt structured information.

5. The method for extracting judicial writing index according to any one of claims 1 to 4, wherein the process of taking Unilm model hidden layer vectors in step 5) and processing the vectors into Gaussian embedding comprises the following steps:

step 5.1) Using a projection network f consisting of fully connected layers and nonlinear activation functions _u And f _∑ Generating a gaussian distribution parameter:

u _i ＝f _u (h _i )；

∑ _i ＝ELU(f _∑ (h _i ))+(1+∈)；

wherein u is _i ，∑ _i Mean covariance and diagonal covariance, respectively, representing gaussian embeddings, i.e. non-zero elements only along the diagonal of the matrix, f _u And f _∑ The method is characterized in that the method consists of single-layer linear layers, relu is taken as an activation function, and e < -14 > is taken for stability epsilon; processing the hidden layer output vector for Gaussian embedding (u) _i ，∑ _i )。

6. The judicial literature index extraction method according to any one of claims 1-5, wherein the step 6) of calculating the Gaussian-embedded contrast learning loss and the step of iteratively updating to obtain the final model comprise the following two substeps of 6.1) and 6.2):

step 6.1) in order to calculate the contrast loss, consider the KL divergence between all valid token pairs in the sampled corpus, if two tokens x _q And x _p Having the same label y _q ＝ _y p, they are considered as positive examples, with N (u) embedded for their gaussian _p ，∑ _p ) And N (u) _q ，∑ _q ) Its KL divergence was calculated:

both directions of KL divergence are calculated because it is not symmetric:

d(p，q)＝1/2(D _KL |N _p ||N _q |+D _KL |N _q ||N _p |)

for positive samples x _p Calculating x _p Gaussian embedding loss relative to other valid tokens in the batch:

χ _p ＝{(χ _q ，y _q )∈χ|y _p ＝y _q ，p≠q}

and finally, calculating the comparative learning loss by utilizing KL divergence and Gaussian embedding loss:

and 6.2) performing back propagation based on the comparison learning loss function, and performing iterative updating on the UniLM model parameters to obtain an information extraction model which can be finally used for judicial literature index extraction.

Images