CN117891957B - Knowledge graph completion method based on pre-training language model - Google Patents
Knowledge graph completion method based on pre-training language model Download PDFInfo
- Publication number
- CN117891957B CN117891957B CN202410289201.1A CN202410289201A CN117891957B CN 117891957 B CN117891957 B CN 117891957B CN 202410289201 A CN202410289201 A CN 202410289201A CN 117891957 B CN117891957 B CN 117891957B
- Authority
- CN
- China
- Prior art keywords
- entity
- training
- language model
- sample
- learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012549 training Methods 0.000 title claims abstract description 69
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000009466 transformation Effects 0.000 claims abstract description 13
- 230000008485 antagonism Effects 0.000 claims abstract description 4
- 230000006870 function Effects 0.000 claims description 38
- 239000013598 vector Substances 0.000 claims description 35
- 239000011159 matrix material Substances 0.000 claims description 33
- 238000012512 characterization method Methods 0.000 claims description 20
- 238000005070 sampling Methods 0.000 claims description 20
- 238000010606 normalization Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 8
- 238000011156 evaluation Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 3
- 125000006850 spacer group Chemical group 0.000 claims description 3
- 230000001502 supplementing effect Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 2
- 238000013507 mapping Methods 0.000 abstract 1
- 238000012545 processing Methods 0.000 abstract 1
- 230000007246 mechanism Effects 0.000 description 5
- 238000011426 transformation method Methods 0.000 description 5
- 238000011176 pooling Methods 0.000 description 4
- 230000000295 complement effect Effects 0.000 description 3
- 210000002445 nipple Anatomy 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 239000008267 milk Substances 0.000 description 2
- 210000004080 milk Anatomy 0.000 description 2
- 235000013336 milk Nutrition 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000001467 acupuncture Methods 0.000 description 1
- 229910052799 carbon Inorganic materials 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Machine Translation (AREA)
Abstract
The invention discloses a knowledge graph completion method based on a pre-training language model, which is characterized in that the prior knowledge graph is subjected to file format processing, and the mapping and formatting of numbers and names of entities and relations are established, so that knowledge triplet data suitable for the model is obtained, and then the pre-training language model is used for extracting the embedding of the entities and the relations in the triplet, so as to perform training learning. The invention designs an antagonism learning method and an entity fine granularity representation method based on relative transformation, which can remarkably improve the learning efficiency of a model and further improve the effect of knowledge graph completion.
Description
Technical Field
The invention relates to the technical field of knowledge graph completion, in particular to a knowledge graph completion method based on a pre-training language model.
Background
Knowledge graph completion is a challenging field, and is mainly reflected in the massive nature of knowledge graph data, namely, the knowledge graph data comprises a large number of entities and a complex network structure. Therefore, modeling the knowledge graph completion model is a complex project.
Along with the development of deep learning and pre-training language models, knowledge graph completion has appeared a brand new direction, namely a text-based knowledge graph completion model. The model mainly utilizes a pre-training language model to model the knowledge graph completion task, and a better effect is obtained.
However, the text-based knowledge graph completion task mostly only focuses on modeling the triplet matching task by using a language model, but ignores characterization information of massive entities, so that training information sources of the model are insufficient and rich; meanwhile, the model has a 'shortcut' problem in the contrast learning task.
Disclosure of Invention
In view of this, in order to solve the above technical problems at least in part, the present invention provides a knowledge graph complement method based on a pre-training language model by using countermeasure learning, and aims to introduce a relative transformation method based on an attention mechanism, so as to alleviate the "shortcut" problem occurring in the model in contrast learning, and make the training of the model more stable.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a knowledge graph completion method based on a pre-training language model, comprising:
Acquiring known knowledge graph data, constructing an input sequence,
Constructing a pre-training language model for extracting feature vectors of the input sequence,
Carrying out knowledge graph completion according to the matching score of the feature vector; wherein,
Determining a contrast loss function based on a learnable disturbance factor matrix, and training a pre-training language model according to the contrast loss function; the contrast loss function is:
Wherein, the number of negative samples in the current batch N, tau is a temperature coefficient, gamma is a fixed positive constant, beta is a super parameter, RTAM hrt is a scalar value of a given positive sample in an anti-disturbance factor matrix; representing scalar values of a given negative sample in an anti-perturbation factor matrix,/> Representing a triplet evaluation function.
Preferably, the triplet evaluation function employs cosine similarity; i.e.。
Preferably, the known knowledge-graph data is obtained and then preprocessed, including text normalization and relationship normalization,
The text normalization is to remove the underlined characters of the text data in the knowledge base of the knowledge graph;
The relationship is normalized to check if there is a surface in the same form and the relationship is normalized to a different form.
Preferably, the input sequence is structured as follows:
Wherein h represents a head entity, r represents a relationship, t represents a tail entity, and h text,rtext and t text represent a text description of the head entity, a text description of the relationship and a text description of the tail entity, respectively; Representing a concatenation operation, [ SEP ] represents a special spacer, seq hr represents the input sequence of the head entity-relation, and Seq t represents the input sequence of the tail entity.
Preferably, the pre-trained language model includes BERT hr、BERTt and BERT A;
the step of extracting feature vectors of the input sequence comprises:
The input of Seq hr into BERT hr yields vector e hr, the input of Seq t into BERT t yields vector e t, and the input of Seq t into BERT A yields vector Ae t.
Preferably, a pre-trained BERT model is built for initializing BERT hr、BERTt and BERT A using the same weights.
Preferably, the expression of each vector is:
wherein mean_pool represents the average pooling operation and normalize represents the normalization operation.
Preferably, the known knowledge-graph data is negatively sampled; to perform feature learning of head entity-relationship pairs and tail entities using a contrast learning paradigm, the negative sampling strategy comprising:
Negative sampling in a batch, and replacing tail entities in a training sample currently input into the model by other tail entities in the batch;
performing negative sampling before batch, and using the head entity-relation feature vector and the tail entity feature vector obtained by the previous round of training as negative samples of the current batch of training samples;
negative sampling is performed, and the tail entity in the current triplet is replaced by the head entity.
Preferably, the learning countermeasure factor matrix obtaining process includes:
calculating the distance between each group of inquiry (h, r) and all other candidate tail entities (t) to obtain a relative transformation matrix;
inputting the relative transformation matrix into a multi-layer perceptron to obtain attention weight;
Multiplying the relative transformation matrix by the attention weight to obtain an anti-disturbance factor matrix.
Preferably, the relative transformation matrix is calculatedWhen the distance calculation formula is adopted as cosine distance,
Where n is the batch size of the current batch training,。
Preferably, based on the entity characterization countermeasure learning method, learning fine-grained entity characterization features, the process includes:
Constructing an antagonism learning virtual sample library;
the vector Ae t is obtained to interact with the sample library, and the positive and negative samples in the updated sample library are used for updating the following steps:
where k i represents the update policy of the negative samples in the sample library, k i + represents the update policy of the positive samples in the sample library, i + represents the selection policy of the positive samples, η is the learning rate of the positive and negative sample updates in the repository, The temperature coefficient updated for the memory bank samples, p (·) represents the probability that the samples in the current sample bank are positive samples relative to the current Ae t, and K is the size of the sample bank.
Preferably, training the pre-training language model with a loss function determined based on a matrix of learnable challenge disturbance factors and a loss function based on an entity characterization challenge learning method, comprises:
preferably, the loss function based on the entity characterization countermeasure learning method is expressed as follows:
。
Compared with the prior art, the knowledge graph completion method based on the pre-training language model is provided, and the learning anti-disturbance factor matrix is obtained by introducing a relative transformation method based on an attention mechanism and utilizing the relation among samples, so that the problem of 'shortcut learning' in the original InfoNCE training model process is relieved, and the training efficiency and accuracy of the knowledge graph completion model are improved.
The invention has the other beneficial effects that the fine-granularity entity characterization countermeasure learning module is introduced, the characterization information of rich entity individuals in the knowledge graph is further utilized, and the fine-granularity entity representation characteristics are learned, so that the representation space of the entity in the knowledge graph is optimized, and the learning efficiency of the model training process and the accuracy and efficiency of the knowledge graph completion are further improved. The method provided by the invention has the advantages of high accuracy and low resource consumption.
The invention has the further beneficial effects that the combination of the relative transformation method based on the attention mechanism and Aler modules is used, and experiments show that the original knowledge graph completion model can achieve the performance under the condition of multiple computing resources under the condition of fewer computing resources, thereby improving the computing efficiency of the computing equipment and having the practical benefits of carbon reduction and energy saving.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a knowledge graph completion method based on a pre-training language model provided by the invention;
FIG. 2 is a diagram illustrating an exemplary training process of a pre-training language model provided by the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1
The embodiment of the application discloses a knowledge graph completion method based on a pre-training language model based on countermeasure learning and the pre-training language model, and the knowledge graph completion method based on the pre-training language model disclosed by the application can be applied to the completion of acupuncture knowledge graphs and can also be applied to the completion of knowledge graphs in other fields.
The complement method steps are as shown in fig. 1, and include:
Acquiring known knowledge graph data, constructing an input sequence,
Constructing a pre-training language model for extracting feature vectors of the input sequence,
Carrying out knowledge graph completion according to the matching score of the feature vector;
determining a contrast loss function based on a learnable disturbance factor matrix, and training a pre-training language model according to the contrast loss function; the contrast loss function is:
Wherein, the number of negative samples in the current batch N, tau is a temperature coefficient, gamma is a fixed positive constant, beta is a super parameter, RTAM hrt is a scalar value of a given positive sample in an anti-disturbance factor matrix; representing scalar values of a given negative sample in an anti-perturbation factor matrix,/> Representing a triplet evaluation function.
In this embodiment, the triplet evaluation function employs cosine similarity; i.e.。
The following describes the execution of each step in detail.
Firstly, acquiring known knowledge graph data and constructing an input sequence;
in this embodiment, preprocessing is performed after acquiring known knowledge graph data, where the preprocessing step includes text normalization and relationship normalization; wherein,
Text normalization refers to removing the underlined characters of the knowledge base of the knowledge graph that are present in the text data;
Relationship normalization refers to checking whether there is a relationship with the same surface form that is normalized to a different form.
Then constructing an input sequence, wherein in the application, the sequence construction refers to splicing text descriptions of corresponding entities and relations in the knowledge graph with the current entities and relations to form a desired input sequence;
in the application, the knowledge graph is composed of triples and is expressed by (h, r, t), wherein h represents a head entity, r represents a relation, and t represents a tail entity. Corresponding text descriptions, namely h text,rtext and t text, respectively represent the text description of the head entity, the text description of the relation and the text description of the tail entity of the three elements in the knowledge graph.
Assuming that the current knowledge graph triplet is related to (acupoint name, efficacy, acupoint efficacy), then the corresponding h is the acupoint name, h text is the text description of the acupoint name, r is the efficacy, t is the efficacy of the acupoint, generally abbreviated text, and t text is the text description of the specific efficacy of the acupoint, generally longer text.
As an exemplary embodiment, there are two kinds of construction formats of the input sequence, respectively:
Wherein, Representing a concatenation operation, [ SEP ] represents a special spacer, seq hr represents the input sequence of the head entity-relation, and Seq t represents the input sequence of the tail entity.
A second step of constructing a pre-training language model for extracting feature vectors of the input sequence,
In the invention, a pre-training language model (BERT-base-uncased) is used for constructing an encoder module of a knowledge-graph triplet, which comprises a head entity-relation encoder BERT hr, a tail entity encoder BERT t and an auxiliary entity encoder BERT A;
Preferably, a pre-trained BERT model is built, and the BERT hr、BERTt and BERT A are initialized by using the same weight; wherein BERT hr does not share weights derived from subsequent training updates with BERT t、BERTA.
In this embodiment, the head entity-relation pair and the tail entity are respectively encoded by using the constructed pre-training language model to obtain respective feature vectors, which specifically includes the following steps:
Inputting the obtained Seq hr into BERT hr via pooling layer and normalizing to obtain head entity-relation encoding vector e hr, inputting the obtained Seq t into BERT t and BERT A via pooling layer and normalizing to obtain tail entity encoding vector e t and auxiliary entity encoding vector Ae t respectively,
Preferably, the coding vectors obtained in the encoder module are averaged and pooled; i.e.
Wherein mean_pool represents the average pooling operation and normalize represents the normalization operation.
In one embodiment, the pre-training language model is trained using a comparative learning paradigm; as shown in fig. 2, includes:
1) Negative sampling is carried out on the known knowledge graph data;
Specifically, three negative sampling strategies, namely intra-batch negative sampling, pre-batch negative sampling and self-negative sampling, are adopted to carry out negative sample sampling, and are used for carrying out feature learning of a head entity-relation pair and a tail entity based on a comparison learning paradigm; wherein,
Negative sampling in a batch, i.e. the training samples currently input to the model, using other tail entities in the batch with (h, r, t) as positive samplesReplacing t in (h, r, t) to obtain a negative sample/>Examples of negative sampling within a batch that may be possible are: (Yangxi, located in the center of the nipple);
The negative sampling before batch, namely, the head entity-relation feature vector and the tail entity feature vector obtained by the previous round of training are used as negative samples of the training samples of the current batch, namely, the possible sample of the negative sampling before batch is: in fact this is a correct triplet (in milk, at the center of the nipple), except that in our training phase, the sequence code of (in milk) is coded to e hr and the sequence code of (at the center of the nipple) is coded to e t in different training sequences, so this is also considered as a negative sample;
Negative samples (h, r, h) are obtained from negative sampling, i.e. for the current positive sample (h, r, t), the tail entity in the current triplet is replaced with the head entity. One example of a self-negative sampling that may be possible is: (Yangxi, located in Yangxi).
2) The traditional comparison file paradigm adopts InfoNCE loss functions as the training loss functions, specifically:
The existing research shows that the traditional InfoNCE loss function can cause the problem of 'shortcut learning' on model training, so the invention uses a relative transformation method based on attention to relieve 'shortcut learning' in the InfoNCE loss function.
The method comprises the following specific steps:
Calculating the distance between each group of inquiry (h, r) and all other candidate tail entities (t) to obtain a relative transformation matrix ；
Will beInput into a multi-layer perceptron to obtain attention weight/>；
Will beAnd/>Multiplying to obtain an anti-disturbance factor matrix/>Namely, a relative transformation matrix based on an attention mechanism, and the specific calculation method comprises the following steps:
Wherein, CD (·) is a distance function, MLP is a multi-layer perceptron, n is the batch size of the current batch training;
preferably, the relative transformation matrix is calculated When the distance function used is the cosine distance, i.e
Wherein,；
Finally, the contrast loss function which is obtained by integrating the relative transformation method based on the attention mechanism and can learn to resist the disturbance factor matrix determination is as follows:
Wherein, the number of negative samples in the current batch N, tau is a temperature coefficient, gamma is a fixed positive constant, beta is a super parameter, RTAM hrt is a scalar value of a given positive sample in an anti-disturbance factor matrix; representing scalar values of a given negative sample in an anti-perturbation factor matrix,/> Representing a triplet evaluation function. In this embodiment, the triplet evaluation function employs cosine similarity; i.e./>。
In this embodiment, n=128, γ=0.2, and β=0.5 are set.
3) The application further provides a method for using Aler to further optimize the representation space of the entity based on the entity characterization fine granularity learning module for resisting learning, which is used for obtaining the loss of the entity characterization fine granularity learning;
Specifically, the process of learning fine-grained entity characterization features based on an entity characterization challenge learning method includes:
Constructing an antagonism learning virtual sample library; as a sample library of entity characterization fine-grained learning modules based on countermeasure learning,
The vector Ae t is used for interacting with a sample library to obtain a feedback value for updating positive and negative samples in the sample library, and the iterative updating and the characterization information of the samples in the repository are synchronized in the training process, wherein the updating steps are as follows:
where k i represents the update policy of the negative samples in the sample library, k i + represents the update policy of the positive samples in the sample library, i + represents the selection policy of the positive samples, η is the learning rate of the positive and negative sample updates in the repository, The temperature coefficient updated for the memory bank samples, p (·) represents the probability that the samples in the current sample bank are positive samples relative to the current Ae t, and K is the size of the sample bank.
In one embodiment, k=65536,=0.04。
The loss function of the entity characterization fine granularity learning finally obtained by the application is as follows:
Further, to enable the fine-grained learning module to be applied to the triad structured information obtained when other modules are trained based on the entity representation of the countermeasure learning, part of the parameters in the BERT t are shared to the BERT A, so that the BERT A has the capability of sensing the triad structured information.
Because the BERT A used in Aler needs to obtain part of the parameters shared in BERT t, other relevant modules are updated implicitly according to the loss function of Aler modules when the final gradient is back propagated according to the chain rule;
Specifically, BERT hr is updated only from the triplet matching training module, BERT t is updated from the two penalty function terms, BERT A is updated from the penalty of BERT t and Aler;
Before each round of loss return, performing parameter sharing of the BERT t->BERTA, and after the return, performing back propagation parameter updating of the L_ Aler;
And all parameters are shared as a percentage of the super parameters, i.e. param _ BERT A = gamma * param_BERTA + (1-gamma) * param_BERTt.
According to the application, the Aler module is introduced, so that the characterization information of rich entity individuals in the knowledge graph is further utilized, and the entity representation characteristics of fine granularity are learned, thereby optimizing the representation space of the entities in the knowledge graph, and further improving the accuracy and efficiency of knowledge graph completion.
In a preferred embodiment, the two loss values are weighted and added to obtain a final loss function of model training, and the loss function is used for training the pre-training language model.
Where θ represents a weight coefficient, specifically, θ=1.4 is selected as the best embodiment.
Thirdly, carrying out knowledge graph completion according to the matching score of the feature vector;
The matching score calculation formula of the feature vector obtained through the normalization operation can be simplified. The simplified matching score calculation formula is the dot product among vectors:
Wherein e x and e y represent two embedded vectors respectively, And/>Representing the modulo lengths of the two embedded vectors, respectively. Since the vector obtained from the encoder is processed via normalize, the calculation formula for calculating the matching score can be simplified as:
I.e., given a query (h, r), the answer (t) of the model prediction can be expressed as:
wherein ɛ is the candidate entity corpus.
Example two
The present embodiment provides a computer readable storage medium, where the storage medium may be a storage medium such as a ROM, a RAM, a magnetic disk, or an optical disk, and the storage medium stores one or more programs, where the programs, when executed by a processor, implement the knowledge graph complement method based on the pre-training language model disclosed in the first embodiment.
Example III
The present embodiment provides a computing device, which may be a desktop computer, a notebook computer, a smart phone, a PDA handheld terminal, a tablet computer, or other terminal devices with display functions, where the computing device includes a processor and a memory, where the memory stores one or more programs, and when the processor executes the programs stored in the memory, the knowledge graph completing method based on the pre-training language model in the first embodiment is implemented.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (6)
1. A knowledge graph completion method based on a pre-training language model, comprising:
Acquiring known knowledge graph data, wherein the knowledge graph consists of triples and is expressed by (h, r, t), wherein h represents a head entity, r represents a relation, and t represents a tail entity; and the input sequence is constructed in the following format,
Seqhr＝(h||htext[SEP]r||rtext)
Seqt＝(t|ttext)
Wherein h text,rtext and t text represent a textual description of the head entity, a textual description of the relationship, and a textual description of the tail entity, respectively; the I represents a concatenation operation, [ SEP ] represents a special spacer, seq hr represents the input sequence of the head entity-relation, and Seq t represents the input sequence of the tail entity;
Constructing a pre-training language model, and constructing an encoder module of a knowledge-graph triplet by using the pre-training language model, wherein the encoder module comprises a head entity-relation encoder BERT hr, a tail entity encoder BERT t and an auxiliary entity encoder BERT A; the method for extracting the feature vector of the input sequence comprises the following steps:
Inputting Seq hr into BERT hr to obtain vector e hr, inputting Seq t into BERT t to obtain vector e t, and inputting Seq t into BERT A to obtain vector Ae t;
Carrying out knowledge graph completion according to the matching score of the feature vector; wherein,
Determining a contrast loss function based on a learnable disturbance factor matrix, and training a pre-training language model according to the contrast loss function; the contrast loss function is:
Wherein, the number of negative samples in the current batch N, tau is a temperature coefficient, gamma is a fixed positive constant, beta is a super parameter, RTAM hrt is a scalar value of a given positive sample in an anti-disturbance factor matrix; RTAM hrt′ denotes the scalar value of a given negative sample in the anti-perturbation factor matrix, Representing a triplet evaluation function;
the learning anti-disturbance factor matrix obtaining process comprises the following steps:
Calculating the distance between each group of inquiry (h, r) and all other candidate tail entities (t) to obtain a relative transformation matrix; when the relative transformation matrix is calculated, the distance calculation formula adopted is cosine distance,
Wherein RTM n*n is the relative transformation matrix, n is the batch size of the current batch training, CD (. Cndot.) is the distance function,
Inputting the relative transformation matrix into a multi-layer perceptron to obtain attention weight;
α＝MLP(RTMn*n)
alpha is attention weight, and MLP is multi-layer perceptron;
multiplying the relative transformation matrix with the attention weight to obtain an anti-disturbance factor matrix;
RTAMn*n＝RTMn*n*α。
2. The method for supplementing knowledge patterns based on a pre-training language model according to claim 1 wherein the pre-processing is performed after the known knowledge pattern data is obtained, including text normalization and relationship normalization,
The text normalization is to remove the underlined characters of the text data in the knowledge base of the knowledge graph;
The relationship is normalized to check if there is a surface in the same form and the relationship is normalized to a different form.
3. The knowledge-graph completion method based on a pre-training language model according to claim 1, wherein known knowledge-graph data is negatively sampled; to perform feature learning of head entity-relationship pairs and tail entities using a contrast learning paradigm, the negative sampling strategy includes:
Negative sampling in a batch, and replacing tail entities in a training sample currently input into the model by other tail entities in the batch;
performing negative sampling before batch, and using the head entity-relation feature vector and the tail entity feature vector obtained by the previous round of training as negative samples of the current batch of training samples;
negative sampling is performed, and the tail entity in the current triplet is replaced by the head entity.
4. The knowledge-graph completion method based on a pre-training language model according to claim 1, wherein the learning of fine-grained entity characterization features based on an entity characterization countermeasure learning method comprises:
Constructing an antagonism learning virtual sample library;
the vector Ae t is obtained to interact with the sample library, and the positive and negative samples in the updated sample library are used for updating the following steps:
Where K i represents the update policy of the negative sample in the sample library, K i + represents the update policy of the positive sample in the sample library, i + represents the selection policy of the positive sample, η is the learning rate of the positive and negative sample update in the memory library, τ' is the temperature coefficient of the memory library sample update, p (·) represents the probability that the sample in the current sample library is the positive sample relative to the current Ae t, and K is the size of the sample library.
5. The knowledge-graph completion method based on a pre-training language model according to claim 4, wherein the pre-training language model is trained by using a loss function determined based on a matrix of learnable challenge disturbance factors and a loss function based on an entity characterization challenge learning method, comprising
Loss＝LTMT+θ·LAler
Where θ represents a weight coefficient, L TMT represents a loss function determined based on a matrix of learnable challenge disturbance factors, and L Aler represents a loss function based on an entity characterization challenge learning method.
6. The knowledge-graph completion method based on a pre-training language model according to claim 5, wherein the loss function of the countermeasure learning method based on entity characterization is expressed as:
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410289201.1A CN117891957B (en) | 2024-03-14 | 2024-03-14 | Knowledge graph completion method based on pre-training language model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410289201.1A CN117891957B (en) | 2024-03-14 | 2024-03-14 | Knowledge graph completion method based on pre-training language model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117891957A CN117891957A (en) | 2024-04-16 |
| CN117891957B true CN117891957B (en) | 2024-05-07 |
Family
ID=90645997
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410289201.1A Active CN117891957B (en) | 2024-03-14 | 2024-03-14 | Knowledge graph completion method based on pre-training language model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117891957B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022135120A1 (en) * | 2020-12-21 | 2022-06-30 | 浙江大学 | Adaptive knowledge graph representation learning method combining graph structure with text information |
| CN116383401A (en) * | 2023-03-09 | 2023-07-04 | 大连理工大学 | Knowledge graph completion method integrating text description and graph convolution mechanism |
| CN117035080A (en) * | 2023-08-18 | 2023-11-10 | 西南交通大学 | Knowledge graph completion method and system based on triplet global information interaction |
| CN117273134A (en) * | 2023-09-28 | 2023-12-22 | 东南大学 | Zero-sample knowledge graph completion method based on pre-training language model |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11593560B2 (en) * | 2020-10-21 | 2023-02-28 | Beijing Wodong Tianjun Information Technology Co., Ltd. | System and method for relation extraction with adaptive thresholding and localized context pooling |
-
2024
- 2024-03-14 CN CN202410289201.1A patent/CN117891957B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022135120A1 (en) * | 2020-12-21 | 2022-06-30 | 浙江大学 | Adaptive knowledge graph representation learning method combining graph structure with text information |
| CN116383401A (en) * | 2023-03-09 | 2023-07-04 | 大连理工大学 | Knowledge graph completion method integrating text description and graph convolution mechanism |
| CN117035080A (en) * | 2023-08-18 | 2023-11-10 | 西南交通大学 | Knowledge graph completion method and system based on triplet global information interaction |
| CN117273134A (en) * | 2023-09-28 | 2023-12-22 | 东南大学 | Zero-sample knowledge graph completion method based on pre-training language model |
Non-Patent Citations (1)
| Title |
|---|
| 基于神经网络的知识推理研究综述;张仲伟;曹雷;陈希亮;寇大磊;宋天挺;;计算机工程与应用;20190325(第12期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117891957A (en) | 2024-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111737476B (en) | Text processing method and device, computer readable storage medium and electronic equipment | |
| CN111755078A (en) | Drug molecule attribute determination method, device and storage medium | |
| CN113837370B (en) | Method and apparatus for training a model based on contrast learning | |
| WO2021238333A1 (en) | Text processing network, neural network training method, and related device | |
| CN110990596B (en) | Multi-mode hash retrieval method and system based on self-adaptive quantization | |
| CN111753190A (en) | Meta learning-based unsupervised cross-modal Hash retrieval method | |
| CN114491039B (en) | Primitive learning few-sample text classification method based on gradient improvement | |
| CN115687571B (en) | Depth unsupervised cross-modal retrieval method based on modal fusion reconstruction hash | |
| CN115080749B (en) | Weak supervision text classification method, system and device based on self-supervision training | |
| CN116932722A (en) | Cross-modal data fusion-based medical visual question-answering method and system | |
| CN115482395A (en) | Model training method, image classification method, device, electronic equipment and medium | |
| CN115827954A (en) | Dynamically weighted cross-modal fusion network retrieval method, system and electronic equipment | |
| CN116383422A (en) | Non-supervision cross-modal hash retrieval method based on anchor points | |
| CN115329120A (en) | Weak label Hash image retrieval framework with knowledge graph embedded attention mechanism | |
| CN113420552B (en) | Biomedical multi-event extraction method based on reinforcement learning | |
| CN113869005A (en) | Pre-training model method and system based on sentence similarity | |
| CN112182144B (en) | Search term normalization method, computing device, and computer-readable storage medium | |
| CN112132269B (en) | Model processing method, device, equipment and storage medium | |
| CN116720519B (en) | Seedling medicine named entity identification method | |
| CN117891957B (en) | Knowledge graph completion method based on pre-training language model | |
| Qin et al. | Deep top similarity hashing with class-wise loss for multi-label image retrieval | |
| CN116796749A (en) | Medical named entity recognition robustness enhancement method and system | |
| CN113204679B (en) | Code query model generation method and computer equipment | |
| CN115273110A (en) | Text recognition model deployment method, device, equipment and storage medium based on TensorRT | |
| CN117371447A (en) | Named entity recognition model training method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |