JP7335300B2 - Knowledge pre-trained model training method, apparatus and electronic equipment - Google Patents

Description

TECHNICAL FIELD The present disclosure relates to language, natural language processing, and deep learning technology fields in the computer technology field, and more particularly to training methods, devices, electronic devices, storage media, and computer program products for knowledge pre-trained models.

Many models today lack common sense reasoning. For example, if the question is "What can you use to copy a document to paper with ink?" You can choose. However, since the copy paper frequently appears with the copy of the question and the paper, the model is more likely to choose the answer copy paper, leading to the model outputting wrong results. Model training methods in related arts cannot achieve joint training of common sense learning and semantic learning, model gain is limited to sample quality, the model needs to be retrained, and flexibility is poor. .

Kind Code: A1 Methods, apparatus, electronics, storage media, and computer program products for training knowledge pretrained models are provided.

According to a first aspect, providing a method for training a knowledge pretrained model, obtaining a training text, said training text comprising a structured knowledge text and corresponding sentences, said structured knowledge text comprising: including a head node, a tail node, and the relationship between the head node and the tail node; and training a knowledge pretrained model to be trained based on the training text.

According to a second aspect, there is provided an apparatus for training a knowledge pre-trained model, an acquisition module for acquiring a training text, said training text comprising a structured knowledge text and corresponding sentences, said structured A knowledge text includes an acquisition module including a head node, a tail node, and the relationship between the head node and the tail node, and a training module for training a knowledge pretrained model to be trained based on the training text. .

According to a third aspect, an electronic device is provided,
at least one processor;
a memory communicatively coupled to the at least one processor;
The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor performs the knowledge according to the second aspect. Run the training method of the pretrained model.

According to a fourth aspect, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, said computer instructions instructing said computer to perform the knowledge pre-trained model according to said first aspect. Execute the training method.

According to a fifth aspect, there is provided a computer program product, comprising a computer program, wherein said computer program, when executed by a processor, trains a knowledge pretrained model according to the first aspect of the present disclosure. implement the method.
According to a sixth aspect, there is provided a computer program, said computer program causing a computer to perform the method of training a knowledge pre-trained model according to said first aspect.

It should be noted that the content described in this section is not intended to limit the key or critical features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the present application will be readily understood from the following description.

The drawings are for better understanding of the present technical solution and do not limit the present application.
4 is a flow chart of a method for training a knowledge pre-trained model according to a first embodiment of the present disclosure; FIG. 4 is a flow chart of obtaining training text in a method for training a knowledge pre-trained model according to a second embodiment of the present disclosure; FIG. FIG. 5 is a schematic diagram of training a knowledge pre-trained model to be trained based on a training text in a method for training a knowledge pre-trained model according to a third embodiment of the present disclosure; 1 is a block diagram of a training device for knowledge pre-trained models according to a first embodiment of the present disclosure; FIG. Fig. 2 is a block diagram of a training device for knowledge pre-trained models according to a second embodiment of the present disclosure; FIG. 3 is a block diagram of an electronic device for implementing the training method of the knowledge pre-trained model of an embodiment of the present disclosure;

Illustrative embodiments of the present application are described below in conjunction with the drawings, and various details of the embodiments of the present application are included therein for ease of understanding and are merely exemplary. should be regarded as Accordingly, those skilled in the art may make various changes and modifications to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and brevity, the following description omits descriptions of well-known functions and constructions.

Speech can include technical fields such as speech recognition, voice interaction, etc., and is an important direction in the field of artificial intelligence.

Voice Recognition is a technology in which a device identifies and understands a voice signal and converts it into corresponding text or instructions, and is mainly divided into three aspects: feature extraction technology, mode matching criteria and model training technology. including.

Voice interaction is a technology in which a device and a user perform interactive actions such as dialogue, communication, and information exchange using voice as an information carrier. It has the advantage of being more comfortable.

Natural Language Processing (NLU) is a science that studies computer systems, especially software systems, that can effectively realize natural language communication, and is an important direction in the fields of computer science and artificial intelligence. be.

Deep learning (DL) is a new research direction in the field of machine learning (ML). It is the science that enables us to recognize data such as characters, images and sounds, and is widely applied to speech and image recognition.

FIG. 1 is a flow chart of a method for training a knowledge pre-trained model according to a first embodiment of the present disclosure.

As shown in FIG. 1, the training method for the knowledge pre-trained model according to the first embodiment of the present disclosure includes steps 101 to 102 below.

Step 101, obtain a training text, the training text includes a structured knowledge text and corresponding sentences, the structured knowledge text includes a head node, a tail node, and the relationship between the head node and the tail node.

It should be noted that the execution subject of the training method of the knowledge pre-trained model of the embodiment of the present disclosure may be a hardware device having data processing capability and/or software necessary for driving the operation of this hardware device. good. Optionally, execution entities may include workstations, servers, computers, user terminals, and other smart devices. Here, user terminals include, but are not limited to, mobile phones, computers, smart voice interactive devices, smart home appliances, vehicle-mounted terminals, and the like.

In embodiments of the present disclosure, a number of training texts can be obtained for training a knowledge pre-trained model to be trained. Training texts include structured knowledge texts and corresponding sentences.

In embodiments of the present disclosure, the structured knowledge text has explicit semantic information, including head nodes, tail nodes, and relationships between header and tail nodes.

For example, the structured knowledge text may be "Zhangsan Graduation University Huazhong University of Science and Technology". The head node of this structured knowledge text is Zhang San, the tail node is Huazhong University of Science and Technology, and the relationship between the head node and the tail node is the graduated university. The semantic information that it is a science and technology university can be clearly obtained.

Or, the structured knowledge text may be "Zhang San friend Li Si", the head node of this structured knowledge text is Zhang San, the tail node is Li Si, the relationship between the head node and the tail node is friend, Through this structured knowledge text, we can clearly obtain the semantic information that Li Si is Zhang San's friend.

It is understood that the structured knowledge text may be any other knowledge text with explicit semantic information and is not limited here.

In an embodiment of the present disclosure, structured knowledge texts and sentences have a correspondence relationship, one structured knowledge text corresponds to at least one sentence, and one sentence corresponds to at least one structured knowledge text. can be done. In addition, in the embodiment of the present disclosure, the content, format, etc. of the text are not limited, and can be obtained by methods such as networks and books. You can get sentences that

For example, if the structured knowledge text is "Zhang San's friend Li Si", the Internet can be searched for sentences containing the semantic information "Li Si is Zhang San's friend". Alternatively, if the structured knowledge text is "Nanjing, the capital of Jiangsu Province", it is possible to search for Jiangsu through the Internet and obtain sentences containing the semantic information that "the capital of Jiangsu Province is Nanjing."

Step 102, train a knowledge pre-trained model to be trained based on the training text.

Although related art often incorporates training entities into a knowledge pretrained model to be trained to learn common sense knowledge, this method uses methods such as TransE to perform initial pretraining on the knowledge pretrained model. After that, it needs to be mixed in the training process of the knowledge pre-trained model, and the joint training of common sense knowledge and semantic knowledge cannot be realized. It is difficult to fully learn the rich contextual information of a training entity, the performance gain of a knowledge pretrained model is limited to the embedding quality of the training entity, and the training entity is static, making the knowledge pretrained model Often need to be retrained.

Embodiments of the present disclosure may train a knowledge pre-trained model to be trained based on training texts, where the training texts include structured knowledge texts and corresponding sentences. Structured knowledge texts have clear semantic information but lack rich linguistic information, sentences have rich linguistic information but lack clear semantic information, structured knowledge texts and corresponding sentences Training the knowledge pre-trained model of the trainee based on the training sample consisting of the knowledge pre-trained model of the trainee learns common sense knowledge and rich semantic knowledge at the same time, so that common sense knowledge and semantic knowledge jointly Training can be realized.

Furthermore, the method does not require embedding the training entity into the knowledge pretrained model of the training target, the performance gain of the knowledge pretrained model is not limited to the embedding quality of the training entity, and the knowledge pretrained model is Rich contextual information can be obtained from the text, and it can be adjusted dynamically and is highly flexible.

Optionally, the knowledge pre-trained model to be trained can be set according to the actual situation.

As described above, according to the method for training an intelligent pre-trained model according to an embodiment of the present disclosure, a training text is obtained, the training text includes a structured knowledge text and corresponding sentences, the structured knowledge text includes a head node, A knowledge pre-trained model of the training target is trained based on the training text, including the tail nodes and the relationship between the head and tail nodes. Therefore, the knowledge pre-trained model of the training target can acquire common sense knowledge and rich semantic knowledge at the same time, and the joint training of common knowledge and semantic knowledge can be realized, and the training entity does not need to be embedded in the knowledge pre-trained model of the training target. , the performance gain of a knowledge pretrained model is not limited to the embedding quality of the training entity. Knowledge pre-trained models are capable of obtaining rich contextual information from the sentences of the training text and are dynamically adjustable and flexible.

On any of the above examples, and as shown in FIG. 2, in step 101, obtaining training text may include steps 201-205 as follows.

Step 201, get a word entry.

In embodiments of the present disclosure, many word entries can be obtained to obtain a large amount of training text.

Note that the embodiments of the present disclosure do not limit the content, format, etc. of word entries. For example, titles include, but are not limited to, person names, place names, and the like. For example, Zhangsan, Beijing, Summer Palace, etc.

Step 202, obtain the corresponding sentence based on the word entry.

In embodiments of the present disclosure, there is correspondence between word entries and sentences, one word entry can correspond to at least one sentence, and one sentence can correspond to at least one word entry. can.

Selectably, obtaining a corresponding sentence based on the word entry can include searching for the word entry over a network and obtaining a corresponding sentence from network search results corresponding to the word entry. For example, if the word entry is Zhang San, a website can be searched based on the search term Zhang San, and the relevant sentences can be obtained from the corresponding search results.

Selectably obtaining a corresponding sentence based on the word entry includes obtaining at least one candidate sentence based on the word entry, obtaining a degree of relevance between each candidate sentence and the word entry, and determining the degree of relevance between each candidate sentence and the word entry. A high candidate sentence may be the sentence corresponding to the word entry. This method can select a sentence with the highest degree of correlation with a word entry from a plurality of sentences.

Step 203, obtain the corresponding target triples based on the word entry and the sentence.

Optionally, obtaining the corresponding triplet based on the word entry and the sentence includes obtaining the corresponding candidate triples from a Knowledge Graph (KG) with the word entry as the head node, the candidate triples being , the head node, the tail node, and the relationship between the head node and the tail node, and determine the candidate triple corresponding to the tail node in the sentence as the target triple.

It is understood that the corresponding candidate triples can be obtained from the knowledge graph with the word entry as the head node, ie the head node of the candidate triple is the word entry. For example, if the word entry is Zhang San, the head node of its corresponding candidate triple is Zhang San.

This allows the method to filter out target triples from the knowledge graph in which the word entry is the head node and the tail node appears in the sentence.

Optionally, the knowledge graph can be set according to the actual situation.

Step 204, textualize the target triples to obtain structured knowledge texts.

It is understood that if the target triples do not have textual structure, the target triples can be textified to obtain a structured knowledge text.

Selectably, textualizing the target triples to obtain the structured knowledge text can include textualizing the target triples to obtain the structured knowledge text based on preset textualization rules. can. Here, the preset text conversion rule can be set according to the actual situation.

For example, if the target triple is (Zhang San, Friend, Li Si), the corresponding structured text may be Zhang San Friend Li Si, and if the target triple is (Jiangsu, Provincial Capital, Nanjing), The corresponding structured text may be Nanjing, Jiangsu Province.

Note that the method of converting the target triples into text may be other formats, and is not limited here.

Step 205, splicing the structured knowledge text and sentences to obtain training text.

Optionally, splicing the structured knowledge text and the sentence can include splicing the structured knowledge text to a preset position in the sentence. Here, the preset position can be set according to the actual situation, including, but not limited to, the date, the position of the tail node in the sentence, and the like.

Thus, the method obtains the corresponding sentence based on the word entry, obtains the corresponding target triple based on the word entry and the sentence, transcribes the target triple, obtains the structured knowledge text, and obtains the structure The training text can be obtained by splicing the knowledge text and sentences.

Under any of the above embodiments, as shown in FIG. 3, in step S102, training a pre-trained model to be trained based on the training text includes steps 301-302 as follows:

Step 301: Input the training text masked with the preset elements into the knowledge pre-trained model to be trained to generate prediction data of the preset elements.

In embodiments of the present disclosure, the training text includes at least one preset element, the at least one preset element in the training text is masked, and the training text with the masked preset element is the knowledge of the training target. It can be fed into a pre-trained model to generate prediction data for preset elements.

After inputting the training text in which the preset elements are masked into the knowledge pre-trained model of the trainee, predicting the preset elements through the knowledge pre-trained model of the trainee, It is understood that predictive data can be obtained.

Selectably, the preset elements may be any of head nodes, tail nodes, and relationships between head and tail nodes in the structured knowledge text, or human words in sentences. . In addition, if the preset element is any one of the head node, the tail node, and the relationship between the head node and the tail node in the structured knowledge text, the knowledge pre-trained model is allowed to learn common sense knowledge. It is understood that the knowledge pre-trained model may be semantically learned if the preset element is any one word in the sentence.

For example, if the text is "Zhang San is 26 years old, he graduated from Huazhong University of Science and Technology, which is a science university, and he is good at photo and video editing," and the structured knowledge text is "Zhang San Huazhong University of Science and Technology", the head node of this structured knowledge text is Zhang San, the tail node is Huazhong University of Science and Technology, the relationship between the head node and the tail node is the graduation school, and the sentence is "Zhang San is 26 years old. He graduated from Huazhong University of Science and Technology, which is a science and technology university. He is good at photo and video editing." The training text with the preset elements masked is "Zhang San's age is 26 years old," Mask "Graduated University is Huazhong University of Science and Technology, Huazhong University of Science and Technology is a college of science, good at photo and video editing. "Zhang San's age is 26 years old. Zhang San "Mask" is from Huazhong University of Science and Technology. Huazhong University of Science and Technology is a college of science and is good at photo and video editing. , "Zhang San is 26 years old. Zhang San graduated from "Mask" and Huazhong University of Science and Technology, specializing in photo and video editing. , "Zhang San's age is 26 years old. Zhang San's graduation academy is Huazhong University of Science and Technology, which is [Mask]. He is good at photo and video editing", including but not limited to.

Step 302, training a knowledge pre-trained model to be trained according to the preset factor prediction data and the preset factor;

Selectably, if there is a difference between the prediction data of the preset element and the preset element, training a knowledge pre-trained model of the training target based on said difference to produce a knowledge pre-trained model converge, or the number of iterations reaches a preset threshold number of iterations, or the model accuracy reaches a preset accuracy threshold, until the training of the knowledge pretrained model is terminated, and the last training can be the trained knowledge pre-trained model. Here, the iteration count threshold and the accuracy threshold can be set according to the actual situation.

Thus, the method inputs the training text with the preset elements masked into the knowledge pre-trained model to be trained to generate the prediction data of the preset elements, and the prediction data of the preset elements. Train a knowledge pre-trained model of the training target based on data and preset factors.

FIG. 4 is a block diagram of a training device for knowledge pre-trained models according to the first embodiment of the present disclosure.

As shown in FIG. 4, a knowledge pre-trained model training device 400 of an embodiment of the present disclosure includes an acquisition module 401 and a training module 402 .

The acquisition module 401 is configured to acquire a training text, said training text comprising a structured knowledge text and corresponding sentences, said structured knowledge text comprising a head node, a tail node, and a head node and a tail node. including the relationship between

A training module 402 is configured to train a knowledge pretrained model to be trained based on the training text.

In one embodiment of the present disclosure, the training module 402 specifically inputs the training text, with preset elements masked, into the knowledge pre-trained model of the trainee so that preset elements are masked. and train a knowledge pre-trained model to be trained based on the prediction data of the preset factor and the preset factor.

In one embodiment of the present disclosure, the preset element is any one of the head node, the tail node, and the relation in the structured knowledge text, or any one in the sentence. one word.

Above, the training device for the knowledge pre-trained model of the embodiment of the present disclosure obtains the training text, the training text includes the structured knowledge text and the corresponding sentences, the structured knowledge text includes the head node, the tail node, and the Train a knowledge pre-trained model of the training target based on the training text, including the relationship between the head node and the tail node. Therefore, the knowledge pre-trained model of the training target can acquire common sense knowledge and rich semantic knowledge at the same time, and the joint training of common knowledge and semantic knowledge can be realized, and the training entity does not need to be embedded in the knowledge pre-trained model of the training target. , the performance gain of a knowledge pretrained model is not limited to the embedding quality of the training entity. Knowledge pre-trained models are capable of obtaining rich contextual information from the sentences of the training text and are dynamically adjustable and flexible.

FIG. 5 is a block diagram of a training device for knowledge pre-trained models according to a second embodiment of the present disclosure.

As shown in FIG. 5, a knowledge pre-trained model training device 500 of an embodiment of the present disclosure includes an acquisition module 501 and a training module 502 .

Here, training module 502 has the same functionality and structure as training module 402 .

In one embodiment of the present disclosure, said obtaining module 501 comprises a first obtaining unit 5011 for obtaining word entries and a second obtaining unit 5012 for obtaining said corresponding sentences based on said word entries. a third obtaining unit 5013 for obtaining corresponding target triples based on the word entry and the sentence; and a textualization unit 5014 for textualizing the target triples to obtain the structured knowledge text. and a splicing unit 5015 for splicing the structured knowledge text and sentences to obtain the training text.

In one embodiment of the present disclosure, the third obtaining unit 5013 specifically obtains corresponding candidate triples from a knowledge graph with the word entry as the head node, wherein the candidate triples are the head node, including the corresponding tail node and the relation, and configured to determine the candidate triple corresponding to the tail node appearing in the sentence as the target triple.

As described above, the training device for the knowledge pre-trained model of the embodiment of the present disclosure obtains the training text, the training text includes the structured knowledge text and the corresponding sentences, the structured knowledge text includes the head node, the tail node and the head node. Train a knowledge pre-trained model of the training target based on the training text, including the relationships between the nodes and the tail nodes. Therefore, the knowledge pre-trained model of the training target can acquire common sense knowledge and rich semantic knowledge at the same time, and the joint training of common knowledge and semantic knowledge can be realized, and the training entity does not need to be embedded in the knowledge pre-trained model of the training target. , the performance gain of a knowledge pretrained model is not limited to the embedding quality of the training entity. Knowledge pre-trained models are capable of obtaining rich contextual information from the sentences of the training text and are dynamically adjustable and flexible.

According to embodiments of the present application, the present application provides an electronic device, a readable storage medium and a computer program product.
According to an embodiment of the present application, the present application provides a computer program, which causes a computer to perform a method of training a knowledge pre-trained model provided by the present application.

As shown in FIG. 6, it is a schematic block diagram of an electronic device of an electronic device 600 according to an embodiment of the present application. Electronic equipment is intended to represent various forms of digital computers such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronics can also represent various forms of mobile devices such as personal digital processors, mobile phones, smart phones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the description and/or required implementation of the application herein.

As shown in FIG. 6, the apparatus 600 includes a computing unit 601, which is a computer program stored in a read only memory (ROM) 602 or a computer program loaded into a random access memory (RAM) 603 from a storage unit 608. Various suitable actions and processes can be performed based on the program. Various programs and data necessary for the device 600 to operate can also be stored in the RAM 603 . Computing unit 601 , ROM 602 and RAM 603 are connected to each other via bus 604 . An input/output (I/O) interface 605 is also connected to bus 604 .

The multiple components in the device 600 are an input unit 606 such as a keyboard, mouse, etc., an output unit 607 such as various types of displays, speakers, etc., a storage unit 608 such as a magnetic disk, optical disk, etc., a network card, a modem, wireless communication. It is connected to an input/output (I/O) interface 605 including a communication unit 609 such as a transceiver. Communication unit 609 enables device 600 to exchange information/data with other devices over computer networks such as the Internet and/or various telecommunications networks.

Computing unit 601 may be various general purpose and/or special purpose processing components having respective processing and computing capabilities. Some examples of computing unit 601 include a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, computing units for various driving equipment learning model algorithms, digital signal processors ( DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, eg, the training method of the knowledge pretrained model described in FIGS. 1-3, for example. For example, in some embodiments, the training method for the knowledge pre-trained model can be implemented as a computer software program tangibly contained in a machine-readable medium such as storage unit 608 . In some embodiments, part or all of a computer program can be loaded and/or installed on electronic device 600 via ROM 602 and/or communication unit 609 . When the computer program is loaded into RAM 603 and executed by computing unit 601, it can perform one or more steps of the method of training a knowledge pre-trained model described above. Alternatively, in other embodiments, computing unit 601 may be configured by any other suitable method (eg, via firmware) as a training method for executing a knowledge pretrained model. .

Various embodiments of the systems and techniques described herein include digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), dedicated integrated circuits (ASICs), dedicated standard products (ASSPs), systems on chip. system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments are embodied in one or more computer programs, which can be executed and/or interpreted on a programmable system including at least one programmable processor. , the programmable processor, which may be a dedicated or shared programmable processor, receives data and instructions from the storage system, at least one input device, and at least one output device, and transmits data and instructions to the storage system, at least It can include transmitting to one input device and at least one output device.

Program code to implement the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be implemented in a general purpose computer, special purpose computer, or other programmable data processing apparatus, to perform the functions/acts specified in the flowchart illustrations and/or block diagrams when executed by a processor or controller. It may be provided in a processor or controller. Program code may be executed entirely on a machine, partially on a machine, as a stand-alone package, partially executed on a machine, and partially executed on a remote machine, or entirely remote machine. or run on a server.

In the context of this application, a machine-readable medium contains a program for use by or in combination with an instruction execution system, apparatus, or device, or It may be a tangible medium capable of being stored. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus or devices, or any suitable combination thereof. More specific examples of machine-readable storage media are electrical connections based on one or more lines, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable reads. memory only (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination thereof.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer that includes a display device (e.g., cathode ray tube (CRT)) for displaying information to the user. ) or LCD (liquid crystal display) monitor), and a keyboard and pointing device (e.g., mouse or trackball) through which a user can provide input to the computer. Other types of devices can also provide interaction with a user, e.g., the feedback provided to the user can be any form of sensing feedback (e.g., visual, auditory, or tactile feedback). may receive input from the user in any form (including acoustic, speech, and tactile input).

The systems and techniques described herein may be computing systems that include back-end components (e.g., data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include front-end components. A system (e.g., a user computer having a graphical user interface or web browser, through which users interact with embodiments of the systems and techniques described herein), or such a backend component , middleware components, and front-end components in any combination. The components of the system can be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include local area networks (LAN), wide area networks (WAN), and the Internet.

The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is created by computer programs running on corresponding computers and having a client-server relationship to each other. The server is a cloud server, also known as a cloud computing server or cloud host, is one host product in the cloud computing service system, and is the same as the traditional physical host and VPS service (abbreviated as "Virtual Private Server" or "VPS" This is because it resolved the flaws that existed in the existing system, which is difficult to manage and weak in business expandability. The server can be a distributed system server or a server that incorporates a blockchain.

According to an embodiment of the present application, the present application further provides a computer program product, comprising a computer program, wherein said computer program, when executed by a processor, performs the steps described in the above embodiments of the present application. We implement a training method for the knowledge pre-trained model that has been modified.

It should be appreciated that steps may be reordered, added, or deleted using the various forms of flow shown above. For example, each step described in this application may be performed in parallel, sequentially, or in a different order, but the technology disclosed in this application There is no limitation herein as long as the desired result of the scheme can be achieved.

The above specific embodiments are not intended to limit the protection scope of the present application. Those skilled in the art can make various modifications, combinations, subcombinations, and substitutions depending on design requirements and other factors. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall all fall within the protection scope of this application.

Claims (11)

A method of training a knowledge pretrained model, comprising:
obtaining a training text, the training text comprising a structured knowledge text and corresponding sentences, the structured knowledge text comprising head nodes, tail nodes, and relationships between the head nodes and the tail nodes; a step comprising
training a knowledge pretrained model to be trained based on the training text;
The training method for the knowledge pre-trained model comprises:
obtaining word entries;
obtaining the corresponding sentence based on the word entry;
obtaining corresponding target triples based on the word entry and the sentence;
textualizing the target triples to obtain the structured knowledge text;
splicing the structured knowledge text and the sentences to obtain the training text;
A training method for a knowledge pre-trained model characterized by: Based on the training text, training a knowledge pretrained model to be trained comprises:
inputting the training text with preset elements masked into the trained knowledge pre-trained model to generate prediction data for the preset elements;
training a knowledge pre-trained model of the training target based on the preset factor prediction data and the preset factor;
The training method according to claim 1, characterized by: the preset element is any one of the head node, the tail node, and the relationship in the structured knowledge text, or any one word in the sentence;
The training method according to claim 2, characterized by: Obtaining corresponding target triples based on the word entry and the sentence includes:
obtaining corresponding candidate triples from a knowledge graph with the word entry as the head node, the candidate triples including the head node, the corresponding tail node, and the relation;
determining the candidate triple corresponding to the tail node appearing in the sentence as the target triple;
The training method according to claim 1, characterized by: A training device for a knowledge pretrained model, comprising:
An acquisition module for acquiring a training text, the training text comprising a structured knowledge text and corresponding sentences, the structured knowledge text comprising a head node, a tail node, and the head node and the tail node. a retrieval module containing a relationship with
a training module for training a knowledge pretrained model to be trained based on the training text;
the acquisition module,
a first obtaining unit for obtaining word entries;
a second obtaining unit for obtaining the corresponding sentence based on the word entry;
a third obtaining unit for obtaining corresponding target triples based on the word entry and the sentence;
a textualization unit for textualizing the target triples to obtain the structured knowledge text;
a splicing unit for splicing the structured knowledge text and the sentences to obtain the training text;
A knowledge pre-trained model training device characterized by: The training module comprises:
inputting the training text with preset elements masked into the knowledge pre-trained model of the training target to generate prediction data for the preset elements;
training a knowledge pre-trained model of the training target based on predictive data of the preset factor and the preset factor;
6. The training device according to claim 5, characterized in that: the preset element is any one of the head node, the tail node, and the relationship in the structured knowledge text, or any one word in the sentence;
The training device according to claim 6 , characterized in that: the third acquisition unit comprising:
obtaining corresponding candidate triples from a knowledge graph with the word entry as the head node, the candidate triples including the head node, the corresponding tail node, and the relation;
determining the candidate triple corresponding to the tail node appearing in the sentence as the target triple;
6. The training device according to claim 5, characterized in that: at least one processor;
a memory communicatively coupled to the at least one processor;
The memory stores instructions executable by the at least one processor, and the instructions allow the at least one processor to perform the training method for the knowledge pre-trained model according to any one of claims 1 to 4. executably executed by the at least one processor;
An electronic device characterized by: A non-transitory computer-readable storage medium having computer instructions stored thereon,
The computer instructions cause a computer to execute the method of training a knowledge pretrained model according to any one of claims 1 to 4,
A non-transitory computer-readable storage medium characterized by: A computer program,
The computer program causes a computer to execute the knowledge pre-trained model training method according to any one of claims 1 to 4,
A computer program characterized by:

Images