CN111832282B - External knowledge fused BERT model fine adjustment method and device and computer equipment - Google Patents
External knowledge fused BERT model fine adjustment method and device and computer equipment Download PDFInfo
- Publication number
- CN111832282B CN111832282B CN202010688347.5A CN202010688347A CN111832282B CN 111832282 B CN111832282 B CN 111832282B CN 202010688347 A CN202010688347 A CN 202010688347A CN 111832282 B CN111832282 B CN 111832282B
- Authority
- CN
- China
- Prior art keywords
- vector
- sentence
- speech
- chinese
- vectors
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Document Processing Apparatus (AREA)
- Machine Translation (AREA)
Abstract
The invention discloses a method, a device and computer equipment for fine tuning a BERT model fused with external knowledge, wherein the method comprises the following steps: if an input Chinese sentence is received, obtaining a sentence vector and a part-of-speech vector of the Chinese sentence according to the BERT model; extracting an semantic set of the Chinese sentence from a preset external knowledge base; inputting the sememes in the sememe set into the BERT model to obtain a sememe vector set of the sememe set; screening out the semantic vector of the Chinese sentence from the semantic vector set; and fusing the sentence vector, the part-of-speech vector and the semantic vector of the Chinese sentence according to a preset fusion rule to finish fine tuning of the BERT model. Based on the natural language processing technology in the artificial intelligence, the invention not only successfully fuses the external knowledge into the BERT model to finish the fine adjustment of the BERT model, improves the accuracy of the BERT model in text analysis, but also can expand the text analysis field of the BERT model according to the external knowledge.
Description
Technical Field
The invention relates to the technical field of classification models, in particular to a method and a device for fine tuning a BERT model fusing external knowledge and computer equipment.
Background
A Pre-training model (PTM) is a network model that is trained on a large data set and then stored. The pre-training model is used for transfer learning and can be used as a feature extraction device. The application of the pre-trained model is generally the following process: firstly, under the condition that the calculation performance is met, a pre-training model is trained by using a certain larger data set, then the pre-training model is reconstructed according to different tasks, and finally, the data set of a new task is used for fine adjustment on the reconstructed pre-training model. The pre-training model has the advantages that the training cost is low, the fast convergence rate can be realized by matching with downstream tasks, the model performance can be effectively improved, particularly, the model has a better effect on tasks with scarce training data, namely, the model is learned based on a better initial state, and better performance can be achieved. The pre-training model is adopted, the whole network structure does not need to be retrained, and only the training is carried out on a plurality of layers of networks.
Currently, there are two methods for applying a pre-training model to a downstream classification task: feature-based (Feature-based) method and Fine-tuning (Fine-tuning) method, where Feature-based refers to using Word vectors trained by a pre-training language model as features, and inputting the features into a downstream target task, an ELMO (embedded from language model) model is mainly adopted, a network structure of the ELMO model adopts a two-layer bidirectional LSTM, and the ELMO provides a Feature form of each Word, namely Word Embedding of context information, to the downstream, and then directly inputs the Word Embedding into other network (GRU, CNN) structures, and parameters to be trained in the downstream task are only weights of each above-mentioned embidding. The Fine-tuning is to add a small amount of Task-specific parameters on the basis of a trained Language Model, and then retrain the Language material for Fine tuning, wherein the Fine-tuning includes a BERT (Bidirectional Encoder retrieval from transformers, bidirectional attention neural network Model) Model, and the feature extraction in the BERT Model uses a Transformer which has stronger feature extraction capability than LSTM, wherein the BERT uses a Mask Language Model (mass Language Model) to make the encor of the Transformer realize fusion of Bidirectional features, but the BERT Model cannot be completely qualified for the analysis Task of knowledge information in a specific field after being pre-trained in a large-scale unmarked corpus and in the subsequent Fine tuning process.
Disclosure of Invention
The embodiment of the invention provides a method and a device for fine tuning a BERT model fusing external knowledge and computer equipment, and solves the problem that the BERT model cannot be completely competent for the analysis task of knowledge information in a specific field in the subsequent fine tuning process.
In a first aspect, an embodiment of the present invention provides a method for fine tuning a BERT model fused with external knowledge, which includes:
if an input Chinese sentence is received, acquiring a sentence vector and a part-of-speech vector of the Chinese sentence according to the BERT model;
extracting an semantic set of the Chinese sentence from a preset external knowledge base;
inputting the sememes in the sememe set into the BERT model to obtain a sememe vector set of the sememe set;
screening out the semantic vector of the Chinese sentence from the semantic vector set;
and fusing the sentence vector, the part-of-speech vector and the primitive sense vector of the Chinese sentence according to a preset fusion rule to finish fine tuning of the BERT model.
In a second aspect, an embodiment of the present invention provides a fine tuning apparatus for a BERT model fusing external knowledge, including:
the first obtaining unit is used for obtaining a sentence vector and a part-of-speech vector of the Chinese sentence according to the BERT model if the input Chinese sentence is received;
the semantic extraction unit is used for extracting a semantic set of the Chinese sentence from a preset external knowledge base;
a second obtaining unit, configured to input an primitive in the primitive set into the BERT model to obtain a primitive vector set of the primitive set;
the screening unit is used for screening out the semantic vector of the Chinese sentence from the semantic vector set;
and the fusion unit is used for fusing the sentence vector, the part-of-speech vector and the sememe vector of the Chinese sentence according to a preset fusion rule so as to finish fine adjustment of the BERT model.
In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the method for fine tuning of the external knowledge fused BERT model according to the first aspect.
In a fourth aspect, the embodiments of the present invention further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the method for fine-tuning a BERT model fusing external knowledge according to the first aspect.
The embodiment of the invention provides a method, a device and computer equipment for fine tuning a BERT model fusing external knowledge. The method for finely adjusting the BERT model fusing the external knowledge successfully fuses the external knowledge into the BERT model to finish fine adjustment of the BERT model, improves the accuracy of the BERT model in analyzing the text, and can also improve the field of analyzing the text by the BERT model according to the external knowledge.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
Fig. 1 is a schematic flowchart of a fine tuning method of a BERT model fused with external knowledge according to an embodiment of the present invention;
FIG. 2 is a sub-flow diagram of a method for fine tuning a BERT model with external knowledge incorporated therein according to an embodiment of the present invention;
FIG. 3 is a schematic view of another sub-flow of a fine tuning method of a BERT model with external knowledge fused according to an embodiment of the present invention;
FIG. 4 is a schematic view of another sub-flow of a fine tuning method of a BERT model with external knowledge fused according to an embodiment of the present invention;
FIG. 5 is another sub-flowchart of the method for fine tuning a BERT model with knowledge fused with external knowledge according to the embodiment of the present invention;
FIG. 6 is a schematic view of another sub-flow of a fine tuning method of a BERT model with external knowledge fused according to an embodiment of the present invention;
FIG. 7 is a schematic block diagram of a fine tuning apparatus of a BERT model fusing external knowledge according to an embodiment of the present invention;
FIG. 8 is a block diagram of a sub-unit schematic diagram of a fine tuning apparatus of a BERT model fusing external knowledge provided by an embodiment of the present invention;
FIG. 9 is a schematic block diagram of another sub-unit of a fine tuning apparatus of a BERT model fusing external knowledge provided by an embodiment of the present invention;
FIG. 10 is a schematic block diagram of another sub-unit of a fine tuning apparatus of a BERT model fusing external knowledge provided by an embodiment of the present invention;
FIG. 11 is a schematic block diagram of another sub-unit of a fine tuning apparatus of a BERT model fusing external knowledge provided by an embodiment of the present invention;
FIG. 12 is a schematic block diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1, fig. 1 is a flow chart illustrating a fine tuning method of a BERT model with external knowledge fused according to an embodiment of the present invention. The method for finely tuning the BERT model fusing the external knowledge is built and operated in a server, after an input Chinese sentence is received in the process of finely tuning the BERT model in the server, sentence vectors and part-of-speech vectors of the Chinese sentence are obtained through a pre-trained BERT model, simultaneously, the sememes of all sentences in the Chinese sentence are extracted from a preset external knowledge base, then the extracted sememes are screened and input into the pre-trained BERT model to obtain the sememe vectors of the Chinese sentence, and the sentence vectors, the part-of-speech vectors and the sememe vectors of the Chinese sentence are fused, so that not only can the external network of the pre-trained BERT model be updated, but also the fine tuning of the pre-trained BERT model can be completed.
As shown in fig. 1, the method includes steps S110 to S150.
S110, if an input Chinese sentence is received, a sentence vector and a part-of-speech vector of the Chinese sentence are obtained according to the BERT model.
And if an input Chinese sentence is received, obtaining a sentence vector and a part-of-speech vector of the Chinese sentence according to the BERT model. The Chinese sentences are sentences for fine tuning the BERT model, the BERT model is a language model which is trained in advance, and the part-of-speech vectors of the Chinese sentences are sentence vectors containing the parts-of-speech of the words in the Chinese sentences. The Chinese sentence is input into a server in a character string mode, and when the server receives the character string of the Chinese sentence, a sentence vector and a part-of-speech vector of the Chinese sentence can be obtained in two modes according to the BERT model. The first method comprises the following steps: performing word segmentation processing on the Chinese sentence to obtain a word of the Chinese sentence, performing part-of-speech tagging on the word to obtain a part-of-speech tagged word, then respectively inputting the Chinese sentence and the part-of-speech tagged word into the BERT model to obtain a sentence vector of the Chinese sentence and a part-of-speech vector of the word, and finally constructing the part-of-speech vector of the Chinese sentence according to the part-of-speech vector of the word; the second method is as follows: the method comprises the steps of performing word segmentation processing on the Chinese sentence to obtain words of the Chinese sentence, performing part-of-speech tagging on the words to obtain words after part-of-speech tagging, then respectively inputting the words and the words after part-of-speech tagging into a BERT model to obtain word vectors and part-of-speech vectors of the words, and finally constructing the sentence vectors and the part-of-speech vectors of the Chinese sentence according to the word vectors and the part-of-speech vectors of the words.
In one embodiment, as shown in fig. 2, the step S110 of obtaining a sentence vector of the chinese sentence according to the BERT model includes steps S111 and S112.
And S111, performing word segmentation processing on the Chinese sentence to obtain words in the Chinese sentence.
And performing word segmentation processing on the Chinese sentence to obtain words in the Chinese sentence. Specifically, in the process of performing word segmentation processing on the chinese sentence, there are generally three methods for performing word segmentation processing on the chinese sentence, including: the method comprises the following steps of (1) performing word segmentation based on character string matching, a word segmentation based on statistics and a word segmentation based on understanding, wherein the word segmentation based on the character string is a word segmentation method for matching a Chinese character string with entries in a dictionary; the word segmentation method based on statistics calculates the adjacent co-occurrence probability of each word by counting the frequency of the combination of adjacent co-occurrence words in the material, thereby judging whether the adjacent words can be formed into words; the understanding-based word segmentation method achieves the effect of recognizing words by enabling a computer to simulate the understanding of the Chinese sentence by a human. In the embodiment of the invention, the Chinese sentence is subjected to word segmentation by adopting a reverse maximum matching method in the character string-based word segmentation method, and the word segmentation process comprises the following steps: and setting the number of Chinese characters contained in the longest entry in a preset dictionary to be L, and starting processing from the tail of the character string of the Chinese sentence. At the beginning of each cycle, the last L characters of the character string are taken as processing objects, and the dictionary is searched. If the dictionary has such an L word, the matching is successful, and the processing object is segmented as a word; if the Chinese sentence is not successful, the first Chinese character of the processing object is removed, the rest character strings are used as new processing objects, matching is carried out again until the segmentation is successful, namely, one round of matching is completed, a word is segmented, and the steps are circulated until all the words in the Chinese sentence are segmented.
For example, the length of the longest word in the dictionary is 6, for a character string of the chinese sentence "computer science and technology", first, 6 characters of "computer science and technology" are taken as the character string to be processed, and the word does not exist in the dictionary, so that matching fails; the first character is removed, the rest 'science and technology' is used as a new character string to be processed, and the matching is failed again; in this way, the "technology" is finally removed as a matching field, if the word exists in the dictionary, the matching is successful, and the first word "technology" is segmented. Then, the remaining character string 'computer science and' in the Chinese sentence is taken, and the second word 'and' at the segmentation position is segmented. By repeating the steps, the final segmentation result is as follows: "computer", "science", "and", "technology".
And S112, inputting the words into the BERT model to obtain word vectors of the words, and constructing sentence vectors of the Chinese sentences according to the word vectors.
And inputting the words into the BERT model to obtain word vectors of the words and constructing sentence vectors of the Chinese sentences according to the word vectors. Specifically, the BERT model is a sentence-level language model, unlike the ELMo model that weights are added to each layer to perform global pooling when the model is spliced with a downstream specific NLP task, the BERT can directly obtain a unique vector representation of an entire sentence. The method adds a special mark [ CL ] in front of each input, then allows a Transformer to carry out deep encoding on the [ CL ], because the Transformer can encode global information into each position regardless of space and distance, and the highest hidden layer of the [ CL ] is directly connected with an output layer of softmax as the representation of a sentence/sentence pair, so that the highest hidden layer can be used as a 'checkpoint' on a gradient back propagation path, and the upper layer characteristics of the whole input can be learned. The process of constructing the sentence vector of the Chinese sentence according to the word vector comprises the following steps: and inputting all words in the Chinese sentence into the BERT model to obtain word vectors of all words in the Chinese sentence, and then performing vector superposition on the word vectors of all words in the Chinese sentence according to the arrangement sequence of characters in the Chinese sentence to obtain a sentence vector of the Chinese sentence.
In one embodiment, as shown in fig. 3, the step S110 of obtaining the part-of-speech vector of the chinese sentence according to the BERT model includes steps S113 and S114.
And S113, performing part-of-speech tagging on the words according to a preset part-of-speech tagging rule to obtain the words after the part-of-speech tagging.
And performing part-of-speech tagging on the words according to a preset part-of-speech tagging rule to obtain the words after the part-of-speech tagging. The part-of-speech tagging rules are rule information for performing part-of-speech tagging on the words to obtain the words after the part-of-speech tagging. The part-of-speech tagging refers to a procedure for tagging the word with a correct part-of-speech, that is, a process for determining whether the word is a noun, a verb, an adjective, or another part-of-speech. The part of speech is the grammatical attribute of the word and is determined according to the grammatical function of the word in the combination. The grammatical attributes of words in Chinese include fourteen attributes of nouns, verbs, adjectives, numerators, quantifiers, pronouns, distinguishments, adverbs, prepositions, conjunctions, auxiliary words, sighs, adversaries and vocabularies. Specifically, in the process of performing part-of-speech tagging on the words, syntactic analysis is performed on the chinese sentences first to determine the position relationships of the words in the sentences, part-of-speech information of the words is acquired from a preset part-of-speech tagging set according to the position relationships of the words in the sentences, and then the words are tagged according to a BIES tagging standard, that is, a character at the beginning of the word is represented by B, a character at the end of the word is represented by E, a character in the middle of the word is represented by I, an individual character becomes a character of the word is represented by S, and the part-of-speech information of the word is represented by X. If the word is a word with two characters, the word is represented as B-E-X; if the word is of a single character, the word is represented as S-X; if the word is a word with three characters, the word is represented as B-I-E-X; if the word is a four character word, the word is represented as B-I-I-E-X.
S114, inputting the words with the parts of speech tagged into the BERT model to obtain part of speech vectors of the words, and constructing the part of speech vectors of the Chinese sentences according to the part of speech vectors of the words.
And inputting the words with the parts of speech tagged into the BERT model to obtain part of speech vectors of the words, and constructing the part of speech vectors of the Chinese sentences according to the part of speech vectors of the words. Specifically, the part-of-speech tagged word carries part-of-speech information of the sentence, the part-of-speech vector of the word is a word vector with part-of-speech tagging obtained after the sentence is subjected to word vectorization in the BERT model, and constructing the part-of-speech vector of the chinese sentence according to the part-of-speech vector of the word is: and performing vector superposition on the part-of-speech vectors of all words in the Chinese sentence according to the arrangement sequence of the characters in the Chinese sentence to obtain the sentence vector of the Chinese sentence.
S120, extracting the semantic set of the Chinese sentence from a preset external knowledge base.
And extracting the semantic source set of the Chinese sentence from a preset external knowledge base. Specifically, the external knowledge base is an open source knowledge network (OpenHowNet), and OpenHowNet is a common knowledge base which uses concepts represented by words of chinese and english as description objects to disclose relationships between concepts and attributes of the concepts as basic contents. The semantic source is the smallest irreparable semantic unit in linguistics, that is, one word can correspond to a plurality of semantic sources, and the semantic source set is the set of the semantic sources in the Chinese sentence. The process of extracting the semantic set of the Chinese sentence from the external knowledge base comprises the following steps: and performing word segmentation processing on the Chinese sentence to obtain a word of the Chinese sentence, acquiring all the sememes corresponding to the word from OpenHowNet according to the mapping relation between the word and the sememes in the OpenHowNet, and taking all the sememes corresponding to the word as a sememe set of the Chinese sentence.
S130, inputting the sememes in the sememe set into the BERT model to obtain a sememe vector set of the sememe set.
And inputting the sememes in the sememe set into the BERT model to obtain a sememe vector set of the sememe set. Specifically, the set of the primitive vectors of the primitive set is a set of the primitive vectors in the primitive set, and the primitives in the primitive set are stored in the server in a character string form before being input into the BERT model, so that all the primitives in the primitive set are input into the BERT model to be vectorized to obtain the primitive vectors of all the primitives in the primitive set, that is, the set of the primitive vectors.
S140, the semantic vector of the Chinese sentence is screened out from the semantic vector set.
And screening out the semantic vector of the Chinese sentence from the semantic vector set. Specifically, because a plurality of sememes in the sememe set may correspond to each word in the chinese sentence, and the sememe vectors in the sememe set are obtained by inputting the sememes in the sememe set into the BERT model, and a plurality of sememe vectors in the sememe vector set correspond to each word vector in the chinese sentence, it is necessary to perform screening from the sememe vector set so that each word in the chinese sentence corresponds to only one sememe vector in the sememe vector set, that is, to perform screening from the sememe vector set according to the part of speech of each word in the chinese sentence to obtain a sememe vector of the chinese sentence.
In one embodiment, as shown in fig. 4, step S140 includes sub-steps S141 and S142.
S141, calculating the similarity between the semantic vectors in the semantic vector set and the part-of-speech vectors of the words.
And calculating the similarity between the semantic vectors in the semantic vector set and the part of speech vectors of the words. Specifically, the similarity is a reaction of distances obtained by calculating distances between a plurality of primitive vectors corresponding to the words in the primitive vector set and the part-of-speech vectors of the words, respectively, the longer the distance is, the lower the similarity between the primitive vectors in the primitive vector set and the part-of-speech vectors of the words is, and otherwise, the higher the similarity is, and the primitive vector with the highest similarity is used as the primitive vector of the corresponding word in the chinese sentence. The distance calculation comprises calculation methods such as Euclidean distance calculation, manhattan distance calculation, chebyshev distance calculation, minkowski distance calculation, standardized Euclidean distance calculation, mahalanobis distance calculation, included angle cosine calculation, hamming distance calculation, jacard similarity coefficient calculation, correlation coefficient calculation and information entropy calculation. In this embodiment, the similarity score is obtained by using a euclidean distance calculation method. The euclidean distance is a commonly used distance definition, and refers to the true distance between two points in an n-dimensional space, or the natural length of a vector. The Euclidean distance calculation formula of the semantic vectors in the semantic vector set and the part-of-speech vectors of the words is as follows:
where n denotes the dimension of the vector, x 1k Is a part-of-speech vector, x, of said word 2k Is the original vector in the set of original vectors.
S142, obtaining the sememe vector of the Chinese sentence from the sememe vector set according to the similarity.
And acquiring the semantic vector of the Chinese sentence from the semantic vector set according to the similarity. Because the similarity is a reaction of distances obtained by calculating the distances between a plurality of primitive vectors corresponding to the words in the primitive vector set and the part-of-speech vectors of the words respectively, the longer the distance is, the lower the similarity between the primitive vectors in the primitive vector set and the part-of-speech vectors of the words is, and otherwise, the higher the similarity is, therefore, all the primitive vectors in the primitive vector set are calculated by the similarity, and when the distance between one primitive vector in the primitive vector set and the part-of-speech vector of the word is the shortest, the highest similarity between the primitive vector and the part-of-speech vector of the word is, namely one primitive vector in the Chinese sentence.
S150, according to a preset fusion rule, fusing the sentence vector, the part-of-speech vector and the primitive sense vector of the Chinese sentence to finish fine tuning of the BERT model.
And fusing the sentence vector, the part-of-speech vector and the primitive sense vector of the Chinese sentence according to a preset fusion rule to finish fine tuning of the BERT model. The fusion rule is rule information for fusing a sentence vector, a word vector and an sememe vector of the Chinese sentence to finish fine tuning of the BERT model. Specifically, the BERT model can be finely adjusted through the fusion rule, an external network of the BERT model can be updated, and therefore the accuracy of the BERT model in classifying sentences in a text is improved.
In one embodiment, as shown in FIG. 4, step S150 includes sub-steps S151 and S152.
And S151, splicing the part-of-speech vector, the primitive sense vector and the sentence vector of the Chinese sentence to obtain a spliced sentence vector.
And splicing the part-of-speech vector, the primitive sense vector and the sentence vector of the Chinese sentence to obtain a spliced sentence vector. Specifically, the part-of-speech vector, the primitive-sense vector and the sentence vector of the Chinese sentence are spliced to be vector superposition of the part-of-speech vector, the primitive-sense vector and the sentence vector of the Chinese sentence, so as to obtain a spliced sentence vector, wherein the spliced sentence vector not only contains semantic information of the Chinese sentence, but also contains primitive-sense information corresponding to the Chinese sentence.
S152, inputting the spliced sentence vectors into a preset first recurrent neural network to finish fine adjustment of the BERT model.
And inputting the spliced sentence vector into a preset first recurrent neural network to finish fine adjustment of the BERT model. Specifically, the first recurrent neural network is a trained external network of the BERT model, the first recurrent neural network may be a GRU recurrent neural network or a BiLSTM recurrent neural network, and the first recurrent neural network may be updated after the spliced sentence vector is input into the first recurrent neural network, that is, the external network of the BERT is updated, thereby completing fine tuning of the BERT model.
In another embodiment, as shown in fig. 6, step S150 includes sub-steps S1501 and S1502.
S1501, the part-of-speech vectors and the primitive sense vectors of the Chinese sentences are respectively input into a preset second cyclic neural network to obtain the part-of-speech sentence vectors and the primitive sense sentence vectors of the Chinese sentences.
And respectively inputting the part-of-speech vector and the original sense vector of the Chinese sentence into a preset second cyclic neural network to obtain the part-of-speech sentence vector and the original sense sentence vector of the Chinese sentence. Specifically, the second recurrent neural network is a trained external network of the BERT model, the second recurrent neural network may be a GRU recurrent neural network or a BiLSTM recurrent neural network, the part-of-speech vector and the primitive-sense vector of the chinese sentence are respectively input into the second recurrent neural network to perform semantic updating on the second recurrent neural network, and the part-of-speech sentence vector and the primitive-sense sentence vector of the chinese sentence output by the second recurrent neural network may be used to analyze the chinese sentence subsequently.
S1502, the part-of-speech sentence vectors, the primitive sense sentence vectors and the sentence vectors of the Chinese sentences are spliced to finish fine adjustment of the BERT model.
And splicing the part-of-speech sentence vector, the primitive sentence vector and the sentence vector of the Chinese sentence to finish fine adjustment of the BERT model. Specifically, the word-wise sentence vector, the primitive sense sentence vector and the sentence vector of the Chinese sentence are spliced, that is, the word-wise sentence vector, the primitive sense sentence vector and the sentence vector of the Chinese sentence are subjected to vector superposition to obtain the semantic vector of the Chinese sentence, so that the fine tuning of the BERT model is completed, and the problem that the subsequent Chinese sentences in the same field cannot be analyzed is solved.
The embodiment of the invention also provides a fine tuning device 100 of the external knowledge fused BERT model, which is used for executing any embodiment of the fine tuning method of the external knowledge fused BERT model. Specifically, referring to fig. 7, fig. 7 is a schematic block diagram of a fine tuning apparatus 100 for a BERT model with external knowledge fusion according to an embodiment of the present invention.
As shown in fig. 7, the fine tuning apparatus 100 for a BERT model fusing external knowledge includes a first obtaining unit 110, an extraction unit 120, a second obtaining unit 130, a filtering unit 140, and a fusing unit 150.
The first obtaining unit 110 is configured to, if an input chinese sentence is received, obtain a sentence vector and a part-of-speech vector of the chinese sentence according to the BERT model.
In another embodiment of the present invention, as shown in fig. 8, the first obtaining unit 110 includes a word segmentation unit 111, a third obtaining unit 112, a part of speech tagging unit 113, and a fourth obtaining unit 114.
And the word segmentation unit 111 is configured to perform word segmentation processing on the chinese sentence to obtain a word in the chinese sentence.
A third obtaining unit 112, configured to input the word into the BERT model to obtain a word vector of the word, and construct a sentence vector of the chinese sentence according to the word vector.
And a part-of-speech tagging unit 113, configured to perform part-of-speech tagging on the word according to a preset part-of-speech tagging rule to obtain a part-of-speech tagged word.
A fourth obtaining unit 114, configured to input the part-of-speech tagged word into the BERT model to obtain a part-of-speech vector of the word, and construct a part-of-speech vector of the chinese sentence according to the part-of-speech vector of the word.
And an semantic extracting unit 120, configured to extract a semantic set of the chinese sentence from a preset external knowledge base.
A second obtaining unit 130, configured to input an primitive in the primitive set into the BERT model to obtain a set of primitive vectors of the primitive set.
A screening unit 140, configured to screen out the semantic vector of the chinese sentence from the semantic vector set.
In other embodiments of the invention, as shown in fig. 9, the screening unit 140 includes a calculating unit 141 and a selecting unit 142.
A calculating unit 141, configured to calculate a similarity between a source vector in the source vector set and a part-of-speech vector of the word.
A selecting unit 142, configured to obtain an semantic vector of the chinese statement from the semantic vector set according to the similarity.
And a fusion unit 150, configured to fuse the sentence vector, the part-of-speech vector, and the primitive sense vector of the chinese sentence according to a preset fusion rule to complete fine tuning of the BERT model.
In other inventive embodiments, as shown in fig. 10, the fusion unit 150 includes a first splicing unit 151 and a first generation unit 152.
The first splicing unit 151 is configured to splice the part-of-speech vector, the primitive sense vector, and the sentence vector of the chinese sentence to obtain a spliced sentence vector.
A first generating unit 152, configured to input the stitched sentence vector into a preset first recurrent neural network to complete fine tuning of the BERT model.
In other inventive embodiments, as shown in fig. 11, the fusion unit 150 includes a second generation unit 1501 and a second splicing unit 1502.
The second generating unit 1501 is configured to input the part-of-speech vector and the primitive-sense vector of the chinese sentence into a preset second recurrent neural network, respectively, to obtain the part-of-speech sentence vector and the primitive-sense sentence vector of the chinese sentence.
A second concatenation unit 1502 is configured to concatenate the part-of-speech sentence vector, the primitive sentence vector, and the sentence vector of the chinese sentence to complete fine tuning of the BERT model.
The external knowledge fused BERT model fine tuning device 100 provided by the embodiment of the invention is used for executing the above steps of obtaining a sentence vector and a part-of-speech vector of a Chinese sentence according to the BERT model if the input Chinese sentence is received; extracting an semantic set of the Chinese sentence from a preset external knowledge base; inputting the sememes in the sememe set into the BERT model to obtain a sememe vector set of the sememe set; screening out an semantic vector of the Chinese sentence from the semantic vector set; and fusing the sentence vector, the part-of-speech vector and the primitive sense vector of the Chinese sentence according to a preset fusion rule to finish fine tuning of the BERT model.
Referring to fig. 12, fig. 12 is a schematic block diagram of a computer device according to an embodiment of the present invention.
Referring to fig. 12, the device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, may cause the processor 502 to perform a method of fine tuning a BERT model that incorporates external knowledge.
The processor 502 is used to provide computing and control capabilities that support the operation of the overall device 500.
The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 can be caused to perform a method for tuning the BERT model that incorporates external knowledge.
The network interface 505 is used for network communication, such as providing transmission of data information. It will be appreciated by those skilled in the art that the configuration shown in fig. 12 is a block diagram of only a portion of the configuration associated with aspects of the present invention and is not intended to limit the apparatus 500 to which aspects of the present invention may be applied, and that a particular apparatus 500 may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following functions: if an input Chinese sentence is received, obtaining a sentence vector and a part-of-speech vector of the Chinese sentence according to the BERT model; extracting an semantic set of the Chinese sentence from a preset external knowledge base; inputting the sememes in the sememe set into the BERT model to obtain a sememe vector set of the sememe set; screening out the semantic vector of the Chinese sentence from the semantic vector set; and fusing the sentence vector, the part-of-speech vector and the primitive sense vector of the Chinese sentence according to a preset fusion rule to finish fine tuning of the BERT model.
Those skilled in the art will appreciate that the embodiment of the apparatus 500 shown in fig. 12 does not constitute a limitation on the specific construction of the apparatus 500, and in other embodiments, the apparatus 500 may include more or fewer components than shown, or some components may be combined, or a different arrangement of components. For example, in some embodiments, the apparatus 500 may only include the memory and the processor 502, and in such embodiments, the structure and function of the memory and the processor 502 are the same as those of the embodiment shown in fig. 12, and are not repeated herein.
It should be understood that, in the embodiment of the present invention, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general-purpose processors 502, a Digital Signal Processor (DSP) 502, an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. The general-purpose processor 502 may be a microprocessor 502 or the processor 502 may be any conventional processor 502 or the like.
In another embodiment of the present invention, a computer storage medium is provided. The storage medium may be a non-volatile computer-readable storage medium. The storage medium stores a computer program 5032, wherein the computer program 5032 when executed by the processor 502 performs the steps of: if an input Chinese sentence is received, obtaining a sentence vector and a part-of-speech vector of the Chinese sentence according to the BERT model; extracting an semantic set of the Chinese sentence from a preset external knowledge base; inputting the sememes in the sememe set into the BERT model to obtain a sememe vector set of the sememe set; screening out the semantic vector of the Chinese sentence from the semantic vector set; and fusing the sentence vector, the part-of-speech vector and the primitive sense vector of the Chinese sentence according to a preset fusion rule to finish fine tuning of the BERT model.
It can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working processes of the above-described devices, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the technical solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method may be implemented in other manners. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only a logical division, and there may be other divisions when the actual implementation is performed, or units having the same function may be grouped into one unit, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a device 500 (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
While the invention has been described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (7)
1. A method for fine tuning a BERT model fused with external knowledge is characterized by comprising the following steps:
if an input Chinese sentence is received, obtaining a sentence vector and a part-of-speech vector of the Chinese sentence according to a BERT model;
extracting an semantic set of the Chinese sentence from a preset external knowledge base;
inputting the sememes in the sememe set into the BERT model to obtain a sememe vector set of the sememe set;
screening out the semantic vector of the Chinese sentence from the semantic vector set;
fusing a sentence vector, a part-of-speech vector and a sememe vector of the Chinese sentence according to a preset fusion rule to finish fine tuning of the BERT model;
obtaining a part-of-speech vector of the Chinese sentence according to the BERT model, wherein the obtaining comprises the following steps:
performing part-of-speech tagging on the words according to a preset part-of-speech tagging rule to obtain the words after the part-of-speech tagging; the part-of-speech tag comprises the position relation of the words in the sentence and the part-of-speech information of the words, and the position relation comprises the beginning of the words, the end of the words, the middle of the words and single word words;
inputting the words with the parts of speech tagged into the BERT model to obtain part of speech vectors of the words and constructing the part of speech vectors of the Chinese sentences according to the part of speech vectors of the words;
the process of fusing the sentence vector, the part-of-speech vector and the primitive-meaning vector of the Chinese sentence according to the preset fusion rule to finish the fine tuning of the BERT model comprises the following steps:
splicing the part-of-speech vector, the primitive sense vector and the sentence vector of the Chinese sentence to obtain a spliced sentence vector; the step of splicing the part-of-speech vector, the primitive sense vector and the sentence vector of the Chinese sentence comprises the following steps: performing vector superposition on the part-of-speech vector, the primitive sense vector and the sentence vector of the Chinese sentence;
inputting the spliced sentence vectors into a preset first recurrent neural network to finish fine adjustment of the BERT model;
or, the process of fusing the sentence vector, the part-of-speech vector, and the primitive sense vector of the chinese sentence according to the preset fusion rule to complete the fine tuning of the BERT model includes: respectively inputting the part-of-speech vector and the original sense vector of the Chinese sentence into a preset second cyclic neural network to obtain the part-of-speech sentence vector and the original sense sentence vector of the Chinese sentence;
splicing the part-of-speech sentence vector, the primitive sentence vector and the sentence vector of the Chinese sentence to finish fine adjustment of the BERT model; the method for splicing the part-of-speech sentence vector, the primitive sense sentence vector and the sentence vector of the Chinese sentence comprises the following steps: and performing vector superposition on the part-of-speech sentence vectors, the original meaning sentence vectors and the sentence vectors of the Chinese sentences.
2. The method of claim 1, wherein obtaining a sentence vector of the chinese sentence according to the BERT model comprises:
performing word segmentation processing on the Chinese sentence to obtain words in the Chinese sentence;
and inputting the words into the BERT model to obtain word vectors of the words and constructing sentence vectors of the Chinese sentences according to the word vectors.
3. The method of claim 1, wherein the step of filtering out the semantic vector of the chinese sentence from the set of semantic vectors comprises:
calculating the similarity between the semantic vectors in the semantic vector set and the part-of-speech vectors of the words;
and acquiring the semantic vector of the Chinese sentence from the semantic vector set according to the similarity.
4. A fine adjustment device of a BERT model fusing external knowledge is characterized by comprising:
the system comprises a first acquisition unit, a second acquisition unit and a processing unit, wherein the first acquisition unit is used for acquiring a sentence vector and a part-of-speech vector of an input Chinese sentence according to a BERT model if the input Chinese sentence is received;
the semantic extraction unit is used for extracting a semantic set of the Chinese sentence from a preset external knowledge base;
a second obtaining unit, configured to input an primitive in the primitive set into the BERT model to obtain a primitive vector set of the primitive set;
the screening unit is used for screening out the semantic vector of the Chinese sentence from the semantic vector set;
the fusion unit is used for fusing the sentence vectors, the part-of-speech vectors and the primitive senses vectors of the Chinese sentences according to a preset fusion rule so as to finish fine adjustment of the BERT model;
the first acquisition unit includes:
the part-of-speech tagging unit is used for carrying out part-of-speech tagging on the words according to a preset part-of-speech tagging rule so as to obtain the words after the part-of-speech tagging; the part-of-speech tag comprises the position relation of the words in the sentence and the part-of-speech information of the words, wherein the position relation comprises the beginning of the words, the end of the words, the middle of the words and single word words;
a fourth obtaining unit, configured to obtain part-of-speech vectors of the words in a BERT model and construct the part-of-speech vectors of the chinese sentences according to the part-of-speech vectors of the words;
the fusion unit includes: the first splicing unit is used for splicing the part-of-speech vector, the primitive sense vector and the sentence vector of the Chinese sentence to obtain a spliced sentence vector; the step of splicing the part-of-speech vector, the primitive sense vector and the sentence vector of the Chinese sentence comprises the following steps: performing vector superposition on the part-of-speech vector, the primitive sense vector and the sentence vector of the Chinese sentence; the first generation unit is used for inputting the spliced sentence vectors into a preset first recurrent neural network so as to finish fine adjustment of the BERT model;
or, the fusion unit includes: the second generation unit is used for respectively inputting the part-of-speech vectors and the primitive sense vectors of the Chinese sentences into a preset second cyclic neural network to obtain the part-of-speech sentence vectors and the primitive sense sentence vectors of the Chinese sentences;
the second splicing unit is used for splicing part-of-speech sentence vectors, primitive sense sentence vectors and sentence vectors of the Chinese sentences to finish fine adjustment of the BERT model; the splicing of the part-of-speech sentence vector, the primitive sentence vector and the sentence vector of the Chinese sentence comprises the following steps: and performing vector superposition on the part-of-speech sentence vector, the original semantic sentence vector and the sentence vector of the Chinese sentence.
5. The external knowledge fused BERT model fine-tuning device according to claim 4, wherein the first obtaining unit comprises:
the word segmentation unit is used for performing word segmentation processing on the Chinese sentence to obtain a word in the Chinese sentence;
and the third acquisition unit is used for inputting the words into the BERT model to obtain word vectors of the words and constructing sentence vectors of the Chinese sentences according to the word vectors.
6. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor when executing the computer program implements the method of fine tuning of the external knowledge fused BERT model as claimed in any of claims 1 to 3.
7. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to carry out the method of fine tuning of an external knowledge fused BERT model according to any one of claims 1 to 3.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010688347.5A CN111832282B (en) | 2020-07-16 | 2020-07-16 | External knowledge fused BERT model fine adjustment method and device and computer equipment |
| PCT/CN2020/118744 WO2021139266A1 (en) | 2020-07-16 | 2020-09-29 | Fine-tuning method and apparatus for external knowledge-fusing bert model, and computer device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010688347.5A CN111832282B (en) | 2020-07-16 | 2020-07-16 | External knowledge fused BERT model fine adjustment method and device and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111832282A CN111832282A (en) | 2020-10-27 |
| CN111832282B true CN111832282B (en) | 2023-04-14 |
Family
ID=72924335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010688347.5A Active CN111832282B (en) | 2020-07-16 | 2020-07-16 | External knowledge fused BERT model fine adjustment method and device and computer equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111832282B (en) |
| WO (1) | WO2021139266A1 (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100383B (en) * | 2020-11-02 | 2021-02-19 | 之江实验室 | Meta-knowledge fine tuning method and platform for multitask language model |
| GB2609768A (en) * | 2020-11-02 | 2023-02-15 | Zhejiang Lab | Multi-task language model-oriented meta-knowledge fine tuning method and platform |
| CN113589957A (en) * | 2021-07-30 | 2021-11-02 | 广州赛宸信息技术有限公司 | Method and system for rapidly inputting professional words of laws and regulations |
| CN114611524B (en) * | 2022-02-08 | 2023-11-17 | 马上消费金融股份有限公司 | Text error correction method and device, electronic equipment and storage medium |
| CN114723008A (en) * | 2022-04-01 | 2022-07-08 | 北京健康之家科技有限公司 | Language representation model training method, device, equipment, medium and user response method |
| CN114970532A (en) * | 2022-05-18 | 2022-08-30 | 重庆邮电大学 | Chinese named entity recognition method based on embedded distribution improvement |
| CN116630386B (en) * | 2023-06-12 | 2024-02-20 | 新疆生产建设兵团医院 | CTA scanning image processing method and system thereof |
| CN117034961B (en) * | 2023-10-09 | 2023-12-19 | 武汉大学 | BERT-based medium-method inter-translation quality assessment method |
| CN118520932B (en) * | 2024-07-25 | 2024-10-15 | 山东海量信息技术研究院 | Visual language model training method, device, medium and computer program product |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019024704A1 (en) * | 2017-08-03 | 2019-02-07 | 阿里巴巴集团控股有限公司 | Entity annotation method, intention recognition method and corresponding devices, and computer storage medium |
Family Cites Families (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100592293C (en) * | 2007-04-28 | 2010-02-24 | 李树德 | Knowledge search engine based on intelligent noumenon and implementing method thereof |
| CN106681981B (en) * | 2015-11-09 | 2019-10-25 | 北京国双科技有限公司 | The mask method and device of Chinese part of speech |
| JP6900190B2 (en) * | 2016-01-14 | 2021-07-07 | キヤノン株式会社 | Cognitive learning device, cognitive learning method and program |
| CN107239444B (en) * | 2017-05-26 | 2019-10-08 | 华中科技大学 | A kind of term vector training method and system merging part of speech and location information |
| CN107368466A (en) * | 2017-06-27 | 2017-11-21 | 成都准星云学科技有限公司 | A kind of name recognition methods and its system towards elementary mathematics field |
| CN108509411B (en) * | 2017-10-10 | 2021-05-11 | 腾讯科技（深圳）有限公司 | Semantic analysis method and device |
| CN108170674A (en) * | 2017-12-27 | 2018-06-15 | 东软集团股份有限公司 | Part-of-speech tagging method and apparatus, program product and storage medium |
| CN108717406B (en) * | 2018-05-10 | 2021-08-24 | 平安科技（深圳）有限公司 | Text emotion analysis method and device and storage medium |
| CN108664473A (en) * | 2018-05-11 | 2018-10-16 | 平安科技（深圳）有限公司 | Recognition methods, electronic device and the readable storage medium storing program for executing of text key message |
| KR102184278B1 (en) * | 2018-11-21 | 2020-11-30 | 한국과학기술원 | Method and system for transfer learning into any target dataset and model structure based on meta-learning |
| CN109740163A (en) * | 2019-01-09 | 2019-05-10 | 安徽省泰岳祥升软件有限公司 | Semantic representation resource generation method and device applied to deep learning model |
| CN109992648B (en) * | 2019-04-10 | 2021-07-02 | 北京神州泰岳软件股份有限公司 | Deep text matching method and device based on word migration learning |
| CN109992671A (en) * | 2019-04-10 | 2019-07-09 | 出门问问信息科技有限公司 | Intension recognizing method, device, equipment and storage medium |
| CN110147446A (en) * | 2019-04-19 | 2019-08-20 | 中国地质大学（武汉） | A kind of word embedding grammar based on the double-deck attention mechanism, equipment and storage equipment |
| CN110347843B (en) * | 2019-07-10 | 2022-04-15 | 陕西师范大学 | Knowledge map-based Chinese tourism field knowledge service platform construction method |
| CN110516055A (en) * | 2019-08-16 | 2019-11-29 | 西北工业大学 | A kind of cross-platform intelligent answer implementation method for teaching task of combination BERT |
| CN110580339B (en) * | 2019-08-21 | 2023-04-07 | 华东理工大学 | Method and device for perfecting medical term knowledge base |
| CN111125356A (en) * | 2019-11-29 | 2020-05-08 | 江苏艾佳家居用品有限公司 | Text classification method and system |
| CN111078836B (en) * | 2019-12-10 | 2023-08-08 | 中国科学院自动化研究所 | Machine reading understanding method, system and device based on external knowledge enhancement |
| CN111125318B (en) * | 2019-12-27 | 2021-04-30 | 北京工业大学 | Method for improving knowledge graph relation prediction performance based on sememe-semantic item information |
| CN111198939B (en) * | 2019-12-27 | 2021-11-23 | 北京健康之家科技有限公司 | Statement similarity analysis method and device and computer equipment |
| CN111209728B (en) * | 2020-01-13 | 2024-01-30 | 深圳市企鹅网络科技有限公司 | Automatic labeling and inputting method for test questions |
| CN111159416B (en) * | 2020-04-02 | 2020-07-17 | 腾讯科技（深圳）有限公司 | Language task model training method and device, electronic equipment and storage medium |
| CN111401077B (en) * | 2020-06-02 | 2020-09-18 | 腾讯科技（深圳）有限公司 | Language model processing method and device and computer equipment |
-
2020
- 2020-07-16 CN CN202010688347.5A patent/CN111832282B/en active Active
- 2020-09-29 WO PCT/CN2020/118744 patent/WO2021139266A1/en active Application Filing
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019024704A1 (en) * | 2017-08-03 | 2019-02-07 | 阿里巴巴集团控股有限公司 | Entity annotation method, intention recognition method and corresponding devices, and computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111832282A (en) | 2020-10-27 |
| WO2021139266A1 (en) | 2021-07-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111832282B (en) | External knowledge fused BERT model fine adjustment method and device and computer equipment | |
| CN109840287B (en) | Cross-modal information retrieval method and device based on neural network | |
| JP5128629B2 (en) | Part-of-speech tagging system, part-of-speech tagging model training apparatus and method | |
| US7295965B2 (en) | Method and apparatus for determining a measure of similarity between natural language sentences | |
| US10719668B2 (en) | System for machine translation | |
| CN117151220B (en) | Entity link and relationship based extraction industry knowledge base system and method | |
| CN112906392B (en) | Text enhancement method, text classification method and related device | |
| CN111444320A (en) | Text retrieval method and device, computer equipment and storage medium | |
| CN111783455B (en) | Training method and device of text generation model, and text generation method and device | |
| CN111291195A (en) | Data processing method, device, terminal and readable storage medium | |
| CN107315734A (en) | A kind of method and system for becoming pronouns, general term for nouns, numerals and measure words standardization based on time window and semanteme | |
| CN111966812A (en) | Automatic question answering method based on dynamic word vector and storage medium | |
| CN109783806A (en) | A kind of text matching technique using semantic analytic structure | |
| CN111985228A (en) | Text keyword extraction method and device, computer equipment and storage medium | |
| CN114265936A (en) | Method for realizing text mining of science and technology project | |
| CN117251524A (en) | Short text classification method based on multi-strategy fusion | |
| Tapsai et al. | Thai Natural Language Processing: Word Segmentation, Semantic Analysis, and Application | |
| US20220292263A1 (en) | Robust name matching with regularized embeddings | |
| CN114298031A (en) | Text processing method, computer device and storage medium | |
| CN111950281B (en) | Demand entity co-reference detection method and device based on deep learning and context semantics | |
| CN116414988A (en) | Graph convolution aspect emotion classification method and system based on dependency relation enhancement | |
| CN113012685A (en) | Audio recognition method and device, electronic equipment and storage medium | |
| CN115688885A (en) | Method and system for fine tuning BERT model based on convolutional neural network | |
| Muaidi | Levenberg-Marquardt learning neural network for part-of-speech tagging of Arabic sentences | |
| CN114676684B (en) | Text error correction method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |