CN114841173A - Academic text semantic feature extraction method and system based on pre-training model and storage medium - Google Patents

Abstract

The invention provides a method, a system and a storage medium for extracting academic text semantic features based on a pre-training model, wherein the method comprises the following steps: acquiring academic resource text data; inputting the acquired academic resource text data into a pre-training model to obtain a multi-dimensional academic text semantic feature vector; the pre-training model is a student pre-training model obtained by finely tuning a Bert pre-training model based on a multiple load sample loss function, and training the student model by knowledge distillation by taking the finely tuned Bert pre-training model as a teacher model; and performing dimension reduction compression on the multidimensional academic text semantic feature vector, and outputting the final academic text semantic feature. The method and the device improve the quality of vector generation, accelerate the speed of vector generation, and are suitable for text vector generation in an academic big data scene.

Description

Method, system and storage medium for academic text semantic feature extraction based on pre-training model

technical field

The invention relates to the technical field of big data, in particular to a method, system and storage medium for extracting semantic features of academic texts based on a pre-training model.

Background technique

Compared with traditional Internet data, academic resources show more complex features. In terms of extracting text representation features of academic resources, word frequency-based statistical models, topic models, or deep learning-based word vector representation methods are usually used. TF-IDF (word frequency - inverse document frequency) is a typical text vector representation method, which uses statistical methods to extract text features, calculates the word frequency by calculating the word frequency, calculates the document word weight with the inverse document frequency, and uses the weight set of all words in the document. Document vector representation. Wu Zhe et al. also proposed the TTF-LDA (Time+TF-IDF+Latent Dirichlet Allocation) algorithm, which is based on TF-IDF and LDA, and uses the method of topic analysis to process the abstracts of academic literature. Mikolov et al. also introduced the Word2Vec model, which uses the continuous bag of words model and the Skip-Gram method to obtain the hidden layer vector representation of a word through the task of predicting the next word.

With the rapid development of artificial intelligence technology, deep learning technology has also been used for feature extraction of academic texts. Autoencoders can effectively learn the semantic representation of text data, so Eisa et al. proposed to use deep autoencoder technology to extract lexical feature sets. The network structure of Recurrent Neural Network (RNN) can process input of different time series, which is very suitable for extracting features from serialized text data, and is widely used in text processing tasks. On this basis, in order to solve the problem of RNN gradient disappearance caused by long distance, people improved RNN and constructed LSTM (Long Short-Term Memory Network) and GRU (Gated Recurrent Unit) models. By integrating memory, forgetting, and output stages Retaining long-distance semantic information enables the model to achieve better results in feature extraction from long texts. Devlin et al. also proposed the Bert (BidirectionalEncoder Representation from Transformers, transformer-based bidirectional encoding representation) pre-training model, which maximizes the use of multi-head self-attention mechanism to capture context semantics, and achieves good results in multiple natural language tasks. Good results.

The Bert pre-training model is an encoder that uses a bidirectional transformer. Compared with the traditional model, the Bert model uses a masked language model (Masked LM) to capture the word-level semantic representation in the text during the pre-training process, and predict based on the next sentence ( Next Sentence Prediction) captures the semantic relationships between sentences and obtains sentence-level semantic representations in the text. In order to ensure the quality of pre-training, the Bert pre-training model will probabilistically select some adjacent and non-adjacent sentences as input to ensure that the model can understand the relationship between different sentences. The bidirectional transformer of the Bert pre-training model introduces a self-attention mechanism, which learns the relationship within the sentence, the relationship within the target sentence, and the relationship between the source sentence and the target sentence. The structure of attention.

The pre-trained Bert model can be fine-tuned under specific NLP (Natural Language Processing) tasks, so that a single model can be adapted to multiple different NLP tasks, saving computing resources. Although the Bert pre-training model can be fine-tuned under NLP tasks, the text vector provided by the current Bert model does not perform well in the scenario of academic resource text feature representation, and the generated vector representation will appear "collapse". Sentences tend to be encoded into a small spatial region, which makes most sentence pairs have high similarity scores, even those that are completely semantically irrelevant. In addition, the vector generation speed of the current Bert model is too slow, which greatly affects the speed of academic text feature extraction.

Therefore, how to accelerate the speed of vector generation while improving the quality of vector generation for text vector generation in academic big data scenarios is an urgent problem to be solved.

SUMMARY OF THE INVENTION

Aiming at the problems existing in the prior art, the present invention proposes a method and system for extracting semantic features of academic texts based on a pre-training model, so as to improve the quality of vector generation and the speed of vector generation.

One aspect of the present invention provides a method for extracting semantic features of academic text based on a pre-training model, the method comprising the following steps:

Obtain academic resource text data;

Input the obtained academic resource text data into the pre-training model to obtain multi-dimensional academic text semantic feature vectors; the pre-training model fine-tunes the Bert pre-training model based on multiple negative sample loss functions, and pre-trains the fine-tuned Bert The model is a student pre-training model obtained by training the student model through knowledge distillation as a teacher model;

The multi-dimensional academic text semantic feature vector is subjected to dimensionality reduction and compression, and the final academic text semantic feature vector is output.

In some embodiments of the present invention, the academic resource data of the webpage is crawled by the crawler technology, and the text data of the academic resource is obtained; in the process of crawling the academic resource text data of the webpage by the scrapy crawler, for the webpage to be crawled with an anti-crawling mechanism, the Extract the document ID of the original URL of the crawled webpage, use the extracted document ID to construct a new URL, and guide the crawler to the detail page without anti-crawling mechanism, thereby obtaining the complete document information of the webpage to be crawled.

In some embodiments of the present invention, the fine-tuning of the Bert pre-training model based on multiple negative sample loss functions, and using the fine-tuned Bert pre-training model as a teacher model to train the student model through knowledge distillation includes: using natural language inference data Set or semantic text similarity benchmark dataset to fine-tune the Bert pre-training model based on multiple negative example loss functions; use the Wikidata set to use the fine-tuned Bert pre-training model as a teacher model to train the student model through knowledge distillation; the Bert The input to the pretrained model is sentence pairs containing relational labels in the natural language inference dataset.

In some embodiments of the present invention, the multiple negative sample loss function satisfies the following formula:

；

;

表示句向量

和

之间的点积，

表示在计算句向量之间的点积时使用的预训练模型，K表示向Bert预训练模型输入的句子对的数量。Among them, u and v represent sentence vector sequences [ u ₁ , …, u _i , …, u _K ] and [ v ₁ , …, v _i , …, v _K ] obtained based on the Bert pre-training model, respectively,

Represents a sentence vector

and

The dot product between ,

represents the pretrained model used when computing the dot product between sentence vectors, and K represents the number of sentence pairs input to the Bert pretrained model.

In some embodiments of the present invention, the loss function used in the training process of the student model is the MSE loss function, and the MSE loss function is expressed as:

；

;

表示教师模型生成的句向量，

表示学生模型生成的句向量；n表示句向量数量。in,

represents the sentence vector generated by the teacher model,

Represents the sentence vector generated by the student model; n represents the number of sentence vectors.

In some embodiments of the present invention, the method further includes: a student pre-training model training step, which includes: using multiple negative sample loss functions to pre-train the Bert model based on a natural language inference dataset or a semantic text similarity benchmark dataset Perform fine-tuning to obtain the fine-tuned Bert pre-training model; take the fine-tuned Bert pre-training model as the teacher model, and train the student model through knowledge distillation to obtain the student pre-training model.

In some embodiments of the present invention, performing dimension reduction and compression on the multi-dimensional academic text semantic feature vector includes: using a principal component analysis dimension reduction algorithm to perform dimension reduction and compression on the multi-dimensional academic text feature vector output by the pre-training model.

In some embodiments of the present invention, the academic resource text data includes structured academic resource text data and/or unstructured academic resource text data.

In some embodiments of the present invention, when the fine-tuned Bert pre-training model is used as a teacher model to train a student model through knowledge distillation, the teacher model includes 12 hidden layers, and the teacher model has the number of [1, 4, 7, 10] The hidden layer is reserved as the hidden layer of the student model.

Another aspect of the present invention further provides a system for extracting semantic features of academic text based on a pre-training model. The system includes a processor and a memory, where computer instructions are stored in the memory, and the processor is configured to execute the memory. Computer instructions stored in the system, when executed by the processor, implement the steps of the method as previously described.

Another aspect of the present invention also provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, implements the steps of the aforementioned method.

The academic text semantic feature extraction method and system based on the pre-training model of the present invention can use any speech-driven speaker head movement synthesis method as input, and the naturalness of the result can be improved.

Additional advantages, objects, and features of the present invention will be set forth in part in the description that follows, and in part will become apparent to those of ordinary skill in the art upon study of the following, or may be learned from practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the description and drawings.

Those skilled in the art will appreciate that the objects and advantages that can be achieved with the present invention are not limited to those specifically described above, and that the above and other objects that can be achieved by the present invention will be more clearly understood from the following detailed description.

Description of drawings

The accompanying drawings described herein are used to provide a further understanding of the present invention, and constitute a part of the present application, and do not constitute a limitation to the present invention.

FIG. 1 is a schematic diagram of fine-tuning a Bert pre-training model according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of model compression of a fine-tuned Bert model through knowledge distillation in an embodiment of the present invention.

FIG. 3 is a schematic flowchart of a method for extracting semantic features of academic text based on a pre-training model according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of academic resource collection, preprocessing and storage in an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

It should be noted that, in the drawings or descriptions in the specification, the same drawing numbers are used for similar or identical parts. In the accompanying drawings, they are marked for simplification or convenience. Furthermore, implementations not shown or described in the accompanying drawings are in the form known to those of ordinary skill in the art. Additionally, although examples of parameters including specific values may be provided herein, it should be understood that the parameters need not be exactly equal to the corresponding value, but rather approximate the corresponding value within acceptable error tolerances or design constraints.

Here, it should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only the structures and/or processing steps closely related to the solution according to the present invention are shown in the drawings, and the related structures and/or processing steps are omitted. Other details not relevant to the invention.

It should be emphasized that the term "comprising/comprising" when used herein refers to the presence of a feature, element, step or component, but does not exclude the presence or addition of one or more other features, elements, steps or components.

Aiming at the problem that the text vector provided by the existing Bert pre-training model (Bert model for short) does not perform well in the scenario of academic resource text feature representation, the present invention proposes a new pre-training model based on the Bert pre-training model, and Based on this new pre-training model, a semantic feature extraction method for academic texts is proposed. The new pre-training model proposed by the present invention is a deep learning model generated by combining model fine-tuning (Finetune), knowledge distillation (Knowledge Distilling) and principal component analysis (Principal Component Analysis, PCA) on the basis of the existing Bert model. , based on the new pre-training model, it can use academic resource text to accurately and quickly extract text semantic features, so as to generate text vectors more accurately and quickly. In the embodiment of the present invention, the proposed new pre-training model is called Bert-FDP pre-training model, where FDP is the abbreviation of Finetune, Distilling and PCA, which means fine-tuning, knowledge distillation and PCA dimensionality reduction optimization.

The Bert-FDP pre-training model proposed in the embodiment of the present invention will be described below.

Model fine-tuning refers to re-training the trained model in a new task scenario to adjust the model parameters, so that the model can achieve better results on the original basis. In an embodiment of the present invention, the Bert model is fine-tuned on a natural language inference (NLI) data set to learn potential semantic implications in the natural language inference data set. Among them, the natural language inference data set as the training data may adopt the training samples in the currently existing natural language inference data set, and these training samples are a large number of sentence pairs. Fig. 1 shows a schematic diagram of fine-tuning the Bert pre-training model in an embodiment of the present invention. As shown in Fig. 1 , the Bert pre-training model is trained on a natural language inference data set using a double-tower structure, and the training data consists of multiple The two sets of inputs of the double-tower structure are sentence sequences respectively, that is, the first set of inputs is a sentence sequence composed of sentence 1 and its subsequent sentences, and the second set of inputs is a sentence composed of sentence 2 and its subsequent sentences. sequence. Among them, sentence 1 and sentence 2 belong to similar sentence pairs, sentence 1 and other sentences in the second group of inputs are not similar, and sentence 2 and other sentences in the first group of inputs are not similar. The input multiple sets of sentence pairs pass through the Bert basic network structure stacked by the encoder of the multi-layer converter. The encoder of each layer of the Bert basic network structure consists of a multi-head attention layer and a forward feedback layer ( Feed-forword), since the Bert basic network structure of the Bert pre-training model, which is the fine-tuned object of the present invention, is an existing structure, it will not be repeated here. The Bert pre-training model is used to capture the bidirectional relationship between sentences. The sentence vectors output by the Bert basic network structure undergo dimension reduction processing through the pooling layer, and output sentence vector sequences [ u ₁ , u ₂ , …, u _n ] and [ v ₁ , v ₂ , …, v _n ], and then perform a similarity calculation on the sentences in the two sentence vector sequences, such as a dot product calculation.

组成，其中

是一对相似的句子，

在i!=j时不相似，该多重负样例损失函数在最小化

之间距离的同时最大化i!=j时的所有

句子对之间的距离。对于自然语言推理数据集，本发明将具有蕴含标签的句子对作为正例输入到本发明的预训练模型中，对于一组句子对构成的批（batch），具体的损失函数

形式如下：In the embodiment of the present invention, the training data of the multiple negative example loss function consists of sentence pairs

composed of

is a pair of similar sentences,

Dissimilar when i!=j, the multiple negative loss function is minimizing

while maximizing the distance between i!=j for all

The distance between sentence pairs. For the natural language inference data set, the present invention inputs sentence pairs with implied labels as positive examples into the pre-training model of the present invention. For a batch composed of a set of sentence pairs, the specific loss function

The form is as follows:

；

;

表示句向量

和

之间的点积，

表示在计算句向量点积时使用的预训练模型，K表示向Bert预训练模型输入的句子对的数量。where u and v represent sentence vector sequences [ u ₁ , …, u _i , …, u _K ] and [ v ₁ , …, v _i , …, v _K ], respectively,

Represents a sentence vector

and

The dot product between ,

Represents the pretrained model used in computing the sentence vector dot product, and K represents the number of sentence pairs input to the Bert pretrained model.

，然后使用多重负样例损失函数对Bert模型进行微调。相比现有模型中使用

和

构造组合向量

，并将该向量作为全连接层构造的Softmax分类器的输入，本发明实施例使用多重负样例损失函数来微调Bert模型，可以获得更好的句子表示效果。In the embodiment of the present invention, the vector generated by the Bert model is used to construct the vector set dot product

, and then fine-tune the Bert model using a multiple negative loss function. compared to existing models used

and

Construct a combined vector

, and the vector is used as the input of the Softmax classifier constructed by the fully connected layer. The embodiment of the present invention uses multiple negative sample loss functions to fine-tune the Bert model, so that better sentence representation effects can be obtained.

While reducing the distance between similar sentence vectors, the multiple negative example loss function can make the sentence vectors between negative examples farther apart, so that the Bert-FDP model can generate more accurate and reasonable sentence vectors.

For the pre-training model fine-tuned by the loss function of multiple negative samples, in order to ensure the speed of sentence embedding generation in the big data scenario, the present invention further uses the means of knowledge distillation to compress the model, generate a student network model, and use the student network model as the pre-training model. Train the model to generate sentence embedding vectors of academic texts (such as scientific research results titles), so as to optimize the speed of vector generation while ensuring the quality of sentence vectors. Knowledge distillation refers to using the output results provided by the teacher network model (referred to as the teacher model) to train the student network model (referred to as the student model) with fewer layers and faster. In the embodiment of the present invention, the teacher network model is a Bert pre-training model fine-tuned by multiple negative example loss functions. FIG. 2 is a schematic diagram of model compression of a fine-tuned Bert model through knowledge distillation in an embodiment of the present invention. In an embodiment of the present invention, the wiki data set is used to train the student model. For example, the teacher model may include 12 hidden layers. In the selection of the student model, the [1, 4, 7, 10] hidden layers of the teacher model are reserved. As the hidden layer of the student model, and on this basis, the external sentence corpus is used to generate sentence vectors, and the MSE (Mean Square Error, mean square error) loss function is used to compare the generated results of the vectors. Here, keeping the [1, 4, 7, 10] hidden layers of the teacher model as the hidden layers of the student model is only an example of the present invention, and the present invention is not limited to this, and more, less or other layer as the hidden layer of the student model.

In an embodiment of the present invention, the MSE loss function is expressed as:

；

;

表示教师模型生成的句向量，

表示学生模型生成的句向量；n表示句向量数量。在本发明实施例中，在微调的Bert预训练模型的基础上通过知识蒸馏生成学生模型后，利用学生模型作为预训练模型进行学术文本语义特征的提取，可以在提高向量生成质量的同时加快向量生成的速度。in,

represents the sentence vector generated by the teacher model,

Represents the sentence vector generated by the student model; n represents the number of sentence vectors. In the embodiment of the present invention, after the student model is generated by knowledge distillation on the basis of the fine-tuned Bert pre-training model, the student model is used as the pre-training model to extract the semantic features of academic text, which can improve the quality of vector generation and speed up the vector Generation speed.

After using the trained student model as the pre-training model to extract the semantic features of the academic text, for the vector output by the student model, in this embodiment of the present invention, a dimensionality reduction algorithm can also be used to further reduce the size of the vector, thereby further reducing the similarity Time consumption for comparison. A dimension reduction algorithm can be selected from the existing dimension reduction algorithms to perform dimension reduction and compression on the feature vector output by the student model. Existing dimension reduction algorithms may include Principal Component Analysis (PCA) dimension reduction algorithms, Independent Component Analysis (ICA) dimension reduction algorithms, Linear Discriminant Analysis (LDA) dimension reduction algorithms, etc., but the present invention is not limited thereto.

Taking the PCA dimensionality reduction algorithm as an example, the PCA dimensionality reduction algorithm mainly reduces the dimension of high-dimensional data through data preprocessing (data cleaning) through linear transformation (decomposing feature matrix), so as to realize the transformation of high-dimensional data in low-dimensional space. projection. Assuming that the data output by the student model is n pieces of d-dimensional data, the data can be arranged to form a matrix X with n rows and d columns. After zero-meaning each column of the matrix X (that is, subtracting the mean of this column), Further find the covariance matrix, obtain the eigenvalues of the covariance matrix and the corresponding eigenvectors, then arrange the eigenvectors into a matrix from top to bottom according to the size of the eigenvalues, and take the first k rows to form a matrix P; PX can get the data reduced to k-dimension. Since the PCA dimensionality reduction algorithm in the embodiment of the present invention may adopt the existing PCA dimensionality reduction algorithm, details are not described herein again.

The embodiment of the present invention can ensure the quality of the sentence embedding generated by the pre-training model by effectively combining the fine-tuning of the pre-training model, performing knowledge distillation on the fine-tuned model, and compressing the vector dimension by using principal component analysis (PCA) on the vector generated by the student model. It can also speed up the speed of sentence embedding generation and the speed of vector similarity calculation.

The present invention verifies the actual effect of Bert-FDP generating sentence vectors on multiple text matching data sets in Chinese and English. Linguistic reasoning) corpus constructed a dataset containing 980,000 sentence pairs as training data to pre-train the fine-tuned Bert model; in the process of knowledge distillation, the Wikidata set containing 7.87 million sentences was used to train the student model, and finally The sentence vectors constructed using PCA dimensionality reduction are compared on a public dataset of semantic similarity matching.

The experimental results show that the Bert-FDP model trained by the present invention has improved correlation coefficient values in each data set compared with the commonly used text feature extraction models.

In the above, using the natural language inference data set as training data to fine-tune the Bert pre-training model based on multiple negative sample loss functions is only an example. In another embodiment of the present invention, a semantic text similarity benchmark (Semantic Textual Similarity Benchmark, STS-B) data set is used as training data to fine-tune the Bert pre-training model based on multiple negative sample loss functions. It is found in experiments that comparing the Bert pre-training model on the STS-B data set can make the feature vector The extraction accuracy is increased by 13%, indicating that the Bert-FDP model can generate better text vectors, effectively alleviating the problem of flattening academic text vectors and low discrimination.

The following describes an implementation method for extracting semantic features of academic texts based on the pre-training model generated above in the embodiments of the present invention. FIG. 3 is a schematic flowchart of a method for extracting semantic features of academic text based on a pre-training model according to an embodiment of the present invention. As shown in Figure 3, the method includes the following steps:

Step S110, acquiring academic resource text data.

As an example, in terms of obtaining academic resource text data, a crawler technology (such as the Scrapy crawler technology) can be used to crawl and organize online network academic data (that is, web page academic resource data). Regular expressions, Xpath or CSS (Cascading Style Sheets) selectors can be used to formulate corresponding crawling rules for different types of web page academic resource data, so as to obtain papers, scholars and patents in the Internet and various knowledge bases , projects and other academic information. For example, as shown in Figure 4, for academic resource data of web pages such as paper index pages and patent disclosure pages, URL parsing and splicing crawling rules can be used to crawl paper details and patent details. For the subject area tree page Types of web resource data can be crawled by recursive crawling of the subject area tree content. In the acquisition and processing of academic resource data, the invention uses scrapy crawler to crawl the academic resource data of the web pages related to scientific research achievements such as papers and patents. Since the original web page often has an anti-crawling mechanism, the data of the original article details page is Item is incomplete. In order to solve this problem, the present invention extracts the document ID of the original URL of the webpage to be crawled during the crawling process to construct a new URL, and guides the crawler to the detail page without anti-crawling mechanism, thereby obtaining the webpage to be crawled. Complete document information, so that you can obtain complete scientific research results data information.

Based on the crawled document content, academic resource text data can be obtained. The academic resource text data can include structured text data and unstructured text data. Here, structured text data refers to a logical representation of a two-dimensional table. Unstructured text data refers to text data that is inconvenient to express logically in two-dimensional tables. For example, for academic resource text data, unstructured text data may include paper abstracts, patent abstracts and other texts; structured text data obtained through report parsing may include scholars, research institutions, subject areas, patents, papers Tabular text data under properties such as , meetings, etc. The scrapy crawler technology can initially realize the screening and crawling of academic resources.

During the crawling process of academic resource text data, the present invention directly saves the acquired structured text data and its attributes into the MySql database, and the structured text data may also include structured entity data; In order to facilitate the retrieval of the text by the knowledge service component, the present invention saves the text data in the Elasticsearch database based on the inverted index. Further, the present invention can also use an information extraction algorithm based on deep semantic representation to extract the academic subject heading entities, subject areas and the association relationship between academic entities in academic texts, and then combine the structured entity data saved in the MySql database and the relationship between entities. relationship, which can construct entity-entity relationship triples, which are further saved in the Neo4j database.

Through step S110 of the present invention, the collection and storage of academic resource text data is designed, including a text data acquisition part and a text data storage part, structured and unstructured academic resource text data can be obtained and stored in the corresponding database, It realizes the structuring of multi-source and multi-field academic achievements. In this step S110, the text information of academic resources such as papers, scholars, patents, projects, etc. in the Internet and various knowledge bases can be obtained.

Step S120, inputting the obtained academic resource text data into a pre-training model to obtain a multi-dimensional academic text semantic feature vector.

In this step, the pre-training model is the student pre-training model obtained through the steps described above. That is, the student pre-training model is a student pre-training model obtained by fine-tuning the Bert pre-training model based on multiple negative example loss functions, and using the fine-tuned Bert pre-training model as a teacher model to train the student model through knowledge distillation.

The academic resource text data obtained in step S110 may be input into the student pre-training model, and the student pre-training model outputs a multi-dimensional academic text semantic feature vector.

In this embodiment of the present invention, the academic resource text data input to the student pre-training model may be acquired structured text data or unstructured text data. For example, when the input data is unstructured text data, the input unstructured text data can be text data such as paper abstract text, patent abstract text, etc., or other academic types of text data; in the input data In the case of unstructured text data, the input structured text data may be text data under a specific attribute, or may be entity content text data.

In step S130, the multi-dimensional academic text semantic feature vector output by the student's pre-training model is subjected to dimension reduction and compression, and the final academic text semantic feature is output.

In this embodiment of the present invention, a dimension reduction algorithm may be selected from existing dimension reduction algorithms to perform dimension reduction and compression on the feature vector. Existing dimension reduction algorithms may include Principal Component Analysis (PCA) dimension reduction algorithms, Independent Component Analysis (ICA) dimension reduction algorithms, Linear Discriminant Analysis (LDA) dimension reduction algorithms, etc., but the present invention is not limited thereto.

As an example, the multidimensional vector data output by the student's pre-training model is dimensionally reduced through the Principal Component Analysis (PCA) dimensionality reduction algorithm to obtain the final academic text semantic feature vector.

By using the Bert pre-training model, the present invention uses a large-scale corpus to pre-train the converter model before executing specific downstream task coding, and then based on the pre-trained parameters under specific downstream tasks based on multiple negative sample loss functions The Bert pre-training model is fine-tuned, and the fine-tuned Bert pre-training model is used as a teacher model to train the student model through knowledge distillation, so that the pre-trained student model is used as the pre-training model used for the semantic feature extraction of academic texts in the present invention. , can get better results, can improve the quality of vector generation while speeding up the speed of vector generation, and is suitable for text vector generation in academic big data scenarios.

In the embodiment of the present invention, since the natural language inference data set is used to train the Bert pre-training model, the proposed pre-training model fine-tuned on the basis of the Bert pre-training model can contain more semantic information, while multiple negative The sample loss function reduces the distance between similar sentence vectors, and at the same time makes the sentence vectors between negative samples farther apart, so that the pre-training model of the present invention can generate more accurate and reasonable sentence vectors. At the same time, PCA dimensionality reduction corrects the distribution of sentence vectors to a certain extent, so that the calculated similarity is more reasonable. Due to the reduced number of model layers, the model also generates vectors faster than the original model.

The embodiments of the present invention effectively combine pre-training model fine-tuning, knowledge distillation and PCA dimensionality reduction. While improving the quality of vector generation, it speeds up the speed of vector generation, and is suitable for text vector generation in academic big data scenarios. Since the Bert pretrained model is fine-tuned on the natural language inference dataset based on the multiple negative example loss function, this significantly improves the semantic vector distribution in academic big data scenarios. The Bert-FDP pre-training model proposed by the present invention not only realizes the accurate vector representation of academic resource texts, but also ensures the speed of sentence embedding generation in big data scenarios.

The experimental results show that the Bert-FDP model trained by the present invention has improved correlation coefficient values in each data set compared with the commonly used text feature extraction models. The Bert-FDP model can generate better text vectors and effectively alleviate the problem. It solves the problem that academic text vectors are flattened and the degree of discrimination is not high.

Corresponding to the above method, the present invention also provides an academic text semantic feature extraction system based on a pre-training model, the system includes a computer device, the computer device includes a processor and a memory, and the memory stores computer instructions, The processor is configured to execute computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps of the aforementioned method.

Embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the foregoing method for deploying an edge computing server. The computer-readable storage medium may be a tangible storage medium such as random access memory (RAM), memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

It should be understood by those of ordinary skill in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is implemented in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, elements of the invention are programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted over a transmission medium or communication link by a data signal carried in a carrier wave.

It is to be understood that the present invention is not limited to the specific arrangements and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above-described embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the sequence of steps after comprehending the spirit of the present invention.

In the present invention, features described and/or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, and/or in combination with or in place of features of other embodiments Features of the implementation.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, various modifications and changes may be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A semantic feature extraction method for academic texts based on a pre-training model is characterized by comprising the following steps:

acquiring academic resource text data;

inputting the acquired academic resource text data into a pre-training model to obtain a multi-dimensional academic text semantic feature vector; the pre-training model is a student pre-training model obtained by finely tuning a Bert pre-training model based on a multiple load sample loss function, and training the student model by knowledge distillation by taking the finely tuned Bert pre-training model as a teacher model;

and performing dimensionality reduction compression on the multidimensional academic text semantic feature vector, and outputting a final academic text semantic feature vector.

2. The method of claim 1, wherein obtaining academic resource text data comprises: crawling webpage academic resource data through a crawler technology to obtain academic resource text data;

in the process of crawling webpage academic resource text data by a script crawler, aiming at a webpage to be crawled with a reverse crawling mechanism, extracting a document ID of an original URL of the webpage to be crawled, constructing a new URL by using the extracted document ID, and guiding the crawler to a detailed page without the reverse crawling mechanism, so that complete document information of the webpage to be crawled is obtained.

3. The method of claim 1, wherein the fine-tuning the Bert pre-trained model based on the multiple negative sample loss function, and training the student model by knowledge distillation using the fine-tuned Bert pre-trained model as a teacher model comprises: fine-tuning the Bert pre-training model by utilizing a natural language reasoning data set or a semantic text similarity benchmark data set based on a multiple heavy sample loss function, and training a student model by utilizing a wiki data set to take the fine-tuned Bert pre-training model as a teacher model through knowledge distillation;

and the input of the Bert pre-training model is a sentence pair containing a relation label in the natural language reasoning data set.

4. The method of claim 3, wherein the multiple heavy load sample loss function satisfies the following equation:

；

wherein,uandvrespectively show the sentence vector sequences obtained based on the Bert pre-training modelu ₁ , …, u _i , …, u _K ]And 2v ₁ , …, v _i , …, v _K ]，

Representing a sentence vector

And

the dot product between the two (C) and (D),

representing a pre-trained model used in computing dot products between sentence vectors,Krepresenting the number of sentence pairs input to the Bert pre-trained model.

5. The method of claim 1, wherein the loss function used in the student model training process is an MSE loss function, and the MSE loss function is expressed as:

；

wherein,

a sentence vector generated by the teacher model is represented,

sentence vectors generated by the student model are represented; n represents the number of sentence vectors.

6. The method of any one of claims 1-5, wherein performing dimension reduction compression on the multidimensional academic text semantic feature vector comprises: and performing dimensionality reduction compression on the multidimensional academic text feature vector output by the pre-training model by using a principal component analysis dimensionality reduction algorithm.

7. The method of any of claims 1-5, wherein the academic resource text data comprises structural chemistry source text data and/or non-structural chemistry source text data.

8. The method of claim 1, wherein when training the student models by knowledge distillation using the trimmed Bert pre-training model as the teacher model, the teacher model comprises 12 hidden layers, and the [1,4,7,10] hidden layer of the teacher model is reserved as the hidden layer of the student model.

9. An academic text semantic feature extraction system based on a pre-trained model, the system comprising a processor and a memory, wherein the memory stores computer instructions, the processor is used for executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system realizes the steps of the method according to any one of claims 1 to 8.

10. A computer storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the steps of the method according to any one of claims 1 to 8.

Images