CN111581476A - Intelligent webpage information extraction method based on BERT and LSTM - Google Patents

Abstract

The invention belongs to the field of Internet information extraction and mining, and discloses an intelligent webpage information extraction method based on BERT and LSTM, which comprises the following steps: s1, crawling a webpage; s2, preprocessing a webpage; s3, webpage target labeling; s4, training a neural network model; s5, deploying a neural network model; s6, after the training is completed, the target information is recognized for the input web page using the obtained model. According to the scheme, the information can be accurately positioned after the page content is updated and the webpage structure is changed, so that an accurate extraction result is obtained, information can be intelligently extracted from the webpage, the webpage information extraction cost is reduced, the webpage information extraction speed is increased, through detection, the identification accuracy of the model to the page in the training set is over 98%, and the recall rate is over 97%.

Description

An Intelligent Webpage Information Extraction Method Based on BERT and LSTM

technical field

The invention relates to the technical field of Internet information extraction and mining, and more particularly, to an intelligent webpage information extraction method based on BERT and LSTM.

Background technique

With the rapid development of science and technology, the Web has become the largest encyclopedia database in the world. Although users can easily obtain information on the Web through applications such as browsers, it is still relatively difficult to retrieve information on the Internet. Web page information is extracted, and the information is structured and stored to form a knowledge map, which can make search more accurate and convenient.

The information extraction based on HTML structure is the information extraction method used by web crawlers in the usual sense. First, on the premise of locating the location of the information in HTML, the combination of regular expressions and HTML selectors is used to write the format that conforms to the current HTML page. The extraction rules are used to extract data from HTML documents, which is also the most widely used extraction method at present. However, this method has poor scalability, requires a lot of manual work, and is very costly.

The concept of deep learning (Deep Learning) originated from the research of artificial neural network. It is a branch of the field of machine learning (Machine Learning) and is a collection of algorithms for modeling high-level abstract data. Compared with Shallow Learning, it can form more abstract high-level features by combining low-level features at multiple levels, so as to realize automatic learning of features without human participation in feature selection.

The rapid development of deep learning in recent years is due to the "Layer-wise Pre-training" (Layer-wise Pre-training) algorithm based on the Deep Belife Network (DBN) proposed by Hinton et al. It solves the problem that the learning effect of deep structure calculation is not ideal, and brings hope to solve the optimization problems related to deep structure.

Bert model (Bidirectional Encoder Representations from Transformers) BERT is a Transformer-based bidirectional encoding representation. The bidirectional meaning means that when processing a word, it can take into account the information of the words before and after the word, so as to obtain the semantics of the context.

Long-short-term memory model (LSTM, Long-short Term Memory) is a specific RNN structure model, its purpose is to solve the long-term dependency problem (Long Term Dependencies) in neural networks, that is to use past sequence information to speculate current sequence information capability. The basic node in LSTM is called a "cell". The state parameter in the cell is responsible for memory information. The input and output interact with the cell through Input Gate and Output Gate respectively. At the same time, Forget Gate is added to the model to discard the expired memory information and forget it. So as to achieve a certain degree of memory effect.

The prior art has the following problems: accurate information extraction requires a lot of manual workload, and the information extracted manually is located by analyzing the page structure. After the structure changes, the problem of extraction failure or inaccurate extraction results caused by the failure of positioning information is prone to occur.

SUMMARY OF THE INVENTION

1. Technical problems to be solved

In view of the problems existing in the prior art, the purpose of the present invention is to provide an intelligent web page information extraction method based on BERT and LSTM, which can accurately locate the information after the page content is updated and the web page structure changes, and obtain accurate extraction results. .

2. Technical solutions

In order to solve the above problems, the present invention adopts the following technical solutions.

An intelligent web page information extraction method based on BERT and LSTM, including the following steps:

S1. Crawling web pages: Crawling the vertical domain web pages that need to be extracted, at least 3000 company page information as a training set, and 500 page information as a test set;

S2, web page preprocessing: preprocess the web page and clean the useless HTML tags;

S3. Web page target labeling: Use manual rules to automatically label web page targets. Manually first observe the HTML tags that need to extract information, and then use xpath to mark the HTML tags with relevant classification tags;

S4. Train the neural network model: first establish the vocabulary table and establish the mapping. After the establishment is completed, the source code of the webpage is divided into words one by one, and then fine-tuned on the basis of the pre-trained BERT model to obtain the word vector;

S5. Deploy the neural network model: perform automatic information extraction on web pages and evaluate the accuracy and recall rate of the model;

S6. After the training is completed, use the obtained model to identify the target information of the input web page, use 500 pages in the test set to test the recognition effect, ignore the tag information during the recognition process, and judge according to the content and context of the text node Type of target information.

Preferably, the company page information includes a unified social credit code, type, business scope, address, and the like.

Preferably, the training neural network model adopts the Nesterov Momentum algorithm as the optimization algorithm, and 100 epoll trainings are performed on the training set, and the loss obtained from the training is less than 0.09.

3. Beneficial effects

Compared with the prior art, the advantages of the present invention are:

(1) This solution is applied to the fields of Internet information extraction and mining, open knowledge graph construction, etc. After the page content is updated and the page structure changes, the information can still be accurately located, and accurate extraction results can be obtained, thereby realizing intelligent extraction of information from web pages. , reduce the cost of web page information extraction and improve the speed of web page information extraction.

(2) After testing, the recognition accuracy rate of the model for the pages in the training set is over 98%, and the recall rate is over 97%.

Description of drawings

FIG. 1 is a schematic diagram of the neural network architecture in the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention; obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. The embodiments of the present invention, and all other embodiments obtained by those of ordinary skill in the art without creative work, fall within the protection scope of the present invention.

In the description of the present invention, it should be noted that the orientations or positional relationships indicated by the terms "upper", "lower", "inner", "outer", "top/bottom", etc. are based on the orientations shown in the drawings or The positional relationship is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed to indicate or imply relative importance.

In the description of the present invention, it should be noted that, unless otherwise expressly specified and limited, the terms "installation", "provided with", "sleeve/connection", "connection", etc., should be understood in a broad sense, such as " Connection", which can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be an internal connection between two components. of connectivity. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in specific situations.

Please refer to Figure 1, an intelligent web page information extraction method based on BERT and LSTM, including the following steps:

S1. Crawling web pages: Crawling the vertical domain web pages that need to be extracted, at least 3000 company page information as a training set, 500 page information as a test set, information extraction aims to achieve through the training of neural networks. The automatic extraction of target information of similar websites saves the trouble of writing extraction rules for each type of website separately;

S2. Web page preprocessing: Preprocess the web page and clean the useless HTML tags. There is a lot of useless information in the HTML code of the enterprise information web page, which is too complicated, and most of them are not helpful for the identification of target information. The web page is preprocessed to simplify the content of the page information. The preprocessed page information structure is very clear, and it is easier to observe the location structure of the target information. For example, the type information follows the "type" node, and the unified social credit code information Immediately after the "Unified Social Credit Code" node, etc., since the location structure is particularly important for the system to determine the type of target information, preprocessing is an important part of the process of adding ordinary web pages to the training set;

S3. Web page target labeling: Use manual rules to automatically label web page targets. Humans first observe the HTML tags that need to extract information, and then use xpath to mark relevant classification tags for the HTML tags. The purpose is to tell the system in the web page structure, which common The information needs to be obtained, so that such information can be identified during training, and the correctly marked pages can be added to the training set as valid samples to ensure the correctness of the training results;

S5. Train the neural network model: first establish a thesaurus table and establish a mapping. In order to identify the key words in the text nodes, identify the features of the text nodes, and generate word vectors for neural network input, it is necessary to identify the words encountered in the web page information. build a thesaurus;

Based on the structural characteristics of the information in the web page, the text in each text node is more like a whole, so a mapping to the thesaurus table is established for each individual Chinese character instead of a phrase, and a separate word is established for each parsed foreign word. The mapping of foreign words (usually English words) in web pages is different from Chinese characters and needs to be composed of letters to have corresponding meanings, so a separate mapping is established for each word;

After the establishment is completed, the source code of the webpage is divided into words one by one, and then fine-tuned on the basis of the pre-trained BERT model to obtain the word vector;

Based on the characteristics of information in Web pages, a multi-level neural network is used to build a model. The Bert and softmax layers are used for word vector representation and result classification of page information, respectively. The middle two layers use LSTM to train and learn the text information between pages. Features, in view of the powerful capabilities of LSTM in natural language processing, combining Web information extraction technology with LSTM can also gain two additional benefits - one saves the step of word segmentation in text processing and the step of building feature engineering in information extraction, Because the feature of Web page text information, that is, the information text, is usually embedded in HTML-formatted tab pages in small units, and most of the text in the tab page already contains independent semantics, so the model is directly based on the tab page. The text node is used as a unit, and the information in the entire web page is not divided into words. At the same time, one of the advantages of the neural network is that it can establish a grasp of the fuzzy relationship between entities, and through this relationship, the description relationship between the page information can be established.

S5. Deploy the neural network model: Automatically extract information from web pages and evaluate the accuracy and recall rate of the model.

S6. After the training is completed, use the obtained model to identify the target information of the input web page, use 500 pages in the test set to test the recognition effect, ignore the tag information during the recognition process, and judge according to the content and context of the text node Type of target information.

Further, the company page information includes the unified social credit code, type, business scope, address, etc.

Further, the Nesterov Momentum algorithm is used as the optimization algorithm to train the neural network model, and 100 epoll training is performed on the training set, and the loss obtained from the training is less than 0.09. The Nesterov Momentum algorithm is used as the optimization algorithm of the neural network, and the momentum optimization factor used is 0.9 to update all Weight, in each iterative training of the network, the softmax cross entropy is used as the loss function of the classification to train the neural network at all levels. During the back propagation of the error value, the Nesterov Momentum algorithm is used to calculate the gradient value, and the calculated gradient value is used. Update the parameters of the network, use the L2 regularization term for all parameters of the network, the weight decay factor is 0.0005, and perform 100poll training on the training dataset.

After testing, the recognition accuracy rate of the model for the pages in the training set is over 98%, and the recall rate is over 97%.

The above description is only a preferred embodiment of the present invention; however, the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical scope of the present invention, according to the technical solution of the present invention and its improvement concept, equivalently replaces or changes, should be covered within the protection scope of the present invention.

Claims (3)

1. An intelligent webpage information extraction method based on BERT and LSTM is characterized in that: the method comprises the following steps:

s1, crawling the webpage: crawling a vertical field webpage to be extracted, crawling at least 3000 company page information as a training set, and crawling 500 page information as a testing set;

s2, webpage preprocessing: preprocessing a webpage, and cleaning useless HTML tags;

s3, webpage target labeling: automatically labeling a webpage target by using a manual rule, manually observing an HTML (hypertext markup language) label needing to extract information, and then marking a relevant classification mark on the HTML label by using xpath;

s5, training a neural network model: firstly establishing a word bank table and mapping, after the establishment is finished, segmenting a webpage source code into words, and then finely tuning on the basis of a pre-trained BERT model to obtain word vectors;

s5, deploying a neural network model: carrying out automatic information extraction on the webpage and evaluating the accuracy and the recall rate of the model;

and S6, after training is completed, identifying the target information of the input webpage by using the obtained model, testing the identification effect by using 500 pages in the test set, neglecting the marking information in the identification process, and judging the type of the target information according to the content and the context of the text node.

2. The intelligent web page information extraction method based on BERT and LSTM as claimed in claim 1, wherein: the corporate page information includes a unified social credit code, type, business range, residence, etc.

3. The intelligent web page information extraction method based on BERT and LSTM as claimed in claim 1, wherein: the training neural network model adopts a Nesterov Momentum algorithm as an optimization algorithm, 100 epoll training is carried out on a training set, and the loss obtained by training is less than 0.09.

Images