WO2019218514A1 - 网页目标信息的提取方法、装置及存储介质 - Google Patents

网页目标信息的提取方法、装置及存储介质 Download PDF

Info

Publication number
WO2019218514A1
WO2019218514A1 PCT/CN2018/102115 CN2018102115W WO2019218514A1 WO 2019218514 A1 WO2019218514 A1 WO 2019218514A1 CN 2018102115 W CN2018102115 W CN 2018102115W WO 2019218514 A1 WO2019218514 A1 WO 2019218514A1
Authority
WO
WIPO (PCT)
Prior art keywords
webpage
target
category
topic
classification
Prior art date
Application number
PCT/CN2018/102115
Other languages
English (en)
French (fr)
Inventor
吴壮伟
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2019218514A1 publication Critical patent/WO2019218514A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/284—Lexical analysis, e.g. tokenisation or collocates
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the present application relates to the field of data processing technologies, and in particular, to a method for extracting webpage target information, an electronic device, and a computer readable storage medium.
  • the present application provides a method for extracting webpage target information, a server, and a computer readable storage medium, the main purpose of which is to improve the accuracy of extracting target information from a target webpage.
  • the present application provides a method for extracting webpage target information, including:
  • a word segmentation step receiving a request for extracting target information from a target webpage, obtaining a webpage source code of the target webpage, and performing word segmentation on the obtained webpage source code to obtain a set of available words of the target webpage;
  • a topic classification step calculating a word vector of the target webpage according to the available word set of the target webpage, inputting the calculated word vector into a predetermined classification model corresponding to each topic category, and identifying a topic category to which the target webpage belongs ;
  • a location prediction step determining a first tag corresponding to the target information, inputting a webpage source code of the target webpage into a location prediction model corresponding to the first tag in the identified topic category, and predicting that the target information appears differently a list of location information for the location;
  • the information extraction step screening a preset number of locations with the highest probability from the location information list, and extracting information from the filtered location as the target information.
  • the present application further provides an electronic device, including: a memory, a processor, and an extracting program for storing webpage target information executable on the processor, where the webpage target is stored
  • an extracting program for storing webpage target information executable on the processor, where the webpage target is stored
  • a word segmentation step receiving a request for extracting target information from a target webpage, obtaining a webpage source code of the target webpage, and performing word segmentation on the obtained webpage source code to obtain a set of available words of the target webpage;
  • a topic classification step calculating a word vector of the target webpage according to the available word set of the target webpage, inputting the calculated word vector into a predetermined classification model corresponding to each topic category, and identifying a topic category to which the target webpage belongs ;
  • a location prediction step determining a first tag corresponding to the target information, inputting a webpage source code of the target webpage into a location prediction model corresponding to the first tag in the identified topic category, and predicting that the target information appears differently a list of location information for the location;
  • the information extraction step screening a preset number of locations with the highest probability from the location information list, and extracting information from the filtered location as the target information.
  • the present application further provides a computer readable storage medium, where the computer readable storage medium includes an extraction program of webpage target information, and when the extraction program of the webpage target information is executed by a processor, Any step in the method of extracting the web page target information as described above is implemented.
  • the method for extracting webpage target information, the electronic device and the computer readable storage medium proposed by the present application improve the classification of the target webpage by using different classification models for different topic categories to construct different classification models.
  • the accuracy of the target page topic classification by constructing different location prediction models for different information categories of different topic categories, using the location prediction models corresponding to different information categories under different topic categories to predict the location information of the location where the target information is located in the target webpage
  • the list improves the accuracy of the location of the predicted target information; selects the location in the location information list with the probability ranking higher and the probability greater than the probability threshold, and extracts the information from the location as the target information, thereby improving the accuracy of the target information extraction.
  • FIG. 1 is a flow chart of a preferred embodiment of a method for extracting webpage target information according to the present application
  • FIG. 2 is a schematic diagram of a preferred embodiment of an electronic device of the present application.
  • FIG. 3 is a schematic diagram of a program module of the extraction procedure of the webpage target information in FIG.
  • the application provides a method for extracting webpage target information.
  • FIG. 1 it is a flowchart of a preferred embodiment of a method for extracting target information of a webpage of the present application.
  • the method can be performed by a device that can be implemented by software and/or hardware.
  • the method for extracting webpage target information includes steps S1-S4:
  • S1 Receive a request for extracting target information from a target webpage, obtain a webpage source code of the target webpage, and perform word segmentation processing on the obtained webpage source code to obtain a set of available words of the target webpage;
  • the information extraction request carries the target webpage information and the target information to be extracted, and the label corresponding to the target information is determined according to the target information to be extracted.
  • the crawler tool to crawl the source code of the target webpage and perform word segmentation on the webpage source of the target webpage.
  • the original data of the webpage source of the target webpage is extracted, and the irrelevant data in the original data is removed by using a regular expression, for example, Javascript script code, CSS style code, and HTML tag data.
  • the retained data is segmented by the word segmentation tool, and a set of initial words separated by spaces is generated.
  • the initial word set is deactivated to determine the available word set, and the available word set is used. Characterize the content of the landing page.
  • the word frequency-inverse document frequency index (TF-IDF) algorithm is used to calculate the importance degree of each word in the available word set of the target webpage, and each word in the available word set of the target webpage is performed according to the order of importance from high to low. Sort.
  • the top N vocabulary in the available word set of the target web page is selected as the keyword of the target web page, where N>0 and N is an integer.
  • a Chinese word vector model (Word2vec model) is generated based on the Chinese Wikipedia corpus, and the word vectors of the N keywords in the available word set of the target web page are respectively calculated by the Word2vec model, and the N keys obtained by the above steps are used.
  • the word vector of the word calculates the word vector for the landing page.
  • the word vector of the target webpage is sequentially input into the classification model corresponding to the different subject categories that are pre-trained, for example, the classification model corresponding to the tourism category, the classification model corresponding to the economic category, and the classification corresponding to the sports category.
  • the model output result of the classification model corresponding to different topic categories indicates the probability that the topic category to which the target web page belongs is each topic category. Therefore, from the output results of the classification models corresponding to the different topic categories, the topic category corresponding to the maximum probability is selected as the topic category to which the target web page belongs.
  • a preset threshold for example, 0.5
  • the maximum probability of the output of each classification model is selected and compared with a preset threshold, when the probability is maximum.
  • the threshold is greater than or equal to the preset threshold
  • the subject category corresponding to the maximum probability is used as the subject category to which the target webpage belongs.
  • the probability maximum value is less than the preset threshold, the user receives the classification instruction of the topic category to which the target webpage belongs, and determines the topic category to which the target webpage belongs according to the topic category included in the classification instruction.
  • the training steps of the predetermined classification model include:
  • Obtaining the source code of the specified webpage respectively segmenting the source code of each specified webpage, obtaining a set of available words for each specified webpage, extracting keywords from the set of available words, and generating a word vector of each specified webpage;
  • the sample data in the set is divided into a training set and a verification set, and the neural network model is trained by using the training set, and the neural network model is verified by using the verification set, and when the verification result satisfies the first preset condition, determining the Classification models corresponding to different topic types.
  • the different second tags represent different subject categories to which the web page belongs, such as travel, economy, sports, politics, and entertainment.
  • the word vectors of the web pages of different subject categories are respectively taken as positive samples corresponding to the subject categories.
  • a negative sample needs to be constructed before the model is trained.
  • the word vector of the second label is a positive type of the web page
  • the second label is a negative sample of the word vector of the webpage of the other category
  • Different subject categories correspond to different classification models, which improves the accuracy of web page topic classification, and lays a good foundation for predicting the location of target information and extracting target information from the target web page.
  • the first tag represents the category of the target information to be extracted.
  • the first tab of the webpage includes: number of days, time, per capita fee, companion, and so on.
  • different first tags of the same subject category correspond to different location prediction models. Therefore, after determining the topic category to which the target webpage belongs according to the above steps, the model file of the location prediction model corresponding to the first label in the topic category is invoked, and the webpage source code of the target webpage is input into the location prediction model, and the model output result is
  • the target information may appear in a list of location information at different locations in the web page source code of the target web page, and the probability that the target information appears in different locations.
  • the training steps of the position prediction model include:
  • Different first tags are respectively marked in the source code of each specified webpage, and the source code of each webpage in each set is respectively divided into sub-collections corresponding to the first tags, as samples corresponding to different first tags in each topic category. Data;
  • the sample data in the subset is divided into a training set and a verification set, and the training set is used to train the cyclic neural network model, and the verification set is used to verify the cyclic neural network model.
  • the verification result satisfies the second preset condition, A position prediction model corresponding to different first labels under each subject category is determined.
  • web pages of the same subject category have a similar web page structure: a label (ie, a first label) and attribute data.
  • a label ie, a first label
  • the first tab of a travel page includes: number of days, time, per capita fee, companion, and subject and body information
  • the first tab of a political web page includes: subject, body, time, media, and related information
  • the first labels include: economic policy, foreign policy, stock information, real estate policy or national policy
  • the first tabs of sports webpages include: star data, team competitions, match time and game scores, etc.
  • Tags include: stars, events, time, etc.
  • the webpage source code of the webpage source code of the specified webpage of the same topic category is marked with the same first label as the first label in the topic category.
  • the sample data of the position prediction model It should be noted that, since the webpage source code of a webpage contains different first tags, the webpage source code of the same webpage may appear in the sample data corresponding to different first tags at the same time. In addition, the sample data includes both positive and negative samples, which will not be described here.
  • 80% of the data of the first tag in the subject category is extracted as a training set, and 20% of the data is used as a verification set.
  • the training set is used to train the cyclic neural network model to construct a position prediction model, and The trained position prediction model is tuned, and the calibrated position prediction model is verified by the verification set until the second preset condition is met (for example, the accuracy is greater than or equal to 95%).
  • the above steps are repeated to determine a position prediction model corresponding to each of the first labels in each subject category.
  • Different topic categories and different first tags correspond to different location prediction models, which improves the accuracy of location prediction and lays a good foundation for subsequent extraction of target information from target web pages.
  • Obtaining the foregoing location information list reading the probability that the target information appears in different locations from the location information list, sorting the different locations according to the probability, and selecting the preset number of presets (for example, three) as the target information.
  • the location and extract the information of the preset number of locations as the target information.
  • a location probability threshold may be preset, and the probability that the target information appears at different positions is read from the location information list, and the preset number of the top is sorted ( For example, three positions with a probability greater than or equal to the position probability threshold are taken as the location where the target information is located, and the information of the position is extracted as the target information.
  • the method for extracting webpage target information by constructing different classification models for webpages of different topic categories, classifying the target webpages by using the classification models corresponding to different topic categories, and improving the accuracy of the target webpage classification classification;
  • Different location prediction models are constructed for different information categories of different subject categories, and position prediction models corresponding to different information categories under different subject categories are used to predict the location information list of the location where the target information is located in the target webpage, thereby improving the location of the predicted target information.
  • Accuracy selecting the position in the position information list with the probability ranking first and the probability greater than the probability threshold, extracting information from the position as the target information, and improving the accuracy of the target information extraction.
  • step S2 may be replaced by:
  • the subject category with the highest similarity is used as the The subject category to which the landing page belongs;
  • the classification instruction for the topic category to which the target webpage belongs is received, and the topic category included in the classification instruction is used as the topic category to which the target webpage belongs.
  • the word vector of the predetermined subject categories is obtained by the following steps:
  • the source code of the webpage of the specified webpage under each topic category is obtained separately, and the source code of the webpage is separately processed into words, and the available word collection of each webpage is obtained.
  • the TF-IDF algorithm the importance degree of each vocabulary in the available word set of each webpage is calculated, and the top N vocabulary with the highest degree of importance is selected as the keyword of the webpage for each webpage.
  • the word vector of the selected N keywords is calculated by the Word2vec model, and the word vector of the web page is calculated by the word vector of the keyword.
  • the word vector of all web pages is calculated in this way.
  • the keywords of all the webpages in each topic category are summarized, and the word frequency of each keyword of each webpage in each topic category is separately counted, and the word frequency reflects the weight of the keyword.
  • Select the M keywords with the highest word frequency as the keywords of each topic category calculate the word vectors of each keyword summarized in the topic category by Word2vec model, and calculate the word vector of the topic category according to the word vector of the keyword and the word frequency.
  • the word vector of the subject category is used as the cluster center corresponding to each topic category.
  • the similarity between the word vector of the target webpage and the word vector of each topic category is calculated by the formula of the cosine similarity calculation, and the similarity of the word vector with the target webpage is selected.
  • the largest word vector for the subject category It can be understood that the higher the similarity, the higher the accuracy of the target page topic classification.
  • a similarity threshold is preset, when the similarity maximum is greater than or equal to the similarity threshold.
  • the subject category corresponding to the similarity maximum value is used as the subject category to which the target webpage belongs; when the similarity maximum value is less than the similarity threshold, the classification instruction for the subject category to which the target webpage belongs is received, according to the theme included in the classification instruction
  • the category is the subject category to which the landing page belongs.
  • the method for extracting webpage target information proposed by the foregoing embodiment uses a clustering method to predetermine a cluster center (word vector) corresponding to each topic category, and calculates a cluster corresponding to each of the predetermined topic categories by calculating a word vector of the target webpage.
  • the similarity of the center selects the topic category corresponding to the maximum similarity of the preset condition as the topic category to which the target webpage belongs, so that the webpage topic classification is more accurate.
  • the application also provides an electronic device.
  • FIG. 2 it is a schematic diagram of a preferred embodiment of the electronic device 1 of the present application.
  • the electronic device 1 may be a terminal device with a data processing function, such as a server, a smart phone, a tablet computer, a portable computer, a desktop computer, etc.
  • the server may be a rack server, a blade server, or a tower. Server or rack server.
  • the electronic device 1 includes a memory 11, a processor 12, a communication bus 13, and a network interface 14.
  • the memory 11 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, and the like.
  • the memory 11 may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1, in some embodiments.
  • the memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in hard disk equipped on the electronic device 1, a smart memory card (SMC), and a secure digital (Secure Digital) , SD) cards, flash cards, etc. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device.
  • the memory 11 can be used not only for storing application software and various types of data installed in the electronic device 1, such as the extraction program 10 of the web page target information, but also for temporarily storing data that has been output or is to be output.
  • the processor 12 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor or other data processing chip for running program code or processing stored in the memory 11.
  • Data such as an extraction program 10 of web page target information, and the like.
  • Communication bus 13 is used to implement connection communication between these components.
  • the network interface 14 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the electronic device 1 and other electronic devices.
  • a standard wired interface such as a WI-FI interface
  • FIG. 2 shows only the electronic device 1 having the components 11-14. It will be understood by those skilled in the art that the structure shown in FIG. 2 does not constitute a limitation on the electronic device 1, and may include fewer or more than the illustration. Multiple components, or a combination of certain components, or different component arrangements.
  • the electronic device 1 may further include a user interface
  • the user interface may include a display, an input unit such as a keyboard, and the optional user interface may further include a standard wired interface and a wireless interface.
  • the display may be an LED display, a liquid crystal display, a touch liquid crystal display, and an Organic Light-Emitting Diode (OLED) touch device.
  • the display may also be referred to as a display screen or display unit for displaying information processed in the electronic device 1 and a user interface for displaying visualizations.
  • the program code of the extraction program 10 storing the webpage target information in the memory 11 as a computer storage medium, when the processor 12 executes the program code of the extraction program 10 of the webpage target information , to achieve the following steps:
  • the word segmentation step receiving a request for extracting target information from the target webpage, obtaining a webpage source code of the target webpage, and performing word segmentation on the obtained webpage source code to obtain a set of available words of the target webpage.
  • the information extraction request carries the target webpage information and the target information to be extracted, and the label corresponding to the target information is determined according to the target information to be extracted.
  • the crawler tool to crawl the source code of the target webpage and perform word segmentation on the webpage source of the target webpage.
  • the original data of the webpage source of the target webpage is extracted, and the irrelevant data in the original data is removed by using a regular expression, for example, Javascript script code, CSS style code, and HTML tag data.
  • the retained data is segmented by the word segmentation tool, and a set of initial words separated by spaces is generated.
  • the initial word set is deactivated to determine the available word set, and the available word set is used. Characterize the content of the landing page.
  • a topic classification step calculating a word vector of the target webpage according to the available word set of the target webpage, inputting the calculated word vector into a predetermined classification model corresponding to each topic category, and identifying a topic category to which the target webpage belongs .
  • the word frequency-inverse document frequency index (TF-IDF) algorithm is used to calculate the importance degree of each word in the available word set of the target webpage, and each word in the available word set of the target webpage is performed according to the order of importance from high to low. Sort.
  • the top N vocabulary in the available word set of the target web page is selected as the keyword of the target web page, where N>0 and N is an integer.
  • a Chinese word vector model (Word2vec model) is generated based on the Chinese Wikipedia corpus, and the word vectors of the N keywords in the available word set of the target web page are respectively calculated by the Word2vec model, and the N keys obtained by the above steps are used.
  • the word vector of the word calculates the word vector for the landing page.
  • the word vector of the target webpage is sequentially input into the classification model corresponding to the different subject categories that are pre-trained, for example, the classification model corresponding to the tourism category, the classification model corresponding to the economic category, and the classification corresponding to the sports category.
  • model output result of the classification model corresponding to different topic categories indicates the probability that the topic category to which the target web page belongs is each topic category.
  • the model output result of the classification model corresponding to different topic categories indicates the probability that the topic category to which the target web page belongs is each topic category. Therefore, from the output results of the classification models corresponding to the different topic categories, the topic category corresponding to the maximum probability is selected as the topic category to which the target web page belongs.
  • a preset threshold for example, 0.5
  • the maximum probability of the output of each classification model is selected and compared with a preset threshold, when the probability is maximum.
  • the threshold is greater than or equal to the preset threshold
  • the subject category corresponding to the maximum probability is used as the subject category to which the target webpage belongs.
  • the probability maximum value is less than the preset threshold, the user receives the classification instruction of the topic category to which the target webpage belongs, and determines the topic category to which the target webpage belongs according to the topic category included in the classification instruction.
  • the training steps of the predetermined classification model include:
  • a second label is marked for the predetermined webpage according to the topic category to which the webpage belongs.
  • the different second tags represent different subject categories to which the web page belongs, such as travel, economy, sports, politics, and entertainment.
  • the web pages of different subject categories and the corresponding word vectors are respectively taken as positive samples corresponding to different subject categories. In order to ensure the accuracy of the classification model, a negative sample needs to be constructed before the model is trained.
  • the word vector of the second label is a positive type of the web page
  • the second label is a negative sample of the word vector of the webpage of the other category
  • the sample set corresponding to the different subject categories [X , Y] where X is a word vector corresponding to a certain topic category webpage, and Y is a topic category corresponding to the word vector.
  • Different subject categories correspond to different classification models, which improves the accuracy of web page topic classification, and lays a good foundation for predicting the location of target information and extracting target information from the target web page.
  • a location prediction step determining a first tag corresponding to the target information, inputting a webpage source code of the target webpage into a location prediction model corresponding to the first tag in the identified topic category, and predicting that the target information appears differently A list of location information for the location.
  • the first tag represents the category of the target information to be extracted.
  • the first tab of the webpage includes: number of days, time, per capita fee, companion, and so on.
  • different first tags of the same subject category correspond to different location prediction models. Therefore, after determining the topic category to which the target webpage belongs according to the above steps, the model file of the location prediction model corresponding to the first label in the topic category is invoked, and the webpage source code of the target webpage is input into the location prediction model, and the model output result is
  • the target information may appear in a list of location information at different locations in the web page source code of the target web page, and the probability that the target information appears in different locations.
  • the training steps of the position prediction model include:
  • Different first tags are respectively marked in the source code of each specified webpage, and the source code of each webpage in each set is respectively divided into sub-collections corresponding to the first tags, as samples corresponding to different first tags in each topic category. Data;
  • the sample data in the subset is divided into a training set and a verification set, and the training set is used to train the cyclic neural network model, and the verification set is used to verify the cyclic neural network model.
  • the verification result satisfies the second preset condition, A position prediction model corresponding to different first labels under each subject category is determined.
  • web pages of the same subject category have a similar web page structure: a label (ie, a first label) and attribute data.
  • a label ie, a first label
  • the first tab of a travel page includes: number of days, time, per capita fee, companion, and subject and body information
  • the first tab of a political web page includes: subject, body, time, media, and related information
  • the first labels include: economic policy, foreign policy, stock information, real estate policy or national policy
  • the first tabs of sports webpages include: star data, team competitions, match time and game scores, etc.
  • Tags include: stars, events, time, etc.
  • the webpage source code of the webpage source code of the specified webpage of the same topic category is marked with the same first label as the first label in the topic category.
  • the sample data of the position prediction model It should be noted that, since the webpage source code of a webpage contains different first tags, the webpage source code of the same webpage may appear in the sample data corresponding to different first tags at the same time. In addition, the sample data includes both positive and negative samples, which will not be described here.
  • 80% of the data of the first tag in the subject category is extracted as a training set, and 20% of the data is used as a verification set.
  • the training set is used to train the cyclic neural network model to construct a position prediction model, and The trained position prediction model is tuned, and the calibrated position prediction model is verified by the verification set until the second preset condition is met (for example, the accuracy is greater than or equal to 95%).
  • the above steps are repeated to determine a position prediction model corresponding to each of the first labels in each subject category.
  • Different topic categories and different first tags correspond to different location prediction models, which improves the accuracy of location prediction and lays a good foundation for subsequent extraction of target information from target web pages.
  • the information extraction step screening a preset number of locations with the highest probability from the location information list, and extracting information from the filtered location as the target information.
  • Obtaining the foregoing location information list reading the probability that the target information appears in different locations from the location information list, sorting the different locations according to the probability, and selecting the preset number of presets (for example, three) as the target information.
  • the location and extract the information of the preset number of locations as the target information.
  • a location probability threshold may be preset, and the probability that the target information appears at different positions is read from the location information list, and the preset number of the top is sorted ( For example, three positions with a probability greater than or equal to the position probability threshold are taken as the location where the target information is located, and the information of the position is extracted as the target information.
  • the electronic device 1 proposed in the above embodiment, by constructing different classification models for web pages of different subject categories, classifying the target webpages by using the classification models corresponding to different topic categories, and improving the accuracy of the target webpage topic classification; Different information categories of different categories are used to construct different position prediction models, and position prediction models corresponding to different information categories under different subject categories are used to predict the position information list of the location where the target information is located in the target webpage, thereby improving the accuracy of the location of the predicted target information. Selecting the position in the position information list with the probability ranking first and the probability greater than the probability threshold, and extracting information from the position as the target information improves the accuracy of the target information extraction.
  • the extraction program 10 of the webpage target information may also be divided into one or more modules, one or more modules being stored in the memory 11 and being processed by one or more processors ( This embodiment is executed by the processor 12) to accomplish the present application.
  • a module referred to herein refers to a series of computer program instructions that are capable of performing a particular function.
  • FIG. 3 it is a block diagram of the extraction program 10 of the webpage target information in FIG. 2.
  • the webpage target information extraction program 10 can be divided into a word segmentation module 110, a topic classification module 120, and a position prediction.
  • the module 130 and the information extraction module 140, the functions or operation steps implemented by the modules 110-140 are similar to the above, and are not described in detail herein, for example, where:
  • the word segmentation module 110 is configured to receive a request for extracting target information from the target webpage, obtain a webpage source code of the target webpage, and perform word segmentation processing on the obtained webpage source code to obtain a set of available words of the target webpage;
  • the topic classification module 120 is configured to calculate a word vector of the target webpage according to the available word set of the target webpage, input the calculated word vector into a predetermined classification model corresponding to each topic category, and identify that the target webpage belongs to Subject category;
  • the location prediction module 130 is configured to determine a first label corresponding to the target information, input the webpage source code of the target webpage into a location prediction model corresponding to the first label in the identified topic category, and predict the target information. a list of location information that appears in different locations;
  • the information extraction module 140 is configured to filter a preset number of locations with the highest probability from the location information list, and extract information from the filtered location as the target information.
  • the embodiment of the present application further provides a computer readable storage medium, where the computer readable storage medium includes an extracting program 10 of webpage target information, and the extracting program 10 of the webpage target information is executed by a processor to implement the following operations. :
  • a word segmentation step receiving a request for extracting target information from a target webpage, obtaining a webpage source code of the target webpage, and performing word segmentation on the obtained webpage source code to obtain a set of available words of the target webpage;
  • a topic classification step calculating a word vector of the target webpage according to the available word set of the target webpage, inputting the calculated word vector into a predetermined classification model corresponding to each topic category, and identifying a topic category to which the target webpage belongs ;
  • a location prediction step determining a first tag corresponding to the target information, inputting a webpage source code of the target webpage into a location prediction model corresponding to the first tag in the identified topic category, and predicting that the target information appears differently a list of location information for the location;
  • the information extraction step screening a preset number of locations with the highest probability from the location information list, and extracting information from the filtered location as the target information.
  • the specific implementation manner of the computer readable storage medium of the present application is substantially the same as the specific implementation manner of the method for extracting the webpage target information, and details are not described herein again.
  • the technical solution of the present application which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM as described above). , a disk, an optical disk, including a number of instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in the various embodiments of the present application.
  • a terminal device which may be a mobile phone, a computer, a server, or a network device, etc.

Abstract

一种网页目标信息的提取方法、电子装置及计算机存储介质，该方法包括：接收从目标网页中提取目标信息的请求，获取所述目标网页的网页源码，对网页源码进行分词处理得到所述目标网页的可用词集合；将根据可用词集合计算的词向量输入分类模型，以确定所述目标网页所属的主题类别；将所述目标网页的网页源码输入预先确定的位置预测模型，预测所述目标信息出现在不同位置的位置信息列表；从所述位置信息列表中筛选出预设数量的所述目标信息出现概率最高的位置，并从筛选出的位置提取信息作为目标信息。利用所述方法、电子装置及计算机存储介质，可以提高从目标网页提取目标信息的准确性。

Description

网页目标信息的提取方法、装置及存储介质
本申请基于巴黎公约申明享有2018年5月14日递交的申请号为CN2018104558405、名称为“网页目标信息的提取方法、装置及存储介质”的中国专利申请的优先权，该中国专利申请的整体内容以参考的方式结合在本申请中。
技术领域
本申请涉及数据处理技术领域，尤其涉及一种网页目标信息的提取方法、电子装置及计算机可读存储介质。
背景技术
随着互联网技术和Web技术的高速发展，互联网上网页的数量正在不断的增加。网络信息的增加大大方便了人们获取信息，但是过大的信息量也给人们处理信息带来了很多的困难。在这一背景下，传统靠人工的信息处理方式已经无法适应大量数据处理的要求。如何在海量的信息中将用户感兴趣的信息类型提取出来逐渐成为大家所关注的研究点。中文网页种类繁多，如何对网页进行自动分类，并准确获取网页中的目标信息，是组织和管理网络资源的关键。
发明内容
鉴于以上内容，本申请提供一种网页目标信息的提取方法、服务器及计算机可读存储介质，其主要目的在于提高从目标网页提取目标信息的准确性。
为实现上述目的，本申请提供一种网页目标信息的提取方法，该方法包括：
分词步骤：接收从目标网页中提取目标信息的请求，获取所述目标网页的网页源码，对获取到的网页源码进行分词处理得到所述目标网页的可用词集合；
主题分类步骤：根据所述目标网页的可用词集合计算所述目标网页的词向量，将计算得到的词向量输入预先确定的各主题类别对应的分类模型，识 别出所述目标网页所属的主题类别；
位置预测步骤：确定所述目标信息对应的第一标签，将所述目标网页的网页源码输入识别出的主题类别中所述第一标签对应的位置预测模型中，预测所述目标信息出现在不同位置的位置信息列表；及
信息提取步骤：从所述位置信息列表中筛选出预设数量的概率最高的位置，并从筛选出的位置提取信息作为目标信息。
此外，本申请还提供一种电子装置，其特征在于，该装置包括：存储器、处理器，所述存储器上存储有可在所述处理器上运行的网页目标信息的提取程序，所述网页目标信息的提取程序被所述处理器执行时，可实现如下步骤：
分词步骤：接收从目标网页中提取目标信息的请求，获取所述目标网页的网页源码，对获取到的网页源码进行分词处理得到所述目标网页的可用词集合；
主题分类步骤：根据所述目标网页的可用词集合计算所述目标网页的词向量，将计算得到的词向量输入预先确定的各主题类别对应的分类模型，识别出所述目标网页所属的主题类别；
位置预测步骤：确定所述目标信息对应的第一标签，将所述目标网页的网页源码输入识别出的主题类别中所述第一标签对应的位置预测模型中，预测所述目标信息出现在不同位置的位置信息列表；及
信息提取步骤：从所述位置信息列表中筛选出预设数量的概率最高的位置，并从筛选出的位置提取信息作为目标信息。
此外，为实现上述目的，本申请还提供一种计算机可读存储介质，所述计算机可读存储介质中包括网页目标信息的提取程序，所述网页目标信息的提取程序被处理器执行时，可实现如上所述网页目标信息的提取方法中的任意步骤。
本申请提出的网页目标信息的提取方法、电子装置及计算机可读存储介质，通过为不同的主题类别的网页构建不同的分类模型，利用不同主题类别对应的分类模型对目标网页进行分类，提高了目标网页主题分类的准确性；通过为不同主题类别的不同信息类别构建不同的位置预测模型，利用不同主题类别下不同信息类别对应的位置预测模型，预测目标网页中目标信息所在 的位置的位置信息列表，提高了预测目标信息所在位置的准确性；选择位置信息列表中概率排序靠前且概率大于概率阈值的位置，从该位置提取信息作为目标信息，提高了目标信息提取的准确性。
附图说明
图1为本申请网页目标信息的提取方法较佳实施例的流程图；
图2为本申请电子装置较佳实施例的示意图；
图3为图2中网页目标信息的提取程序的程序模块示意图。
本申请目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
具体实施方式
应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
本申请提供一种网页目标信息的提取方法。参照图1所示，为本申请网页目标信息的提取方法较佳实施例的流程图。该方法可以由一个装置执行，该装置可以由软件和/或硬件实现。
在本实施例中，网页目标信息的提取方法包括步骤S1-S4：
S1、接收从目标网页中提取目标信息的请求，获取所述目标网页的网页源码，对获取到的网页源码进行分词处理得到所述目标网页的可用词集合；
信息提取请求中携带目标网页信息及待提取的目标信息，根据待提取的目标信息确定目标信息对应的标签。
利用爬虫工具爬取该目标网页的网页源码，并对目标网页的网页源码进行分词处理。具体地，提取目标网页的网页源码的原始数据，利用正则表达式去除原始数据中的无关数据，例如，Javascript脚本代码、CSS样式代码和HTML标签数据等。对保留的数据通过分词工具进行分词，生成以空格分隔的初始词汇集合，按照预设的停用词词表，对初始词汇集合进行去停用词处理确定可用词集合，将可用词集合用于表征目标网页的内容。
S2、根据所述目标网页的可用词集合计算所述目标网页的词向量，将计算得到的词向量输入预先确定的各主题类别对应的分类模型，识别出所述目 标网页所属的主题类别；
具体地，根据词频-逆文档频率指数(TF-IDF)算法计算目标网页的可用词集合中各个词汇的重要程度，根据重要程度由高到低的顺序对目标网页的可用词集合中各个词汇进行排序。选择目标网页的可用词集合中的排序靠前的N个词汇作为目标网页的关键词，其中，N＞0，且N为整数。另外，基于中文维基百科语料库生成中文语料的词向量模型(Word2vec模型)，通过该Word2vec模型分别计算目标网页的可用词集合中的N个关键词的词向量，并利用上述步骤得到的N个关键词的词向量计算目标网页的词向量。
确定目标网页的词向量后，将目标网页的词向量依次输入预先训练好的不同主题类别对应的分类模型中，例如，旅游类对应的分类模型、经济类对应的分类模型、体育类对应的分类模型、政治类对应的分类模型、娱乐类对应的分类模型等，然后根据模型输出结果确定所述目标网页所属的主题类别。
需要说明的是，不同主题类别对应的分类模型的模型输出结果表示目标网页所属的主题类别为各主题类别的概率。因此，从不同主题类别对应的分类模型的输出结果中，选择概率最大值对应的主题类别，作为目标网页所属的主题类别。
可以理解的是，为了提高目标网页主题分类的准确性，预先设置一个预设阈值(例如，0.5)，选择各分类模型的输出结果中概率最大值与预设阈值进行比对，当概率最大值大于或等于预设阈值时，将概率最大值对应的主题类别，作为目标网页所属的主题类别。相反，当概率最大值小于预设阈值时，接收用户对目标网页所属主题类别的分类指令，根据分类指令中包含的主题类别确定目标网页所属的主题类别。
作为一种实施方式，所述预先确定的分类模型的训练步骤包括：
获取指定网页的网页源码，分别对每个指定网页的网页源码进行分词，得到每个指定网页的可用词集合，从可用词集合中提取关键词，并生成每个指定网页的词向量；
分别为每个指定网页标注第二标签，将所述词向量划分至不同第二标签对应的集合中，作为不同主题类别的样本数据；及
将所述集合中的样本数据划分为训练集及验证集，利用训练集对神经网络模型进行训练，利用验证集对神经网络模型进行验证，当验证结果满足第 一预设条件时，确定所述不同主题类型对应的分类模型。
具体地，不同的第二标签表示网页所属的不同主题类别，例如，旅游类、经济类、体育类、政治类、及娱乐类等。分别将不同主题类别的网页的词向量作为各主题类别对应的正样本。为了保证分类模型的准确性，在模型训练之前，还需构建负样本。以政治类网页为例，将第二标签为政治类的网页的词向量作为正样本，将第二标签为其他类别的网页的词向量作为负样本，最终确定不同主题类别对应的样本集合[X,Y]，其中，X为某一主题类别网页对应的词向量，Y为词向量对应的主题类别。
从每个主题类别的样本集中抽取80％的数据作为训练集[X1，Y1]，剩下20％的数据作为验证集[X2，Y2]，利用训练集[X1，Y1]对深度神经网络模型进行训练，构建分类模型，并对经过训练后的分类模型进行调优，利用验证集[X2，Y2]对调优后的分类模型进行验证，直到满足第一预设条件(例如，准确率大于或等于95％)为止。重复上述步骤，确定每个主题类别对应的分类模型。不同主题类别对应不同的分类模型，提高了网页主题分类的准确性，为后续从目标网页中预测目标信息的位置、提取目标信息打下良好的基础。
S3、确定所述目标信息对应的第一标签，将所述目标网页的网页源码输入识别出的主题类别中所述第一标签对应的位置预测模型中，预测所述目标信息出现在不同位置的位置信息列表；
具体地，第一标签表示待提取的目标信息的类别。以旅游类网页为例，该类网页的第一标签包括：天数、时间、人均费用、同伴等。在本实施例中，同一主题类别不同第一标签对应不同的位置预测模型。因此，根据上述步骤确定目标网页所属的主题类别后，调用该主题类别中该第一标签对应的位置预测模型的模型文件，并将目标网页的网页源码输入该位置预测模型中，模型输出结果为目标信息可能出现在目标网页的网页源码中的不同位置的位置信息列表，及目标信息出现在不同位置的概率。
作为一种实施方式，所述位置预测模型的训练步骤包括：
分别为每个指定网页标注所述第二标签，根据第二标签将所述指定网页的网页源码划分至不同主题类别对应的集合中；
分别在每个指定网页的网页源码中标注不同的第一标签，分别将每个集合中的网页源码划分至各第一标签对应的子集合中，作为各主题类别下不同 第一标签对应的样本数据；及
将所述子集合中的样本数据划分为训练集及验证集，利用训练集对循环神经网络模型进行训练，利用验证集对循环神经网络模型进行验证，当验证结果满足第二预设条件时，确定各主题类别下不同第一标签对应的位置预测模型。
需要说明的是，相同主题类别的网页有着类似的网页结构：标签(即为第一标签)及属性数据。例如，旅游类网页的第一标签包括：天数、时间、人均费用、同伴，以及主题和正文信息等；政治类网页的第一标签包括：主题、正文、时间、媒体以及相关信息；经济类网页的第一标签包括：经济政策、外交政策、股票信息、房产政策或者国家政策；体育类网页的第一标签包括：球星数据，球队比赛，比赛时间和比赛比分等；娱乐类网页的第一标签包括：明星，事件，时间等。因此，分别为上述指定网页的网页源码标注多个第一标签后，将某一主题类别的指定网页的网页源码中标注了同一第一标签的网页源码作为该主题类别中该第一标签对应的的位置预测模型的样本数据。需要说明的是，鉴于一个网页的网页源码中包含不同的第一标签，因此，同一个网页的网页源码可能同时出现在不同第一标签对应的样本数据中。另外，样本数据既包括正样本也包括负样本，这里不再说明。
从该主题类别中该第一标签的样本数据中抽取80％的数据作为训练集，剩下20％的数据作为验证集，利用训练集对循环神经网络模型进行训练，构建位置预测模型，并对经过训练后的位置预测模型进行调优，利用验证集对调优后的位置预测模型进行验证，直到满足第二预设条件(例如，准确率大于或等于95％)为止。重复上述步骤，确定每个主题类别中每个第一标签对应的位置预测模型。不同主题类别、不同的第一标签对应不同的位置预测模型，提高了位置预测的准确性，为后续从目标网页中提取目标信息打下良好的基础。
S4、从所述位置信息列表中筛选出预设数量的概率最高的位置，并从筛选出的位置提取信息作为目标信息。
获取上述位置信息列表，从位置信息列表中读取目标信息出现在不同位置的概率，根据概率对不同的位置进行排序，选择排序靠前的预设数量(例如，3个)的位置作为目标信息所在的位置，并提取该预设数量的位置的信息作为目标信息。
在其他实施例中，为了提高预测目标信息所在位置的准确性，可以预先设置一个位置概率阈值，从位置信息列表中读取目标信息出现在不同位置的概率，将排序靠前的预设数量(例如，3个)、且概率大于或等于位置概率阈值的位置作为目标信息所在的位置，并提取该位置的信息作为目标信息。
上述实施例提出的网页目标信息的提取方法，通过为不同的主题类别的网页构建不同的分类模型，利用不同主题类别对应的分类模型对目标网页进行分类，提高目标网页主题分类的准确性；通过为不同主题类别的不同信息类别构建不同的位置预测模型，利用不同主题类别下不同信息类别对应的位置预测模型，预测目标网页中目标信息所在的位置的位置信息列表，提高了预测目标信息所在位置的准确性；选择位置信息列表中概率排序靠前且概率大于概率阈值的位置，从该位置提取信息，作为目标信息，提高了目标信息提取的准确性。
基于上述实施例，还提出本申请网页目标信息的提取方法的另一较佳实施例。
在本实施例中，所述步骤S1、S3及S4的实施方式与上述实施例中的内容一致，与上述实施例的区别在于，所述步骤S2可以替换为：
分别计算所述目标网页的词向量与预先确定的各主题类别的词向量之间的相似度，当相似度最大值大于或等于预设相似度阈值时，将相似度最高的主题类别作为所述目标网页所属的主题类别；
当相似度最大值小于预设相似度阈值时，接收针对目标网页所属的主题类别的分类指令，根据分类指令中包含的主题类别作为目标网页所属的主题类别。
其中，所述预先确定的各主题类别的词向量通过以下步骤得到：
分别获取各主题类别下指定网页的网页源码，分别对所述网页源码进行分词处理，得到各网页的可用词集合。根据TF-IDF算法计算各网页的可用词集合中各个词汇的重要程度，针对每个网页选择重要程度最高的前N个词汇作为该网页的关键词。针对每个网页，通过Word2vec模型计算选择出的N个关键词的词向量，通过关键词的词向量计算网页的词向量。按照这种方式计算得到所有网页的词向量。
将每个主题类别中的所有网页的关键词汇总，分别统计各主题类别中所有网页的各关键词的词频，词频体现了该关键词的权重。选择M个词频最大的关键词作为各主题类别的关键词，通过Word2vec模型分别计算主题类别中汇总的各个关键词的词向量，根据关键词的词向量和词频计算主题类别的词向量，将各主题类别的词向量作为各主题类别对应的聚类中心。
在确定各主题类别的词向量后，通过余弦相似度的计算公式，分别计算目标网页的词向量与上述各主题类别的词向量之间的相似度，并筛选出与目标网页的词向量相似度最大的主题类别的词向量。可以理解的是，相似度越高，目标网页主题分类准确性也越高，为了提高目标网页主题分类的准确性，预先设置一个相似度阈值，当相似度最大值大于或等于该相似度阈值时，将该相似度最大值对应的主题类别作为目标网页所属的主题类别；当相似度最大值小于该相似度阈值时，接收针对目标网页所属的主题类别的分类指令，根据分类指令中包含的主题类别作为目标网页所属的主题类别。
上述实施例提出的网页目标信息的提取方法，利用聚类方法，预先确定各主题类别对应的聚类中心(词向量)，通过计算目标网页的词向量与预先确定的各主题类别对应的聚类中心的相似度，选择满足预设条件的相似度最大值对应的主题类别作为目标网页所属的主题类别，使网页主题分类更准确。
本申请还提供一种电子装置。参照图2所示，为本申请电子装置1较佳实施例的示意图。
在本实施例中，电子装置1可以是服务器、智能手机、平板电脑、便携计算机、桌上型计算机等具有数据处理功能的终端设备，所述服务器可以是机架式服务器、刀片式服务器、塔式服务器或机柜式服务器。
该电子装置1包括存储器11、处理器12，通信总线13，及网络接口14。
其中，存储器11至少包括一种类型的可读存储介质，所述可读存储介质包括闪存、硬盘、多媒体卡、卡型存储器(例如，SD或DX存储器等)、磁性存储器、磁盘、光盘等。存储器11在一些实施例中可以是所述电子装置1的内部存储单元，例如该电子装置1的硬盘。存储器11在另一些实施例中也可以是所述电子装置1的外部存储设备，例如该电子装置1上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD) 卡，闪存卡(Flash Card)等。进一步地，存储器11还可以既包括该电子装置1的内部存储单元也包括外部存储设备。
存储器11不仅可以用于存储安装于该电子装置1的应用软件及各类数据，例如网页目标信息的提取程序10等，还可以用于暂时地存储已经输出或者将要输出的数据。
处理器12在一些实施例中可以是一中央处理器(Central Processing Unit,CPU)、控制器、微控制器、微处理器或其他数据处理芯片，用于运行存储器11中存储的程序代码或处理数据，例如网页目标信息的提取程序10等。
通信总线13用于实现这些组件之间的连接通信。
网络接口14可选的可以包括标准的有线接口、无线接口(如WI-FI接口)，通常用于在该电子装置1与其他电子设备之间建立通信连接。
图2仅示出了具有组件11-14的电子装置1，本领域技术人员可以理解的是，图2示出的结构并不构成对电子装置1的限定，可以包括比图示更少或者更多的部件，或者组合某些部件，或者不同的部件布置。
可选地，该电子装置1还可以包括用户接口，用户接口可以包括显示器(Display)、输入单元比如键盘(Keyboard)，可选的用户接口还可以包括标准的有线接口、无线接口。
可选地，在一些实施例中，显示器可以是LED显示器、液晶显示器、触控式液晶显示器以及有机发光二极管(Organic Light-Emitting Diode，OLED)触摸器等。其中，显示器也可以称为显示屏或显示单元，用于显示在电子装置1中处理的信息以及用于显示可视化的用户界面。
在图2所示的电子装置1实施例中，作为一种计算机存储介质的存储器11中存储网页目标信息的提取程序10的程序代码，处理器12执行网页目标信息的提取程序10的程序代码时，实现如下步骤：
分词步骤：接收从目标网页中提取目标信息的请求，获取所述目标网页的网页源码，对获取到的网页源码进行分词处理得到所述目标网页的可用词集合。
信息提取请求中携带目标网页信息及待提取的目标信息，根据待提取的目标信息确定目标信息对应的标签。
利用爬虫工具爬取该目标网页的网页源码，并对目标网页的网页源码进行分词处理。具体地，提取目标网页的网页源码的原始数据，利用正则表达式去除原始数据中的无关数据，例如，Javascript脚本代码、CSS样式代码和HTML标签数据等。对保留的数据通过分词工具进行分词，生成以空格分隔的初始词汇集合，按照预设的停用词词表，对初始词汇集合进行去停用词处理确定可用词集合，将可用词集合用于表征目标网页的内容。
主题分类步骤：根据所述目标网页的可用词集合计算所述目标网页的词向量，将计算得到的词向量输入预先确定的各主题类别对应的分类模型，识别出所述目标网页所属的主题类别。
具体地，根据词频-逆文档频率指数(TF-IDF)算法计算目标网页的可用词集合中各个词汇的重要程度，根据重要程度由高到低的顺序对目标网页的可用词集合中各个词汇进行排序。选择目标网页的可用词集合中的排序靠前的N个词汇作为目标网页的关键词，其中，N＞0，且N为整数。另外，基于中文维基百科语料库生成中文语料的词向量模型(Word2vec模型)，通过该Word2vec模型分别计算目标网页的可用词集合中的N个关键词的词向量，并利用上述步骤得到的N个关键词的词向量计算目标网页的词向量。
确定目标网页的词向量后，将目标网页的词向量依次输入预先训练好的不同主题类别对应的分类模型中，例如，旅游类对应的分类模型、经济类对应的分类模型、体育类对应的分类模型、政治类对应的分类模型、娱乐类对应的分类模型等，然后根据模型输出结果确定所述目标网页所属的主题类别。
需要说明的是，不同主题类别对应的分类模型的模型输出结果表示目标网页所属的主题类别为各主题类别的概率。
需要说明的是，不同主题类别对应的分类模型的模型输出结果表示目标网页所属的主题类别为各主题类别的概率。因此，从不同主题类别对应的分类模型的输出结果中，选择概率最大值对应的主题类别，作为目标网页所属的主题类别。
可以理解的是，为了提高目标网页主题分类的准确性，预先设置一个预设阈值(例如，0.5)，选择各分类模型的输出结果中概率最大值与预设阈值进行比对，当概率最大值大于或等于预设阈值时，将概率最大值对应的主题类别，作为目标网页所属的主题类别。相反，当概率最大值小于预设阈值时， 接收用户对目标网页所属主题类别的分类指令，根据分类指令中包含的主题类别确定目标网页所属的主题类别。
作为一种实施方式，所述预先确定的分类模型的训练步骤包括：
获取指定网页的网页源码，利用上述步骤计算预先确定的网页的词向量。然后，根据网页所属的主题类别为预先确定的网页标注第二标签。具体地，不同的第二标签表示网页所属的不同主题类别，例如，旅游类、经济类、体育类、政治类、及娱乐类等。分别将不同主题类别的网页及对应的词向量作为不同主题类别对应的正样本。为了保证分类模型的准确性，在模型训练之前，还需构建负样本。以政治类网页为例，将第二标签为政治类的网页的词向量作为正样本，将第二标签为其他类别的网页的词向量作为负样本，最终确定不同主题类别对应的样本集合[X,Y]，其中，X为某一主题类别网页对应的词向量，Y为词向量对应的主题类别。
从每个主题类别的样本集中抽取80％的数据作为训练集[X1，Y1]，剩下20％的数据作为验证集[X2，Y2]，利用训练集[X1，Y1]对深度神经网络模型进行训练，构建分类模型，并对经过训练后的分类模型进行调优，利用验证集[X2，Y2]对调优后的分类模型进行验证，直到满足第一预设条件(例如，准确率大于或等于95％)为止。重复上述步骤，确定每个主题类别对应的分类模型。不同主题类别对应不同的分类模型，提高了网页主题分类的准确性，为后续从目标网页中预测目标信息的位置、提取目标信息打下良好的基础。
位置预测步骤：确定所述目标信息对应的第一标签，将所述目标网页的网页源码输入识别出的主题类别中所述第一标签对应的位置预测模型中，预测所述目标信息出现在不同位置的位置信息列表。
具体地，第一标签表示待提取的目标信息的类别。以旅游类网页为例，该类网页的第一标签包括：天数、时间、人均费用、同伴等。在本实施例中，同一主题类别不同第一标签对应不同的位置预测模型。因此，根据上述步骤确定目标网页所属的主题类别后，调用该主题类别中该第一标签对应的位置预测模型的模型文件，并将目标网页的网页源码输入该位置预测模型中，模型输出结果为目标信息可能出现在目标网页的网页源码中的不同位置的位置信息列表，及目标信息出现在不同位置的概率。
作为一种实施方式，所述位置预测模型的训练步骤包括：
分别为每个指定网页标注所述第二标签，根据第二标签将所述指定网页的网页源码划分至不同主题类别对应的集合中；
分别在每个指定网页的网页源码中标注不同的第一标签，分别将每个集合中的网页源码划分至各第一标签对应的子集合中，作为各主题类别下不同第一标签对应的样本数据；及
将所述子集合中的样本数据划分为训练集及验证集，利用训练集对循环神经网络模型进行训练，利用验证集对循环神经网络模型进行验证，当验证结果满足第二预设条件时，确定各主题类别下不同第一标签对应的位置预测模型。
需要说明的是，相同主题类别的网页有着类似的网页结构：标签(即为第一标签)及属性数据。例如，旅游类网页的第一标签包括：天数、时间、人均费用、同伴，以及主题和正文信息等；政治类网页的第一标签包括：主题、正文、时间、媒体以及相关信息；经济类网页的第一标签包括：经济政策、外交政策、股票信息、房产政策或者国家政策；体育类网页的第一标签包括：球星数据，球队比赛，比赛时间和比赛比分等；娱乐类网页的第一标签包括：明星，事件，时间等。因此，分别为上述指定网页的网页源码标注多个第一标签后，将某一主题类别的指定网页的网页源码中标注了同一第一标签的网页源码作为该主题类别中该第一标签对应的的位置预测模型的样本数据。需要说明的是，鉴于一个网页的网页源码中包含不同的第一标签，因此，同一个网页的网页源码可能同时出现在不同第一标签对应的样本数据中。另外，样本数据既包括正样本也包括负样本，这里不再说明。
从该主题类别中该第一标签的样本数据中抽取80％的数据作为训练集，剩下20％的数据作为验证集，利用训练集对循环神经网络模型进行训练，构建位置预测模型，并对经过训练后的位置预测模型进行调优，利用验证集对调优后的位置预测模型进行验证，直到满足第二预设条件(例如，准确率大于或等于95％)为止。重复上述步骤，确定每个主题类别中每个第一标签对应的位置预测模型。不同主题类别、不同的第一标签对应不同的位置预测模型，提高了位置预测的准确性，为后续从目标网页中提取目标信息打下良好的基础。
信息提取步骤：从所述位置信息列表中筛选出预设数量的概率最高的位置，并从筛选出的位置提取信息作为目标信息。
获取上述位置信息列表，从位置信息列表中读取目标信息出现在不同位置的概率，根据概率对不同的位置进行排序，选择排序靠前的预设数量(例如，3个)的位置作为目标信息所在的位置，并提取该预设数量的位置的信息作为目标信息。
在其他实施例中，为了提高预测目标信息所在位置的准确性，可以预先设置一个位置概率阈值，从位置信息列表中读取目标信息出现在不同位置的概率，将排序靠前的预设数量(例如，3个)、且概率大于或等于位置概率阈值的位置作为目标信息所在的位置，并提取该位置的信息作为目标信息。
上述实施例提出的电子装置1，通过为不同的主题类别的网页构建不同的分类模型，利用不同主题类别对应的分类模型对目标网页进行分类，提高目标网页主题分类的准确性；通过为不同主题类别的不同信息类别构建不同的位置预测模型，利用不同主题类别下不同信息类别对应的位置预测模型，预测目标网页中目标信息所在的位置的位置信息列表，提高了预测目标信息所在位置的准确性；选择位置信息列表中概率排序靠前且概率大于概率阈值的位置，从该位置提取信息，作为目标信息，提高了目标信息提取的准确性。
可选地，在其他的实施例中，网页目标信息的提取程序10还可以被分割为一个或者多个模块，一个或者多个模块被存储于存储器11中，并由一个或多个处理器(本实施例为处理器12)所执行，以完成本申请，本申请所称的模块是指能够完成特定功能的一系列计算机程序指令段。例如，参照图3所示，为图2中网页目标信息的提取程序10的模块示意图，该实施例中，网页目标信息的提取程序10可以被分割为分词模块110、主题分类模块120、位置预测模块130及信息提取模块140，所述模块110-140所实现的功能或操作步骤均与上文类似，此处不再详述，示例性地，例如其中：
分词模块110，用于接收从目标网页中提取目标信息的请求，获取所述目标网页的网页源码，对获取到的网页源码进行分词处理得到所述目标网页的可用词集合；
主题分类模块120，用于根据所述目标网页的可用词集合计算所述目标网页的词向量，将计算得到的词向量输入预先确定的各主题类别对应的分类模型，识别出所述目标网页所属的主题类别；
位置预测模块130，用于确定所述目标信息对应的第一标签，将所述目标网页的网页源码输入识别出的主题类别中所述第一标签对应的位置预测模型中，预测所述目标信息出现在不同位置的位置信息列表；及
信息提取模块140，用于从所述位置信息列表中筛选出预设数量的概率最高的位置，并从筛选出的位置提取信息作为目标信息。
此外，本申请实施例还提出一种计算机可读存储介质，所述计算机可读存储介质中包括网页目标信息的提取程序10，所述网页目标信息的提取程序10被处理器执行时实现如下操作：
分词步骤：接收从目标网页中提取目标信息的请求，获取所述目标网页的网页源码，对获取到的网页源码进行分词处理得到所述目标网页的可用词集合；
主题分类步骤：根据所述目标网页的可用词集合计算所述目标网页的词向量，将计算得到的词向量输入预先确定的各主题类别对应的分类模型，识别出所述目标网页所属的主题类别；
位置预测步骤：确定所述目标信息对应的第一标签，将所述目标网页的网页源码输入识别出的主题类别中所述第一标签对应的位置预测模型中，预测所述目标信息出现在不同位置的位置信息列表；及
信息提取步骤：从所述位置信息列表中筛选出预设数量的概率最高的位置，并从筛选出的位置提取信息作为目标信息。
本申请之计算机可读存储介质的具体实施方式与上述网页目标信息的提取方法的具体实施方式大致相同，在此不再赘述。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、装置、物品或者方法不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、装置、物品或者方法所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、装置、物品或者方法中还存在另外的相同要素。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在如上所述的一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本申请各个实施例所述的方法。
以上仅为本申请的优选实施例，并非因此限制本申请的专利范围，凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其它相关的技术领域，均同理包括在本申请的专利保护范围内。

Claims (20)

  1. 一种网页目标信息的提取方法，应用于电子装置，其特征在于，所述方法包括：
    分词步骤：接收从目标网页中提取目标信息的请求，获取所述目标网页的网页源码，对获取到的网页源码进行分词处理得到所述目标网页的可用词集合；
    主题分类步骤：根据所述目标网页的可用词集合计算所述目标网页的词向量，将计算得到的词向量输入预先确定的各主题类别对应的分类模型，识别出所述目标网页所属的主题类别；
    位置预测步骤：确定所述目标信息对应的第一标签，将所述目标网页的网页源码输入识别出的主题类别中所述第一标签对应的位置预测模型中，预测所述目标信息出现在不同位置的位置信息列表；及
    信息提取步骤：从所述位置信息列表中筛选出预设数量的概率最高的位置，并从筛选出的位置提取信息作为目标信息。
  2. 根据权利要求1所述的网页目标信息的提取方法，其特征在于，所述“识别出所述目标网页所属的主题类别”的步骤包括：
    选择所述分类模型的输出结果中概率最高值对应的主题类别，作为所述目标网页所属的主题类别。
  3. 根据权利要求2所述的网页目标信息的提取方法，其特征在于，所述主题分类步骤可以替换为：
    分别计算所述目标网页的词向量与预先确定的各主题类别的词向量之间的相似度，当相似度最大值大于或等于预设相似度阈值时，将相似度最高的主题类别作为所述目标网页所属的主题类别；及
    当相似度最大值小于预设相似度阈值时，接收针对目标网页所属的主题类别的分类指令，根据分类指令中包含的主题类别作为目标网页所属的主题类别。
  4. 根据权利要求1所述的网页目标信息的提取方法，其特征在于，所述分类模型的训练步骤包括：
    获取指定网页的网页源码，分别对每个指定网页的网页源码进行分词，得到每个指定网页的可用词集合，从可用词集合中提取关键词，并生成每个 指定网页的词向量；
    分别为每个指定网页标注第二标签，将所述词向量划分至不同第二标签对应的集合中，作为不同主题类别的样本数据；及
    将所述集合中的样本数据划分为训练集及验证集，利用训练集对神经网络模型进行训练，利用验证集对神经网络模型进行验证，当验证结果满足第一预设条件时，确定所述不同主题类型对应的分类模型。
  5. 根据权利要求4所述的网页目标信息的提取方法，其特征在于，所述位置预测模型的训练步骤包括：
    分别为每个指定网页标注所述第二标签，根据第二标签将所述指定网页的网页源码划分至不同主题类别对应的集合中；
    分别在每个指定网页的网页源码中标注不同的第一标签，分别将每个集合中的网页源码划分至各第一标签对应的子集合中，作为各主题类别下不同第一标签对应的样本数据；及
    将所述子集合中的样本数据划分为训练集及验证集，利用训练集对循环神经网络模型进行训练，利用验证集对循环神经网络模型进行验证，当验证结果满足第二预设条件时，确定各主题类别下不同第一标签对应的位置预测模型。
  6. 根据权利要求5所述的网页目标信息的提取方法，其特征在于，所述“识别出所述目标网页所属的主题类别”的步骤包括：
    选择所述分类模型的输出结果中概率最高值对应的主题类别，作为所述目标网页所属的主题类别。
  7. 根据权利要求6所述的网页目标信息的提取方法，其特征在于，所述主题分类步骤可以替换为：
    分别计算所述目标网页的词向量与预先确定的各主题类别的词向量之间的相似度，当相似度最大值大于或等于预设相似度阈值时，将相似度最高的主题类别作为所述目标网页所属的主题类别；及
    当相似度最大值小于预设相似度阈值时，接收针对目标网页所属的主题类别的分类指令，根据分类指令中包含的主题类别作为目标网页所属的主题类别。
  8. 一种电子装置，其特征在于，该装置包括：存储器、处理器，所述存 储器上存储有可在所述处理器上运行的网页目标信息的提取程序，所述网页目标信息的提取程序被所述处理器执行时，可实现如下步骤：
    分词步骤：接收从目标网页中提取目标信息的请求，获取所述目标网页的网页源码，对获取到的网页源码进行分词处理得到所述目标网页的可用词集合；
    主题分类步骤：根据所述目标网页的可用词集合计算所述目标网页的词向量，将计算得到的词向量输入预先确定的各主题类别对应的分类模型，识别出所述目标网页所属的主题类别；
    位置预测步骤：确定所述目标信息对应的第一标签，将所述目标网页的网页源码输入识别出的主题类别中所述第一标签对应的位置预测模型中，预测所述目标信息出现在不同位置的位置信息列表；及
    信息提取步骤：从所述位置信息列表中筛选出预设数量的概率最高的位置，并从筛选出的位置提取信息作为目标信息。
  9. 根据权利要求8所述的电子装置，其特征在于，所述“识别出所述目标网页所属的主题类别”的步骤包括：
    选择所述分类模型的输出结果中概率最高值对应的主题类别，作为所述目标网页所属的主题类别。
  10. 根据权利要求9所述的电子装置，其特征在于，所述主题分类步骤可以替换为：
    分别计算所述目标网页的词向量与预先确定的各主题类别的词向量之间的相似度，当相似度最大值大于或等于预设相似度阈值时，将相似度最高的主题类别作为所述目标网页所属的主题类别；
    当相似度最大值小于预设相似度阈值时，接收针对目标网页所属的主题类别的分类指令，根据分类指令中包含的主题类别作为目标网页所属的主题类别。
  11. 根据权利要求10所述的电子装置，其特征在于，所述分类模型的训练步骤包括：
    获取指定网页的网页源码，分别对每个指定网页的网页源码进行分词，得到每个指定网页的可用词集合，从可用词集合中提取关键词，并生成每个指定网页的词向量；
    分别为每个指定网页标注第二标签，将所述词向量划分至不同第二标签对应的集合中，作为不同主题类别的样本数据；及
    将所述集合中的样本数据划分为训练集及验证集，利用训练集对神经网络模型进行训练，利用验证集对神经网络模型进行验证，当验证结果满足第一预设条件时，确定所述不同主题类型对应的分类模型。
  12. 根据权利要求11所述的电子装置，其特征在于，所述位置预测模型的训练步骤包括：
    分别为每个指定网页标注所述第二标签，根据第二标签将所述指定网页的网页源码划分至不同主题类别对应的集合中；
    分别在每个指定网页的网页源码中标注不同的第一标签，分别将每个集合中的网页源码划分至各第一标签对应的子集合中，作为各主题类别下不同第一标签对应的样本数据；及
    将所述子集合中的样本数据划分为训练集及验证集，利用训练集对循环神经网络模型进行训练，利用验证集对循环神经网络模型进行验证，当验证结果满足第二预设条件时，确定各主题类别下不同第一标签对应的位置预测模型。
  13. 根据权利要求12所述的电子装置，其特征在于，所述“识别出所述目标网页所属的主题类别”的步骤包括：
    选择所述分类模型的输出结果中概率最高值对应的主题类别，作为所述目标网页所属的主题类别。
  14. 根据权利要求13所述的电子装置，其特征在于，所述主题分类步骤可以替换为：
    分别计算所述目标网页的词向量与预先确定的各主题类别的词向量之间的相似度，当相似度最大值大于或等于预设相似度阈值时，将相似度最高的主题类别作为所述目标网页所属的主题类别；及
    当相似度最大值小于预设相似度阈值时，接收针对目标网页所属的主题类别的分类指令，根据分类指令中包含的主题类别作为目标网页所属的主题类别。
  15. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中包括网页目标信息的提取程序，所述网页目标信息的提取程序被所述处理 器执行时，可实现如下步骤：
    分词步骤：接收从目标网页中提取目标信息的请求，获取所述目标网页的网页源码，对获取到的网页源码进行分词处理得到所述目标网页的可用词集合；
    主题分类步骤：根据所述目标网页的可用词集合计算所述目标网页的词向量，将计算得到的词向量输入预先确定的各主题类别对应的分类模型，识别出所述目标网页所属的主题类别；
    位置预测步骤：确定所述目标信息对应的第一标签，将所述目标网页的网页源码输入识别出的主题类别中所述第一标签对应的位置预测模型中，预测所述目标信息出现在不同位置的位置信息列表；及
    信息提取步骤：从所述位置信息列表中筛选出预设数量的概率最高的位置，并从筛选出的位置提取信息作为目标信息。
  16. 根据权利要求15所述的计算机可读存储介质，其特征在于，所述“识别出所述目标网页所属的主题类别”的步骤包括：
    选择所述分类模型的输出结果中概率最高值对应的主题类别，作为所述目标网页所属的主题类别。
  17. 根据权利要求16所述的计算机可读存储介质，其特征在于，所述主题分类步骤可以替换为：
    分别计算所述目标网页的词向量与预先确定的各主题类别的词向量之间的相似度，当相似度最大值大于或等于预设相似度阈值时，将相似度最高的主题类别作为所述目标网页所属的主题类别；
    当相似度最大值小于预设相似度阈值时，接收针对目标网页所属的主题类别的分类指令，根据分类指令中包含的主题类别作为目标网页所属的主题类别。
  18. 根据权利要求15所述的计算机可读存储介质，其特征在于，所述分类模型的训练步骤包括：
    获取指定网页的网页源码，分别对每个指定网页的网页源码进行分词，得到每个指定网页的可用词集合，从可用词集合中提取关键词，并生成每个指定网页的词向量；
    分别为每个指定网页标注第二标签，将所述词向量划分至不同第二标签 对应的集合中，作为不同主题类别的样本数据；及
    将所述集合中的样本数据划分为训练集及验证集，利用训练集对神经网络模型进行训练，利用验证集对神经网络模型进行验证，当验证结果满足第一预设条件时，确定所述不同主题类型对应的分类模型。
  19. 根据权利要求18所述的计算机可读存储介质，其特征在于，所述位置预测模型的训练步骤包括：
    分别为每个指定网页标注所述第二标签，根据第二标签将所述指定网页的网页源码划分至不同主题类别对应的集合中；
    分别在每个指定网页的网页源码中标注不同的第一标签，分别将每个集合中的网页源码划分至各第一标签对应的子集合中，作为各主题类别下不同第一标签对应的样本数据；及
    将所述子集合中的样本数据划分为训练集及验证集，利用训练集对循环神经网络模型进行训练，利用验证集对循环神经网络模型进行验证，当验证结果满足第二预设条件时，确定各主题类别下不同第一标签对应的位置预测模型。
  20. 根据权利要求19所述的计算机可读存储介质，其特征在于，所述主题分类步骤可以替换为：
    分别计算所述目标网页的词向量与预先确定的各主题类别的词向量之间的相似度，当相似度最大值大于或等于预设相似度阈值时，将相似度最高的主题类别作为所述目标网页所属的主题类别；及
    当相似度最大值小于预设相似度阈值时，接收针对目标网页所属的主题类别的分类指令，根据分类指令中包含的主题类别作为目标网页所属的主题类别。
PCT/CN2018/102115 2018-05-14 2018-08-24 网页目标信息的提取方法、装置及存储介质 WO2019218514A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810455840.5 2018-05-14
CN201810455840.5A CN108629043B (zh) 2018-05-14 2018-05-14 网页目标信息的提取方法、装置及存储介质

Publications (1)

Publication Number Publication Date
WO2019218514A1 true WO2019218514A1 (zh) 2019-11-21

Family

ID=63693220

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/102115 WO2019218514A1 (zh) 2018-05-14 2018-08-24 网页目标信息的提取方法、装置及存储介质

Country Status (2)

Country Link
CN (1) CN108629043B (zh)
WO (1) WO2019218514A1 (zh)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111124916A (zh) * 2019-12-23 2020-05-08 北京云聚智慧科技有限公司 一种基于动作语义向量的模型训练方法和电子设备
CN111832298A (zh) * 2020-07-14 2020-10-27 北京百度网讯科技有限公司 病历的质检方法、装置、设备以及存储介质
CN112101819A (zh) * 2020-10-28 2020-12-18 平安国际智慧城市科技股份有限公司 食品风险预测方法、装置、设备及存储介质
CN112328833A (zh) * 2020-11-09 2021-02-05 腾讯科技（深圳）有限公司 标签处理方法、装置及计算机可读存储介质
CN115618291A (zh) * 2022-10-14 2023-01-17 吉林省吉林祥云信息技术有限公司 一种基于Transformer的web指纹识别方法、系统、设备以及存储介质
CN116975410A (zh) * 2023-09-22 2023-10-31 北京中关村科金技术有限公司 网页数据采集方法、装置、电子设备及可读存储介质

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111191095A (zh) * 2018-11-14 2020-05-22 中国移动通信集团河北有限公司 网页数据获取方法、装置、设备及介质
CN109634922A (zh) * 2018-12-06 2019-04-16 苏州科创风云信息技术有限公司 共享货架中资源的分类方法及装置
CN109657710B (zh) * 2018-12-06 2022-01-21 北京达佳互联信息技术有限公司 数据筛选方法、装置、服务器及存储介质
CN109960725A (zh) * 2019-01-17 2019-07-02 平安科技（深圳）有限公司 基于情感的文本分类处理方法、装置和计算机设备
CN109992344A (zh) * 2019-03-29 2019-07-09 珠海豹好玩科技有限公司 网页处理方法、系统、设备及计算机可读存储介质
CN110110127B (zh) * 2019-05-05 2023-07-18 深圳劲嘉集团股份有限公司 一种识别专色混合油墨的基色油墨的方法以及电子设备
CN110427618B (zh) * 2019-07-22 2021-03-16 清华大学 对抗样本生成方法、介质、装置和计算设备
CN111401935B (zh) * 2020-02-21 2023-04-07 中国平安财产保险股份有限公司 资源分配方法、装置及存储介质
CN111428489B (zh) * 2020-03-19 2023-08-29 北京百度网讯科技有限公司 一种评论生成方法、装置、电子设备及存储介质
CN113268651B (zh) * 2021-05-27 2023-06-06 清华大学 一种搜索信息的摘要自动生成方法及装置
CN113254751B (zh) * 2021-06-24 2021-09-21 北森云计算有限公司 一种复杂网页结构化信息精确提取方法、设备及存储介质
TWI827984B (zh) * 2021-10-05 2024-01-01 台灣大哥大股份有限公司 網站分類系統及方法
CN114996622B (zh) * 2022-08-02 2022-11-11 北京弘玑信息技术有限公司 信息获取方法、值网络模型的训练方法及电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103678310A (zh) * 2012-08-31 2014-03-26 腾讯科技（深圳）有限公司 网页主题的分类方法及装置
CN106156204A (zh) * 2015-04-23 2016-11-23 深圳市腾讯计算机系统有限公司 文本标签的提取方法和装置
US20180039696A1 (en) * 2016-08-08 2018-02-08 Baidu Usa Llc Knowledge graph entity reconciler
CN107862039A (zh) * 2017-11-06 2018-03-30 工业和信息化部电子第五研究所 网页数据获取方法、系统和数据匹配推送方法

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101094194B (zh) * 2006-06-19 2010-06-23 腾讯科技（深圳）有限公司 一种提取Web页面中用户所需Web信息的方法
CN101593200B (zh) * 2009-06-19 2012-10-03 淮海工学院 基于关键词频度分析的中文网页分类方法
CN101794311B (zh) * 2010-03-05 2012-06-13 南京邮电大学 基于模糊数据挖掘的中文网页自动分类方法
CN105589913A (zh) * 2015-06-15 2016-05-18 广州市动景计算机科技有限公司 一种提取页面信息的方法及装置
CN105786951A (zh) * 2015-12-31 2016-07-20 北京金山安全软件有限公司 一种网页中内容块的提取方法、装置及服务器

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103678310A (zh) * 2012-08-31 2014-03-26 腾讯科技（深圳）有限公司 网页主题的分类方法及装置
CN106156204A (zh) * 2015-04-23 2016-11-23 深圳市腾讯计算机系统有限公司 文本标签的提取方法和装置
US20180039696A1 (en) * 2016-08-08 2018-02-08 Baidu Usa Llc Knowledge graph entity reconciler
CN107862039A (zh) * 2017-11-06 2018-03-30 工业和信息化部电子第五研究所 网页数据获取方法、系统和数据匹配推送方法

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111124916A (zh) * 2019-12-23 2020-05-08 北京云聚智慧科技有限公司 一种基于动作语义向量的模型训练方法和电子设备
CN111124916B (zh) * 2019-12-23 2023-04-07 北京云聚智慧科技有限公司 一种基于动作语义向量的模型训练方法和电子设备
CN111832298A (zh) * 2020-07-14 2020-10-27 北京百度网讯科技有限公司 病历的质检方法、装置、设备以及存储介质
CN111832298B (zh) * 2020-07-14 2024-03-01 北京百度网讯科技有限公司 病历的质检方法、装置、设备以及存储介质
CN112101819A (zh) * 2020-10-28 2020-12-18 平安国际智慧城市科技股份有限公司 食品风险预测方法、装置、设备及存储介质
CN112328833A (zh) * 2020-11-09 2021-02-05 腾讯科技（深圳）有限公司 标签处理方法、装置及计算机可读存储介质
CN112328833B (zh) * 2020-11-09 2024-03-26 腾讯科技（深圳）有限公司 标签处理方法、装置及计算机可读存储介质
CN115618291A (zh) * 2022-10-14 2023-01-17 吉林省吉林祥云信息技术有限公司 一种基于Transformer的web指纹识别方法、系统、设备以及存储介质
CN115618291B (zh) * 2022-10-14 2023-09-29 吉林省吉林祥云信息技术有限公司 一种基于Transformer的web指纹识别方法、系统、设备以及存储介质
CN116975410A (zh) * 2023-09-22 2023-10-31 北京中关村科金技术有限公司 网页数据采集方法、装置、电子设备及可读存储介质
CN116975410B (zh) * 2023-09-22 2023-12-19 北京中关村科金技术有限公司 网页数据采集方法、装置、电子设备及可读存储介质

Also Published As

Publication number Publication date
CN108629043A (zh) 2018-10-09
CN108629043B (zh) 2023-05-12

Similar Documents

Publication Publication Date Title
WO2019218514A1 (zh) 网页目标信息的提取方法、装置及存储介质
CN109325165B (zh) 网络舆情分析方法、装置及存储介质
CN109145215B (zh) 网络舆情分析方法、装置及存储介质
CN107992596B (zh) 一种文本聚类方法、装置、服务器和存储介质
WO2021068339A1 (zh) 文本分类方法、装置及计算机可读存储介质
WO2019227710A1 (zh) 网络舆情的分析方法、装置及计算机可读存储介质
WO2020000717A1 (zh) 网页分类方法、装置及计算机可读存储介质
WO2017167067A1 (zh) 网页文本分类的方法和装置，网页文本识别的方法和装置
WO2020237856A1 (zh) 基于知识图谱的智能问答方法、装置及计算机存储介质
WO2019041521A1 (zh) 用户关键词提取装置、方法及计算机可读存储介质
WO2015149533A1 (zh) 一种基于网页内容分类进行分词处理的方法和装置
US20130073514A1 (en) Flexible and scalable structured web data extraction
CN107209861A (zh) 使用否定数据优化多类别多媒体数据分类
CN110390044B (zh) 一种相似网络页面的搜索方法及设备
CN112632278A (zh) 一种基于多标签分类的标注方法、装置、设备及存储介质
CN113626607B (zh) 异常工单识别方法、装置、电子设备及可读存储介质
US20140297628A1 (en) Text Information Processing Apparatus, Text Information Processing Method, and Computer Usable Medium Having Text Information Processing Program Embodied Therein
WO2021068681A1 (zh) 标签分析方法、装置及计算机可读存储介质
CN113051356A (zh) 开放关系抽取方法、装置、电子设备及存储介质
CN114021577A (zh) 内容标签的生成方法、装置、电子设备及存储介质
CN109271624B (zh) 一种目标词确定方法、装置及存储介质
CN103488707B (zh) 一种基于贪心策略和启发式算法搜索候选类别的方法
CN114416998A (zh) 文本标签的识别方法、装置、电子设备及存储介质
CN112818200A (zh) 基于静态网站的数据爬取及事件分析方法及系统
CN106570196B (zh) 视频节目的搜索方法和装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18918623

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18918623

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 01.03.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 18918623

Country of ref document: EP

Kind code of ref document: A1