CN104317903A - Chapter type text chapter integrity identification method and device - Google Patents

Abstract

The present invention provides a method and device for identifying chapter integrity of a chapter-style text. The method includes: respectively identifying the catalog page and multiple content pages of the chapter-style text from multiple sites, wherein each site corresponds to a catalog page, each catalog page corresponds to a plurality of content pages; according to the multiple content pages corresponding to each catalog page, determine the catalog page sets of the chapter text on different sites; analyze the catalog pages and content of each catalog page in the catalog page set /or the content page corresponding to each catalog page, and identify the chapter completeness of each catalog page in the catalog page set according to the analysis result. The technical solution provided by the invention can flexibly and quickly identify the chapter completeness of the chapter text, and the recognition result is accurate and objective.

Description

Method and device for identifying chapter integrity of chapter text

technical field

The invention relates to the technical field of the Internet, in particular to a method and a device for identifying chapter integrity of a chapter text.

Background technique

With the increasing popularity of computers and computer networks, the Internet has penetrated into all areas of people's work, study and life, and has become an important way for people to publish and obtain information.

At present, there are a large number of chapter-style texts on the Internet, and the same text may be reproduced in large quantities by different websites. Due to the influence of some objective factors when reprinting, the content of the text may be incomplete or even false on some websites. . Taking novel texts as an example, reading novels is a strong demand of Internet users, especially on mobile devices. There are a large number of novel websites, but the quality varies. The same online novel will be reproduced in large quantities by different websites, but affected by some objective factors, the content of the novel may be incomplete (such as missing chapters) on some websites. Even the content is false (putting together false chapters). When indexing these novel sites, search engines need to make judgments on the integrity of the chapters of the novel, try to present the site with complete content to users, improve the quality of novel content obtained by users, and improve user experience.

In the related technology, the manual configuration templates of different novel sites are used to judge the integrity of the chapters. Although the accuracy of this method is very high, the disadvantages are also obvious: the number of websites covered by manpower is limited, it is not smart enough, and the response to changes in website templates is not timely. . Therefore, how to flexibly, quickly and accurately identify the chapter completeness of a chapter-style text has become a technical problem to be solved urgently.

Contents of the invention

In view of the above problems, the present invention is proposed to provide a method for identifying chapter integrity of a chapter text and a corresponding device that overcomes the above problems or at least partially solves the above problems.

According to one aspect of the present invention, a method for identifying chapter integrity of a chapter-style text is provided, including: respectively identifying the catalog page and multiple content pages of the chapter-style text from multiple sites, wherein each site corresponds to a Catalog pages, each catalog page corresponds to multiple content pages; according to the multiple content pages corresponding to each catalog page, determine the catalog page sets of the chapter text on different sites; analyze each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identify the chapter completeness of each catalog page in the catalog page set according to the result obtained from the analysis.

Optionally, according to the plurality of content pages corresponding to each catalog page, determining the set of catalog pages of the chapter-style text on different sites includes: calculating the intersection between the content pages corresponding to every two catalog pages, and using it as The intersection of every two catalog pages; according to the intersection of every two catalog pages, determine the set of catalog pages of the chapter text on different sites.

Optionally, the calculation of the intersection between the content pages corresponding to every two catalog pages includes: extracting the text feature vector of each content page in the content pages corresponding to multiple catalog pages; The pages are clustered to generate multiple content page groups; according to the multiple content page groups and the mapping relationship between each catalog page and its corresponding content page, the intersection between the content pages corresponding to every two catalog pages is calculated.

Optionally, according to the intersection of every two catalog pages, determining the set of catalog pages of the chapter text on different sites includes: merging every two catalog pages whose number of elements in the intersection is greater than or equal to a preset threshold , to obtain a merged result; the merged result is used as a set of catalog pages of the chapter text on different sites.

Optionally, analyzing each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identifying the chapter integrity of each catalog page in the catalog page set according to the analysis results, including: calculating the The average number of elements of the intersection of every two catalog pages in the set of catalog pages; if the number of elements of the intersection of a certain catalog page and multiple other catalog pages in the set of catalog pages is less than the average , it is determined that the chapter corresponding to the table of contents page is incomplete.

Optionally, analyzing each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identifying the chapter integrity of each catalog page in the catalog page set according to the analysis results, including: if a certain The content page corresponding to a catalog page includes content pages corresponding to multiple other catalog pages in the catalog page set, and there are other content pages in the content page corresponding to the catalog page, then it is determined that the other content pages are the latest chapters , and the content page has the ability to continuously contribute new chapters.

Optionally, analyzing each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identifying the chapter integrity of each catalog page in the catalog page set according to the analysis results, including: if a certain A certain content page corresponding to a catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, and the length of the content page does not belong to the range corresponding to the average length of the content pages corresponding to the catalog page, then Determine that the content page is a fake content page.

Optionally, analyzing each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identifying the chapter integrity of each catalog page in the catalog page set according to the analysis results, including: if a certain A certain content page corresponding to a catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, the length of the content page belongs to the range corresponding to the average length of the content pages corresponding to the catalog page, and the catalog page If the page does not have the ability to continuously contribute new chapters, it is determined that the content page is a false content page.

Optionally, identifying the catalog page and the plurality of content pages of the chapter text from multiple sites includes: searching for web pages related to the chapter text from multiple sites; identifying the A table of contents page with chapter-style text and multiple content pages.

Optionally, identifying the catalog page and multiple content pages of the chapter-style text from the searched webpage includes: parsing the searched webpage into a text object model tree structure; Classify each node of the webpage to determine the structural block of the web page; extract the catalog page and multiple content pages of the chapter text according to the structural block.

According to another aspect of the present invention, a device for identifying the completeness of chapters of a chapter text is also provided, including:

The acquisition module is adapted to identify the chapter-style text content page and multiple content pages from multiple sites, wherein each site corresponds to a content page, and each content page corresponds to multiple content pages;

The determination module is adapted to determine the set of catalog pages of the chapter text on different sites according to the plurality of content pages corresponding to each catalog page;

The identification module is adapted to analyze each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identify the chapter integrity of each catalog page in the catalog page set according to the analysis result.

Optionally, the determining module is further adapted to: calculate the intersection between the content pages corresponding to every two catalog pages, and use it as the intersection of every two catalog pages; determine the chapter according to the intersection of every two catalog pages A collection of catalog pages on different sites with the same text.

Optionally, the determination module is further adapted to: extract the text feature vector of each content page in the content pages corresponding to multiple catalog pages; cluster the content pages with the same text feature vector to generate multiple content page groups ; According to the plurality of content page groups and the mapping relationship between each catalog page and its corresponding content page, calculate the intersection between the content pages corresponding to every two catalog pages.

Optionally, the determination module is further adapted to: merge every two catalog pages whose number of elements in the intersection is greater than or equal to a preset threshold to obtain a merged result; use the merged result as the chapter text in different A collection of catalog pages on the site.

Optionally, the identification module is further adapted to: calculate the average value of the number of elements in the intersection of every two catalog pages in the catalog page set; If the number of elements in the intersection of the catalog pages is less than the average value, it is determined that the chapter corresponding to the catalog page is incomplete.

Optionally, the identification module is further adapted to: if the content page corresponding to a certain catalog page contains content pages corresponding to multiple other catalog pages in the catalog page set, and the content page corresponding to the catalog page still exists other content pages, it is determined that the other content pages are the content pages of the latest chapter, and the content page has the ability to continuously contribute new chapters.

Optionally, the identification module is further adapted to: if a certain content page corresponding to a certain catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, and the length of the content page does not belong to the catalog page If the interval range corresponding to the average length of the content page corresponding to the page is determined, the content page is determined to be a false content page.

Optionally, the identification module is further adapted to: if a certain content page corresponding to a certain catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, the length of the content page belongs to the content page corresponding to the catalog page The range corresponding to the average length of the content page, and the content page does not have the ability to continuously contribute new chapters, then it is determined that the content page is a false content page.

Optionally, the acquiring module is further adapted to: search for webpages related to the chapter text from multiple sites; identify the catalog page and multiple content pages of the chapter text from the searched web pages.

Optionally, the acquisition module is further adapted to: parse the searched webpage into a text object model tree structure; classify each node in the text object model tree structure to determine the structural blocks of the webpage ; Extract the catalog page and multiple content pages of the chapter text in blocks according to the structure.

According to the technical solution provided by the present invention, the catalog pages and multiple content pages of the chapter-style text are respectively identified from multiple sites, and then according to the multiple content pages corresponding to each catalog page, the catalogs of the chapter-style text on different sites are determined. page collection. Then analyze each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identify the chapter integrity of each catalog page in the catalog page set according to the analysis result. It can be seen that the present invention realizes the automatic processing of the acquisition of data sources (the catalog pages and multiple content pages of chapter text on multiple sites), the determination of the catalog page set and the analysis of the catalog page set, Therefore, the problem of low efficiency caused by manual configuration of templates to judge the integrity of chapters in related technologies is solved. Moreover, the present invention can flexibly acquire data sources, and then determine the set of catalog pages, and analyze the set of catalog pages, which solves the problem of untimely response to website template changes in the related art. In addition, the catalog page and the content page can accurately and objectively reflect the chapter integrity of the chapter-style text, and the present invention analyzes the chapter-style text in the catalog page sets on different sites and/or the corresponding content of each catalog page. content pages, and then identify the chapter completeness of each catalog page in the catalog page set according to the analysis result, so that the identification result is more accurate. To sum up, the technical solution provided by the present invention can flexibly and quickly identify the integrity of the chapters of the chapter text, and the recognition result is accurate and objective.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Those skilled in the art will be more aware of the above and other objects, advantages and features of the present invention according to the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same parts. In the attached picture:

FIG. 1 shows a flowchart of a method for identifying chapter completeness of a chapter text according to an embodiment of the present invention; and

Fig. 2 shows a schematic structural diagram of an apparatus for identifying chapter integrity of a chapter text according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

In order to solve the above technical problems, an embodiment of the present invention provides a method for identifying chapter integrity of a chapter-style text, and FIG. 1 shows a flow chart of a method for identifying chapter integrity of a chapter-style text according to an embodiment of the present invention . As shown in Fig. 1, the method at least includes the following steps S102 to S106.

Step S102 , identifying chapter-style text content pages and multiple content pages from multiple sites, wherein each site corresponds to a content page, and each content page corresponds to multiple content pages.

Step S104, according to the plurality of content pages corresponding to each catalog page, determine the catalog page sets of the chapter text on different sites.

Step S106 , analyzing each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identifying the chapter integrity of each catalog page in the catalog page set according to the analysis result.

According to the technical solution provided by the present invention, the catalog pages and multiple content pages of the chapter-style text are respectively identified from multiple sites, and then according to the multiple content pages corresponding to each catalog page, the catalogs of the chapter-style text on different sites are determined. page collection. Then analyze each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identify the chapter integrity of each catalog page in the catalog page set according to the analysis result. It can be seen that the present invention realizes the automatic processing of the acquisition of data sources (the catalog pages and multiple content pages of chapter text on multiple sites), the determination of the catalog page set and the analysis of the catalog page set, Therefore, the problem of low efficiency caused by manual configuration of templates to judge the integrity of chapters in related technologies is solved. Moreover, the present invention can flexibly acquire data sources, and then determine the set of catalog pages, and analyze the set of catalog pages, which solves the problem of untimely response to website template changes in the related art. In addition, the catalog page and the content page can accurately and objectively reflect the chapter integrity of the chapter-style text, and the present invention analyzes the chapter-style text in the catalog page sets on different sites and/or the corresponding content of each catalog page. content pages, and then identify the chapter completeness of each catalog page in the catalog page set according to the analysis result, so that the identification result is more accurate. To sum up, the technical solution provided by the present invention can flexibly and quickly identify the integrity of the chapters of the chapter text, and the recognition result is accurate and objective.

The chapter text mentioned above in step S102 refers to a text composed of several chapters, such as a novel, a thesis, and the like. The catalog page refers to the catalog of chapter-style texts. For example, when users search for novels, they usually look for the catalog page of novels. A content page refers to the specific content of a chapter of a chapter-style text. The present invention provides a preferred scheme for identifying chapter-style text catalog pages and multiple content pages from multiple sites. A table of contents page of chapter-style text and multiple content pages are identified in the web page of .

Further, identifying and extracting the catalog pages and multiple content pages of the chapter text from the searched web pages can be carried out by manually writing rules to identify and extract the catalog pages and content pages from the searched web pages. Or, based on the marked template, the best matching template can be found in the template library for each extraction, and then the content page and content page of the template can be identified and extracted. In addition, in order to improve the recognition efficiency, the present invention can also parse the searched webpage into a text object model tree structure, and classify each node in the text object model tree structure to determine the structural blocks of the webpage, and then according to the structure Extract the table of contents page and multiple content pages of chapter text in blocks. A preferred scheme for classifying each node in the tree structure of the text object model to determine the structural blocks of the webpage is provided here. In this scheme, the tree structure of the text object model can be traversed to obtain each node in the tree structure of the text object model. The content of each node is then entered into the decision tree according to the preset rules, and each node is classified by the decision tree. Alternatively, the text object model tree structure can be traversed to obtain the dimensional characteristics of each node in the text object model tree structure, and then the dimensional characteristics of each node are input into the decision tree according to preset rules, and the decision tree classifies each node.

The decision tree is based on the known statistical data of various dimensional features in various blocks, and uses the dimensional features of each node to obtain the block type corresponding to each node through training the decision tree. The decision tree will be introduced in detail below to classify each node in the tree structure of the text object model of the webpage, so as to determine the scheme of dividing the structure of the webpage into blocks.

First, determine the dimensional features used for block segmentation. In the embodiment of the present invention, there are as many as 105 dimensional features that can be used, mainly involving the following contents: text length, number of hyperlinks, hyperlink text length, highlighted text length (including bold and bold text), Chinese character length, English character length, number character length, specific keywords, specific punctuation marks, etc. That is, a type of block can be determined by one or more of the 105 dimensional features taking specific values. It should be noted that the dimension features determined according to the actual situation are not limited to 105, and can be expanded in the subsequent process.

Secondly, input the determined dimensional features for block into the decision tree for training and constructing the decision tree.

Furthermore, according to preset rules, the content of each node in the text object model tree structure of the web page is input into the decision tree, and the content of each node is analyzed by the decision tree to obtain the dimensional characteristics of each node, and then according to the dimensional characteristics of each node. sort.

Multiple implementations of acquiring data sources in step S102 (content pages of chapter text and multiple content pages on multiple sites) have been introduced in detail above, and one or more implementations of determining a set of catalog pages will be introduced below.

In the above step S104, according to the plurality of content pages corresponding to each catalog page, the set of catalog pages of the chapter text on different sites is determined. The present invention provides a preferred solution, in which every two catalog pages are calculated The intersection between the content pages corresponding to the page is used as the intersection of every two catalog pages, and then according to the intersection of every two catalog pages, the set of catalog pages of the chapter text on different sites is determined.

Further, in the preferred solution of the present invention, the idea of clustering is used to calculate the intersection between the content pages corresponding to every two catalog pages, which may be to extract the text features of each content page in the content pages corresponding to multiple catalog pages vector, and then cluster the content pages with the same text feature vector to generate multiple content page groups, and then calculate the The intersection between the corresponding content pages of the page. For example, the multiple sites are sites A, B, and C, and the corresponding catalog pages of chapter-style texts are catalog pages A, B, and C. The multiple content pages corresponding to catalog page A are content pages A1, A2, A3, the multiple content pages corresponding to catalog page B are content pages B1, B2, and the multiple content pages corresponding to catalog page C are content pages C1, C2, C3, C4. The text feature vectors of each content page in the extracted content pages A1, A2, A3, B1, B2, C1, C2, C3, and C4 are a, b, c, a, b', a, b, c, d, respectively, Cluster the content pages with the same text feature vector to generate multiple content pages grouped into {a, a, a}, {b, b}, {b'}, {c, c}, {d}. Then, according to the grouping of multiple content pages and the mapping relationship between each catalog page and its corresponding content page, calculate the intersection between the content pages corresponding to every two catalog pages, that is, the intersection of the content pages corresponding to catalog page A and catalog page B The intersection between content pages corresponding to catalog page A and catalog page C is {a}, the intersection set between content pages corresponding to catalog page A and catalog page C is {a, b, c}, and the intersection set between content pages corresponding to catalog page B and catalog page C is {a }.

At this time, according to the intersection of every two catalog pages, determine the set of catalog pages of the chapter text on different sites, which can be merged every two catalog pages whose number of elements in the intersection is greater than or equal to the preset threshold value to obtain the merged As a result, the combined results are collections of catalog pages on different sites as chapter-style text. Still taking the above example as an example, the intersection between the content pages corresponding to every two catalog pages is taken as the intersection of every two catalog pages, that is, the intersection of catalog page A and catalog page B is {a}, and catalog page A and catalog page The intersection of page C is {a, b, c}, and the intersection of catalog page B and catalog page C is {a}. Take the preset threshold value as 1, merge every two catalog pages with the number of intersection elements greater than or equal to 1, and obtain the merged results as catalog pages A, B, and C, then the catalog pages of the chapter text on different sites The collection is catalog pages A, B, C.

After step S104 above determines the catalog page sets of chapter text on different sites according to the multiple content pages corresponding to each catalog page, step S106 analyzes each catalog page and/or the content corresponding to each catalog page in the catalog page set pages, and identify the chapter completeness of each catalog page in the catalog page set according to the result obtained by the analysis. The present invention provides various analysis methods, which will be described in detail below.

The first one is to calculate the average number of elements in the intersection of every two catalog pages in the catalog page set, if the number of elements in the intersection of a certain catalog page and multiple other catalog pages in the catalog page set is less than the average value, it is determined that the chapter corresponding to the table of contents page is incomplete. Taking the above example as an example, the set of catalog pages is catalog pages A, B, and C, the number of elements in the intersection of catalog page A and catalog page B is 1, and the number of elements in the intersection of catalog page A and catalog page C is 3. If the number of elements in the intersection of catalog page B and catalog page C is 1, the average value is 5/3, and the number of catalog page B, catalog page A, and catalog page C is 1, which is less than the average value of 5/3. Then it is determined that the chapter corresponding to the content page B is incomplete.

Second, if the content page corresponding to a catalog page contains content pages corresponding to multiple other catalog pages in the catalog page set, and there are other content pages in the content page corresponding to the catalog page, then determine the other content pages as The content page of the latest chapter, and the table of contents has the ability to continuously contribute new chapters. Here, the latest chapter refers to the latest published chapter of the chapter-type text, for example, the latest published chapter of a serial novel. Fiction users usually follow books, that is, once the latest chapter is published by the author, users want to see it immediately. The faster the latest chapter is released, the more popular the novel site is. Taking the above example as an example, the content pages A1, A2, and A3 corresponding to catalog page A (their text feature vectors are respectively a, b, and c), and the content pages B1 and B2 corresponding to catalog page B (their text feature vectors are respectively a , b'), content pages C1, C2, C3, and C4 corresponding to catalog page C (their text feature vectors are respectively a, b, c, and d). It can be seen that the content page corresponding to catalog page A includes the content pages corresponding to catalog page A and catalog page B, and there are other content pages (ie content page C4) in catalog page C, then determine that content page C4 is the content page of the latest chapter , and the table of contents page C has the ability to continuously contribute new chapters.

Third, if a content page corresponding to a catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, and the length of the content page does not belong to the average length of the content pages corresponding to the catalog page interval range, it is determined that the content page is a false content page. Taking the above example as an example, the content pages A1, A2, and A3 corresponding to catalog page A (their text feature vectors are respectively a, b, and c), and the content pages B1 and B2 corresponding to catalog page B (their text feature vectors are respectively a , b'), content pages C1, C2, C3, and C4 corresponding to catalog page C (their text feature vectors are respectively a, b, c, and d). It can be seen that the content page B2 corresponding to the catalog page B does not exist in the content pages corresponding to the catalog pages A and C, if the length of the content page B2 does not belong to the range corresponding to the average length of the content pages corresponding to the catalog page B, then determine the content page B2 is a false content page.

Fourth, if a content page corresponding to a catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, the length of the content page belongs to the range corresponding to the average length of the content pages corresponding to the catalog page , and the content page does not have the ability to continuously contribute new chapters, then it is determined that the content page is a false content page. Taking the above example as an example, the content pages A1, A2, and A3 corresponding to catalog page A (their text feature vectors are respectively a, b, and c), and the content pages B1 and B2 corresponding to catalog page B (their text feature vectors are respectively a , b'), content pages C1, C2, C3, and C4 corresponding to catalog page C (their text feature vectors are respectively a, b, c, and d). It can be seen that the content page B2 corresponding to the catalog page B does not exist in the content pages corresponding to the catalog pages A and C, and the length of the content page B2 belongs to the range corresponding to the average length of the content pages corresponding to the catalog page B. If the catalog page B does not have If there is no ability to continuously contribute new chapters, the content page B2 is determined to be a false content page.

It should be noted that the above four analysis methods can be used alone for the analysis of chapter integrity, and any one or more of these four analysis methods can be combined for the analysis of chapter integrity. For example, the first method is used to determine that the chapter corresponding to catalog page B is incomplete, and then the third or fourth method is used for further analysis, and it is determined that the content page B2 corresponding to catalog page B is a false content page, making the recognition result more accurate ,objective. In addition, the above example (that is, the multiple sites are sites A, B and C, the catalog pages of the corresponding chapter texts are catalog pages A, B and C. The multiple content pages corresponding to catalog page A are content pages A1 and A2 , A3, the multiple content pages corresponding to catalog page B are content pages B1 and B2, and the multiple content pages corresponding to catalog page C are content pages C1, C2, C3, C4) are only illustrative and do not limit the present invention .

Based on the same inventive concept, an embodiment of the present invention also provides a device for identifying chapter integrity of a chapter-style text, so as to realize the above-mentioned method for identifying chapter integrity of a chapter-style text.

Fig. 2 shows a schematic structural diagram of an apparatus for identifying chapter integrity of a chapter text according to an embodiment of the present invention. Referring to FIG. 2 , the device at least includes: an acquisition module 210 , a determination module 220 and an identification module 230 .

The functions of each component or device and the connection relationship between the various parts of the identification device of the chapter integrity of the chapter text of the embodiment of the present invention are now introduced:

The acquiring module 210 is adapted to separately identify a chapter-style text catalog page and multiple content pages from multiple sites, wherein each site corresponds to a catalog page, and each catalog page corresponds to multiple content pages;

The determination module 220, coupled with the acquisition module 210, is adapted to determine the set of catalog pages of the chapter text on different sites according to the multiple content pages corresponding to each catalog page;

The identifying module 230, coupled with the determining module 220, is adapted to analyze each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identify each catalog page in the catalog page set according to the analysis result chapter completeness.

In one embodiment of the present invention, the determination module 220 is further adapted to: calculate the intersection between the content pages corresponding to every two catalog pages, and use it as the intersection of every two catalog pages; , to determine the set of catalog pages on different sites for the chapter-style text.

In an embodiment of the present invention, the determination module 220 is further adapted to: extract the text feature vector of each content page in the content pages corresponding to multiple catalog pages; cluster the content pages with the same text feature vector to generate multiple content page groups; according to the multiple content page groups and the mapping relationship between each catalog page and its corresponding content page, calculate the intersection between the content pages corresponding to every two catalog pages.

In one embodiment of the present invention, the determination module 220 is further adapted to: merge every two catalog pages whose number of elements in the intersection is greater than or equal to a preset threshold to obtain a merged result; use the merged result as the A collection of catalog pages on different sites for chapter-style text.

In one embodiment of the present invention, the identification module 230 is further adapted to: calculate the average value of the number of elements in the intersection of every two catalog pages in the catalog page set; if a certain catalog page and the catalog page If the number of elements in the intersection of multiple other catalog pages in the collection is less than the average value, it is determined that the chapter corresponding to the catalog page is incomplete.

In an embodiment of the present invention, the identification module 230 is further adapted to: if the content page corresponding to a certain catalog page contains content pages corresponding to multiple other catalog pages in the catalog page set, and the corresponding content page of the catalog page If there are other content pages in the content page, it is determined that the other content pages are the content pages of the latest chapter, and the content page has the ability to continuously contribute new chapters.

In an embodiment of the present invention, the identification module 230 is further adapted to: if a certain content page corresponding to a certain catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, and the content page If the length does not belong to the range corresponding to the average length of the content pages corresponding to the catalog page, then it is determined that the content page is a false content page.

In an embodiment of the present invention, the identification module 230 is further adapted to: if a certain content page corresponding to a certain catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, the length of the content page If it belongs to the range corresponding to the average length of the content page corresponding to the content page, and the content page does not have the ability to continuously contribute new chapters, then the content page is determined to be a false content page.

In one embodiment of the present invention, the acquisition module 210 is further adapted to: search for webpages related to the chapter text from multiple sites; identify the catalog page and multiple contents of the chapter text from the searched web pages Page.

In an embodiment of the present invention, the acquisition module 210 is further adapted to: parse the searched webpage into a text object model tree structure; classify each node in the text object model tree structure to determine the Structural blocks of the webpage; extracting the catalog page and multiple content pages of the chapter text according to the structural blocks.

According to any one of the above preferred embodiments or a combination of multiple preferred embodiments, the embodiments of the present invention can achieve the following beneficial effects:

According to the technical solution provided by the present invention, the catalog pages and multiple content pages of the chapter-style text are respectively identified from multiple sites, and then according to the multiple content pages corresponding to each catalog page, the catalogs of the chapter-style text on different sites are determined. page collection. Then analyze each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identify the chapter integrity of each catalog page in the catalog page set according to the analysis result. It can be seen that the present invention realizes the automatic processing of the acquisition of data sources (the catalog pages and multiple content pages of chapter text on multiple sites), the determination of the catalog page set and the analysis of the catalog page set, Therefore, the problem of low efficiency caused by manual configuration of templates to judge the integrity of chapters in related technologies is solved. Moreover, the present invention can flexibly acquire data sources, and then determine the set of catalog pages, and analyze the set of catalog pages, which solves the problem of untimely response to website template changes in the related art. In addition, the catalog page and the content page can accurately and objectively reflect the chapter integrity of the chapter-style text, and the present invention analyzes the chapter-style text in the catalog page sets on different sites and/or the corresponding content of each catalog page. content pages, and then identify the chapter completeness of each catalog page in the catalog page set according to the analysis result, so that the identification result is more accurate. To sum up, the technical solution provided by the present invention can flexibly and quickly identify the integrity of the chapters of the chapter text, and the recognition result is accurate and objective.

The invention also discloses:

A1. A method for identifying chapter completeness of a chapter text, comprising:

Identify chapter-style text catalog pages and multiple content pages from multiple sites, where each site corresponds to a catalog page, and each catalog page corresponds to multiple content pages;

According to multiple content pages corresponding to each content page, determine the set of content pages of the chapter text on different sites;

Analyzing each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identifying the chapter integrity of each catalog page in the catalog page set according to the analysis result.

A2, according to the method described in A1, wherein, according to a plurality of content pages corresponding to each catalog page, determine the catalog page set of the chapter text on different sites, including:

Calculate the intersection between the content pages corresponding to every two catalog pages, and use it as the intersection of every two catalog pages;

According to the intersection of every two catalog pages, a set of catalog pages of the chapter text on different sites is determined.

A3. The method according to A1 or A2, wherein the calculation of the intersection between the content pages corresponding to every two catalog pages includes:

Extracting the text feature vector of each content page in the content pages corresponding to the plurality of catalog pages;

Cluster content pages with the same text feature vector to generate multiple content page groups;

According to the plurality of content page groups and the mapping relationship between each catalog page and its corresponding content page, the intersection between the content pages corresponding to every two catalog pages is calculated.

A4. The method according to any one of A1 to A3, wherein, according to the intersection of every two catalog pages, determining the set of catalog pages of the chapter text on different sites includes:

Merge every two catalog pages whose number of elements in the intersection is greater than or equal to the preset threshold, and obtain the merged result;

The merged result is used as a set of catalog pages of the chapter text on different sites.

A5. The method according to any one of A1 to A4, wherein each catalog page in the catalog page set and/or the content page corresponding to each catalog page is analyzed, and the content page in the catalog page set is identified according to the analysis results. Section completeness of each table of contents page, including:

calculating the average value of the number of elements in the intersection of every two catalog pages in the catalog page set;

If the number of elements in the intersection of a certain catalog page and multiple other catalog pages in the catalog page set is less than the average value, it is determined that the chapter corresponding to the catalog page is incomplete.

A6. The method according to any one of A1 to A5, wherein each catalog page in the catalog page set and/or the content page corresponding to each catalog page is analyzed, and the content page in the catalog page set is identified according to the analysis result. Section completeness of each table of contents page, including:

If the content page corresponding to a certain catalog page includes content pages corresponding to multiple other catalog pages in the catalog page set, and there are other content pages in the content page corresponding to the catalog page, then it is determined that the other content pages are The content page of the latest chapter, and the table of contents has the ability to continuously contribute new chapters.

A7. The method according to any one of A1 to A6, wherein each catalog page in the catalog page set and/or the content page corresponding to each catalog page is analyzed, and the content page in the catalog page set is identified according to the analysis results. Section completeness of each table of contents page, including:

If a content page corresponding to a certain catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, and the length of the content page does not belong to the range corresponding to the average length of the content pages corresponding to the catalog page , it is determined that the content page is a fake content page.

A8. The method according to any one of A1 to A7, wherein each catalog page in the catalog page set and/or the content page corresponding to each catalog page is analyzed, and the content page in the catalog page set is identified according to the analysis results. Section completeness of each table of contents page, including:

If a certain content page corresponding to a certain catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, the length of the content page belongs to the range corresponding to the average length of the content pages corresponding to the catalog page, and If the content page does not have the ability to continuously contribute new chapters, then it is determined that the content page is a false content page.

A9. The method according to any one of A1 to A8, wherein the catalog page and multiple content pages of the chapter text are identified from multiple sites, including:

Web pages related to chapter text are searched from multiple sites;

A content page and a plurality of content pages of the chapter text are identified from the searched web pages.

A10. The method according to any one of A1 to A9, wherein the catalog page and multiple content pages of the chapter text are identified from the searched web pages, including:

Parse the searched webpage into a text object model tree structure;

Classify each node in the tree structure of the text object model to determine the structural blocks of the webpage;

The content page and multiple content pages of the chapter text are extracted in blocks according to the structure.

B11. A device for identifying chapter integrity of a chapter text, comprising:

The acquisition module is adapted to identify the chapter-style text content page and multiple content pages from multiple sites, wherein each site corresponds to a content page, and each content page corresponds to multiple content pages;

The determination module is adapted to determine the set of catalog pages of the chapter text on different sites according to the plurality of content pages corresponding to each catalog page;

The identification module is adapted to analyze each catalog page in the catalog page set and/or the content page corresponding to each catalog page, and identify the chapter integrity of each catalog page in the catalog page set according to the analysis result.

B12. The device according to B11, wherein the determining module is further adapted to:

Calculate the intersection between the content pages corresponding to every two catalog pages, and use it as the intersection of every two catalog pages;

According to the intersection of every two catalog pages, a set of catalog pages of the chapter text on different sites is determined.

B13. The device according to B11 or B12, wherein the determination module is further adapted to:

Extracting the text feature vector of each content page in the content pages corresponding to the plurality of catalog pages;

Cluster content pages with the same text feature vector to generate multiple content page groups;

According to the plurality of content page groups and the mapping relationship between each catalog page and its corresponding content page, the intersection between the content pages corresponding to every two catalog pages is calculated.

B14. The device according to any one of B11 to B13, wherein the determination module is further adapted to:

Merge every two catalog pages whose number of elements in the intersection is greater than or equal to the preset threshold, and obtain the merged result;

The merged result is used as a set of catalog pages of the chapter text on different sites.

B15. The device according to any one of B11 to B14, wherein the identification module is further adapted to:

calculating the average value of the number of elements in the intersection of every two catalog pages in the set of catalog pages;

If the number of elements in the intersection of a certain catalog page and multiple other catalog pages in the catalog page set is less than the average value, it is determined that the chapter corresponding to the catalog page is incomplete.

B16. The device according to any one of B11 to B15, wherein the identification module is further adapted to:

If the content page corresponding to a certain catalog page includes content pages corresponding to multiple other catalog pages in the catalog page set, and there are other content pages in the content page corresponding to the catalog page, then it is determined that the other content pages are The content page of the latest chapter, and the table of contents has the ability to continuously contribute new chapters.

B17. The device according to any one of B11 to B16, wherein the identification module is further adapted to:

If a content page corresponding to a certain catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, and the length of the content page does not belong to the range corresponding to the average length of the content pages corresponding to the catalog page , it is determined that the content page is a fake content page.

B18. The device according to any one of B11 to B17, wherein the identification module is further adapted to:

If a certain content page corresponding to a certain catalog page does not exist in the content pages corresponding to other catalog pages in the catalog page set, the length of the content page belongs to the range corresponding to the average length of the content pages corresponding to the catalog page, and If the content page does not have the ability to continuously contribute new chapters, then it is determined that the content page is a false content page.

B19. The device according to any one of B11 to B18, wherein the acquisition module is further adapted to:

Web pages related to chapter text are searched from multiple sites;

A content page and a plurality of content pages of the chapter text are identified from the searched web pages.

B20. The device according to any one of B11 to B19, wherein the acquisition module is further adapted to:

Parse the searched webpage into a text object model tree structure;

Classify each node in the tree structure of the text object model to determine the structural blocks of the webpage;

The content page and multiple content pages of the chapter text are extracted in blocks according to the structure.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline this disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to realize some or some of all components in the device for identifying chapter integrity of a chapter text according to an embodiment of the present invention Or full functionality. The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

So far, those skilled in the art should appreciate that, although a number of exemplary embodiments of the present invention have been shown and described in detail herein, without departing from the spirit and scope of the present invention, the disclosed embodiments of the present invention can still be used. Many other variations or modifications consistent with the principles of the invention are directly identified or derived from the content. Accordingly, the scope of the present invention should be understood and deemed to cover all such other variations or modifications.

Claims (10)

1. a recognition methods for the chapters and sections integrality of chapters and sections formula text, comprising:

The catalogue page of chapters and sections formula text and multiple content pages is identified respectively from multiple website, wherein, the corresponding catalogue page of each website, the corresponding multiple content pages of each catalogue page;

The multiple content pages corresponding according to each catalogue page, determine the catalogue page set of described chapters and sections formula text on different website;

Analyze each catalogue page in described catalogue page set and/or content pages corresponding to each catalogue page, according to the chapters and sections integrality analyzing the result obtained and identify each catalogue page in described catalogue page set.

2. method according to claim 1, wherein, the multiple content pages corresponding according to each catalogue page, determine the catalogue page set of described chapters and sections formula text on different website, comprising:

Calculate the common factor between content pages corresponding to every two catalogue pages, and as the common factor of every two catalogue pages;

According to the common factor of every two catalogue pages, determine the catalogue page set of described chapters and sections formula text on different website.

3. method according to claim 1 and 2, wherein, the common factor between the content pages that every two catalogue pages of described calculating are corresponding, comprising:

Extract the Text eigenvector of each content pages in content pages corresponding to multiple catalogue page;

The content pages possessing same text proper vector is carried out cluster, generates the grouping of multiple content pages;

According to the mapping relations of the content pages that described multiple content pages is divided into groups and each catalogue page is corresponding with it, calculate the common factor between content pages corresponding to every two catalogue pages.

4. the method according to any one of claims 1 to 3, wherein, according to the common factor of every two catalogue pages, determine the catalogue page set of described chapters and sections formula text on different website, comprising:

Every two catalogue pages element number of common factor being more than or equal to predetermined threshold value merge, and obtain amalgamation result;

Using described amalgamation result as the catalogue page set of described chapters and sections formula text on different website.

5. the method according to any one of Claims 1-4, wherein, analyze each catalogue page in described catalogue page set and/or content pages corresponding to each catalogue page, according to the chapters and sections integrality analyzing the result obtained and identify each catalogue page in described catalogue page set, comprising:

Calculate the mean value of the element number of the common factor of every two catalogue pages in described catalogue page set;

If the number of the element of the common factor of other catalogue pages multiple is all less than described mean value in a certain catalogue page and described catalogue page set, then determine that the chapters and sections that this catalogue page is corresponding are imperfect.

6. the method according to any one of claim 1 to 5, wherein, analyze each catalogue page in described catalogue page set and/or content pages corresponding to each catalogue page, according to the chapters and sections integrality analyzing the result obtained and identify each catalogue page in described catalogue page set, comprising:

If the content pages that a certain catalogue page is corresponding includes the content pages that in described catalogue page set, other catalogue pages multiple are corresponding, and also there is other guide page in content pages corresponding to this catalogue page, then determine that described other guide page is the content pages of up-to-date chapters and sections, and this catalogue page possesses the ability continuing the new chapters and sections of contribution.

7. the method according to any one of claim 1 to 6, wherein, analyze each catalogue page in described catalogue page set and/or content pages corresponding to each catalogue page, according to the chapters and sections integrality analyzing the result obtained and identify each catalogue page in described catalogue page set, comprising:

If certain content pages that a certain catalogue page is corresponding is not present in the content pages that in described catalogue page set, other catalogue pages are corresponding, and this content pages length does not belong to interval range corresponding to the average length of content pages corresponding to this catalogue page, then determine that this content pages is false content pages.

8. the method according to any one of claim 1 to 7, wherein, analyze each catalogue page in described catalogue page set and/or content pages corresponding to each catalogue page, according to the chapters and sections integrality analyzing the result obtained and identify each catalogue page in described catalogue page set, comprising:

If certain content pages that a certain catalogue page is corresponding is not present in the content pages that in described catalogue page set, other catalogue pages are corresponding, this content pages length belongs to interval range corresponding to the average length of content pages corresponding to this catalogue page, and this catalogue page does not possess the ability continuing the new chapters and sections of contribution, then determine that this content pages is false content pages.

9. the method according to any one of claim 1 to 8, wherein, describedly identifies the catalogue page of chapters and sections formula text and multiple content pages respectively from multiple website, comprising:

The webpage relevant from multiple site search to chapters and sections formula text;

The catalogue page of described chapters and sections formula text and multiple content pages is identified from the webpage searched.

10. a recognition device for the chapters and sections integrality of chapters and sections formula text, comprising:

Acquisition module, is suitable for identifying the catalogue page of chapters and sections formula text and multiple content pages respectively from multiple website, wherein, and the corresponding catalogue page of each website, the corresponding multiple content pages of each catalogue page;

Determination module, is suitable for the multiple content pages corresponding according to each catalogue page, determines the catalogue page set of described chapters and sections formula text on different website;

Identification module, is suitable for analyzing each catalogue page in described catalogue page set and/or content pages corresponding to each catalogue page, according to the chapters and sections integrality analyzing the result obtained and identify each catalogue page in described catalogue page set.