JP2008217047A - Method, system and program for dividing web document - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To divide a document on a Web site into appropriate units of block documents according to a type of the document. <P>SOLUTION: A document dividing process part 13 includes: a first dividing part 135 for creating block documents by dividing a document on a Web site into units of blank-line tags; a character counting part 136 for counting the number of all the characters of the block documents and the number of hyperlinked characters; a determination part 137 for calculating the content rate of the number of hyperlinked characters in the block documents and determining whether or not the block documents should be subdivided according to the content rate; and a second dividing part 138 for subdividing the block documents when the determination part 137 determines that they should be subdivided. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a Web document dividing method, system, and program for dividing a document on a Web site into appropriate sections.

  For example, a document update determination system that automatically determines whether or not a new document has been posted on a WWW (World Wide Web) site on the Internet has been studied. This document update determination system compares two documents acquired at different times from the same Web site (Web page) and determines whether or not both are the same document. As the determination method, a full text comparison method, a hash value comparison method, a morpheme analysis method, a time information acquisition method, a ciphertext comparison method, and the like are known.

In the above document comparison, it is important for an accurate document update determination to divide a document on a Web site into appropriate block document units and compare them with appropriate block documents. Conventionally, as a method for extracting a document (character string) to be compared, for example, Patent Document 1 discloses a method of extracting a specific character string from character string information such as an e-mail stored in a memory. ing. Patent Document 2 discloses a method for acquiring a specific WWW document from a WWW site.
Japanese Patent Laid-Open No. 11-272703 Japanese Patent No. 2867986

  However, none of those disclosed in the above-mentioned patent documents fully consider the characteristics of documents on a Web site. That is, a document on a Web site includes text data mainly composed of only characters, text data including a hyperlink portion, text data composed mostly of a hyperlink portion, and the like. In accordance with the nature of these documents, a method for dividing a document on a Web site in an appropriate block document unit has not been proposed so far, and there has been a disadvantage that accurate document update determination cannot be performed.

  The present invention has been made in view of such circumstances, and provides a Web document dividing method, system, and program capable of dividing the document into appropriate block document units according to the document type on the Web site. For the purpose.

  The Web document dividing method according to claim 1 of the present invention creates a block document by dividing a document on a Web site in units of first tags indicating blank lines, and the number of all characters included in the block document. If the ratio of the number of link characters to which the second tag for hyperlinks to other websites is given is determined, and the ratio of the number of link characters is higher than a predetermined first threshold, The block document is subdivided in units of a third tag indicating a line feed included in the block document.

  According to this configuration, the document on the Web site is first divided with the blank line portion as a delimiter, and a block document is created. And the content rate of the hyperlink character in this block document is calculated | required. When the content rate of the hyperlink character is high, there is a high possibility that the block document is an enumerated document such as a news title with a link. In this case, it is presumed that the block document includes a plurality of documents having different contents. Therefore, by subdividing the block document in units of the third tag indicating a line break, an appropriate document unit is obtained. Can be separated. On the other hand, when the content rate of the hyperlink character is low, there is a high possibility that the block document is a single news article, blog document, bulletin board document, or the like. If such a block document is divided by the unit of the third tag, a unitary document is subdivided, and in this case, no re-division is performed. Therefore, it is possible to divide the document on the Web site in an appropriate unit according to the content.

  According to a second aspect of the present invention, there is provided a Web document dividing method for generating a block document by dividing a document on a Web site by a unit of a first tag indicating a blank line. If the ratio of the number of link characters to which the second tag for hyperlink to the Web site is given and the ratio of the number of link characters is higher than a predetermined first threshold, If the block document is re-divided in units of third tags indicating line breaks included in the image, and the ratio of the number of link characters is higher than a predetermined second threshold value that is higher than the first threshold value, The block document is subdivided in units of the second tag included in the block document.

  According to this configuration, in addition to the document subdivision requirement, when the content ratio of the hyperlink character is higher than a predetermined second threshold value that is higher than the first threshold value, the block document is used for hyperlinks. Are subdivided in units of the second tag. It is highly likely that a document with a very high hyperlink character content is a block document in which a plurality of link destinations are arranged in one line. In this case, if the block document is subdivided in units of the third tag indicating a line break, it may not be divided into appropriate document units. However, if the subdocument is subdivided in units of the second tag, Can be subdivided into

  In any one of the above configurations, it is desirable that the markup language describing the Web site is in an HTML format. According to this configuration, the Web site document can be easily and appropriately divided using the <br> tag, the <a> tag, and the like.

  According to a fourth aspect of the present invention, there is provided a Web document dividing system including a first document dividing unit that generates a block document by dividing a document on a Web site in units of first tags indicating blank lines, and all the documents included in the block document. Counting means for counting the number of characters and the number of link characters provided with second tags for hyperlinks to other websites, and the ratio of the number of link characters to the number of all characters Determining means for determining whether to re-divide the block document based on the ratio, and a second document dividing means for re-dividing the block document when the determining means determines to re-divide, The determination means determines that the block document is to be subdivided when the ratio of the number of link characters is higher than a predetermined first threshold, 2 Document dividing means is characterized by subdividing the block document in units of third tag indicating a line break included in the block document.

  In this case, it is desirable that the first threshold value is 30%. According to this configuration, it is possible to classify an enumerated document such as a news title with a link and a single news article with a high probability.

  According to a fifth aspect of the present invention, there is provided a Web document dividing system including a first document dividing unit that generates a block document by dividing a document on a Web site in units of first tags indicating blank lines, and all the documents included in the block document. Counting means for counting the number of characters and the number of link characters provided with second tags for hyperlinks to other websites, and the ratio of the number of link characters to the number of all characters Determining means for determining whether to re-divide the block document based on the ratio, and a second document dividing means for re-dividing the block document when the determining means determines to re-divide, The determination means includes a first content rate in which the ratio of the number of link characters is higher than a predetermined first threshold value, and a predetermined value higher than the first threshold value. The block document is determined to be subdivided when the second content rate is higher than two thresholds, and the second document dividing unit indicates a line feed included in the block document when the first content rate is reached. The block document is subdivided in units of third tags, and in the case of the second content rate, the block documents are subdivided in units of the second tag included in the block documents.

  In this case, it is desirable that the first threshold value is 30% and the second threshold value is 85%. According to this configuration, it is possible to determine an enumerated document such as a news title with a link and one news article with a high probability. Further, a block document in which a plurality of link destinations are arranged in one line can be determined with a high probability.

  In any one of the configurations described above, it is preferable that the information processing apparatus further includes character conversion means for converting the second tag and the third tag into special characters. According to this configuration, it is possible to prevent erroneous determination even when the same character as the tag character is included in the block document.

  A Web document dividing program according to claim 9 divides a document on a Web site into units of a first tag indicating a blank line and a block document on a computer capable of analyzing character information and tag information on the Web site. Creating a ratio of the number of link characters provided with second tags for hyperlinks to other websites with respect to the total number of characters included in the block document; and Determining whether the number ratio is higher than a predetermined first threshold; and, if higher than the first threshold, the block document in units of a third tag indicating a line break included in the block document And a step of subdividing.

  A Web document dividing program according to claim 10 divides a document on a Web site into units of a first tag indicating a blank line on a computer capable of analyzing character information and tag information on the Web site. Creating a ratio of the number of link characters provided with second tags for hyperlinks to other websites with respect to the total number of characters included in the block document; and Determining whether the ratio of the number is a first content rate higher than a predetermined first threshold and whether the second content rate is higher than a predetermined second threshold value higher than the first threshold value; In the case of the first content rate, the block document is subdivided in units of a third tag indicating a line feed included in the block document, and in the case of the second content rate. , Characterized in that to execute a step of subdividing said block document in units of the second tag included in the block document.

  According to the Web document dividing method, system, and program as described above, a document mainly composed of only characters, a document including many hyperlink parts or a document composed mostly of hyperlink parts. Can be divided appropriately. Accordingly, it is possible to accurately execute the document update determination performed by dividing the document on the Web site in units of appropriate block documents and comparing the block documents with each other.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a configuration diagram showing a hardware configuration of a document update determination system S to which a Web document dividing method according to the present invention is applied. FIG. 2 is a flowchart schematically showing the overall operation of the document update determination system S. The Web document dividing method according to the present invention can be applied to various uses. Here, an example in which the Web document dividing method is incorporated into the document update determination system S will be given.

  The document update determination system S is connected to the Internet line IN, and the server device 1 that can browse the Web site 3 (WWW site), the personal computer 21 that can communicate with the server device 1, and the mobile phone 22. And the like.

  The Web site 3 is a place where a plurality of various Web contents such as HTML files and image files recorded in a computer (Web server) connected to the Internet line IN are registered. The Web site 3 provides a corresponding HTML file, image file, or the like in response to a user request such as keyword search from the server device 1. FIG. 1 schematically illustrates a website A regarding “news”, a website B regarding “economic news”, a website C as a “bulletin board”, and a website D as a “blog”.

  The operation of the document update determination system S will be described with reference to FIG. The server apparatus 1 is set with an appropriate sampling period (step S1), and performs a cyclic search for the website 3 at predetermined sampling times (YES in step S1) using a predetermined keyword on the Internet. In step S2, document data (HTML file or the like) including the keyword is extracted from various Web contents. Thereafter, the document extracted by the method described in detail later is divided into appropriate block documents (step S3), and it is determined whether or not the updated document (new document) is posted on the website 3 (step S4). ).

  When an updated document is detected, the server device 1 distributes the update date / time, URL (location description information of the document), etc. of the document to the terminal device 2 (step S5). The terminal device 2 is used for confirming the contents of the updated document or performing statistical analysis based on the delivered URL. The holders of the personal computer 21 and the mobile phone 22 that constitute the terminal device 2 are It is possible to immediately know the existence and content of the updated document and perform statistical analysis and the like.

  Here, an outline of the update determination method employed in the document update determination system S of the present embodiment will be described with reference to FIG. As shown in FIG. 3A, it is assumed that there are a first document 41 and a second document 42 to be compared now. For example, if the existing document is the first document 41 and it is determined whether the second document 42 contains any update information (updated document or new document) with respect to the first document 41, the first document 41 Becomes the comparison source document, and the second document 42 becomes the comparison destination document. In this case, if the identity between the first document 41 and the second document 42 is strictly determined, the constituent characters of both documents may be compared with each other (full text comparison method).

  However, the full text comparison method inevitably increases the processing time. Therefore, characters may be extracted from both documents and compared by the following method. That is, as shown in FIG. 3B, the first comparison characters 41a to 41e are extracted from the first document 41 as the comparison source based on a predetermined character extraction condition, and the second document 42 as the comparison destination. The second comparison characters 42a to 42e are extracted based on similar character extraction conditions. The character extraction condition here is an arithmetic expression of intermittent sampling that depends on the number of constituent characters (data length L; variable) of the first document 41 and the second document 42 and the number of extracted characters (number of extracted characters C). Determined by Then, the first comparison characters 41a to 41e are arranged in order to generate the first comparison character array 410, and the second comparison characters 42a to 42e are arranged in order to obtain the second comparison character array 420. Generated.

  Thereafter, as shown in FIG. 3C, the first comparison character 41a in the first comparison character array 410 and the second comparison character 42a in the second comparison character array 420 are used as the first comparison character 41a. The same comparison characters in the comparison character array 410 and the second comparison character array 420 are compared with each other. As a result, if all of the first comparison character array 410 and the second comparison character array 420 are the same, it is determined that the first document 41 and the second document 42 are the same document (no update). On the other hand, if there is a difference between the first comparison character array 410 and the second comparison character array 420, the first document 41 and the second document 42 are different documents (updated; generated including an updated portion). Update document or new document).

  As a simple method for determining the identity between the first document 41 and the second document 42, there is a method of simply comparing the number of constituent characters of both documents. This is because, if the second document 42 is a document created by adding some writing to the first document 41, there is a difference in the number of characters between the two documents. Therefore, by incorporating an element for comparing the number of characters into the update determination method shown in FIG. 3, a more rational update determination can be performed.

  The Web document dividing method according to the present invention is used here to divide a document on a Web site into block document units suitable for comparison, such as the first document 41 and the second document 42 described above. In other words, no matter how good a document update determination method is introduced, if a document to be compared cannot be extracted properly, an accurate update determination cannot be performed in the first place. For this reason, a technique capable of accurately dividing a Web document is important. Hereinafter, a Web document dividing method according to an embodiment of the present invention will be described in detail.

  FIG. 4 is a functional block diagram showing a functional configuration of the server device 1 (Web document dividing system). The server device 1 includes a transmission / reception unit 11, a website search unit 12, a document division processing unit 13, an update determination processing unit 14, and an overall control unit 100.

  The transmission / reception unit 11 is data for enabling data communication between the server device 1 and the terminal device 2 via a predetermined line such as a LAN, and between the Web site 3 via the Internet line IN. It is a communication part. In the present embodiment, the transmission / reception unit 11 transmits search condition information such as a search keyword to the Web site 3 and receives the search result. In addition, when it is determined that there is a document update, a URL indicating the location of the document is transmitted to the terminal device 2.

  The Web site search unit 12 performs a cyclic search of the Web site 3 by setting a predetermined search condition for each sampling time given from the overall control unit 100. For example, for the purpose of regularly monitoring whether a corporation XYZ ○ △ company posts articles related to the company on the Internet, for example, search keywords “XYZ ○ △”, “XYZ”, “○ △”, “ A search for the Web site 3 is performed by setting “X ○” or the like. As a result, the documents 1-1, 1-2, documents 2-1, 2-2, 2-, which are included in the websites A, B, C, D... Of the website 3 shown in FIG. Documents including the keyword are extracted from 3. By performing such a search, it is possible to quickly collect social and customer evaluation information for the corporation XYZ ○ △, and it is possible to quickly respond to unexpected reputations.

The document division processing unit 13 mainly performs the following processing. Details thereof will be described in detail with reference to FIGS.
(1) First division: A document on the Web site 3 is divided into blank line tags (first tags) to create a block document.
(2) Content rate determination: Hyperlink tags (second tags) to other websites are provided for the total number of characters (hereinafter simply referred to as “characters”) such as characters and symbols included in the block document. The ratio of the number of characters (link characters) is obtained.
(3) Second division A; when the ratio of the number of hyperlink characters is higher than a predetermined first threshold (determination 1), the unit is a unit of a line feed tag (third tag) included in the block document. Subdivide the block document.
(4) Second division B: included in the block document when the ratio of the number of hyperlink characters is higher than a predetermined second threshold set higher than the first threshold (decision 2) The block document is subdivided in units of hyperlink tags.
(5) Maintenance of the first division; when the ratio of the number of hyperlink characters is lower than the predetermined first threshold (determination 0), the block document is not subdivided.

  The update determination processing unit 14 performs a document update determination process using the algorithm described above with reference to FIG.

  The overall control unit 100 includes a CPU (Central Processing Unit) and the like, and manages operations of various functional units in the server device 1. For example, the overall control unit 100 gives a search instruction signal to the Web site search unit 12 at a predetermined sampling period to search the Web site 3 via the transmission / reception unit 11, or the document division processing unit 13 and the update determination processing unit. 14 to execute division processing and determination processing according to a predetermined sequence, or to distribute the URL information of the update document to the terminal device 2 when the update determination processing unit 14 detects an update document. Is to do.

  FIG. 5 is a block diagram showing the functional configuration of the document division processing unit 13 in detail. The document division processing unit 13 includes a RAM (Random Access Memory) 131, a division condition setting unit 132, a threshold setting unit 133, a tag conversion unit 134 (character conversion unit), a first division unit 135 (first document division unit), the number of characters A counting unit 136 (counting unit), a determining unit 137 (determining unit), and a second dividing unit 138 (second document dividing unit) are provided.

  The RAM 131 temporarily stores document data of documents hit by the search by the website search unit 12 and various data generated during the division process.

  The division condition setting unit 132 stores the definition of tag information that is a key when performing the first division, the second division A, and the second division B. FIG. 6 is a table format showing an example of the definition of such tag information. Here, a case where the markup language describing the Web site 3 is in the HMTL format is illustrated. As shown in FIG. 6, in the present embodiment, “first division” is performed in units of “<br> <br> tag” (first tag) indicating “blank row”, and “second division A”. Is performed in units of “<br> tag” (second tag) indicating “line feed”, and “<a href> tag” (third tag) unit in which “second division B” indicates “hyperlink” It shows an example that is set to be performed in.

  The threshold value setting unit 133 defines a threshold value for the hyperlink character content rate (link character content rate) m, which is a criterion for determining whether only the first division or the second division A or the second division B is performed. Remember. FIG. 7 is a table format showing an example of the threshold definition of the content ratio m. Here, when m = 30% (first threshold value) and m = 85% (second threshold value) are selected as the threshold values, and “determination 0” in which only the first division is performed is “m <30%”. When “determination 1” in which the second division A is performed is “30% ≦ m ≦ 85%”, and “determination 2” in which the second division B is performed is “85% <m”. It shows an example where each is set to be done.

  The numerical values (30% and 85%) of the first threshold value and the second threshold value are obtained as a result of document structure analysis on various well-known Internet websites (various news sites, blog sites, bulletin board sites). Parameters. This numerical value is an example, and can be set to an appropriate numerical value based on the relationship of the first threshold value <the second threshold value. However, as will be described later, from the viewpoint of accurately determining the document classification based on the hyperlink character content rate, the first threshold value is in the range of 20% to 40%, preferably 25% to 35%. The threshold value is desirably selected from the range of 75% to 95%, preferably 80% to 90%. Note that the first threshold value and the second threshold value may be variable according to the type of website, the search keyword, the sampling period, and the like.

  The tag conversion unit 134 performs character conversion processing on the “<br> <br> tag”, “<br> tag”, and “<a href> tag”. “<br> <br> tag” is a tag that is a key for document division (first division) that is always performed for any document. In order to recognize this tag reliably, for example, “<br> <br> tag” can be converted into a character string “_empty_line_” (hereinafter referred to as “special character A”) and used as a delimiter. Like that.

  Further, the tag conversion unit 134 prevents the “<br> tag” and the “<a href> tag” from being erroneously determined even when the same character is included in the block document, and If necessary, convert it to a special character so that it can be used as a delimiter later. As this special character, a character string that does not appear in a normal document is selected. For example, “<br> tag” is a character string “_Kai_line_” (hereinafter referred to as “special character C”), and “<a href> tag” is a character string “_Lin_Ku_”. (Hereinafter referred to as “special character B”).

  The first dividing unit 135 divides the document hit by the search by the Web site searching unit 12 in units of “<br> <br> tag” that specifies display of blank lines, and creates a block document. FIG. 8 is a schematic diagram showing a document dividing operation by the first dividing unit 135. Here, an example is shown in which a web page 50 describing economic news includes a first document 51, a second document 52, a third document 53, and a fourth document 54 across a blank line. The first to fourth documents 51 to 54 are documents describing news articles having different contents. The first dividing unit 135 reads “special character A” described as a blank line display designation between the first to fourth documents 51 to 54 as a delimiter and reads the document in units of the “special character A”. The first to fourth documents 51 to 54 are divided into first to fourth block documents on the document data. The first to fourth block documents divided in this way are temporarily stored in the RAM 131.

  When a plurality of descriptive documents having different contents are posted across a blank line, such as the first to fourth documents 51 to 54 of the Web page 50 shown in FIG. 8, the first to fourth documents 51 are no more. It can be said that there is no need to subdivide .about.54. On the other hand, for example, if the subdivision is performed in units of “<br> tag” indicating a line feed, the following problem occurs.

  FIG. 9 shows an HTML document 51A representing the first document 51 of FIG. 8 in the HTML description format. In the HTML document 51A, a blank line tag 511 is described at the beginning and end of the first document 51, and a line feed tag 512 is described at a position corresponding to the line feed position of the first document 51. In this case, if it is divided by the blank line tag 511 and then re-divided by the line feed tag 512, a group of documents (topics) is subdivided into a plurality of block documents.

  If such subdivision is performed, when the first document 51 is updated between a plurality of line feed tags 512 including the new publication of the first document 51 in the subsequent document update determination, In spite of a group of documents, there is a high possibility that it will be determined that a plurality of documents have been updated. As a result, the user cannot be notified accurately and the accuracy of statistical analysis or the like is reduced.

  In addition, a problem occurs when a Web document hit by the search by the Web site search unit 12 is further searched using a keyword. For example, attention is paid to the keyword KE1 (FIG. 8) “image data” included in the first document 51. Since the keyword KE1 happens to straddle a line, it is separated by <br> in the line feed tag 512 when viewed in the HTML document 51A (FIG. 9). Therefore, even if a search is performed with the keyword KE1 “image data”, the first document 51 does not hit.

  Further, for example, a problem occurs when a search is performed using an AND condition using a keyword KE2 “image sensor” and a keyword KE2 “digital camera”. That is, since the keywords KE2 and KE3 belong to another document when the first document 51 is divided in units of line feed tags 512, the AND condition is not satisfied and the first document 51 is hit in such a search. Result. From the above, it is desirable that a document such as the first document 51 should not be subdivided after the first division that is divided in units of blank line tags 511.

  However, it is not always possible to divide only by blank line tag units. FIG. 10 is a schematic diagram showing a Web page 60 in which a plurality of news titles are listed in different rows. The web page 60 includes first to fourth news titles 61 to 64. Each title is given a click operation by the user, and another web page (for example, a web page describing the details of the news shown in FIG. 8). A hyperlink (underlined part in FIG. 10) that jumps to is set. For example, in the first news title 61, a hyperlink is incorporated in the letters “ABC develops next-generation XYZ technology”. The hyperlink is not incorporated in the characters “22 days 15:30”.

  FIG. 11 shows an HTML document page 60A representing the Web page 60 of FIG. 10 in the HTML description format. The HTML document page 60A includes first to fourth HTML documents 61A to 64A corresponding to the first to fourth news titles 61 to 64. Also, blank line tags 611 are described at the beginning and end of the group of the first to fourth news titles 61 to 64, and line feed tags 614 are described at positions corresponding to the line feed positions of the titles 61 to 64, respectively. Further, hyperlink tags 612 and 613 are described in the first to fourth HTML documents 61A to 64A, respectively.

  In this case, the first to fourth news titles 61 to 64 are title documents having different contents. If such a document is left divided in units of blank line tags 611, a plurality of uncoordinated documents will be treated as one block document. Therefore, if only one of the first to fourth news titles 61 to 64 is updated, it is determined that the document has been updated in the subsequent document update determination. Therefore, for example, the occurrence of a document having contents such as the first to third news titles 63 is targeted for update notification, but the occurrence of a document having contents such as the fourth news title 64 is not targeted for update notifications. The inconvenience that it is determined as “updated” by sampling after the fourth news title 64 is added to the page 60 occurs.

  From the above, in title documents such as the first to fourth news titles 61 to 64, it is desirable to divide them in units of blank line tags 611 and then subdivide them in units of line feed tags 614. By such subdivision, a plurality of documents having different contents can be appropriately divided.

  By the way, in many cases, title documents with different contents, such as the first to fourth news titles 61 to 64, include documents with hyperlinks, that is, many hyperlink characters. It can be said that it is a document. On the other hand, the news article document as shown in FIG. 8 contains almost no hyperlink characters. Therefore, whether or not the document is a title document can generally be determined by the content ratio of the hyperlink character with respect to the number of characters of the entire block document.

  Regarding the hyperlink character content rate, if the discrimination threshold (first threshold) is set too low, there is a high possibility that a coherent document will be subdivided. For example, a case where hyperlinks HP1 to HP4 are incorporated only for a predetermined keyword as in a block document 71 shown in FIG. As described above, there are many document forms on a website that link to a predetermined keyword and jump to another web page that displays detailed information about the keyword when a click operation is given.

  In this case, the characters (underlined portions) to which the hyperlinks HP1 to HP4 are attached are only the character strings “ABC company”, “imaging device”, “image processing”, and “digital camera”, and the entire block document 71 From the number of characters, the ratio is small. However, since hyperlink characters are included, if the first threshold value is set too low, such a block document 71 is also subdivided in units of line feed tags. In order to prevent such a problem, as described above, the first threshold value is desirably about 30%.

  On the other hand, even when the content rate of the hyperlink character is higher than the first threshold, a problem may occur if the block document is re-divided in units of line feed tags. FIG. 13 is a schematic diagram showing a Web page 72 in which a plurality of linked sites are listed in one line. Here, an example is shown in which the link destinations of a plurality of newspaper companies are listed. The link destinations of six newspaper companies in the first row 721 and the link destinations of six newspaper companies in the second row 722 are shown. Are listed.

  In this way, in the case of a linked document that contains different character strings on one line, even if the block document is subdivided in units of line feed tags, Don't be. Therefore, in the case of a linked document as shown in FIG. 13, the block document may be subdivided in units of hyperlink tags. Since the hyperlink tag is described for each link destination, for example, a block document composed of the first row 721 and the second row 722 is subdivided into units of “AAA newspaper”, “BBB civil news”, and so on. can do.

  Such a linked document can be said to have a very high content of hyperlink characters. Therefore, it is desirable to determine whether the document is a linked document based on the second threshold set to a level considerably higher than the first threshold. For this reason, the second threshold is set to about 85% as described above. Is desirable.

  Returning to FIG. 5, in view of the above points, the character count unit 136 determines the number of characters of each block document created by the division process by the first division unit 135 and the hyperlink characters included in the block document. Count the number of characters.

  The determination unit 137 obtains the ratio of hyperlink characters to the total number of characters (hyperlink character content ratio m) based on the count result of the character count unit 136, and determines whether to re-divide the block document according to the percentage. decide. The determination unit 137 refers to the first threshold and the second threshold defined in the threshold setting unit 133 during the determination operation. Therefore, after determining the content ratio m of one block document, the determination unit 137 determines “determination 0”, “determination 1”, and “determination 2” for the block document based on the reference shown in FIG. Do.

  When the determination unit 137 outputs “determination 1”, the second division unit 138 subdivides the block document in units of special characters C corresponding to line feed tags, and when the determination unit 137 outputs “determination 2”. The block document is subdivided in units of special characters B corresponding to hyperlink tags. On the other hand, the second dividing unit 138 does not perform re-division processing when the determining unit 137 outputs “determination 0”. In this case, the block document maintains a state of being divided by the special character A corresponding to the blank line tag.

  The flow of the Web document dividing process according to the present embodiment described above will be described based on the flowcharts shown in FIGS. First, one of the Web site documents (HTML document) hit by the search by the Web site search unit 12 is read from the RAM 131 by the tag conversion unit 134 (step S11).

  Next, the tag conversion unit 134 performs processing for converting “<br> <br> tag” to “special character A” (step S12). Subsequently, the tag conversion unit 134 converts “<a href> tag” to “special character B” (step S13) and “<br> tag” to “special character C” (step S14). ) And are executed. Then, after processing (step S15) for deleting other tags (comment tags, line feed codes, etc.), tabs, half-width / full-width spaces, etc. included in the HTML document is performed, the document data is converted into the first division unit. It is output to 135.

  In response to this, the first dividing unit 135 divides the document data in units of the “special character A” converted in step S12 to create one or a plurality of block documents (step S16). Further, the first dividing unit 135 performs numbering 1 to n on the obtained block document (step S17), and data is temporarily stored in the RAM 131 in association with the number.

  The counter in the computer is set to K = 1 (step S18), the first block document K is read from the RAM 131, and the determination process shown in FIG. 15 is performed. In this determination processing, first, the number-of-characters counting unit 136 counts the number of characters of the first block document K and the number of hyperlink characters included in the block document K. And the content rate m of a hyperlink character is calculated | required by the determination part 137 (step S19).

  Next, the necessity of re-division is determined based on the obtained content rate m (step S20). When the content ratio m satisfies the condition of “m> 85%” (YES in step S20), the determination unit 137 outputs “determination 2” to the second division unit 138. In response to this, the second dividing unit 138 subdivides the block document K in units of “special character B” (step S21). After that, the process of deleting “special character A”, special character B ”, and“ special character C ”from the block document K is performed (step S24), and then the update determination processing unit 14 in the next stage stores the block document K. Data is sent.

  On the other hand, when the content rate m does not satisfy the condition “m> 85%” (NO in step S20), the determination unit 137 subsequently determines whether the condition “m ≧ 30%” is satisfied. (Step S22). When the condition of “m ≧ 30%” is satisfied (YES in step S22), the determination unit 137 outputs “determination 1” to the second division unit 138. In response to this, the second dividing unit 138 subdivides the block document K in units of “special character C” (step S23). Thereafter, step S24 described above is executed. On the other hand, when the condition “m ≧ 30%” is not satisfied (NO in step S22), the determination unit 137 outputs “determination 0” to the second division unit 138. In response to this, the second dividing unit 138 passes through the block document K without performing re-division processing. That is, the process skips to step S24.

  Thereafter, it is determined whether or not the counter is K = n (step S25). If K = n is not satisfied (NO in step S25), since another block document remains for one read HTML document, the counter is incremented by 1 as K = K + 1 (step S26). Returning to step S19, the same processing is repeated for the next block document K (second block document).

  On the other hand, if K = n (YES in step S25), it is determined whether or not the division process is to be executed for another Web site document (step S27). If another Web site document exists (YES in step S27), the process returns to step S11 in FIG. 14 and the same processing as described above is repeated for the Web site document. On the other hand, if there is no other Web site document (NO in step S27), the process ends.

  According to the Web document dividing method according to the present embodiment described above, a document mainly composed only of characters, a document including many hyperlink parts or a document composed mostly of hyperlink parts. Can be divided appropriately. Accordingly, it is possible to accurately execute the document update determination performed by dividing the document on the Web site in units of appropriate block documents and comparing the block documents with each other.

  As mentioned above, although embodiment was described about this invention, this invention is not limited to this, For example, the deformation | transformation embodiment as shown below can be taken.

[1] In the above embodiment, the case where the markup language for describing the Web site is in the HTML format has been described as an example. The present invention can of course be applied to other markup languages, for example, an XML format.

[2] In the above embodiment, the example in which the Web document dividing method according to the present invention is incorporated in the document update determination system S has been described. In addition to this, the present invention can be applied to various document retrieval, document data analysis, statistical processing, and the like.

[3] The Web document dividing method performed by the document update determination system S can be provided as a program. Such a program can be recorded on a computer-readable recording medium such as a flexible disk, a CD-ROM, a ROM, a RAM, and a memory card attached to the computer and provided as a program product. Alternatively, the program can be provided by being recorded on a recording medium such as a hard disk built in the computer. A program can also be provided by downloading via a network.

It is a block diagram which shows the hardware constitutions of the document update determination system S to which the Web document division | segmentation method based on this invention was applied. It is a flowchart which shows roughly the whole operation | movement of the document update determination system S which concerns on this embodiment. It is explanatory drawing for demonstrating the outline | summary of the document update determination in this embodiment. It is a functional block diagram which shows the function structure of a server apparatus. It is a block diagram which shows the function structure of a document division process part in detail. It is a figure of a table format which shows an example of a definition of tag information. It is a figure of the table format which shows an example of the threshold value definition of the content rate m. It is a schematic diagram which shows an example of a document division | segmentation operation | movement. It is a schematic diagram which shows the HTML document 51A which represented the 1st document 51 of FIG. 8 in the HTML description format. It is a mimetic diagram showing a Web page where a plurality of news titles are listed in different rows. It is a schematic diagram which shows the HTML document page which represented the Web page of FIG. 10 in the HTML description format. It is a schematic diagram which shows an example of a block document. It is a mimetic diagram showing a Web page where a plurality of linked sites are listed in one line. It is a flowchart which shows the flow of the Web document division | segmentation process which concerns on embodiment of this invention. It is a flowchart which shows the flow of the Web document division | segmentation process which concerns on embodiment of this invention.

Explanation of symbols

1 Server device (Web document splitting system)
11 Transmission / Reception Unit 12 Web Site Search Unit 13 Document Division Processing Unit 131 RAM
132 Division condition setting unit 133 Threshold setting unit 134 Tag conversion unit (character conversion means)
135 1st division part (1st document division means)
136 Character count section (counting means)
137 determination part (determination means)
138 Second division unit (second document division unit)
14 Update determination processing unit 2 Terminal device 3 Web site (WWW site)

Claims (10)

A block document is created by dividing a document on a website in units of first tags indicating blank lines,
Determining the ratio of the number of link characters provided with the second tag for hyperlinks to other websites to the total number of characters included in the block document;
When the ratio of the number of link characters is higher than a predetermined first threshold, the block document is subdivided in units of a third tag indicating a line break included in the block document. Split method. A block document is created by dividing a document on a website in units of first tags indicating blank lines,
Determining the ratio of the number of link characters provided with the second tag for hyperlinks to other websites to the total number of characters included in the block document;
When the ratio of the number of link characters is higher than a predetermined first threshold, the block document is re-divided in units of a third tag indicating a line feed included in the block document,
When the ratio of the number of link characters is higher than a predetermined second threshold value that is higher than the first threshold value, the block document is subdivided in units of the second tag included in the block document. A Web document dividing method characterized by the above.   3. The Web document dividing method according to claim 1, wherein a markup language for describing the Web site is in an HTML format. First document dividing means for dividing a document on a Web site by a unit of a first tag indicating a blank line and creating a block document;
Counting means for counting the number of all characters included in the block document and the number of link characters provided with second tags for hyperlinks to other Web sites;
Determining means for determining the ratio of the number of linked characters to the total number of characters and determining whether to re-divide the block document based on the ratio;
A second document dividing unit that subdivides the block document when the determination unit determines to subdivide,
The determination means determines that the block document is subdivided when the ratio of the number of link characters is higher than a predetermined first threshold,
The Web document dividing system, wherein the second document dividing unit subdivides the block document in units of a third tag indicating a line feed included in the block document. First document dividing means for dividing a document on a Web site by a unit of a first tag indicating a blank line and creating a block document;
Counting means for counting the number of all characters included in the block document and the number of link characters provided with second tags for hyperlinks to other Web sites;
Determining means for determining the ratio of the number of linked characters to the total number of characters and determining whether to re-divide the block document based on the ratio;
A second document dividing unit that subdivides the block document when the determination unit determines to subdivide,
The determination means has a first content rate in which the ratio of the number of link characters is higher than a predetermined first threshold value, and a second content rate that is higher than a predetermined second threshold value that is higher than the first threshold value. In some cases, the block document is decided to be subdivided,
The second document dividing means subdivides the block document by a unit of a third tag indicating a line feed included in the block document in the case of the first content rate, and the block in the case of the second content rate. A Web document dividing system, wherein the block document is subdivided in units of the second tag included in the document.   The Web document dividing system according to claim 4, wherein the first threshold is 30%.   6. The Web document dividing system according to claim 5, wherein the first threshold value is 30% and the second threshold value is 85%.   8. The Web document dividing system according to claim 4, further comprising character conversion means for converting the second tag and the third tag into special characters. A computer that can analyze character information and tag information on a website.
Dividing a document on a website in units of first tags indicating blank lines to create a block document;
Obtaining a ratio of the number of link characters provided with a second tag for hyperlink to another website to the total number of characters included in the block document;
Determining whether the ratio of the number of link characters is higher than a predetermined first threshold;
Re-dividing the block document in units of a third tag indicating a line break included in the block document if higher than the first threshold;
Web document segmentation program characterized by causing A computer that can analyze character information and tag information on a website.
Dividing a document on a website in units of first tags indicating blank lines to create a block document;
Obtaining a ratio of the number of link characters provided with a second tag for hyperlink to another website to the total number of characters included in the block document;
It is determined whether or not the ratio of the number of link characters is a first content rate that is higher than a predetermined first threshold value, and whether or not it is a second content rate that is higher than a predetermined second threshold value that is higher than the first threshold value. And steps to
In the case of the first content rate, the block document is subdivided in units of a third tag indicating a line feed included in the block document, and in the case of the second content rate, the block document is included in the block document. Subdividing the block document in units of the second tag
Web document segmentation program characterized by causing

Images