JP5564442B2 - Text search device - Google Patents

Description

  The present invention relates to an apparatus for searching and outputting a document on a network, and more particularly to a sentence search apparatus that can extract only main sentences from a document at the time of output.

  Currently, there are a large number of documents (hereinafter referred to as Web documents) that are specified in the URL format and described mainly in the HTML format on the network. However, many Web documents are used to navigate to advertisements, links to related sites, and other Web documents in the same site, in addition to the main text (called the main text) of the Web document. Includes links.

  When displaying such a Web document on a terminal with a limited size such as a portable terminal or a signage display installed in a public space, only the text may be extracted and output. desired.

  Also, when searching for a web document on a network by entering a keyword using a search engine, the keyword is included in an advertisement, a link to a related site, or a link for navigating to another web document in the same site. Since hitting does not necessarily match the purpose of the search, it is desirable that only the text is extracted.

However, in HTML that describes a Web document, there is no particular difference in the structure of tags based on the text, advertisements, navigation links, etc., so it is not always easy to extract only the text.

  As a conventional technique that enables such a request, according to Patent Document 1, a technique for extracting a portion estimated to be a text by converting a Web document data into plain text and applying a statistical method is disclosed. Has been.

  As a related background technology, a technology called RSS is generally known. RSS is an abbreviation of “RDF Site Summary” or “Rich Site Summary” or “Really Simple Syndication”, and there are differences in standards depending on multiple versions like Reference Sites 1 and 2, but what is common is that It is a technology that delivers an outline of the delivered Web document in XML format. As an outline, in most cases, the title of the text of the Web document and a partial excerpt of the text are included. At present, RSS is also distributed on a large number of sites, especially those that are frequently updated, such as news sites and blog sites. In addition, as a technology similar to RSS, there is ATOM of reference site 3, and it is possible to describe and distribute an outline of a Web document in XML format as with RSS. Hereinafter, RSS versions and ATOM are collectively referred to as RSS. Since RSS stipulates which tags describe items such as titles and text excerpts, unlike HTML, it is easy to extract each item by analyzing the tag structure. Note that there are differences between RSS versions, but each version's RSS (including ATOM) describes the version itself, so that information is obtained and analyzed according to the described version. By performing the above, the problem of differences due to versions can be avoided.

Reference site 1: http://web.resource.org/rss/1.0/spec
Reference site 2: http://www.rssboard.org/rss-specification
Reference site 3: http://www.ietf.org/wg/concluded/atompub

JP 2006-338364 A

  However, since the method of Patent Document 1 uses a statistical method, it is necessary to collect sufficient data to perform statistically significant estimation. Or when these data are not enough, an effective estimation result cannot be obtained. In addition, there is a problem that even if sufficient data is collected, there is an inevitable estimation error.

When RSS is used, there is no problem due to such a statistical method because data distributed by XML may be analyzed according to the standard. However, in many cases, only a text excerpt is delivered by RSS, and only a small part of the text content is included. That is, there is a problem that it is not possible to acquire all the information of the text only from RSS.

  Moreover, most current Web documents include a design designation called CSS, and when a user browses a Web document, it is viewed in a state displayed based on the designation. On the other hand, in any conventional technique, only data related to text data of Web documents is mainly extracted. For this reason, the design of the original Web document cannot be applied to the results extracted by these conventional techniques. Therefore, when the user sees the display of the extracted results, the results are extracted from the original Web document. There is a problem that it is difficult to recognize the result.

  In order to solve these problems, the RSS information distributed with the Web document is used, and the part very similar to the “title” described in the RSS feed is converted to the tree structure data (DOM) converted from HTML. The text of the Web document is extracted by searching with and specifying it while considering its style as necessary.

The first aspect of the present invention includes an outline document acquisition unit having a function of reading data distributed in a predetermined format including an access destination and a title related to a document on the network and retrieving the access destination and the title related to the document; A tree structure data storage for accessing an access destination of a document extracted by the summary document acquisition unit, acquiring document data and style data of the document, and converting the acquired document data into tree structure data and storing it And the text retrieved at the node of the tree-structured data stored in the tree-structured data storage unit based on the measured similarity Title node search unit having a function of specifying a title on the data of the tree structure, and the title node While tracing the nodes from the title of the data of the identified tree structure to the root of the tree structure by the cord section, by the style node style based on the data to determine whether or not a predetermined condition is satisfied, the A sentence search device comprising a sentence node search unit having a function of specifying a node including a main sentence on tree-structured data is used as a solution means.

The second aspect of the present invention is a summary document acquisition unit having a function of reading data distributed in a predetermined format including an access destination and a title relating to a document on the network and retrieving the access destination, title and summary relating to the document. A tree structure that accesses an access destination of the document extracted by the summary document acquisition unit, acquires document data and style data of the document, and converts the acquired document data into tree-structured data and stores it The similarity between the data storage unit, the title extracted by the summary document acquisition unit, and the text data in the node of the tree structure data stored in the tree structure data storage unit is measured, and the measured similarity is obtained. A title node search unit having a function of specifying a title on the data of the tree structure based on the title; While tracing the node from over de search on the data of the identified tree structure of the unit title to the root of the tree structure, from the beginning of the text of the to be compared with the acquired outline of text by summary document acquisition unit node A sentence node search unit having a function of identifying a node including a main sentence on the data of the tree structure based on a Levenshtein distance from a text cut out in accordance with a predetermined rule. A text search device is used as a solution.

  According to a third aspect of the present invention, in the first or second aspect of the present invention, by deleting or hiding a node that is not in a parent-child relationship with the node on the tree structure data specified by the sentence node search unit A sentence retrieval apparatus comprising an output unit having a function of displaying only the main sentence of the document is used as the solving means.

  According to the present invention, it is possible to extract a main sentence in a Web document with high accuracy without depending on a statistical method. In addition, since it can be specified on the DOM, it is possible to obtain an extraction result while maintaining the design.

It is a lineblock diagram of a text search device concerning a 1st embodiment. It is a figure which shows the processing flow (1) of the summary document acquisition part 11. FIG. It is a figure which shows the processing flow (2) of the summary document acquisition part. It is a figure which shows the processing flow (1) of the title node search part. It is a figure which shows the processing flow (2) of the title node search part 13. FIG. It is a figure which shows an HTML sample. It is a figure which shows the example of a display of an HTML sample. It is a figure which shows the DOM conversion example of an HTML sample. 1 is a configuration diagram of a text search device 1 including an output unit 15. FIG. It is a figure which shows the example of a display before applying this Embodiment. It is a figure which shows the example of a display after applying this Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is an overall configuration diagram of a text search apparatus according to an embodiment of the present invention.

  The text search apparatus 1 acquires a summary document acquisition unit 11 that acquires summary data of the Web document distributed by RSS, acquires the Web document data, and converts the acquired document data into tree-structured data. A tree structure data storage unit 12 to store, a title node search unit 13 for searching and specifying a node that is a title of a main sentence of a Web document from tree structure data, and a tree structure starting from the node that is the specified title And a text node search unit 14 for searching and specifying a main text node from the above data.

  This embodiment can be implemented by running a program for operating the functions of each part as described below on a general-purpose computer connected to a network and having a CPU, memory, HDD, etc. Is possible. Each unit can be executed on the same computer, or each unit can function on an individual computer, and necessary parameters can be transmitted by network communication to be executed on a plurality of computers. It is.

(Operation of the first embodiment)
The processing flow of the summary document acquisition unit 11 is shown in FIGS.

  First, a request is made by HTTP for a URL designated for acquiring RSS, and a response (RSS data) is received (S1). Regarding the specification of the URL, many websites specify a link for acquiring RSS related to the Web document distributed there, so there is a method in which the user sets the link destination URL in advance. . Also, in many current Web documents, the RSS acquisition destination may be embedded in the meta tag, so it can be configured to analyze and access the contents of the meta tag. .

  When accessed, the RSS data is returned in XML format, and the XML data is analyzed (S3). For example, if you use the library software called Universal Feed Parser disclosed at http://www.feedparser.org/, the difference between RSS versions (including ATOM) is absorbed, and the acquired data is analyzed. It becomes possible. In the implementation of the present invention, the use of Universal Feed Parser is not essential. However, the tag names of items that should be obtained by analysis differ slightly depending on the RSS version. In the following explanation, the Universal Feed Parser document [http://www.feedparser.org/docs/reference.html] Write according to the name used in.

  RSS contains information about multiple Web documents. Information of each Web document can be acquired as an array called entries, and the title (entries [i] .title) and access URL (entries [i] .link) are acquired from the information of the i-th Web document. To do. There are various items specified in RSS, but there are not necessarily many items that are actually set at the time of distribution. However, this title and access URL are included in almost all RSS distribution data.

  There are several implementation methods for the detailed implementation of the operation after the analysis, depending on the purpose of use of this embodiment, and three examples will be described below.

If the URL to be extracted has been determined in advance (the URL is set as Target_URL), search for an entry with matching entries [i] .link from the array, along with entries [i] .title, and the title node search unit 13 is passed as a parameter. (Figure 2: S5, S7)
If not particularly determined, for example, entries [0] .title and entries [0] .link at the beginning of the array may be passed as parameters to the title node search unit 13 (not shown).

  Alternatively, entries [i] .title and entries [i] .link may be sequentially passed as parameters to the title node search unit 13 while i is incremented by 1 after a certain time (FIG. 3: S11 to S15). .

  The tree structure data storage unit 12 makes a request to the designated URL by HTTP, and acquires HTML as a response. The acquired HTML is converted into a tree-like data structure according to the specification of Document Object Model (DOM) [http://www.w3.org/DOM/DOMTR], and stored in a memory or the like. In the DOM data having a tree structure, each tag data of HTML constitutes a node of the tree structure, and the node by the BODY tag is the highest as displayed data. Furthermore, Cascading Style Sheet (CSS), which is part of the DOM rules, is also analyzed, and styles such as size in each data structure are converted. There are cases where CSS is described directly in the HTML of the Web document, and where the CSS acquisition URL is written in the HTML and needs to be acquired by accessing the URL. Such conversion including reading of external CSS such as the latter is a well-known technique that is realized in most current browser software.

  For example, the following data search operation can be easily performed on a Web document expanded into tree-structured data called DOM by using a technique generally known as JavaScript.

・ Search the upper node and lower node of a specific node according to the tree structure ・ Search the node data on the DOM with a specific tag name ・ Search the node data on the DOM with a specific attribute value -Search for attribute values of a specific DOM node-Search for text data belonging to a specific DOM node-Search for text data belonging to all nodes under a specific DOM node-Text data that normally belongs Search for the tag's alt attribute value data as an alternative to text data in a tag that does not work (eg, img tag)-Search for the display / non-display status of a specific DOM node-Display a specific DOM node In this state, the size (height and width) on the display is searched. Also in this embodiment, the tree structure data storage unit is obtained by these known techniques. 12 can provide the above-described operation functions to other components of the present invention.

  The processing flow of the title node search unit 13 is shown in FIGS.

  The operation of the title node search unit 13 in the present embodiment can be performed by either processing flow, but first, a simpler FIG. 4 will be described.

  The title node search unit 13 is activated by passing the title (title) and the access destination URL (link) as parameters from the summary document acquisition unit 11 (S101).

  First, the DOM data of the URL passed as link is acquired by accessing the tree structure data storage unit 12 (S103). If the DOM data of the URL is not yet stored in the tree structure data storage unit 12, the tree structure data storage unit 12 may be instructed to obtain and store it via the network.

  Text delivered by RSS as a title is often described by any one of tags H1, H2, H3, H4, and H5 (hereinafter collectively referred to as a heading tag) in HTML. Therefore, first, the heading tag node is extracted from the DOM, and the text information contained under each extracted node is acquired. When the tag is img, the alt attribute value may be used as a text alternative. These operations can be performed using the functions provided by the tree structure data storage unit 12.

  Next, the text data acquired in this way is compared with the title passed as a parameter to search for data with high similarity.

  In calculating the similarity, first, each text data is normalized. Here, normalization refers to the following processing, which reduces the difference in notation as text data.

・ Unify text encoding (for example, all unicode conversion)
・ Unify full-width and half-width display of alphanumeric symbols such as katakana (for example, all half-width conversion)
・ Unify uppercase and lowercase letters (for example, convert all lowercase letters)
・ Unify white space (for example, convert all double-byte spaces, tabs, and line feeds to single-byte spaces, replace two or more consecutive single-byte spaces with single-byte single-byte spaces. )
After normalization in this way, the text data of the title and heading tag are compared, and matching heading tag data is obtained.

Depending on the site, some characters may be added during RSS distribution, or some characters may be deleted if the number of characters exceeds the number determined by each site. May be replaced with another character. In that case, even if normalization does not result in the same text, relax the condition of matching, for example,
・ One text is included in the other text ・ Heading tag data that meets one or more of the following conditions: The Levenshtein distance between two text data is calculated and the distance is smaller than a predetermined condition May be obtained. Note that the Levenshtein distance is a numerical value that measures how many character units can be converted from one text data to the other text data. That is, the smaller the value, the more similar the two text data are. It can be judged. It can be calculated by a method disclosed in [http://www.merriampark.com/ld.htm].

  In this way, the heading tag node determined to be “high in similarity” is stored in the temporary arrangement area as a candidate for the title node to be specified (S105, S1051, S1052).

  The acquired heading tag node is not necessarily one. That is, a plurality of nodes may be stored in the candidate sequence area. Therefore, when a plurality of heading tag nodes are acquired, an optimum node is selected by the following method, for example, based on the CSS of each data (S107).

・ The one with the largest font size ・ If there are multiple ones with the largest font size, the one on the upper side when placed on the screen The node selected in this way is the data representing the title of the text Then, it is sent as a parameter to the next sentence node search unit 14 (S109).

  Next, FIG. 5 which is a preferred embodiment from FIG. 4 will be described.

  In the flow of FIG. 4, there is a possibility that there is no heading tag data that matches the condition in determining the similarity. In HTML, there is a case where it is described so that it can be recognized as a title for the user by setting an appropriate CSS by using the TD tag in the DIV tag or TABLE tag without using the heading tag. Think mainly. In order to deal with such a case, it is only necessary to compare the normalized text and select the optimum tag as described above for all tags in the display state, not limited to the heading tag. Whether or not it is in the display state can be determined using the function of the tree structure data storage unit 12. If such a process is performed, the possibility that none of them will be selected is reduced, but the target tag data increases, which may cause a processing time problem. Therefore, an implementation method in which processing is performed on the heading tag first, and processing is performed on all display tag data only when it is not found from the heading tag is more preferable. Even in the process for all display tag data, if the title cannot be searched, it is determined that the HTML does not correspond to RSS for some reason, and the process is terminated as an error.

  The flow diagram of FIG. 5 is an implementation of this. That is, from step S101 to S1052, the same processing as in FIG. 4 is performed, and the length of the array temporarily stored as a candidate is checked in step S111. When the length is 1 or more, the same processing as in FIG. 4 is performed, but when the length is 0, the processing proceeds to step S113 and subsequent steps. In steps S113, S1131, and S1132, the same processing as in FIG. 4 is not limited to the heading tag node, and processing is performed on tag nodes in all display states.

  That is, Steps S1131 and 1132 described later are performed on all display state tags on the DOM (S113). In step S1131, the similarity between the text information under the target tag and the parameter title is calculated. In subsequent step S1132, if the similarity is greater than a predetermined value, the target tag is added to the “candidate” array.

  Next, in step S115, the length of the array temporarily stored as a candidate is checked. When the length is 1 or more, the same processing as in FIG. 4 is performed (S107, S109). If the length is 0, the process is terminated with an error (S117).

  The text node search unit 14 receives the title position on the DOM searched by the title node search unit 13 and extracts the subsequent text. In the present invention, a portion where the title and the text are combined is called a main sentence. For that purpose, the DOM that has a tree-like data structure is sequentially processed from the title node to the BODY tag node that forms the root of the tree structure data displayed in the Web document. The size of the traced node is acquired from the tree structure data storage unit 12. The title and the text are usually close to each other on the DOM data, that is, they exist as a node on the DOM integrated as the main text while following the BODY node from the title node, and the text is included in the title. By adding, the size (particularly the height) on the display suddenly increases. For this reason, for example, when the following conditions are satisfied, it is possible to stop the sequential processing and specify a node on the DOM at that time as a node of a main sentence to be extracted.

Condition example 1: Exceeding a predetermined height (300 pixels or more, etc.)
Condition example 2: The ratio between the height of the title and the height of the node is obtained, and exceeds a predetermined ratio (eg, more than 5 times the height of the title).
Condition example 3: When processing nodes sequentially toward BODY, the ratio of the area with the immediately preceding node is obtained, and exceeds a predetermined ratio (such as 5 times the area of the immediately preceding node).
The above processing will be described with reference to HTML samples shown in FIGS.

  FIG. 7 is a display example when the HTML of FIG. 6 is displayed by a normal browser. A tag whose class is specified by the class attribute is indicated by “.” (Dot) notation. For example, the node of the DOM data specified by <DIV class = "bttl"> is expressed as “DIV.bttl”.

  FIG. 8 shows tree structure data when expanded to DOM. Among these, the shaded “H2” node is the title specified by the title node search unit 13.

  The sentence node search unit 14 follows the tree structure from “H2” to “DIV.ettl”, “DIV.ent”, “DIV.left”, and “BODY”. First, move from "H2" to "DIV.ettl" and compare the height.

  In this case, since the height hardly changes, the predetermined condition is not satisfied, so it is determined that the sentence is not the main sentence, and the process proceeds to the next “DIV.ent”. Since the height of “DIV.ent” greatly increases, a predetermined condition is satisfied, and this “DIV.ent” portion is determined to be the main sentence.

  According to the embodiment described above, it is possible to extract a main sentence that combines the title and text of a Web document. Further, as described above, since the title part is specified in the middle of the process, it is possible to specify and extract a part of the extracted main sentence excluding the title part as the text.

  As an effect of the present embodiment, a usage example of the main sentence extracted as described above will be described.

  For example, in a system for displaying a Web document, such as a Web browser, after converting HTML data to DOM along with CSS data, the parent-child relationship between the nodes on the DOM of the main sentence extracted by this embodiment (upper level) By deleting or hiding all nodes that are not in the subordinate relationship), it is possible to provide a Web document display device for only the main text without displaying advertisements or navigation links. In addition, in this case, since the CSS specification of the original Web document is correctly left, the design of font, size, color, etc. is applied correctly, so that it is possible to extract the text including such a design. FIG. 9 shows a configuration diagram of the text search apparatus 1 including the output unit 15 having a function of deleting or hiding a node that is not in a parent-child relationship with the main text as described above. 10 and 11 show an example in which the present embodiment is actually applied. In this example, the URL “http://www.ntt.co.jp/RD/OFIS/keyword/” and the title “advanced technology understood by keywords” are acquired by RSS, and the technology of the present invention is applied. The display as shown in FIG. 11 can be obtained by hiding the DOM other than the main text of the article.

  As another example, when it is used for a search engine, only text data belonging to the main sentence specified by the present embodiment is dealt with and stored in a search engine database, etc. It is possible to avoid search results that do not.

(Second Embodiment)
In the second embodiment, first, the summary document acquisition unit 11 acquires entries [i] .summary in addition to the items described in the first embodiment. This summary usually contains the text or part of the text (especially the first few lines) as plain text or HTML source.

  The tree structure data storage unit 12 and the title node search unit 13 are configured in the same manner as in the first embodiment.

  The text node search unit 14 is activated by receiving the title position on the DOM extracted by the title node search unit 13 and the summary obtained from the summary document acquisition unit 11 as parameters. In the second embodiment as well, the process of sequentially following the tree structure (tree) from the title to the BODY node is performed as in the first embodiment. The conditions for terminating the process are as follows. It is not confirmed by design as in the form, but by comparing with the summary received as a parameter, if it is determined that the node matches well with the summary, the processing is aborted, and the main sentence is the processing target when it is terminated To be extracted as a node.

(Operation of Second Embodiment)
As an example of a more specific implementation method, first, when the summary is an HTML source, it is converted to plain text by deleting the tag part, and the normalization described in the first embodiment is performed. If the summary is plain text, just normalize it. The result of normalization is set to text 1.

  On the other hand, text data existing under the node is extracted for the node to be compared. At this time, the text data under the node already extracted as the title is excluded as the title text. The text data excluding the title is also normalized, and is set as text 2. The texts 1 and 2 are compared. For example, when the text 1 is included in the text 2, it is determined that the text 1 is a node, and the processing is terminated. Or, considering the possibility of not being included strictly, text for the number of characters from the beginning of text 2 to R times the number of characters in text 1 (R is determined in advance at about 1.2) When a similarity is found such that the Levenshtein distance between the cutout and the text 1 is equal to or less than a predetermined distance, it may be determined that “substantially included”. The length of the condition determined from the beginning is cut out from the beginning when the RSS is originally summarized, and a certain number of characters are cut out mainly from the beginning, and some text processing is performed as necessary. This is because there are many cases where the Levenstein distance from the entire text 2 is obtained, and the distance for the text data after the cut-out is added to prevent the Levenstein distance from becoming unnecessarily large.

  The second embodiment also has the same effect as the first embodiment. Moreover, you may comprise so that the output part 15 may also be provided similarly to 1st Embodiment.

DESCRIPTION OF SYMBOLS 1 ... Text search device 11 ... Summary document acquisition part 12 ... Tree structure data storage part 13 ... Title node search part 14 ... Text node search part 15 ... Output part

Claims (4)

An overview document acquisition unit having a function of reading data distributed in a predetermined format including an access destination and a title related to a document on the network and retrieving the access destination and the title related to the document;
A tree structure data storage for accessing an access destination of a document extracted by the summary document acquisition unit, acquiring document data and style data of the document, and converting the acquired document data into tree structure data and storing it And
Measure the similarity between the title extracted by the summary document acquisition unit and the text data in the node of the tree structure data stored in the tree structure data storage unit, and based on the measured similarity, the tree structure A title node search unit having a function of specifying a title on the data of
Determining whether the style of the node satisfies a predetermined condition based on the style data while tracing the nodes from the title on the tree structure data specified by the title node search unit to the root of the tree structure And a sentence node search unit having a function of specifying a node including a main sentence on the tree-structured data. An outline document acquisition unit having a function of reading data distributed in a predetermined format including an access destination and a title related to a document on the network, and retrieving an access destination, the title, and an outline related to the document;
A tree structure data storage for accessing an access destination of a document extracted by the summary document acquisition unit, acquiring document data and style data of the document, and converting the acquired document data into tree structure data and storing it And
Measure the similarity between the title extracted by the summary document acquisition unit and the text data in the node of the tree structure data stored in the tree structure data storage unit, and based on the measured similarity, the tree structure A title node search unit having a function of specifying a title on the data of
The beginning of the text of the node to be compared with the summary text acquired by the overview document acquisition unit while tracing the nodes from the title on the tree structure data specified by the title node search unit to the root of the tree structure A sentence node search unit having a function of identifying a node including a main sentence on the tree-structured data based on a Levenshtein distance from a text cut out according to a predetermined rule from Sentence search device. An output unit having a function of displaying only a main sentence of the document by deleting or hiding a node not having a parent-child relationship with the node on the tree structure data specified by the sentence node search unit; The sentence search device according to claim 1, wherein:   A computer program for causing a computer to function as the text search device according to claim 1.