JP2005190074A - Document dividing device and method, program and index preparing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a document dividing device and method, program and an index preparing device for dividing a structured document including a plurality of unit documents whose contents are different, and to improve document retrieving precision to the structured document including the plurality of unit documents. <P>SOLUTION: When retrieving a structured document including a plurality of unit documents, the structured document is preliminarily divided into every unit documents, and an index is prepared based on the divided division document. The divided document is prepared by a path formula table 46 in which start path formula showing the start point of the unit document and end path formula showing the end point of the unit document are stored, a document structure specifying part 32 for storing tags appearing at the time of reading the structured document, and for specifying the document structure at each tag appearance position, a coincidence determination part 34 for determining whether or not the specified document structure is coincident with path formula stored in the path formula table and a processing execution part 36 for executing processing to newly output the document from the tag appearing position matched with start path formula to the tag appearing position matched with end path formula as another document. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a document dividing apparatus and method, program, and index creating apparatus for dividing a structured document including a plurality of unit documents having different contents.

  In the collection-type document search apparatus, an index in which words included in a search target document are associated with the identification information of the document is created in advance, and the document search is performed based on the index. Document information recorded in the index includes a document ID serving as document identification information and an address indicating the location of the document. Normally, one document ID or address is defined for each piece of document data.

JP 2000-259660 A

  Incidentally, some documents include a plurality of topics (topics) recorded as unit documents in one document data. For example, an electronic newspaper article provided on the Web or an electronic manual in a programming language. In the case of such a document, even words appearing in different unit documents are recorded in the index in association with the same document ID, address, and the like. If an AND search that designates a plurality of words as a search term is performed on a document including a plurality of unit documents, a search result may be contrary to the user's intention.

  For example, in an electronic newspaper article, a newspaper article (unit document) for “stock price of company A” and a newspaper article (unit document) for “new product of company B” may be recorded in the same document data. In this case, words such as “Company A”, “Stock Price”, “Company B”, “New Product” are recorded in the index in association with the same document ID. Then, if a user who wants to search for a newspaper article about "A company's new product" performs an AND search using "A company" and "New product" as search terms, "A company" and "New product" Since it is recorded in association with the same document ID (address), this document is output as a search result. However, this document is a document regarding “stock price of company A” and “new product of company B”, and is not a document intended by the user.

  As described above, when one document data includes unit documents for a plurality of topics, there is a problem that the search accuracy is lowered.

  Further, when the amount of documents recorded in the document data is large, it is difficult to understand where the searched word appears in the document. Therefore, the user needs to search the position of the search word again for the document output as the search result, which is complicated.

  By the way, many of such documents are often described as structured documents such as HTML. A structured document is a document in which a tag indicating the document structure is incorporated in the text that is the document body. By analyzing this tag, the structure of the document can be analyzed.

  As an apparatus for handling such a structured document, Patent Document 1 discloses an attribute name and an attribute value (text of a partial structure) assigned to each of a plurality of partial structures (for example, chapters and paragraphs) constituting the structured document. ) Is extracted. According to this, it is not necessary to be aware of the difference between these various expressions for the various expressions of the structured document, and necessary attribute names and attribute values can be extracted by simple designation.

  However, this does not take into account improvement in search accuracy for a structured document including a plurality of unit documents as described above.

  Therefore, an object of the present invention is to provide a document segmentation apparatus, a method thereof, a program, and an index creation apparatus that can improve the search accuracy for a structured document including a plurality of unit documents.

  A document dividing apparatus according to the present invention is a document dividing apparatus that includes a plurality of unit documents and divides a structured document including a tag that defines a document structure and text that is the body of the document, and indicates a starting point of the unit document. A path expression table that stores a start path expression that represents the document structure as an array of tags, an end path expression that represents the end structure of a unit document as an array of tags, and a structured document that is read and appears The document structure specifying means for storing the tags to be specified and specifying the document structure at each tag appearance position, and the match determining means for determining whether the specified document structure matches the path expression stored in the path expression table Processing execution means for executing processing for newly outputting a document from the tag appearance position where the start point path expression and the document structure match to the tag appearance position where the end point path expression and the document structure match; It is characterized by having .

  In a preferred aspect, the path expression table further stores an attribute path expression indicating the document structure indicating the attribute of the unit document by an array of tags, and the process execution means outputs a new document. The text following the tag appearance position where the attribute path expression matches the document structure is output as the attribute of the new document. In another preferred aspect, the path expression table further stores an appearance position path expression indicating the document structure indicating the appearance position of the unit document by an array of tags, and the process execution means outputs a new document. In this case, the text following the tag appearance position where the appearance position path expression matches the document structure is output as the appearance position of a new document.

  Here, the structured document refers to a document in which the document structure of the document is defined by a tag, such as an HTML document or an XML document. The unit document is a document describing one topic (topic). For example, one case article document in an electronic newspaper article or one matter document in an electronic manual can be cited.

  According to the present invention, a structured document can be divided into unit documents. Therefore, if an index is created based on a document divided for each unit document, a document search can be performed with higher accuracy.

  Hereinafter, an index creating apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a hardware configuration of an index creating apparatus 10 according to an embodiment of the present invention. The index creation device 10 stores a central processing unit (hereinafter referred to as “CPU”) 12 having a function of controlling each unit, a memory 14 composed of a ROM, a RAM, and the like, a structured document to be searched, document information thereof, and the like. A hard disk 16 for input, an input unit 20 for giving search conditions and various instructions with a keyboard, a mouse, etc., a display unit 22 for displaying a search result, etc. composed of a CRT or a liquid crystal display, etc. A (FDD) 24, a CD-ROM drive 26 for reading data from a CD-ROM, a communication unit 18 for exchanging signals and data with other communication devices, and the like are connected by buses.

  FIG. 2 is a block diagram showing a functional configuration of the index creation device 10. The index creating apparatus 10 includes a document reading unit 30, a document structure specifying unit 32, a coincidence determining unit 34, a process executing unit 36, and an index creating unit 38, and each is controlled by the CPU 12. In addition, a document storage unit 40 for storing a structured document to be searched, a temporary storage unit 42, a divided document storage unit 44, and the like are provided on a hard disk of another computer communicated via the hard disk 16 or the communication unit 18. Yes.

  The document storage unit 40 stores a large number of structured documents to be searched. Here, the structured document is a document in which the document structure is represented by a tag, for example, an HTML document or an XML document.

  The stored structured document is appropriately read by the document reading unit 30. The document reading unit 30 reads a structured document and outputs the read sentence to the document structure specifying unit 32. The document structure specifying unit 32 determines the presence or absence of a tag in the sentence input from the document reading unit 30, and if a tag is present, further specifies the document structure represented by the tag. The identified document structure is output to the coincidence determination unit 34 as a path expression.

  Here, the path expression is an expression of the document structure as an array of paths, in which tags defining the document structure of the portion are arranged in order. For example, in the case of the HTML document shown in FIG. 3, if the document structure at <BODY> on the second line is expressed by a path expression, “HTML / BODY” is obtained. Also, the document structure of <B> on the fourth line is “HTML / BODY / B”. This indicates that the document structure in the portion <B> is defined by three tags <HTML>, <BODY>, and <B>.

  The coincidence determination unit 34 determines whether a path expression input from the document structure specifying unit 32 matches a path expression recorded in a path expression table 46 described later. The determination result is output to the process execution unit 36. The process execution unit 36 executes various processes based on the determination result and the path expression table 46. The contents of the process to be executed are recorded in the path expression table 46, and there are processes such as temporarily storing the character string and outputting the temporarily stored character string as a divided document. The temporary storage of the character string is performed in the temporary storage unit 42, and the output of the divided document is performed in the divided document storage unit 44.

  The index creation unit 38 refers to the divided document storage unit 44 and creates an index that associates the words included in each divided document with the identification information of the document. The processing in the index creation unit 38 can use a conventional index creation technique.

  Next, an important document dividing process in the index creating apparatus 10 will be described. FIG. 4 shows an example of a structured document (HTML document) to be divided, and FIG. 5 shows a display example of the structured document. In FIG. 4, the number of lines is shown on the left side for the sake of explanation, but there is no such number of lines in an actual structured document.

  Here, an electronic manual of a programming language is taken as an example as a structured document to be divided. The electronic manual is described in an HTML document as shown in FIG. 4 and is stored in the document storage unit 40. FIG. 4 shows a document explaining the read () method and the close () method belonging to the Stream class, but the electronic manual also describes a structured document explaining various classes and methods. It has a lot. All the structured documents have almost the same document structure.

  The HTML document shown in FIG. 4 is displayed as shown in FIG. 5 when displayed on a Web browser. As is apparent from FIG. 5, this document includes a title (unit document T1), a stream class description (unit document T2), a read () method description (unit document T3), and a close () method description (unit document). T4), four unit documents (topics) are recorded. Each unit document is divided by a dividing line 50.

  Here, for example, when the word “A” appears in the unit document T3 and the word “B” appears in the unit document T4, that is, the read () method is the word “A”, and the close () method is the word “B”. When related only to the above, the conventional index creation method has the following problems. For example, if the user searches for a method related to both the words “A” and “B” and performs an AND search with “A” and “B”, the document of FIG. 4 is also output as a search result. This is because the document of FIG. 4 includes both the word “A” and the word “B”. In practice, however, the read () method is not associated with the word “B” and the close () method is not associated with the word “A”. Therefore, the document of FIG. 4 is a document different from the user's intention.

  Also, in FIG. 4, the explanation of each method is omitted, so that the total amount of documents is small, and all the documents can be viewed at once even if they are displayed on a Web browser. However, an actual electronic manual has a long explanation and a large amount of documents. Therefore, for example, even if the word “B” is searched and this document is output as a search result, it cannot be immediately found where the word “B” appears. For this reason, the user needs to perform processing such as searching for the appearance position of a word for the document output as the search result, which is complicated.

  Therefore, in the present embodiment, processing for dividing a document into unit documents is performed as described below. FIG. 6 shows a flowchart of document division processing.

  When document division is performed, first, the document reading unit 30 extracts a stored structured document and reads a character string (S10). Next, the document structure specifying unit 32 determines whether a tag is included in the read character string (S12). The presence or absence of a tag can be determined by the presence or absence of “<” or “>”.

  If there is a tag, the tag is stored in the memory (S14). If a tag has already been stored in the memory, a stack structure for storing the tag is added. Then, the document structure at the appearance position of the read tag is specified from the tag stack stored in the memory (S16). The identified document structure is represented as a path expression that is an array of tags. The path expression is an enumeration of tags that define the document structure of the part.

  For example, if the read sentence is the first line, only the tag “HTML” is stacked in the memory. Therefore, the document structure in the first line is “HTML” when expressed by a path expression. On the other hand, if the read sentence is the second line, the tag “HTML” and the tag “BODY” are stacked in the memory. Therefore, the document structure in the second line can be expressed by a path expression “HTML / BODY”.

  If the document structure is specified as a path expression, it is next determined whether or not the path expression recorded in the path expression table matches the specified path expression (S18). Here, as shown in FIGS. 7 and 8, the path expression table is a table in which a path expression 52 indicating a predetermined document structure and a process 54 based on the document structure indicated by the path expression are recorded in association with each other. . In this path formula table, at least path formulas P1 and P2 indicating the document structure indicating the start point of each unit document and a path formula P3 of the document structure indicating the end point of each unit document are recorded. Processes are associated and recorded.

  For example, in the case of the document shown in FIG. 4, the unit document T1 starts from the second line, that is, the document structure indicated by “HTML / BODY”. The unit document T2 starts from the sixth line, that is, the document structure indicated by “HTML / BODY / DL / DT / B”. In the path expression table, a process of “temporarily storing a subsequent character string” is associated with the path expressions P1 and P2 indicating the start point of each unit document.

  Each unit document has “HTML / BODY / HR” indicating a delimiter as an end point (the fifth line, the eighth line, etc. in FIG. 4). In the path formula table, a process of “outputting the temporarily stored character string as separate document data” is associated with the path formula P3 “HTML / BODY / HR” indicating the end point of each unit document.

  The coincidence determination unit 34 determines whether the path expression recorded in the path expression table matches the path expression specified by the document structure specifying unit 32 (S18), and outputs the determination result to the process execution unit 36. .

  If the path expressions match, the process execution unit 36 executes a process associated with the path expression based on the path expression table (S20). For example, when the second line of the document in FIG. 4 is read, the document structure is represented by a path expression “HTML / BODY”. This path formula matches the path formula P2 recorded in the path formula table. In this case, the process associated with the path expression P2, “temporarily store the subsequent character string” is executed. That is, thereafter, the character string to be read is stored in the temporary storage unit 42. In the example of FIG. 4, the character string “<A HREF=“#read( )”> read () </A>...” Read in the third and subsequent lines is temporarily stored in the temporary storage unit 42. To remember. Here, all the read character strings, that is, all the character strings including the tag are temporarily stored, but only the text portion excluding the tag (“read ()”, “close ()”) is temporarily stored. May be.

  Next, it is determined whether the read sentence is the end of the structured document (S22). If the sentence continues, not the end, continue reading the string. Then, the presence / absence of the appearance of the tag, the matching with the path expression, and the process execution are performed. If the identified path expression matches the path expression recorded in the path expression table, the process is executed based on the path expression table.

  For example, when the fifth line is read, the document structure in the fifth line is represented by a path expression “HTML / BODY / HR”. This coincides with the path formula P3 recorded in the path formula table. Since the path expression P3 is associated with the process “output the temporarily recorded character string as separate document data”, the process execution unit 36 replaces the character string stored in the temporary storage unit 42 with another new character string. The document data is output to the divided document storage unit 44 as a divided document. At this time, the document ID and address of the original document data (document data in FIG. 4) are also recorded in the output divided document.

  Such processing is performed on the entire text of the structured document, and when the end of the document is reached, the document division processing ends (S22).

  As described above, in the path formula table, the path formulas P1 and P2 indicating the start point of each unit document have a process of “temporarily storing the subsequent character strings”, and the path formula P3 indicating the end point of each unit document is included in the path formula table. , “A temporarily stored character string is output as separate document data” is associated. Then, by performing processing based on the path expression table, the structured document can be output as separate document data (divided document) for each unit document.

  If a structured document can be output as a divided document for each unit document, there are the following advantages. For example, by creating an index based on this divided document, erroneous search in AND search can be reduced. That is, the word included in each divided document is associated with the document information (document ID, etc.) and the original document information (original document ID, original document address, etc.) that is the document information of the structured document that is the source of the divided document. Create an index. In this index, even if the words are included in the same structured document, the words included in different unit documents are associated with different document IDs. Therefore, if a search is performed based on this index, an erroneous search in an AND search can be prevented. In this case, as the actual search result, the original document (structured document) is output as the search result.

  Further, the path expression table used in the present embodiment includes a path expression P6 (see FIG. 8) indicating attributes in addition to the path expressions P1, P2, and P3 indicating the start and end points of the unit document, and the appearance position of the unit document. A path expression P7 (see FIG. 8) indicating the document structure, a path expression P4 (see FIG. 7) indicating the change of the document structure, and the like are recorded. These will be described.

  The structure of the structured document shown in FIG. 4 greatly changes after the ninth line. That is, up to the eighth line is a document structure for explaining the class, but from the ninth line is a document structure for explaining the method.

  In such a case, it is better to use a different path expression table instead of the same path expression table before the eighth line and after the ninth line. Therefore, a path formula P4 indicating the document structure of such a part that greatly changes the document structure is recorded in the path formula table, and a process called “change path formula table” is associated with the path formula P4. Then, a second path expression table (FIG. 8) suitable for the document structure on the ninth and subsequent lines is prepared, and processing is performed on the ninth and subsequent lines based on the second path expression table.

  Each unit document may include a word that is an attribute of the unit document and a word that indicates the appearance position of the unit document. A path expression indicating the appearance of such a word may be recorded in a path expression table so that attributes and appearance positions can be temporarily stored.

  For example, in the structured document shown in FIG. 4, the word “read ()” on the 15th line is a word indicating the appearance position of the unit document T3. That is, as is well known, the tag <A NAME="(keyword)"/> is a tag for setting a keyword for setting a link to that part of the structured document. When a link is set by describing <A HERF=“(original document URL)#keyword”>, the link can move to the keyword appearance position of the document. Therefore, for example, when the URL of the structured document in FIG. 4 is “sample.html”, if a link is set as <A HERF=“sample.html#read( )”>, the structured document 15 It is possible to move to the line, that is, the top position of the unit document T3.

  Therefore, a path expression P7 indicating the appearance position of the unit document is recorded in the second path expression table, and a process of temporarily storing the character string following the path expression P7 as the appearance position of the unit document is associated. By doing so, read () is temporarily stored as the appearance position of the unit document T3. When the document structure portion corresponding to the path expression P9 indicating the end point of the unit document is read, the temporarily stored content including the word indicating the appearance position is output as a divided document.

  When creating an index, a word indicating the appearance position is stored in association with a word included in each divided document. When the search position is associated with the corresponding search term when the search is executed, the search result is displayed in a state of being moved to the appearance position. Thereby, even when a document with a large amount of documents is searched, it is possible to display the vicinity of the search word as a search result. Therefore, the user does not need to search for the appearance position of the search word again with respect to the search result, and can easily obtain desired information.

  Each unit document may include a word indicating the attribute of the unit document. In the structured document shown in FIG. 4, a link to “read.htm” is set in the unit document T3 (19th line). This link destination can be set as an attribute of the unit document T3. In this case, the path expression P6 indicating this attribute and the process of “temporarily storing a character string as an attribute” are recorded in the second path expression table in association with each other. By doing so, “read.htm” can be extracted as an attribute of the unit document T3. This attribute is also associated with the index and stored.

  Thereby, in the search, it is possible to search not only by a search word but also by an attribute (where it is linked in the present embodiment). Thereby, more versatile search can be performed. In the present embodiment, the link destination is extracted as an attribute, but naturally other contents may be extracted as the attribute.

  As described above, in this embodiment, a structured document can be divided for each unit document. Then, by creating an index based on the divided divided document, it is possible to provide a search result corresponding to a structured document including a plurality of unit documents. That is, a document search with higher accuracy can be performed. Further, by storing the appearance position and attribute of each unit document in the divided document and reflecting it in the index, it is possible to perform a document search with better operability.

  In the present embodiment, an electronic manual in a programming language has been described as an example of the document division target. However, as long as it is a structured document including a plurality of unit documents, another structured document may be the target. Further, the path expression table may be appropriately changed according to the configuration of the document to be divided. For example, in the present embodiment, a structured document in which a unit document is divided by a dividing line (<HR />) is a target of division. It may be set as a path expression.

It is a block diagram which shows the hardware constitutions of the index production apparatus which is embodiment of this invention. It is a block diagram which shows the function structure of an index production apparatus. It is a figure which shows an example of a structured document. It is a figure which shows an example of the structured document of a division | segmentation object. It is a figure which shows the display of the structured document of FIG. It is a flowchart which shows the process of document division. It is a figure which shows an example of a path | pass type | formula table. It is a figure which shows another example of a path | pass type | formula table | surface.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Index production device, 30 Document reading part, 32 Document structure specification part, 34 Match determination part, 36 Process execution part, 38 Index creation part, 40 Document storage part, 42 Temporary storage part, 44 Division | segmentation document storage part, 46 Path type table.

Claims (6)

A document dividing apparatus that includes a plurality of unit documents and divides a structured document including a tag that defines a document structure and a text that is a body of the document,
A path expression table storing a start point path expression that represents the document structure indicating the start point of the unit document with an array of tags, and an end point path expression that represents the document structure indicating the end point of the unit document with an array of tags;
Document structure specifying means for reading a structured document, storing tags that appear, and specifying the document structure at each tag appearance position;
A match determination means for determining whether the identified document structure matches the document structure indicated by the path expression stored in the path expression table;
Processing execution means for executing processing for newly outputting a document from the tag appearance position that matches the document structure indicated by the start point path expression to the tag appearance position that matches the document structure indicated by the end point path expression;
A document dividing apparatus characterized by comprising: The document dividing device according to claim 1,
The path expression table further stores an attribute path expression that represents a document structure indicating an attribute of a unit document by an array of tags.
A document dividing apparatus, wherein when a new document is output, the process execution means outputs the text following the tag appearance position that matches the document structure indicated by the attribute path expression as an attribute of the new document. The document dividing device according to claim 1 or 2,
The path expression table further stores an appearance position path expression in which the document structure indicating the appearance position of the unit document is represented by an array of tags.
The process executing unit outputs a text following a tag appearance position that matches the document structure indicated by the appearance position path expression as a new document appearance position when outputting a new document. .   A computer program for causing a computer to function as the document dividing device according to any one of claims 1 to 3. A document dividing method that includes a plurality of unit documents and divides a structured document including a tag that defines a document structure and a text that is a body of the document,
The path expression creation means stores a start point path expression that represents the document structure indicating the start point of the unit document as an array of tags and an end point path expression that represents the document structure indicating the end point of the unit document as an array of tags. Creating a table;
A document structure specifying unit that reads a structured document, stores tags that appear, and specifies a document structure at each tag appearance position; and
A match determination step in which the match determination means determines whether the identified document structure matches the path expression stored in the path expression table;
Process execution means for executing a process of newly outputting a document from the tag appearance position where the document structure matches the start point path expression to the tag appearance position where the document structure matches the end path expression as another document Process,
A document dividing method characterized by comprising:   An index creation apparatus that creates an index in which words included in each document output by the document dividing apparatus according to any one of claims 1 to 3 and identification information of the document are stored in association with each other.

Images