JP2013175053A - Xml document retrieval device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently retrieve a portion conforming to a complicated retrieval condition in XPath.SOLUTION: An XML document retrieval device analyzes an XML document set to be retrieved, and generates both a sequence S for storing a shape of DOM tree based on structure information for the XML document and a sequence group T consisting of one or more sequences that store a type of structure path corresponding to each node of the DOM tree. Then, the device calculates a portion conforming to the structure path that is a retrieval query, by scanning the sequence S and the sequence group T.

Description

  The present invention relates to an apparatus and a program for searching a location that matches a search condition specified by a user from a plurality of XML (Extensible Markup Language) documents.

  XML is widely used as a method for describing data used for data exchange and storage. By using XML, various data can be described with a high degree of freedom in a text format that is easy to process mechanically. FIG. 1 shows an example of an XML document. The content of this document is divided into a plurality of parts by tags surrounded by “<” and “>”. The tag includes a start tag 101 written in the format “<tag name>” and an end tag 102 written in the format “</ tag name>”. An area delimited by a start tag and an end tag having the same tag name is called an element or an XML element.

  If there is no tag or text region sandwiched between the start tag and the end tag, XML can use a tag of the form “<tag name />” instead of the start tag and the end tag. In this specification, such a tag is handled in the same manner as when “<tag name> </ tag name>” is described. In the start tag, an attribute value can be given to the tag in the format of “<tag name attribute 1 = attribute value 1 attribute 2 = attribute value 2...>”. In this specification, when such a tag is given, “<tag name> <@attribute 1> attribute value 1 <@attribute 1> <@ attribute 2> attribute value 2 </ @ attribute 2>” Treat as if written. Further, in this specification, the text area is regarded as an element whose tag name is “#”. An element having no parent is called a root element. Each XML document has only one root element.

  In XML, a complicated data structure can be described by nesting a plurality of elements. The tag name of the end tag must be the same as the tag name of the start tag that appears immediately before the start tag for which the corresponding end tag does not yet appear when the XML text is read from the beginning to the end. . Thus, any two elements are limited to either one completely containing the other or no overlap at all. An element A is said to be a child of element A when "element B contains" and "no other element C is contained in element A and contains element B" Element A is said to be the parent of element B. Elements that are reachable by traversing the parent sequentially are called ancestors, and elements that are reachable by traversing the children sequentially are called descendants. An element that is a child of the same parent is called a sibling.

  This relationship between elements is generally represented by a tree structure. FIG. 2 shows an example of a tree structure. The tree structure shown in FIG. 2 is obtained by preparing a node 201 corresponding to each element and extending a directed edge 202 from a node corresponding to the parent element to a node corresponding to the child element. This tree structure is called a DOM tree (document object model tree). A character string obtained by concatenating tag names of elements on the route from the root of the DOM tree to each node with “/” interposed therebetween and adding “/” at the head is referred to as a structure path in this specification. . For example, in the case of FIG. 2, the rightmost “c” structure path is “/ a / c”. The number of tags included in the structure path is defined as the depth of the element.

JP 2006-228155 A

Toshiyuki Shimizu, Makoto Onizuka, Masaharu Eda, Masatoshi Yoshikawa, XML data management and stream processing technology, IEICE Transactions J90-D (2): 159-184, 2007. R. Kaushik, R. Krishnamurthy, J.F.Naughton, R. Ramakrishnan, On the integration of structure indexes and inverted lists, Proc.ACM SIGMOD, pp 779-790, 2004. Eda Yasuharu, Onizuka Makoto, Yamamuro Masashi, High-speed XPath processing method using summary information of XML data, IEICE Transactions D, J89-D (2): 139-150, 2006. Kazuhito Washio, Shuichi Mitarai, Akira Ishino, Masayuki Takeda, Fast and Lightweight XML Document Filtering Based on Incremental Path Trial Construction, DBSJ Letters 6 (2): 1-4, 2007. Navarro, G. and Makinen, V., Compressed full-text indexes, ACM Computing Surveys 39 (1): Article 2, 2007. Managing Gigabytes, I. Witten, A. Moffat, and T. Bell, Morgan Kaufmann

  As a description method of a search query for searching an XML document, a standard called XPath is widely used (Non-patent Document 1). For example, XPath writes “a / b” as a search query that specifies “element b which is a child of element a”. In XPath, such a search query description method is standardized. In addition, in a search query described in XPath, a combination of a plurality of elements having a relationship such as a parent, a child, an ancestor, a descendant, and a sibling can be specified.

  In a search process using a search query described in XPath, as a method for improving the search process, (1) an index that records the structure that appears in the XML data to be searched and (2) the appearance position of each structure Are used in advance, and a method for searching for a location matching the search query by referring to them is widely used (see Patent Document 1, Non-Patent Documents 2 and 3). In addition, in the search process using the search query described in XPath, a method of analyzing the structure appearing in the XML document to be searched at the time of search execution in real time is not executed (Non-patent Document). 4). However, this method is disadvantageous in terms of search speed compared to the method of calculating the search index in advance.

  In order to efficiently execute a search by a search query described in XPath, the structure information appearing in the XML document is analyzed and the location corresponding to each structure path is recorded, and the parent, child, ancestor, A data structure that can efficiently calculate the relationship between elements such as descendants and siblings is required. Furthermore, when handling a large-scale XML document, it is necessary that such data can be expressed with a data size as small as possible and can be read at high speed.

  Therefore, the present invention reduces the data size of the search data used in the XML document search process as much as possible, and makes it possible to calculate at high speed a search for a location that satisfies the conditions specified by the search query. To this end, the present invention provides an XML document analysis unit in which (1) a first numerical sequence S including a numerical sequence representing the depth of an element as a partial sequence in the order of appearance of the element, and (2) a DOM tree Search data for recording the shape of a DOM tree of an XML document given by a sequence group T composed of one or more sequences that record the type of structure path corresponding to each node is created. Then, the number sequence S and the number sequence group T are scanned, and a portion that matches the structure path given as a search query is calculated.

  According to the present invention, it is possible to speed up a search process using a search query described in XPath. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

The figure which shows an example of XML data. The figure which shows an example of a DOM tree. The figure explaining the element number allocated to an XML element. The figure which shows the block structural example of the XML document search apparatus which concerns on a 1st form example. The figure explaining the cooperation between each structure at the time of the pre-processing execution which concerns on a 1st form example. The flowchart explaining the flow of the pre-processing operation | movement which concerns on a 1st form example. The figure which shows the concept of the pre-processing operation | movement which concerns on a 1st example. The flowchart explaining the flow of the processing operation (analysis operation) performed by the XML document analysis part which concerns on a 1st form example. The figure explaining the cooperation between each structure at the time of the search execution which concerns on a 1st form example. The flowchart explaining the flow of the search process by the XML document search apparatus which concerns on a 1st form example. The figure explaining the process example of the search process which concerns on a 1st form example. The figure explaining a rank operation and a select operation. The flowchart explaining the flow of operation | movement of the structure path | pass analysis part which concerns on a 1st form example. The flowchart explaining the flow of operation | movement of the element search part which concerns on a 1st form example. The figure explaining a bit vector. The figure explaining a Wavelet tree. The figure which shows the block structural example of the XML document search apparatus which concerns on a 2nd form example. The figure explaining the cooperation between each structure at the time of the pre-processing execution which concerns on a 2nd form example. The flowchart explaining the flow of the pre-process which concerns on a 2nd form example. The flowchart explaining the flow of the operation | movement which calculates a parent element in the 3rd form example. The flowchart explaining the flow of the operation | movement which calculates the previous sibling element in a 3rd form example. It is a flowchart explaining the flow of the operation | movement which calculates the next sibling element in a 3rd form example.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, this invention is not limited to the form example mentioned later, A various deformation | transformation is possible in the range of the technical thought.

[First embodiment]
The XML document search apparatus according to this embodiment collates search data created by preprocessing an XML document set with a search query, and outputs an element that matches a structure path specified by the search query as a search result. The search result is output based on element numbers assigned to all elements (XML elements) appearing in the XML document.

[Element number]
FIG. 3 shows an example of element number assignment for each element (XML element) constituting the XML document. In FIG. 3, the numbers corresponding to the element numbers 301 are shown in a rectangular frame. The element number 301 is a number uniquely identifying each element that is assigned to all XML elements in the pre-processing of the search process. The element number is a total value of the number of elements and documents that have been met so far when the XML document is scanned from the top. The element number of the j-th element of the i-th XML document is E (i-1) + 1 + j, where E (i-1) is the maximum element number of the i-1th XML document. Note that E (0) = 0.

[Device configuration]
FIG. 4 shows a block configuration of an XML document search apparatus 400 according to this embodiment. The XML document search device 400 includes a CPU (Central Processing Unit) 401, a main storage device 402, an auxiliary storage device 403, a removable drive 404, a user interface 406, and a network interface 407. Each component is connected to each other by an internal bus or the like.

  The XML document search apparatus 400 is connected to an external storage device 430 via a network 440 such as a LAN (Local Area Network). The present embodiment is not limited to the type of the network 440. The network 440 may be a wired connection or a wireless connection.

  The CPU 401 is an arithmetic device that executes a program stored in the main storage device 402. By executing the program stored in the main storage device 402 by the CPU 401, the functions of the XML document search device 400 are realized. In the following description, the processing operation described with the program as the subject is realized through execution of the corresponding program on the CPU 401.

  The main storage device 402 stores a program executed by the CPU 401 and information necessary for executing the program. The main storage device 402 is assumed to be a memory such as a RAM (Random Access Memory).

  The main storage device 402 stores an XML document analysis unit 410, a structure path analysis unit 412 and an element search unit 413 as programs, and as data, an XML document set 420, a path trie 421, a DOM tree 422 converted into a sequence, and a text Data 424 is stored.

  The XML document analysis unit 410 parses each XML document included in the XML document set 420, recognizes a tag, and extracts text data 424. Then, based on the analysis result, the XML document analysis unit 410 generates a path trie 421 and a sequenced DOM tree 422.

  The structural path analysis unit 412 calculates the depth and path type of the structural path that is the search query using the path trie 421. The path type is an identification number assigned to identify each structural path. Details thereof will be described later.

  Based on the depth and path type calculated by the structural path analysis unit 412, the element search unit 413 enumerates all locations that match the search query from the XML document set 420 and sets them as search results.

  The XML document set 420 is data of one or more XML documents to be searched. The path trie 421 is a summary of the structure information included in the XML document set 420. Details thereof will be described later. The digitized DOM tree 422 is obtained by extracting the structure information included in the XML document set 420 in an easily searchable format, and details thereof will be described later. The text data 424 is obtained by extracting text information sandwiched between tags in the XML document set 420.

  Note that the XML document set 420 need not be stored in the main storage device 402, and may be stored in the auxiliary storage device 403, a removable medium, or the external storage device 430, for example. In this case, the CPU 401 reads the XML document set 420 from the auxiliary storage device 403, the removable medium or the external storage device 430 and stores it in the main storage device 402.

  Similarly, the path trie 421, the digitized DOM tree 422, and the text data 424 need not be stored in the main storage device 402, and may be stored in the auxiliary storage device 403 and a removable medium, for example. In this case, the CPU 401 reads these data from the auxiliary storage device 403 and the removable medium 404 as necessary.

  In this embodiment, the XML document analysis unit 410, the structure path analysis unit 412, and the element search unit 413 are all realized by a program, but the present invention is not limited to this. For example, these functions may be realized as dedicated hardware. That is, the XML document search device 400 may include an XML document analysis device, a structure path analysis device, and an element search device.

  The auxiliary storage device 403 is a device capable of permanently storing information, and may be an HDD (Hard Disk Drive), for example. The removable drive 404 is a device that executes data write processing and data read processing on a removable medium. The removable media includes optical disks such as CD-ROM and DVD, and magnetic disks such as floppy (registered trademark) disks.

  The user interface 406 is an interface used by the user of the XML document search apparatus 400 for inputting data and outputting processing results. The user interface 406 includes a display device, a keyboard, a mouse, and the like. The network interface 407 is an interface for connecting to an external device via the network 440.

  Next, specific processing contents of the XML document search apparatus 400 will be described. However, in the following description, it is assumed that the XML document set 420 is stored in the auxiliary storage device 403.

[Inter-configuration linkage during pre-processing]
FIG. 5 shows a cooperative operation between components when the XML document search apparatus 100 according to the present embodiment preprocesses an XML document.

  First, the user of the XML document search apparatus 400 instructs the start of processing using the user interface 406 (step S101).

  Receiving the process start instruction, the CPU 401 reads the XML document set 420 from the auxiliary storage device 403 (step S102). The read XML document set 420 is stored in the main storage device 402.

  Next, the XML document analysis unit 410 (CPU 401) analyzes each XML document included in the XML document set 420, and generates a path trie 421, a digitized DOM tree 422, and text data 424 (step S103). Thereafter, the CPU 401 outputs the path trie 421, the numbered DOM tree 422, and the text data 424 to the auxiliary storage device 403 (step S104).

  In step S101, the user may directly input the XML document set 420. In this case, step S102 can be omitted. The XML document set 420 may be read from the external storage device 430.

[Overview of preprocessing]
FIG. 6 is a flowchart for explaining the flow of preprocessing executed by the XML document search apparatus 400 according to this embodiment before search.

  When the XML document set 420 is input, the CPU 401 executes analysis processing by the XML document analysis unit 410 (step S201). The XML document analysis unit 410 analyzes each XML document included in the XML document set 420, and generates a path trie 421, a DOM tree 422 converted into a sequence, and text data 424. Details of the preprocessing executed by the XML document analysis unit 410 will be described later. When the processing is completed, the CPU 401 outputs the path trie 421, the digitized DOM tree 422, and the text data 424 to the auxiliary storage device 403 (step S202).

[Sequenced DOM tree]
With reference to FIG. 7, an example of the DOM tree 422 converted into a number sequence generated by the XML document analysis unit 410 will be described. FIG. 7 also shows an example of a path trie 421. The connection relationship (that is, the shape of the DOM tree) of each node appearing in the DOM tree is recorded in the sequence S. Numeric values are stored in the sequence S according to the following rules.
The information (partial number sequence) corresponding to each document starts with a numerical value 0.
The numerical value constituting the information (partial number sequence) corresponding to each document represents the depth position of the element corresponding to the start tag that is found when the XML document is read sequentially from the top. As described above, the text is processed in the same way as when it is surrounded by the start tag / end tag whose tag name is “#”.

  In the example shown in FIG. 7, when the first XML document (upper left side) is read from the top, <a>, “text 1”, <b>, “text 2”, </ b>, <c>, Tags and text appear in the order of “text 3”, </ c>, </a>. The depth of elements other than the end tag is 1,2,2,3,2,3. Therefore, in consideration of 0 representing the head of the XML document, 0, 1, 2, 2, 3, 2, 3 are added to the sequence S. Similarly, 0, 1, 2, 3, 2, 2, 3 are added to the second XML document (upper right side).

  In this way, the sequence S can record the shape of the DOM tree. However, in order to use it as search data, information (path type) specified by the tag type is also required. Therefore, the path type identified by the number assigned to an arbitrary structure path is recorded in the numerical sequence T [d] corresponding to the DOM tree structure by depth. The numerical value giving the path type is a number uniquely assigned to the structure path having the same depth.

  The path type number is recorded based on the path trie 421. The path trie 421 is tree-structured data constructed so as to include all of the structural paths that appear for all the XML documents constituting the XML document set 420. The past trie 421 can be constructed by a known method (for example, Non-Patent Document 4). In the case of this embodiment, when it becomes necessary to add a new node during the construction of the path trie 421, a processing function for assigning a new path type number to the new node is added to a known construction function. A method for assigning numbers will be described later. In the case of FIG. 7, a numerical value 703 enclosed in parentheses corresponds to a path type number.

[Generation of sequences S and T]
FIG. 8 shows details of the analysis operation executed in the XML document analysis unit 410 according to this embodiment. In this analysis operation, a number sequence S and a number sequence T [d] are created.

  First, the XML document analysis unit 410 initializes the sequence S, T [1],..., T [D] to an empty sequence (step S300). Here, [D] represents the depth of the element at the deepest position in the XML document set 420. Also, the XML document analysis unit 410 initializes all the elements R [1],..., R [D] of the array R to “1” (step S300). Further, the XML document analysis unit 410 initializes the path trie 421 so as to have only the root node 701 (FIG. 7).

  Next, the XML document analysis unit 410 determines whether or not processing of all the documents included in the XML document set 420 has been completed (step S301). Until a positive result is obtained, steps S302 to S309 described later are repeatedly executed.

  If a negative result is obtained in step S301, the XML document analysis unit 410 reads an unprocessed document (step S302). Hereinafter, this document is referred to as X. In step S302, the XML document analysis unit 410 adds “0” representing the beginning of the document to the sequence S. Also, the XML document analysis unit 410 prepares a variable d and sets “0” as an initial value. Further, the XML document analysis unit 410 prepares a variable v and initializes it to point to the root node 701 of the path trie.

  Next, the XML document analysis unit 410 determines whether or not the document X has been read to the end (step 303). Until a positive result is obtained, steps S304 to S309 described later are repeatedly executed.

  If a negative result is obtained in step S303, the XML document analysis unit 410 determines whether or not the tag at the current reading position is an “end tag” (step S304). If a positive result is obtained, the XML document analysis unit 410 changes the variable v to point to the parent node on the path trie 421, and subtracts 1 from the variable d (step S304-1). Then, the XML document analysis unit 410 advances the reading position to immediately after the end tag, and returns to step S304.

  If a negative result is obtained in step S304, the XML document analysis unit 410 determines whether or not the current reading position in the document X is “text” instead of “tag” (step S305). If a positive result is obtained, the XML document analysis unit 410 reads the text and adds the content to the text data 424 (step S305-1). Further, the XML document analysis unit 410 sets “#” in the variable t, and proceeds to step S307 (step S305-1).

  If a negative result is obtained in step S305, the XML document analysis unit 410 reads the start tag and advances the reading position to just after that. Also, the XML document analysis unit 410 sets the tag name of this start tag in the variable t (step S306).

  After step S305-1 or after step S306, the XML document analysis unit 410 determines whether or not a child node v ′ whose tag name matches the value of the variable t exists in the node v on the path trie 421 ( Step S307). If a negative result is obtained, the XML document analysis unit 410 newly creates a child node (hereinafter referred to as “v ′”), sets the path type of v ′ as the value of R [d], and then selects R [d ] Is added to [] (step S307-1).

  If a positive result is obtained in step S307, the XML document analysis unit 410 updates the variable v to point to the child node v 'of v, and adds 1 to the variable d (step S308).

  After step S307-1, or after step S308, the XML document analysis unit 410 adds the value of the variable d to the sequence S, and further adds the updated path type of the variable v to the sequence T [d]. The process returns to S303 (step S309).

[Inter-configuration linkage during search operations]
FIG. 9 shows a cooperative operation between components when the XML document search apparatus 400 according to this embodiment searches for an XML document.

  First, the XML document search apparatus 400 reads in advance the path trie 421 and the numbered DOM tree 422 used for the search from the auxiliary storage device 403 to the main storage device 402 (step S401). These data are data created in advance by the preprocessing described above.

  A user of the XML document search apparatus 400 inputs a structure path as a search query through the user interface 406 (step S402). Receiving the search query, the CPU 401 uses the path trie 421 to calculate the depth d and the path type t of the element corresponding to the structural path included in the search query in the XML document set 420 (step S403).

  Next, the element search unit 413 (CPU 401) uses the numbered DOM tree 422 to enumerate all element numbers that match the structure path included in the search query (step S404). Thereafter, the CPU 401 transmits the obtained element number to the user (step S405).

[Details of search operation]
FIG. 10 shows the flow of processing when the XML document search apparatus 400 according to this embodiment searches for an XML document. It is assumed that the CPU 401 has previously read the path trie 421, the numbered DOM tree 422, and the text data 424 used for the search from the auxiliary storage device 403 to the main storage device 402.

  First, the user inputs a structure path as a search query to the XML document search apparatus 400 through the user interface 406. Thereafter, the structural path analysis unit 412 accesses the path trie 421 created for the XML document set 420, and calculates the depth d and the path type t of the element corresponding to the structural path specified as the search query (step S501). ).

  Next, the element search unit 413 accesses the numbered DOM tree 422 and lists the numbers of elements corresponding to the structure path (step S502).

  FIG. 11 shows an outline of the above-described search process. FIG. 11 shows the concept of calculating the number of an element in which a structural path appears when a structural path represented by “/ a / b” is given as a search query.

Since this structure path is “/ a / b” and b is the second tag name, the depth is “2”. Further, in the path trie 421, when “a” and “b” are traced from the root node 701, the node written as “b (2)” is reached. “(2)” indicates that the path type of this node is “2”. Therefore, in order to know all the appearance positions of “/ a / b”, it is only necessary to calculate the element numbers for all the elements having the path type “2” at the depth “2”. For this purpose, a rank operation and a select operation described later are executed.
(1) rank (A, c, i) = number of c in the i-th element of the sequence A (2) select (A, c, j) = position of c appearing j in the sequence A

  FIG. 12 shows an example of rank operation and select operation. In the example of FIG. 12, rank (X, 3, 10) that gives the number of “3” in the elements up to the tenth in the sequence X is “2”. Further, select (X, 3, 2) which gives the position where “3” appears second in the sequence X is “7”.

  Returning to the description of FIG. The processing operation for searching for “/ a / b” described above is processing for extracting an element having a depth of “2” and a path type of “2”.

  First, the total number n of elements having a depth of “2” and a path type of “2” can be calculated by n = rank (T [2], 2, | T [2] |). However, | T [d] | is the number of elements of the sequence T [d].

  Next, k ′ = select (T [2], 2, k) is calculated for all integers k where 1 ≦ k ≦ n. The set of values k ′ obtained by this calculation represents the order in which the element of the path type “2” appears when the depth is limited to the element of “2”. In the example of FIG. 11, they are the second and sixth.

  Further, if k ″ = select (S, 2, k ′) is calculated, in the sequence S representing the shape of the DOM tree of the XML document set 420 to be searched, k among the elements having a depth of “2”. It is possible to calculate the position of the element where the 'th occurrence appears. In the example of FIG. 11, they are the fourth and thirteenth. These “4” and “13” are the search results of the element search unit 413.

[Structural path analysis]
FIG. 13 shows a structure path analysis operation performed by the structure path analysis unit 412. Here, the d-th tag name from the left included in the structure path is P [d], and the total number of tags is | P |.

  First, the structural path analysis unit 412 sets the variable v to the route 701 of the path trie 421, and sets the variable d to “0” (step S601).

  Next, the structural path analysis unit 412 determines whether or not d ≧ | P | (step S602). The structural path analysis unit 412 repeats the processes of steps S603 to S605 until a positive result is obtained in step S602. Incidentally, when an affirmative result is obtained (when d ≧ | P |), the structure path analysis unit 412 outputs “d” as the information that gives the depth, and the variable v points to the information that gives the path type 703. The node path type is output, and the analysis process ends (step S606).

  On the other hand, when a negative result is obtained in step S602, the structural path analysis unit 412 adds “1” to the variable d (step S603).

  Subsequently, the structural path analysis unit 412 determines whether or not there is a tag name P [d] as a child of the node indicated by the variable v (step S604). When a negative result is obtained (when there are no children), the structure path analysis unit 412 outputs “no such structure” and ends the search process itself (step S607).

  On the other hand, when a positive result is obtained in step S604 (when a child exists), the structure path analysis unit 412 changes the variable v to a child whose tag name is P [i] (step S605).

[Element search operation]
FIG. 14 shows details of the search operation executed in the element search unit 413.

  The element search unit 413 first sets the “total number” of elements having a depth of d and a path type of t as a variable n. This total number is given as a calculated value of rank (T [d], t, | T [d] |). Furthermore, the element search unit 413 sets the variable k to the initial value “0”.

  Next, the element search unit 413 determines whether k> n is satisfied (step S702). Until a positive result is obtained, the element search unit 413 repeatedly executes the processes of steps S703 to S706 described later.

  The element search unit 413 calculates, by select (T [d], t, k), the position where the next element that matches the search query is located in the element of depth d, and the calculation result is calculated. A variable k ′ is set (step S703).

  Next, the element search unit 413 selects select (S, d) as to which element in the sequence S corresponding to the entire XML document the next element that matches the search query (the same element as in step S703) is located. , K ′), and the calculation result is set to a variable k ″ (step S704).

The element search unit 413 outputs the value of the variable k ″ (step S705).
Thereafter, the element search unit 413 adds “1” to the variable k and returns to step S702 (step S706).

[Acceleration of calculation processing]
In order to speed up the calculation of the XML element that matches the structure path by the above-described processing, it is necessary to process the rank operation and the select operation at high speed.

  First, in the present embodiment, the rank operation is used only for counting how many values t exist in T [d] in step S701. In order to speed up the processing, in this embodiment, when reading the DOM tree 422 that has been converted into a sequence from the auxiliary storage device 403, the number of times each element has appeared in advance is counted to determine the path type (value t). A two-dimensional array N 702 (FIG. 7) in which the number of appearances of the structure path is arranged in this order is created. Among N, the contents of the array N [d] corresponding to a specific depth d are given by (number of values 1 in T [d], number of values 2,..., Number of values t,...). For example, in the case of FIG. 7, in the sequence T [2] representing the path type structure with the depth “2”, the value 1 is 2 times, the value 2 is 2 times, the value 3 is 1 time, and the value 4 is 1 time. Appear. Therefore, the d = 2 portion of the two-dimensional array N is created as N [2] = (2, 2, 1, 1).

  In the subsequent select operation in step S703, the k values change in order 1, 2, 3,..., And the points where the value is t in T [d] are calculated in order. This processing can be realized by simply reading the sequence T [d] in order from the top and calculating the location where the value t appears. If the value t appears frequently in the sequence T [d], the calculation efficiency of this process is sufficiently good. However, if the value t does not appear frequently in the sequence T [d], the processing time for scanning the sequence may cause performance degradation. Therefore, when it is predicted that the value t does not appear frequently in the sequence T [d], it is preferable to use the method described in the second embodiment described later. Note that the number of repetitions n may be calculated by the above method and used in step S702. Instead of determining whether k> n, the element search unit 413 is read when all the elements of the sequence T [d] have been read. You may employ | adopt the method of complete | finishing.

  The select operation in step S704 can also be calculated by reading the sequence in order from the top and obtaining the position where the value d appears k '(= select (T [d], t, k)).

  Here, if 1 <k ≦ n, the value of k ′ = select (T [d], t, k) increases monotonously for k. For this reason, the following inequality holds.

select (T [d], t, k-1) <select (T [d], t, k)
Similarly, the value of select (S, d, k ′) also increases monotonously for k ′. For this reason, the following inequality holds.

select (S, d, select (T [d], t, k-1))
<Select (S, d, select (T [d], t, k))

  For this reason, in step S704, select (S, d, select (T [d], t, k-1)), which is the last output value, is stored, and the sequence S is scanned from that position to select ( If S, d, select (T [d], t, k)) is calculated, the calculation time can be shortened.

[Modification]
The various software exemplified in this embodiment can be stored in various recording media such as electromagnetic, electronic, and optical, and can be downloaded to a computer through a communication network such as the Internet.

  In the above description, the method of reading the entire DOM tree 422 converted into a sequence into the main storage device 402 and processing it has been described. However, in actual operation, in order to be able to process a huge amount of data that cannot fit in the main storage device 402, the data is arranged in the auxiliary storage device 403, and a part necessary for processing is read into the main storage device 402 each time. It is preferable.

[Effect of the first embodiment]
If the XML document search apparatus 400 according to this embodiment is used, it is possible to speed up the XML document search process using a search query described in XPath.

[Second Embodiment]
[Wavelet tree]
A Wavelet tree is known as a data structure that can efficiently calculate a rank operation and a select operation while compressing a number sequence in a compact manner (for example, Navarro, G. and Makinen, V., Compressed full). -text indexes, ACM Computing Surveys 39 (1): Article 2, 2007.).

  The Wavelet tree is constructed using a data structure called a bit vector that is an arbitrary sequence of 0s and 1s. An example of the bit vector is shown in FIG. In order to efficiently perform the rank operation and the select operation when the bit vector is viewed as a sequence, the results of the rank operation and the select operation are sampled and stored at regular intervals, and the value of the unsampled portion is a short bit vector. There is known a technique for calculating at high speed by using a table such as 1501 in which the results of the rank operation and the select operation are pre-calculated and stored (see Non-Patent Document 5, for example).

  FIG. 16 shows a structural example of a Wavelet tree for the sequence S “012223230123223”. A Wavelet tree is a data structure in which a bit vector is stored in a node of the tree and information equivalent to a numerical sequence can be recorded as a whole.

  The root node 1601 stores a bit vector B1 that records which group each value belongs to when the values stored in the numerical sequence are divided into two groups. In the example of FIG. 16, grouping is performed depending on whether the value is “2” or not “2”. In a node 1602 below the root node 1601, the values other than “2” grouped by the root node are further grouped, and the bit vector B2 is grouped by “3” or not “3”. Is stored. Similarly, the lower node 1603 distinguishes the remaining “0” and “1”.

  The rank operation on the WEB tree is performed by tracing the tree from the root node 1601. For example, the case of calculating rank (S, 3, 7) will be described as follows.

  First, “3” is grouped into “1” in the root node 1601. Therefore, rank (B1, 1, 7) is calculated to obtain “4”. This result means that there are a total of four, 0, 1, 3 by the seventh in the sequence S.

  In the next bit vector B2, “3” is grouped into “1”. Therefore, rank (B2, 1, 4) is calculated to obtain “2”. This result indicates that among the four 0, 1, 3 appearing up to the seventh in the sequence S, “3” is a total of two. Thus, it can be seen that “2” was correctly calculated as the result of rank (S, 3, 7).

  On the other hand, the select operation is calculated by tracing the tree from the leaf to the root node. For example, the case of calculating select (S, 3, 2) will be described as follows.

  First, in the node 1602 that is the parent node of the leaf representing “3”, the position corresponding to the second “3” is calculated. “3” is grouped into “1” in the node 1602. Therefore, “4” is obtained by calculating select (B2, 1, 2). This result means that the second “3” is the fourth value in the partial sequence from which only 0, 1, 3 are extracted.

  Further, in order to determine what position this value is in the sequence S, select (B1, 1, 4) may be calculated. In this case, “7” is obtained. This result represents that the position where 0, 1, or 3 appears fourth is the seventh value as a whole. That is, “7” is the calculation result of select (S, 3, 2).

  As described above, when a Wavelet tree is used, a rank operation or a select operation on a bit vector can be realized by repeating the processing for the number of times equal to the height of the tree at most. That is, by executing the number of calculation processes equal to the height of the tree at the maximum, it is possible to obtain a solution of rank operation and select operation for the sequence.

  The height of the wavelet tree is much smaller than the length of the sequence, especially when the sequence is very long and the frequency of the second argument of the rank operation or select operation is small. It is more efficient than scanning. Further, it is known that the Wavelet tree can be compressed if the appearance frequency of each value is biased in the stored numerical sequence.

[Device configuration]
In this embodiment, a means for applying a Wavelet tree to a place where a rank operation and a select operation are executed in the first embodiment is provided.

  FIG. 17 shows a block configuration of an XML document search apparatus 1700 according to this embodiment. In FIG. 17, parts corresponding to those in FIG. The XML document search apparatus 1700 is different from the first embodiment in that it has a Wavelet tree construction unit 411 and a Wavelet tree group 423.

  FIG. 18 shows a cooperative operation between components when the XML document search apparatus 1700 according to this embodiment pre-processes an XML document.

  First, the user of the XML document search apparatus 400 instructs the start of processing using the user interface 406 (step S801).

  Receiving the processing start instruction, the CPU 401 reads the XML document set 420 from the auxiliary storage device 403 (step S802). The read XML document set 420 is stored in the main storage device 402.

  Next, the XML document analysis unit 410 (CPU 401) analyzes each XML document included in the XML document set 420, and generates a path trie 421, a digitized DOM tree 422, and text data 424 (step S803). The steps so far are the same as those in the first embodiment.

  The Wavelet tree construction unit 411 converts each number sequence included in the numbered DOM tree 422 generated by the XML document analysis unit 410 into the Wavelet tree group 423 (step S804). The Wavelet tree construction unit 411 converts the DOM tree 422 converted into a sequence into a Wavelet tree group 423 using a known method (see, for example, Non-Patent Document 5). Then, the Wavelet tree group 423 is output to the auxiliary storage device instead of the DOM tree 422 converted into a sequence, and the path trie 421 and the text data 424 are also output (step S805). The sequenced DOM tree 422 may be deleted after the Wavelet tree group 423 is constructed.

[Overview of preprocessing]
FIG. 19 is a flowchart for explaining the flow of preprocessing executed by the XML document search apparatus 400 according to this embodiment before search.

  When the XML document set 420 is input, the CPU 401 executes analysis processing by the XML document analysis unit 410 (step S901). The XML document analysis unit 410 analyzes each XML document included in the XML document set 420, and generates a path trie 421, a DOM tree 422 converted into a sequence, and text data 424.

  When this process ends, the CPU 401 constructs the Wavelet tree group 423 as described above (step S902).

  Thereafter, the CPU 401 outputs the path trie 421, the wavelet tree group 123, and the text data 424 to the auxiliary storage device 403 (step S903).

[Search processing]
The XML document search apparatus 1700 according to the present embodiment first reads out the path trie 421 and the Wavelet tree group 423 from the auxiliary storage device 403 during the search. As described above, the DOM tree 422 converted into a number sequence is not necessary.

  The search operation is executed according to FIG. 14 as in the first embodiment. The difference from the first embodiment is that a rank operation in step S701 (FIG. 14) executed after a structural path as a search query is given, and a select operation in steps S703 (FIG. 14) and S704 (FIG. 14). A point where a Wavelet tree is used for the two-dimensional array N702 (FIG. 7) is unnecessary. However, since the array reference processing is faster than the rank operation, the two-dimensional array N702 may be used also in this embodiment.

[Effect of the second embodiment]
If the XML document search apparatus 1700 according to this embodiment is used, even when the value t does not appear frequently in the sequence T [d], the XML document search process using the search query described in XPath can be executed at high speed. .

[Third embodiment]
In a search using XPath, there are cases where a constraint condition related to a parent element, a child element, a sibling element, or the like is included in a search query. Therefore, in this embodiment, a function for searching for a parent element, a child element, a sibling element, and the like and a function for checking whether or not any two elements i and j are in a specified relationship will be described. . In the following description, the depths of the elements i and j are di and dj, and the path types are ti and tj, respectively.

(1) Calculation of parent element of element i When the depth di ≦ 1, there is no parent element of element i. Otherwise, the parent element is the element whose element number is less than i and closest to i, given depth di-1. Therefore, the element number of the parent element can be obtained by calculating select (S, di-1, rank (S, di-1, i)).

(2) Calculation of the first child element of element i If there is a child element, the element number is given by i + 1. However, a child element may not exist, and in this case, the depth of the element with element number i + 1 is other than di + 1. As described in the first embodiment, when the DOM tree 422 converted into a sequence is used, it may be determined whether S [i + 1] = di + 1, but as described in the second embodiment. If only the Wavelet tree group 423 can be used, it may be determined whether rank (S, di + 1, i + 1) = rank (S, di + 1, i) +1. In the latter case, the CPU 401 determines the presence of a child element according to the processing procedure shown in FIG. 20 (step S1001), and outputs the element number i + 1 if it exists (step S1002). “No element” is output (step S1003).

(3) Calculation of sibling element of element i When there is a sibling element before element i, the previous sibling element has the same depth di as element i, and is the element number closest to i less than i It becomes the element of. Therefore, the element number j can be given as j = select (S, di, rank (S, di, i-1)).

  On the other hand, when there is a sibling element after the element i, the next sibling element has the same depth di as the element i, and is an element having an element number greater than i and closest to i. Therefore, the element number j can be given as j = select (S, di, rank (S, di, i) +1).

  On the other hand, when such a sibling element does not exist, in any case, only the parent element of either i or j exists between the element i and the element j. Therefore, if rank (S, di-1, i) = rank (S, di-1, j), j is a sibling element, otherwise the sibling element does not exist.

  This determination process is represented by flowcharts in FIGS. 21 and 22. FIG. 21 is a flowchart for searching for the previous sibling element, and FIG. 22 is a flowchart for searching for the next sibling element.

  First, the flowchart shown in FIG. 21 will be described. As described above, the previous sibling element j has the same depth di as the element i, and is an element having an element number closest to i but less than i. Accordingly, in step S1101, j = select (S, di, rank (S, di, i-1)) is calculated. Next, the number up to the i-th (= rank (S, di-1, i) and the number up to the j-th (= rank (S, di-1), which is one depth less than the depth di of the element i. , I) are compared (step S1102) If the two values are the same, a child of the same parent element is output, and j calculated in step S1101 is output (step S1103). “None” is output (step S1104).

  Next, the flowchart shown in FIG. 22 will be described. As described above, the next sibling element j has the same depth di as the element i, and is an element having an element number greater than i and closest to i. Accordingly, in step S1201, j = select (S, di, rank (S, di, i) +1) is calculated. Next, the number up to the i-th (= rank (S, di-1, i)) and the number up to the j-th (= rank (S, di-1, j)) are compared with the depth one shallower than the element i. (Step S1202) If the two values are the same, since it is a child of the same parent element, output j calculated in Step S1201 (Step S1203) If the two values are different, output that there is no sibling element (Step S1202). Step S1204).

(4) Determination of whether element i is the parent of element j Calculate the parent element of element j by the method of (1) above, and determine whether the calculated value of the parent element matches element i .

(5) Determination of whether element i is an ancestor of j If the following conditions are satisfied simultaneously, it is determined that the element i is an ancestor.
(5-1) i <j
If this condition is satisfied, the start tag for element i appears before the start tag for element j.
(5-2) rank (S, i, di) = rank (S, j, di)
If this condition is satisfied, there is no element other than element i having a depth of di or less between element i and element j.

(6) Determining whether or not the elements i and j are sibling elements The parent elements of the elements i and j are calculated, and it is determined whether or not they match.

  If rank (S, i, di-1) = rank (S, j, dj-1), it may be determined that the element is a sibling element.

[Fourth embodiment]
A search query defined in XPath may match a plurality of structure paths. For example, the search query “/ a // text ()” matches all text nodes that are descendants of the root element whose tag name is “a”. When such a search query is given, all the structural paths that match the query are calculated on the path trie, and the union of those search results may be taken.

[Fifth Embodiment]
A search query defined in XPath may include a condition related to text. For example, the search query ""/a//text()[contains(.,"abc")]" matches text that is a descendant of the root element with the tag name "a" and contains "abc". To do. When such a search query is given, the search result relating to the XML element described above and the text search result for the text data 424 may be collated, and a location that satisfies both conditions may be used as the search result.

  Any known technique can be used for the text search process (see, for example, Non-Patent Document 6).

[Other examples]
The above-described embodiments are examples of application of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the embodiments described above. Various modifications can be made without departing from the scope of the present invention. For example, the present invention does not have to include all the constituent elements of the respective embodiments described above. In the phone invention, a part of one embodiment can be replaced with the structure of another embodiment, and the structure of another embodiment can be added to the structure of one embodiment.

  Moreover, you may implement | achieve some or all of each structure, a function, a process part, a process means, etc. which were mentioned above as an integrated circuit or other hardware, for example. In addition, information such as programs, tables, and files that realize each processing function can be stored in a storage device such as a memory, a hard disk, or a solid state drive (SSD), or a storage medium such as an IC card, an SD card, or a DVD. .

400 XML document search device 401 CPU
402 Main storage device 403 Auxiliary storage device 404 Removable drive 406 User interface 407 Network interface 410 XML document analysis unit 411 Wavelet tree construction unit 412 Structure path analysis unit 413 Node search unit 420 XML document set 421 Path trie 422 Numbered DOM tree 423 Wavelet tree 424 Text data 430 External storage device 440 Network 1700 XML document retrieval device

Claims (8)

A structure path having a processor, a main storage device, and an input / output device for inputting / outputting an XML document, and as a search query, lists all elements of the XML document and ancestor elements of the elements in order from the root element. Given an XML document search device that searches for a location that matches the structure path,
The input / output device receives an input of an XML document set to be searched,
The XML document search device
An XML document analysis unit that analyzes the XML document, recognizes a tag type and an inclusion relationship, converts the tag document into a sequence of numbers, and constructs a path trie;
Using the path trie, a structure path analysis unit that calculates the depth and path type of the structure path that is a search query;
An element search unit that calculates a location where an element that matches the structure path that is a search query appears based on the depth and path type of the structure path;
The XML document analysis unit
In order to record the shape of the DOM tree of the XML document, a first number sequence S including a numerical sequence representing the depth of the element as a partial sequence in the order of appearance of the element in the XML document;
A sequence group T composed of one or more sequences that record the type of the structure path corresponding to each node of the DOM tree, and a sequence path T [d] included in the sequence group T has a depth d When recording the type of
The element search unit includes:
An XML document search apparatus characterized in that by scanning the number sequence S and the number sequence group T, a location that matches a structure path as a search query is calculated. The XML document search device according to claim 1,
The element search unit includes:
The position at which the value c appears in the sequence A is denoted as select (A, c, j), and the process for calculating this value is referred to as a select operation. The depth of the structure path of the search query is d, the path When the type is t and the total number of occurrences of the structure path is n, the value of k ″ obtained by applying the formula (1) is calculated for the integer k where 1 ≦ k ≦ n. An XML document search apparatus, wherein a part that matches a search query is searched from the XML document set.
[Equation 1]
k ″ = select (S, d, select (T [d], t, k)) (1) The XML document search device according to claim 2,
Subsequent to the processing of the XML document analysis unit, a Wavelet tree construction unit that converts each number sequence included in the number sequence S and the number sequence group T into a Wavelet tree,
The XML document search apparatus, wherein the Wavelet tree is used for the select operation. The XML document search device according to claim 1,
In order to record the shape of the DOM tree of the XML document, means for constructing a numerical sequence including a numerical sequence representing the depth of the element as a partial sequence in the order of appearance of the element in the XML document,
The number of values c up to the i-th in the sequence S is expressed as rank (S, c, i), and the process of calculating this value is called a rank operation.
The position where the value c appears in the sequence S in the sequence S is expressed as select (S, c, j), and the process of calculating this value is called a select operation.
For the element of the XML document corresponding to the i-th value in the sequence, when the depth of the structure path of the element is d,
The number of the element that is the parent of the element is calculated by equation (2),
Calculate the number of the element that is the first child of the element by equation (3),
Calculate the number of the preceding sibling element closest to the element by equation (4);
An XML document search apparatus characterized in that the number of a subsequent sibling element closest to the element is calculated according to formula (5).
[Equation 2]
select (S, d-1, rank (S, d-1, i)) (2)
[Equation 3]
i + 1 Formula (3)
[Equation 4]
select (S, d, rank (S, d, i-1)) (4)
[Equation 5]
select (S, d, rank (S, d, i) +1) (5) The XML document search device according to claim 4,
Subsequent to the processing of the XML document analysis unit, a Wavelet tree construction unit that converts each number sequence included in the number sequence S and the number sequence group T into a Wavelet tree,
The XML document search apparatus, wherein the Wavelet tree is used for the rank operation and the select operation. When a search path is provided with a structure path that lists all elements of an XML document and ancestor elements of the element in order from the root element, the computer is caused to execute a process of searching for a location that matches the structure path. In the program
The program is
A first process of analyzing the XML document, recognizing a tag type and an inclusion relation, converting the XML document into a sequence of numbers, and constructing a path trie;
A second process of calculating the depth and path type of the structural path that is the search query using the path trie;
Based on the depth of the structural path and the path type, the computer is caused to execute a third process for calculating a location where an element that matches the structural path that is the search query appears,
The first process includes
In order to record the shape of the DOM tree of the XML document, each node in the DOM tree includes a first numerical sequence S including a numerical sequence representing the depth of the element as a partial sequence in the order of appearance of the element in the XML document. And a sequence group T composed of one or more sequences that record the type of the structure path corresponding to, and records the types of structure paths in which the sequence T [d] included in the sequence group T has a depth d When
The third process includes
A program that scans the sequence S and the sequence group T to calculate a location that matches a structural path that is a search query. The program according to claim 6,
The third process includes
The position at which the value c appears in the sequence A is denoted as select (A, c, j), and the process for calculating this value is referred to as a select operation. The depth of the structure path of the search query is d, the path By calculating the value of k ″ obtained by applying Equation (6) for an integer k where 1 ≦ k ≦ n, where t is the type and n is the total number of occurrences of the structure path, A program that searches the XML document set for a location that matches a search query.
[Equation 6]
k ″ = select (S, d, select (T [d], t, k)) (6) The program according to claim 7,
Subsequent to the first process, there is a fourth process for converting each number sequence included in the number sequence S and the number sequence group T into a Wavelet tree,
The program using the Wavelet tree for the select operation.

Images