JP2014238863A - Retrieval apparatus, retrieval method, and retrieval program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately extract data corresponding to a retrieval condition from compressed data while keeping the compressed data.SOLUTION: When compressing document data, a compression processing part 10a compresses the document data, by converting character strings included in the document data and break characters included in the document data into codes, separately. Since the compressed document data are compressed separately for the character strings and the segmentation characters, a retrieval processing part 10c can obtain proper retrieval result even if the compressed data is extracted as it is, on the basis of a retrieval expression.

Description

  The present invention relates to a search device and the like.

  In recent years, a technique has been devised for extracting records and items to be searched from compressed data in a compressed state. Thus, a technique for extracting records and items in a compressed state is referred to as compressed data extraction. Further, the compressed data is referred to as compressed data, and the data before being compressed is referred to as original data.

  If the compressed data extraction is used, it is not necessary to decompress the compressed data, so that the data area can be saved. In addition, by using a specific compression method, a scan-type algorithm such as an AC machine (Aho-Corasick machines) can search at a higher speed when searching for compressed data than when searching for original data.

  FIG. 49 is a diagram for explaining the basic idea of conventional compressed data extraction. As a precondition, it is assumed that the original data 10a in FIG. 49 is compressed into compressed data 10b by BPE (Byte Pair Encoding) compression technology. In conventional compressed data extraction, when an extraction condition is specified, the appearance position of a keyword corresponding to the extraction condition is specified. In the compressed data extraction, records, items, elements, and the like included in the keyword appearance positions are extracted from the compressed data as they are and output.

  For example, assume that the extraction condition “ENTRY / DDB / update-data [./= 19990205]” is designated. The meaning of the extraction condition is to extract the update-data element including “19990205”. In the conventional compressed data extraction, when the keyword appearance position corresponding to the extraction condition is specified using the existing compression search technique, the keyword appearance position is “<up · address · 0205 profit p · drinking” of the compressed data 10b. " In the conventional compressed data detection, “<up · address · 0205 profit p · drink” is output as it is as a search result. In the search process, no access to the original data 10a occurs.

  By the way, as a technique for compressing data, a technique for inserting a delimiter at an arbitrary position in the middle of compressing text data has been disclosed. By inserting a delimiter at an arbitrary position, the processing load can be reduced by decompressing only the partial character string sandwiched between delimiters without decompressing the entire character string.

  In addition, as a technique related to an image compression method, a technique is also known in which an image is divided into predetermined areas, and an area portion of the divided image and a boundary portion of the divided image are independently compressed. By compressing the region portion and the boundary portion independently, it is possible to extract image data of a desired region without decompressing all the images.

JP 2000-22552 A JP 2008-113267 A

T.Kida, T.Matsumoto, M.Takeda, A.Shinohara, and S.Arikawa.Multiple pattern matching algorithms on collage system, In Proc.12th Annual Symposium on Combinatorial Pattern Matching (CPM2001) M.Takeda, S.Miyamoto, T.Kida, A.Shinohara, S.Fukamachi, T.Shinohara, and S.Arikawa.Processing Text Files as Is: Pattern Matching over Compressed Texts, Multi-Byte Character Texts, and Semi- Structured Texts. In Proc. 9th International Symposium on String Processing and Information Retrieval (SPIRE2002)

  If the original data is directly compressed into compressed data, delimiters such as records, items, elements, and XML (Extensible Markup Language) tags may be collectively replaced with another code. As described above, when compression is performed without considering the data delimiter, there is a problem that data corresponding to the extraction condition cannot be extracted as compressed data.

  For example, in the compressed data 10b of FIG. 49, when the pre-compression data corresponding to “drink” is “> <”, the extraction condition “ENTRY / DDB / update-data [./= 19990205]” is specified. In addition, the search result “<up / address / 0205 profit / drink” is output as the search result. When the search result is decompressed, “<update-date> 19990202 </ update-date> <” is obtained, and the search result is not correct.

  When the extraction condition “ENTRY / DDB / update-data [./= 19990205]” is specified, the correct search result after decompression is “<update-date> 19990202 </ update-date>”. If an incorrect search result is input to the XML data processing system, an error occurs.

  Decompress the search result "<up / address / 0205 profit p / drink" and correct the search result so that the correct search result is <update-date> 19990202 </ update-date>. Can be obtained, but an extra calculation cost is required.

  Here, when data is compressed using conventional technology, if a delimiter is inserted, the record / item / element can be distinguished from the delimiter, so the data corresponding to the extraction condition is extracted as compressed data. It becomes possible. However, if a delimiter is inserted for each delimiter, the amount of compressed data increases, which is not practical.

  In addition, when compressing data, if the boundary part is compressed independently, the record / item / element can be distinguished from the delimiter, so the data corresponding to the extraction condition can be transferred to the compressed data without increasing the amount of data. It is possible to extract as it is. However, the technology that compresses the boundary part independently is applicable only when the boundary part and other areas are fixed, so the source that contains records, items, elements, and delimiters at random. It is difficult to apply such compression techniques to data.

  It is an object of the present invention to provide a search device, a search method, and a search program that can accurately extract data corresponding to a search condition from compressed data while being compressed.

  In the first proposal, among the data including the first data and the second data indicating the attribute of the first data, at least the second data is encoded with a code smaller in size than the second data. When a search condition including data indicating the attribute of the specified data is received from the replacement data replaced with the digitized data, the search device stores the relationship before and after the replacement of the second data. Replace with reference to the storage unit, a replacement processing unit that replaces the data indicating the attribute of the specified data with the encoded data, and a search condition in which the data indicating the attribute of the specified data is replaced with the encoded data And a search processing unit that acquires data included in an area on replacement data specified by a search condition from the data.

  According to the present invention, data corresponding to a search condition can be accurately extracted from compressed data while being compressed.

FIG. 1 is a diagram illustrating the configuration of the search system according to the first embodiment. FIG. 2 is a diagram illustrating the configuration of the search device according to the second embodiment. FIG. 3 is a diagram illustrating an example of the data structure of XML data. FIG. 4 is a diagram illustrating an example of a data structure of encoded data. FIG. 5 is a diagram illustrating an example of the data structure of the correspondence table. FIG. 6 is a diagram illustrating an example of the data structure of the AC machine. FIG. 7 is a diagram illustrating an example of the data structure of the state structure. FIG. 8 is a diagram illustrating an example of a data structure of compression dictionary data. FIG. 9 is a diagram illustrating an example of a data structure of the compressed data AC machine. FIG. 10 is a diagram illustrating an operation example of character string compression. FIG. 11 is a diagram illustrating an example of a data structure of character string compressed data. FIG. 12 is a diagram (1) for explaining the tri-T construction process. FIG. 13 is a diagram (2) for explaining the tri-T construction process. FIG. 14 is a diagram (3) for explaining the tri-T construction process. FIG. 15 is a diagram (1) for explaining the failer transition addition process. FIG. 16 is a diagram (2) for explaining the failer transition addition process. FIG. 17 is a diagram (3) for explaining the failer transition addition process. FIG. 18 is a diagram (4) for explaining the failer transition addition process. FIG. 19 is a diagram (5) for explaining the failer transition addition process. FIG. 20 is a diagram (6) for explaining the failer transition addition process. FIG. 21 is a diagram (7) for explaining the failer transition addition process. FIG. 22 is a diagram (8) for explaining the failer transition addition process. FIG. 23 is a diagram (9) for explaining the failer transition addition process. FIG. 24 is a diagram (1) for explaining the compressed data AC machine construction processing. FIG. 25 is a diagram (2) for explaining the compressed data AC machine construction processing. FIG. 26 is a diagram (3) for explaining the compressed data AC machine construction processing. FIG. 27 is a diagram (1) for explaining the collation processing. FIG. 28 is a diagram (2) for explaining the collation processing. FIG. 29 is a diagram (3) for explaining the collation processing. FIG. 30 is a diagram (4) for explaining the collation processing. FIG. 31 is a diagram (5) for explaining the collation processing. FIG. 32 is a diagram (6) for explaining the collation processing. FIG. 33 is a flowchart illustrating a processing procedure of compression processing. FIG. 34 is a flowchart showing the procedure of the character string counting process. FIG. 35 is a flowchart showing a processing procedure of dictionary construction / character string replacement processing. FIG. 36 is a flowchart illustrating a processing procedure of tag name replacement processing. FIG. 37 is a flowchart illustrating the processing procedure of the construction processing. FIG. 38 is a flowchart illustrating the processing procedure of the extraction condition reception processing. FIG. 39 is a flowchart showing the processing procedure of the AC machine construction processing. FIG. 40 is a flowchart illustrating a processing procedure of processing for constructing a cocoon tri-T. FIG. 41 is a flowchart showing a processing procedure of pattern registration processing. FIG. 42 is a flowchart (1) illustrating a processing procedure of processing for adding a failer transition to a trie (Π). FIG. 43 is a flowchart (2) illustrating a processing procedure of a process of adding a failer transition to a trie (Π). FIG. 44 is a flowchart of a process procedure for adding a skip transition to the AC machine AC (Π). FIG. 45 is a flowchart of a process procedure of the first skip transition creation process. FIG. 46 is a flowchart showing the processing procedure of the second skip transition creation processing. FIG. 47 is a flowchart showing the processing procedure of the extraction position calculation process. FIG. 48 is a diagram illustrating a hardware configuration of a computer constituting the search device according to the embodiment. FIG. 49 is a diagram for explaining the basic idea of conventional compressed data extraction.

  Embodiments of a search device, a search method, and a search program disclosed in the present application will be described below in detail with reference to the drawings. Note that the present invention is not limited to the embodiments.

  FIG. 1 is a diagram illustrating the configuration of the search system according to the first embodiment. As shown in FIG. 1, the search device 10 includes a compression processing unit 10a, a replacement processing unit 10b, and a search processing unit 10c. Among these, the compression processing unit 10a acquires a data set in which each data is divided by the division information indicating the attribute of the data, replaces the data set with encoded data based on the data included in the data set, and further, It is a processing unit that compresses a data set by replacing with encoded data based only on delimiter information indicating attributes.

  When the replacement processing unit 10b obtains a search condition having attributes and data to be searched for the data set compressed by the compression processing unit 10a, the replacement processing unit 10b and the partition information before replacement The processing unit replaces the attribute of the search condition with the encoded data based on the relationship.

  The search processing unit 10c specifies an area on the compressed data set specified in the search condition based on the search condition in which the attribute of the search condition is replaced with the encoded data and the compressed data set, Output data contained in the specified area.

  According to the search system 10 described above, when the compression processing unit 10a compresses the data set, the data set is compressed based on the delimiter information included in the data set. Since the compressed data set is compressed based on the delimiter information, the search processing unit 10c can obtain an accurate search result even if the compressed data is extracted as it is based on the search condition.

  FIG. 2 is a diagram illustrating the configuration of the search device according to the second embodiment. As illustrated in FIG. 2, the search device 100 includes an input unit 110, an output unit 120, a communication control IF unit 130, an input / output control IF unit 140, a storage unit 150, and a control unit 160. The search device 100 is connected to a terminal device (not shown) via a network.

  The input unit 110 is an input unit that inputs various types of information, and includes a keyboard, a mouse, a microphone, and the like. In addition, a monitor (output unit 120), which will be described later, also realizes a pointing device function in cooperation with the mouse.

  The output unit 120 is an output unit that outputs various types of information, and includes a monitor (or display, touch panel), a speaker, and the like. The communication control IF unit 130 is a processing unit that controls communication with a terminal device (not shown). The input / output control IF unit 140 is a unit that controls input / output of data by the input unit 110, the output unit 120, the communication control IF unit 130, the storage unit 150, and the control unit 160.

  The storage unit 150 is a storage unit that stores data and programs necessary for various processes performed by the control unit 160. The storage unit 150 includes XML data 150a, encoded data 150b, correspondence table 150c, AC machine 150d, compression dictionary 150e, compressed data AC machine 150f, extraction condition 150g, and post-conversion extraction condition 150h.

  The XML data 150a is document data having a hierarchical structure in which elements are separated by element identifiers “<”, “</”, and the like. FIG. 3 is a diagram illustrating an example of the data structure of XML data. The encoded data 150b is compressed XML data. FIG. 4 is a diagram illustrating an example of the data structure of the encoded data 150b.

  The correspondence table 150c is a table in which the tag name of the XML data 150a is associated with the code corresponding to the tag name. FIG. 5 is a diagram illustrating an example of the data structure of the correspondence table 150c.

  The AC machine 150d is data including a combination of a finite number of states and transitions. By inputting text data (character string) to the AC machine 150d, it is possible to collate whether or not a specific keyword is included in the text data. FIG. 6 is a diagram illustrating an example of the data structure of the AC machine 150d. The AC machine 150d is an AC machine that collates whether or not the keywords AC, BA, BB, BAA, and BACD are included in the text data.

  As shown in FIG. 6, AC machine 150d has states 0-8. Each state is classified into an initial state, a normal state, and a collation state. The initial state is a state in which text data is first collated. The normal state is a state that is collated with text data after the second. The collation state is a state that transitions when the text data hits a specific keyword. In the example shown in FIG. 6, the initial state is state 1, the normal state is states 1 and 3, and the collation state is states 2 and 4-8.

  The AC machine sequentially checks the characters of the text data from the initial state 0, and repeats the normal transition and the failure transition to determine the keyword included in the text data. Here, the normal transition indicates a transition when there is a transition destination by a character to be compared with text data in a state to be collated. For example, in state 1, when the character to be compared with text data is C, a normal transition is made. When the character of the text data is C, a normal transition is made from state 1 to state 2.

  On the other hand, if the conditions for the normal transition are not met, a failer transition occurs. For example, in the state 1, when the character to be compared with the text data is other than C, a failer transition is made. When a failer transition occurs in state 1, state 0 is entered. There is one AC machine in each state. It is assumed that the failer transitions omitted in FIG. 6 are all directed to the initial state 0.

  Here, a data structure of a state (state structure) included in the AC machine shown in FIG. 6 will be described. FIG. 7 is a diagram illustrating an example of the data structure of the state structure. As shown in FIG. 7, this state structure has a state ID for identifying each state, a pattern list, a pointer to a normal transition destination, a pointer to a normal transition destination, and a pointer to a failer transition destination. .

  Among these, the pattern list stores character strings from the initial state 0 to the transition destination state. The pattern list is associated with a pointer to the corresponding normal transition destination. For example, in the state structure of state 1, if the pointer of the normal transition destination that transitions from state 1 to state 2 is stored in g [1], plist [1] contains the initial state 0 to state 2 The character string “AC” is stored.

  The compression dictionary data 150e is a table in which character sets are associated with codes associated with the character sets. FIG. 8 is a diagram illustrating an example of the data structure of the compressed dictionary data 150e. For example, as shown in FIG. 8, a pair of characters B and A corresponds to the code X. The compressed dictionary data 150e is used when generating the compressed data AC machine 150f from the AC machine 150d.

  The compressed data AC machine 150f is an AC machine in which a skip transition is added to the AC machine 150d in order to check whether or not a specific keyword is included in the compressed data. FIG. 9 is a diagram illustrating an example of a data structure of the compressed data AC machine 150f.

  Here, the skip transition is a normal transition that skips the state. For example, if the state to be collated is state 0, and the character to be compared in the compressed data is X, state 3 is skipped and the state transitions to state 4.

  The extraction condition 150g is a condition for data extracted from the XML data 150a. For example, the extraction condition 150g is “/ AAA / BBB =“ BACD ””. Here, “/ AAA / BBB” is an item name that specifies a search position. The search condition "/ AAA / BBB =" BACD "" is a character string that includes the character string "BACD" among the character strings sandwiched between the delimiter characters "BBB" that exist in the lower hierarchy of the delimiter character "AAA". Search conditions for searching.

  The post-conversion extraction condition 150h is an extraction condition in which a tag name is converted based on the correspondence table 150c. For example, the post-conversion extraction condition 150h is “/ 1/2 =“ BACD ””. The AC machine 150d is generated based on the keyword included in the post-conversion extraction condition 150h.

  The control unit 160 has an internal memory for storing programs defining various processing procedures and control data, and executes various processes using these. As shown in FIG. 2, the control unit 160 includes a data management unit 160a, a data compression processing unit 160b, an AC machine construction unit 160c, a collation processing unit 160d, and a cut-out processing unit 160e.

  The data management unit 160 a is a processing unit that stores the acquired XML data and extraction conditions in the storage unit 150 when the XML data and extraction conditions are acquired from the input unit 110 and the communication control IF unit 130.

  The data compression processing unit 160b is a processing unit that executes processing for compressing the XML data 150a to generate encoded data 150b, and processing for converting the extraction condition 150g to generate a post-conversion extraction condition 150h.

  First, processing for compressing the XML data 150a and generating encoded data 150b will be described. The process of generating the encoded data 150b includes a process of compressing a character string sandwiched between tags of the XML data 150a and a process of replacing a tag name.

  In the process of compressing the character string, the data compression processing unit 160b refers to the character string and identifies the adjacent character string of length 2. In the following description, adjacent character strings of length 2 are referred to as character pairs. The data compression processing unit 160b compresses the character string by counting the number of appearances of the character pairs included in the character string and assigning the character pair having the maximum appearance number to unused characters (codes). The data compression processing unit 160b repeatedly executes the above processing until the number of appearances of character pairs included in the character string is less than the count threshold.

  FIG. 10 is a diagram illustrating an operation example of character string compression. In the description of FIG. 10, the character string to be compressed is “ABABACBACD”. Further, the count threshold α is 2, and the character set is “A, B, C, D, X, Y, Z”. The data compression processing unit 160b refers to the character string “ABABACBACD” and identifies a character pair. Character pairs included in the character string “ABABACBACD” are “A, B”, “B, A”, “A, C”, “C, B”, “C, D”.

  The data compression processing unit 160b compares the character string with each character pair, counts the number of appearances of each character pair, and registers it in the counting table. It is assumed that the count table is held by the data compression processing unit 160b. The number of appearances of the character pair “A, B” is 2, the number of appearances of the character pair “B, A” is 3, the number of appearances of the character pair “A, C” is 2, and the number of appearances of the character pair “C, B” is 1. The number of appearances of the character pair “C, D” is 1. See step S1.

  Subsequently, the data compression processing unit 160b specifies a character pair having the maximum number of appearances among the character pairs registered in the counting table, and assigns an unused character to the specified character pair. The relationship between the character pair and the unused character is registered in the compressed dictionary data 150e.

  Referring to the counting table created in step S1, the number of appearances of the character pair “B, A” is maximized, so the data compression processing unit 160b assigns an unused character “X” to the character pair “B, A”. . By assigning the unused character “X” to the character pair “B, A”, the character string “ABABACBACD” becomes the character string “AXXCXCD”. The data compression processing unit 160b registers the relationship between the character pair “B, A” and the unused character (symbol) “X” in the compression dictionary data 150e.

  The data compression processing unit 160b refers to the character string “AXXCXCD” and identifies a character pair. Character pairs included in the character string “AXXCXCD” are “A, X”, “X, X”, “X, C”, “C, X”, and “C, D”.

  The data compression processing unit 160b compares the character string with each character pair, counts the number of appearances of each character pair, and registers it in the counting table. The number of appearances of the character pair “A, X” is 1, the number of appearances of the character pair “X, X” is 1, the number of appearances of the character pair “X, C” is 2, and the number of appearances of the character pair “C, X” is 1. The number of appearances of the character pair “C, D” is 1. See step S2.

  Referring to the counting table created in step S2, the number of appearances of the character pair “X, C” is maximized, so the data compression processing unit 160b assigns an unused character “Y” to the character pair “X, C”. . By assigning unused characters to the character pair “X, C”, the character string “AXXCXCD” becomes the character string “AXYYD”. The data compression processing unit 160b registers the relationship between the character pair “X, C” and the unused character (symbol) “Y” in the compression dictionary data 150e.

  The data compression processing unit 160b refers to the character string “AXYYD” and identifies a character pair. Character pairs included in the character string “AXYYD” are “A, X”, “X, Y”, “Y, Y”, and “Y, D”.

  The data compression processing unit 160b compares the character string with each character pair, counts the number of appearances of each character pair, and registers it in the counting table. The number of appearances of the character pair “A, X” is 1, the number of appearances of “X, Y” is 1, the number of appearances of “Y, Y” is 1, and the number of appearances of “Y, D” is 1. See step S3.

  The data compression processing unit 160b ends the compression of the character string when the number of appearances of each character pair is less than the count threshold α or when there are no unused characters in the character set. The data compression processing unit 160b outputs the compressed character string and the compressed dictionary data 150e when the compression of the character string is completed. See step S4.

  The data compression processing unit 160b performs the above-described compression processing on all character strings included in the XML data 150a. In the following description, XML data obtained by compressing a character string is referred to as character string compressed data. FIG. 11 is a diagram illustrating an example of a data structure of character string compressed data. Comparing the XML data 150a in FIG. 3 with the character string compressed data in FIG. 11, the character string “ABABACBACD” is compressed into the character string “AXYYD”.

  Next, processing for replacing a tag name will be described. The data compression processing unit 160b scans the character string compressed data and registers all tag names appearing in the character string compressed data in the correspondence table 150c. The data compression processing unit 160b assigns a unique code to each tag name registered in the correspondence table 150c. Then, the data compression processing unit 160b generates the encoded data 150b by comparing the character string compressed data with the correspondence table 150c and replacing the tag name of the character string compressed data with the corresponding code. .

  For example, the character string compressed data is the character string compressed data shown in FIG. 11, and the correspondence table 150c is the correspondence table shown in FIG. In this case, the data compression processing unit 160b replaces the tag names “AAA”, “BBB”, and “CCC” in the character string compressed data with the codes “1”, “2”, and “3”, respectively. The encoded data shown in FIG.

  Next, a process for converting the extraction condition 150g to generate the post-conversion extraction condition 150h will be described. The data compression processing unit 160b compares the extraction condition 150g with the correspondence table 150c, and replaces the tag name included in the extraction condition 150g with a code to generate a post-conversion extraction condition 150h.

  For example, the correspondence table 150c is the correspondence table shown in FIG. 5, and the extraction condition 150c is “/ AAA / BBB =“ BACD ””. In this case, the data compression processing unit 160b replaces the tag names “AAA” and “BBB” of the extraction condition 150c with “1” and “2”, respectively, and the post-conversion extraction condition 150h “/ 1/2 =“ BACD ” Is generated.

  Next, the AC machine construction unit 160c shown in FIG. 2 will be described. Based on the keyword included in the post-conversion extraction condition 150h, the AC machine construction unit 160c generates an AC machine 150d that matches the keyword. Then, the AC machine construction unit 160c generates a compressed data AC machine 150f by adding a skip transition to the AC machine 150d based on the compressed dictionary data 150e.

  The AC machine construction unit 160c will be specifically described below. After describing the process of generating the AC machine 150d by the AC machine construction unit 160c, the process of generating the AC machine 150f for compressed data will be described. Further, a set of keywords included in the post-conversion extraction condition 150h is a pattern set Π, and for convenience of explanation, a pattern set Π = {AC, BA, BB, BAA, BACD}.

  The process of generating the AC machine 150d includes a process of building a trie and a process of adding a failer transition to the trie. 12 to 14 are diagrams for explaining the tri-T construction process.

  In FIG. 12, the AC machine construction unit 160c creates the initial state 0, and sets all the normal transition destinations in the state structure in the initial state to the initial state (FIG. 12, step S10). Then, the AC machine construction unit 160c extracts the pattern “AC” from the pattern set Π. The AC machine construction unit 160 c selects the character “A” and sets the normal transition destination of the initial state 0 by the character A to the normal state 1.

  The AC machine construction unit 160c selects the character “C” and sets the normal transition destination of the normal state 1 by the character C to the collation state 2. Also, the AC machine construction unit 160c sets Plist [C] = AC in the pattern structure corresponding to the pointer g [C] to the collation state 2 in the state structure in the normal state 1 (FIG. 12, step S11). .

  The AC machine construction unit 160c returns to the initial state 0, and extracts the pattern “BA” from the pattern set Π. The AC machine construction unit 160 c selects the character “B” and sets the normal transition destination of the initial state 0 by the character B to the normal state 3.

  The AC machine construction unit 160 c selects the character “A” and sets the normal transition destination of the normal state 3 by the character A to the collation state 4. Also, the AC machine construction unit 160c sets Plist [A] = BA in the pattern structure corresponding to the pointer g [A] to the matching state 4 in the state structure in the normal state 3 (FIG. 12, step S12). .

  In FIG. 13, the AC machine construction unit 160 c returns to the initial state 0, extracts the pattern “BB” from the pattern set Π, and selects the character “B”. Here, the normal transition destination of the initial state 0 by the letter B is the normal state 3, and since it has already been created, the current state is transitioned to the normal state 3.

  The AC machine construction unit 160 c selects the character “B” and sets the normal transition destination of the normal state 3 by the character B to the collation state 5. Also, the AC machine construction unit 160c sets Plist [B] = BB in the state structure of state 3 corresponding to the pointer g [B] to the collation state 5 (FIG. 13, step S13).

  The AC machine construction unit 160c returns to the initial state 0, extracts the pattern “BAA” from the pattern set Π, and selects the character “B”. Here, the normal transition destination of the initial state 0 by the letter B is the normal state 3, and since it has already been created, the current state is transitioned to the normal state 3.

  The AC machine construction unit 160c selects the letter “A”. Here, since the normal transition destination of the normal state 3 by the character A is the collation state 4 and has already been created, the current state is transited to the collation state 4.

  The AC machine construction unit 160c selects the letter “A”. The AC machine construction unit 160 c sets the normal transition destination of the collation state 4 with the character “A” to the collation state 6. Further, the AC machine construction unit 160c sets Plist [A] = BAA in the state structure of state 4 corresponding to the pointer g [a] to the collation state 6 (FIG. 13, step S14).

  In FIG. 14, the AC machine construction unit 160 c returns to the initial state 0, extracts the pattern “BACD” from the pattern set Π, and selects the character “B”. Here, the normal transition destination of the initial state 0 by the letter B is the normal state 3, and since it has already been created, the current state is transitioned to the normal state 3.

  The AC machine construction unit 160c selects the letter “A”. Here, since the normal transition destination of the normal state 3 by the character A is the state 4 and has already been created, the current state is transitioned to the collation state 4.

  The AC machine construction unit 160c selects the character “C”. The AC machine construction unit 160c sets the normal transition destination of the collation state 4 with the character C to the normal state 7. The AC machine construction unit 160c selects the character “D”. The AC machine construction unit 160c sets the normal transition destination of the normal state 7 by the character D to the collation state 8 (FIG. 14, step S15). At the stage where step S15 is completed, registration of all patterns included in the pattern set 終了 ends, and the construction process of the trie T ends.

  Next, a process in which the AC machine construction unit 160c adds a failer transition to a trie will be described. 15 to 23 are diagrams for describing the failer transition addition process. The AC machine construction unit 160c determines a state that is a normal transition destination from the initial state 0, and registers the determined state in a queue. The AC machine construction unit 160c sets the failer transition destination registered in the queue to the initial state 0. Here, since the normal transition destination of the initial state 0 is the normal states 1 and 3, 1 and 3 are registered in the queue. In addition, the failer transition destination in the normal states 1 and 3 is set to the initial state 0. See FIG.

  The AC machine construction unit 160c extracts the first state 1 of the queue, and sets the extracted state 1 to the state s. The AC machine construction unit 160c extracts all the characters a that satisfy g [code (a)] ≠ Null in the state s, and stores them in the set X. In this case, the AC machine construction unit 160c extracts the character C and stores the character C in the set X.

  The AC machine construction unit 160c extracts the character C from the set X, and adds the state 2 that is the normal transition destination of the state s to the end of the queue. When the AC machine construction unit 160c transitions from the state 1 to the initial state 0 that has undergone a failer transition, and determines the normal transition destination for the character C, it determines the initial state 0 when determining the failer transition destination of the state next. The AC machine construction unit 160c determines the state next to be transitioned by the character C from the state s (normal state 1), and sets the failer transition destination of the determined state (collation state 2) to the initial state 0. See FIG.

  The AC machine construction unit 160c extracts the state 3 at the head of the queue, and sets the extracted state 3 to the state s. The AC machine construction unit 160c extracts all the characters a that satisfy g [code (a)] ≠ Null in the state s, and stores them in the set X. In this case, the AC machine construction unit 160c extracts the characters A and B and stores the characters A and B in the set X.

  The AC machine construction unit 160c extracts the character A from the set X, and adds the state 4 that is the normal transition destination of the character A in the state s to the tail of the queue. When the AC machine construction unit 160c transitions from the state s to the initial state 0 in which the failure transition is performed and determines the normal transition destination for the character A, the state transitions to the state 1 when the failure transition destination of the state next is determined. The AC machine construction unit 160c determines the state next to be transitioned by the letter A from the state s (normal state 3), and sets the failer transition destination of the determined state (collation state 4) to state 1. See state 4 in FIG.

  The AC machine construction unit 160c extracts the character B from the set X and adds the state 5 that is the normal transition destination of the character B in the state s (normal state 3) to the tail of the queue. When the AC machine construction unit 160c transitions from the state s to the initial state 0 in which the failure transition is performed and determines the normal transition destination for the character B, the state transitions to the state 3 when the failure transition destination of the state next is determined. The AC machine construction unit 160c determines the state next to be transitioned from the state s by the letter B, and sets the failer transition destination of the determined state (collation state 5) to the state 3. See state 5 in FIG.

  The AC machine construction unit 160c extracts the state 2 at the head of the queue, and sets the extracted state 2 to the state s. The AC machine construction unit 160c extracts all the characters a that satisfy g [code (a)] ≠ Null in the state s, and stores them in the set X. Since there is no normal transition destination in the state s, the process proceeds to the next step. See FIG.

  The AC machine construction unit 160c extracts the state 4 at the head of the queue, and sets the extracted state 4 to the state s. The AC machine construction unit 160c extracts all the characters a that satisfy g [code (a)] ≠ Null in the state s, and stores them in the set X. In this case, the AC machine construction unit 160c extracts the characters A and C and stores the characters A and C in the set X.

  The AC machine construction unit 160c extracts the character A from the set X, and adds the state 6 that is the normal transition destination of the character A in the state s (collation state 4) to the tail of the queue. The AC machine construction unit 160c transitions from the state s to the state 1 in which a failer transition is made. In state 1, the normal transition destination for character A is Null, so a failer transition is made again, and an initial state 0 is entered.

  Then, in the initial state 0, when the normal transition destination for the character A is determined, and the failer transition destination of the state next is determined, the state 1 is obtained. The AC machine construction unit 160c determines the state next to be transitioned by the letter A from the state s (normal state 4), and sets the failure transition destination of the determined state (collation state 6) to state 1. See state 6 in FIG.

  The AC machine construction unit 160c extracts the character C from the set X, and adds the state 7 that is the normal transition destination of the character C in the state s (collation state 4) to the tail of the queue. When the AC machine construction unit 160c transitions from the state s to the state 1 in which the failure transition is performed and determines the normal transition destination for the character C, the state transitions to the state 2 when the failure transition destination of the state next is determined. The AC machine construction unit 160c determines the state next to be transitioned by the character C from the state s, and sets the failer transition destination of the determined state (normal state 7) to the state 2.

  Further, the AC machine construction unit 160c sets the failure transition destination of the state 7 to the collation state 2. In this case, the AC machine construction unit 160c sets the pattern list corresponding to the pointer g [C] to the state 7 to Plist [C] = AC in the state structure of the state 4, and checks the state 7 from the normal state. Change to state. See states 4 and 7 in FIG.

  The AC machine construction unit 160c takes out the first state 5 of the queue and sets the taken-out state 5 to the state s. The AC machine construction unit 160c extracts all the characters a that satisfy g [code (a)] ≠ Null in the state s, and stores them in the set X. Since there is no normal transition destination in the state s, the process proceeds to the next step. See FIG.

  The AC machine construction unit 160c extracts the state 6 at the head of the queue, and sets the extracted state 6 to the state s. The AC machine construction unit 160c extracts all the characters a that satisfy g [code (a)] ≠ Null in the state s, and stores them in the set X. Since there is no normal transition destination in the state s, the process proceeds to the next step. See FIG.

  The AC machine construction unit 160c takes out the head state 7 of the queue and sets the taken state 7 to state s. The AC machine construction unit 160c extracts all the characters a that satisfy g [code (a)] ≠ Null in the state s, and stores them in the set X. In this case, the AC machine construction unit 160c extracts the character D and stores the character D in the set X.

  The AC machine construction unit 160c extracts the character D from the set X, and adds the state 8 that is the normal transition destination of the character D in the state s (collation state 7) to the tail of the queue. The AC machine construction unit 160c transitions from the state s to the state 2 in which a failer transition is made. In state 2, the normal transition destination for character D is Null, so a failer transition occurs again, and the state transitions to initial state 0.

  When the AC machine construction unit 160c determines the failure transition destination of the state next by determining the normal transition destination for the character D in the initial state 0, the state becomes the initial state 0. The AC machine construction unit 160c determines the state next to be transitioned by the character D from the state s (collation state 7), and sets the failer transition destination of the determined state (collation state 8) to the initial state 0. See state 8 in FIG.

  The AC machine construction unit 160c takes out the head state 8 of the queue and sets the taken state 8 to state s. The AC machine construction unit 160c extracts all the characters a that satisfy g [code (a)] ≠ Null in the state s, and stores them in the set X. Since there is no normal transition destination in the state s, the process proceeds to the next step. Then, when the state no longer exists in the queue, the AC machine of the pattern set IV is completed. See FIG.

  Next, a process in which the AC machine construction unit 160c generates a compressed data AC machine 150f by adding a skip transition to the AC machine 150d will be described. When the character pair of the compressed dictionary data 150e is a and b, the AC machine construction unit 160c scans the path of the AC machine 150d and determines a place where the transition of the character a and the transition of the character b are continuous. Then, the AC machine construction unit 160c adds a skip transition to the corresponding part of the AC machine 150d.

  Further, the AC machine construction unit 160c scans the path of the AC machine 150d, and determines a location where the transition of the character a and the transition of the character b sandwich the failer transition. Then, the AC machine construction unit 160c adds a skip transition to the corresponding part of the AC machine 150d.

  24 to 26 are diagrams for explaining the compressed data AC machine construction processing. The AC machine construction unit 160c adds a skip transition corresponding to the character pair registered in the compressed dictionary data 150e to the AC machine 150d.

  First, the AC machine construction unit 160c adds a skip transition corresponding to the character pair “B, A” registered in the first row of the compressed dictionary data 150e to the AC machine 150d. The AC machine construction unit 160c scans all paths included in the AC machine 150d, and determines a place where the transition of the letter B and the transition of the letter A are continuous.

  The AC machine construction unit 160c sets the normal transition destination by the letter B of the state s to the state t (sg [code (B)] = t) for the state s and t on the path of the AC machine 150d, and When the normal transition destination of the character A in the state t is the state u (sg [code (A)] = u), a skip transition X is created in the state s (sg [code (X)] = u). At this time, the pattern list (s.Plist [code (X)]) of the skip transition X of the state s is s.Plist [code (X)] = s.Plist [code (B)] ∪t.Plist [code (A)].

  Among all paths of the AC machine 150d, the set of the state s, the state t, and the state u that satisfy the conditions (sg [code (B)] = t) and (tg [code (A)] = u) State 0, normal state 3, and verification state 4 are entered. Accordingly, the AC machine construction unit 160c sets the matching state 4 to the normal transition destination pointer (g [code (X)]) with the character X in the initial state 0, and sets the matching pattern list (Plist [X]). Register “BA”. See initial state 0 in FIG.

  Subsequently, the AC machine construction unit 160c scans the path of the AC machine 150d and determines a location where the transition of the character B and the transition of the character A sandwich the failer transition. The AC machine construction unit 160c sets the normal transition destination by the letter B of the state s to the state t (sg [code (B)] = t) for the state s and t on the path of the AC machine 150d, and All the states t in which the normal transition destination by the character A in the state t is Null (tg [code (A)] = Null) are stored in the set F.

  Since the state t satisfying the condition (sg [code (B)] = t) and (tg [code (A)] = Null) in all paths of the AC machine 150d is the collation state 5, the AC machine The construction unit 160c stores the collation state 5 in the set F. In the set F, the AC machine construction unit 160c performs a failer transition from the state t, and the normal transition destination by the character A in the state of the failer transition destination is Null (t.fail.g [code (A)] = Null). Delete state t. Since the collation state 5 does not satisfy the condition (t.fail.g [code (A)] = Null), it remains in the set F.

  The AC machine construction unit 160c makes a failer transition from the state t to an arbitrary state t of the set F, and sets the normal transition destination by the character A of the state of the failer transition destination to the state u (t.fail.g [code (A )] = u). Further, the AC machine construction unit 160c sets the state s as the state satisfying the condition (s.g [code (B)] = t). At this time, the pattern list (s.Plis [code (X)]) of the skip transition X of the state s is expressed as s.Plist [code (X)] = s.Plist [code (B)] ∪t.fail.Plist [code (A)]

  The state s that satisfies (t.fail.g [code (A)] = u) and (s.g [code (B)] = t) is the normal state 3. The AC machine construction unit 160c sets the matching state 4 to the normal transition destination pointer (g [code (X)]) with the character X in the normal state 3 and sets “BB” in the corresponding pattern list (Plist [X]). , BA ". See normal state 3 in FIG.

  Subsequently, the AC machine construction unit 160c adds a skip transition corresponding to the character pair “X, C” registered in the second stage of the compressed dictionary data 150e to the AC machine 150d. The AC machine construction unit 160c adds a skip transition to the corresponding part of the AC machine 150d.

  The AC machine construction unit 160c sets the normal transition destination by the character X of the state s to the state t (sg [code (X)] = t) with respect to the state s and the state t on the path of the AC machine 150d, and When the normal transition destination of the character C in the state t is the state u (sg [code (C)] = u), a skip transition Y is created in the state s (sg [code (Y)] = u). At this time, the pattern list (s.Plist [code (Y)]) of the skip transition Y of the state s is s.Plist [code (Y)] = s.Plist [code (X)] ∪t.Plist [code (C)].

  Among all paths of the AC machine 150d, the set of the state s, the state t, and the state u that satisfy the condition (sg [code (X)] = t) and (sg [code (C)] = u) A set of state 0, collation state 4 and collation state 7 and a pair of normal state 3, collation state 4 and collation state 7 are set.

  The AC machine construction unit 160c sets the matching state 7 to the normal transition destination pointer (g [code (Y)]) with the letter Y in the initial state 0, and sets “BA” to the corresponding pattern list (Plist [Y]). , AC ". See initial state 0 in FIG.

  Also, the AC machine construction unit 160c sets the matching state 7 to the normal transition destination pointer (g [code (Y)]) with the letter Y in the normal state 3, and sets the matching pattern list (Plist [Y]). Register “BB, BA, AC”. See normal state 3 in FIG.

  Subsequently, the AC machine construction unit 160c scans the path of the AC machine 150d and determines a location where the transition of the character X and the transition of the character C sandwich the failer transition. In the AC machine 150d shown in FIG. 26, there is no place where the transition of the character X and the transition of the character C sandwich the failer transition, so the AC machine shown in FIG. 26 is the compressed data AC machine 150f.

  Next, the verification processing unit 160d shown in FIG. 2 will be described. When the text to be collated is given, the matching processing unit 160d compares the given text with the compressed data AC machine 150f to determine whether or not the text contains a specific keyword. It is a processing unit.

  27 to 32 are diagrams for explaining the collation processing. Here, as an example, the text to be collated is “AXYYD”. The verification processing unit 160d sets the current state s to the initial state 0 of the compressed data AC machine 150f. See FIG.

  The matching processing unit 160d reads the first character “A” of the text and determines the normal transition destination of the initial state 0 by the character A. Since the transition destination of the initial state 0 by the letter A is the normal state 1, the current state s is set to the normal state 1. See FIG.

  The matching processing unit 160d reads the second character “X” of the text and determines the transition destination of the normal state 3 by the character X. Since there is no normal transition destination of the normal state 3 by the letter X, a failer transition is made from the normal state 3 and the state s is set to the initial state 0.

  The verification processing unit 160d determines the normal transition destination of the initial state 0 by the character X. Since the normal transition destination of the initial state 0 by the character X is the collation state 4, the current state s is set to the collation state 4. Since the matching state 4 is hit, the matching processing unit 160d registers the pattern list Plist [X] = {BA} associated with g [code (X)] in the initial state 0 in the set R. See FIG.

  The matching processing unit 160d reads the third character “Y” of the text and determines the normal transition destination of the matching state 4 by the letter Y. Since there is no normal transition destination of the collation state 4 by the letter Y, a failer transition is made from the collation state 4 and the state s is set to the normal state 1.

  The verification processing unit 160d determines the normal transition destination of the normal state 1 by the character Y. Since there is no normal transition destination of the normal state 1 due to the letter Y, a failure transition is made from the normal state 1 and the state s is set to the initial state 0.

  The verification processing unit 160d determines the normal transition destination of the initial state 0 by the letter Y. Since the normal transition destination of the initial state 0 by the letter Y is the collation state 7, the current state s is set to the collation state 7. Since the matching state 7 is hit, the matching processing unit 160d registers the pattern list Plist [X] = {BA, AC} associated with g [code (Y)] in the initial state 0 in the set R. . Since the BA is already registered in the set R, the matching processing unit 160d registers AC in the set R. See FIG.

  The matching processing unit 160d reads the fourth character “Y” of the text and determines the normal transition destination of the matching state 7 by the character Y. Since there is no normal transition destination of the collation state 7 by the letter Y, the failer transitions from the collation state 7 and the state s is set to the collation state 2.

  The matching processing unit 160d determines the normal transition destination of the matching state 2 by the letter Y. Since there is no normal transition destination of the collation state 2 by the letter Y, a failer transition is made from the collation state 2 and the state s is set to the initial state 0.

  The verification processing unit 160d determines the normal transition destination of the initial state 0 by the letter Y. Since the normal transition destination of the initial state 0 by the letter Y is the collation state 7, the current state s is set to the collation state 7. Since the matching state 7 is hit, the matching processing unit 160d registers the pattern list Plist [X] = {BA, AC} associated with g [code (Y)] in the initial state 0 in the set R. . Note that the collation processing unit 160d does not register anything in the set R because BA and AC are already registered in the set R. See FIG.

  The matching processing unit 160d reads the fifth character “D” of the text and determines the normal transition destination of the matching state 7 by the character D. Since the normal transition destination of the collation state 7 by the letter D is the collation state 8, the current state s is set to the collation state 8. Since the matching state 8 is hit, the matching reasoning unit 160d registers the pattern list Plist [D] = {BACD} associated with g [code (D)] in the matching state 7 in the set R. See FIG.

  As illustrated in FIGS. 27 to 32, the collation processing unit 160 d reads out the text character by character, transitions between the states of the compressed data AC machine 150 f, and executes the collation processing. It is determined that “AXYYD” includes a specific keyword “BA, AC, BADC”. It should be noted that the collation processing unit 160d executes the collation process in the same manner as in FIGS. 27 to 32 even when the text “AXYYD” is acquired collectively instead of acquiring the text one character at a time.

  Next, the cutting processing unit 160e shown in FIG. 2 will be described. The cutout processing unit 160e is a processing unit that cooperates with the matching processing unit 160d to calculate the position of the encoded data 150b corresponding to the post-conversion extraction condition 150h. In the following description, the position of the encoded data 150b corresponding to the post-conversion extraction condition 150h is referred to as a cutout position list.

  The cut-out processing unit 160e cuts out the position information corresponding to the cut-out position list from the encoded data 150b, and outputs the cut-out information to the output unit 120 or the communication control IF unit 130 as a search condition answer.

  Hereinafter, the processing of the cutout processing unit 160e will be specifically described. The cutout processing unit 160e scans the encoded data 150b character by character and executes various processes according to the character type. Here, a case where the scanned character is a character string portion, a start tag “<>”, and an end tag “</>” will be described separately.

  A case where the scanned character is a character string portion will be described. When the scanned character is a character string portion, the cutout processing unit 160e outputs the scanned character to the collation processing unit 160d, and performs collation processing on information indicating whether or not the compressed data AC machine 150f has transitioned to the collation state. From the unit 160d.

  In addition to the above-described processing, the collation processing unit 160d described above transitions the current state s when a character is acquired from the cut-out processing unit 160e, and extracts information indicating whether or not the state has transitioned to the collation state. It is assumed to notify the unit 160e. The cut-out processing unit 160e sets the value of the keyword flag k to 1 when acquiring information indicating the transition to the collation state.

  Next, a case where the scanned character is a start tag will be described. When the scanned character is a start tag, the extraction processing unit 160e stacks the tag code and the current position (the number up to “<” of the corresponding tag code counted from the first character of the encoded data 150b; start). Register with S.

  Further, the cutout processing unit 160e determines whether or not the tag code registered in the stack S matches the item part of the post-conversion extraction condition 150h. If they match, the item flag f is set to 1. Set.

  Next, a case where the scanned character is an end tag will be described. The cutout processing unit 160e determines whether the keyword flag k and the item flag f are 1 when the scanned character is an end tag. When the keyword flag k and the item flag f are 1, the cutout processing unit 160e registers the current position (the number up to “>” of the corresponding tag code counted from the first character of the encoded data 150b) in end. Then, the start of the last element of the stack S is acquired. The cutout processing unit 160e registers the acquired combination of start and end (start, end) in the cutout position list. The cut-out processing unit 160e extracts the tag code from the stack S and sets the keyword flag k to 0.

  Here, assuming that the encoded data 150b is the encoded data shown in FIG. 4 and the post-conversion extraction condition “/ 1/2 =“ BACD ””, the process of the cutout processing unit 160e will be described. As an initial setting of the extraction processing unit 160e, the keyword flag k and the item flag f are set to 0, the stack S is an empty stack, and the extraction position list R is an empty list.

  The cut-out processing unit 160e scans the character “<1>” of the encoded data 150b. The character “<1>” is a start tag. The cutout processing unit 160 e registers the tag code “<1>” and the current position “1” in the stack S in the stack S. Stack S = {(<1>, 1)}. Since the tag code “<1>” of the stack S does not match the item part “/ 1/2” of the post-conversion extraction condition, the cutout processing unit 160e keeps the item flag f at 0.

  The cut-out processing unit 160e scans the character “<2>” of the encoded data 150b. The character “<2>” is a start tag. The cutout processing unit 160 e registers the tag code “<2>” and the current position “4” in the stack S in the stack S. Stack S = {(<1>, 1), (<2>), 4}. The cutout processing unit 160e sets the item flag f to 1 because the tag codes “<1>” and “<2>” of the stack S match the item part “/ 1/2” of the post-conversion extraction condition. To do.

  The cut-out processing unit 160e scans the character string “AXYYD” of the encoded data 150b. The cutout processing unit 160e outputs the character string “AXYYD” to the collation processing unit 160d, and acquires the collation result. When the verification result includes the fact that the verification state has been changed, the extraction processing unit 160e sets the keyword flag k to 1.

  The cut-out processing unit 160e scans the character “<2 />” of the encoded data 150b. The character “<2 />” is an end tag, and the item flag f and the keyword flag k are 1. In this case, the extraction processing unit 160e registers a set of end = 15 and the start value (= 4) of the last element of the stack S in the extraction position list R. R = {(4,15)}. The extraction processing unit 160e extracts (hops) (<2>, 4) from the stack S. Stack S = {(<1>, 1)}. The cutout processing unit 160e sets the keyword flag k to 0.

  The cut-out processing unit 160e scans the character “<3>” of the encoded data 150b. The character “<3>” is a start tag. The cut-out processing unit 160e registers the tag code “<3>” and the current position “16” in the stack S in the stack S. Stack S = {(<1>, 1), (<3>), 16}. The cutout processing unit 160e sets the item flag f to 0 because the tag codes “<1>” and “<3>” of the stack S do not match the item part “/ 1/2” of the post-conversion extraction condition. To do.

  The cut-out processing unit 160e scans the character string “AXYYD” of the encoded data 150b. The cutout processing unit 160e outputs the character string “AXYYD” to the collation processing unit 160d, and acquires the collation result. When the verification result includes the fact that the verification state has been changed, the extraction processing unit 160e sets the keyword flag k to 1.

  The cut-out processing unit 160e scans the character “</ 3>” of the encoded data 150b. The character “</ 3>” is an end tag. Since the keyword flag k is 1 but the item flag f is 0, the cutout processing unit 160e does not register the cutout position list R. The cutout processing unit 160e takes out (pops) (<3>, 16) from the stack S. Stack S = {(<1>, 1)}. The cutout processing unit 160e sets the keyword flag k to 0.

  The cut-out processing unit 160e scans the characters “</ 1>” of the encoded data 150b. The character “</ 1>” is an end tag. Since the keyword flag k and the item flag f are 0, the cutout processing unit 160e does not register the cutout position list R. The cutout processing unit 160e extracts (<1>, 1) from the stack S. Stack S = empty stack. The cutout processing unit 160e sets the keyword flag k to 0.

  Since the extraction processing unit 160e has scanned the encoded data 150b to the end, the extraction processing unit 160e extracts information from the encoded data 150b based on the extraction position list R = {(4,15)}. In this case, the cutout processing unit 160e cuts out the character “<2> AXYYD </ 2>” positioned between the fourth and sixteenth characters from the first character of the encoded data 150b.

  Next, a procedure of the search apparatus 100 according to the second embodiment will be described. First, compression processing executed by the data compression processing unit 160b will be described. FIG. 33 is a flowchart illustrating a processing procedure of compression processing.

  As shown in FIG. 33, the data compression processing unit 160b executes a character string counting process (step S101), and executes a dictionary construction / character string replacement process (step S102). Then, the data compression processing unit 160b executes tag name replacement processing (step S103).

  Here, the procedure of the character string counting process shown in step S101 of FIG. 33 will be described. FIG. 34 is a flowchart showing the procedure of the character string counting process. As shown in FIG. 34, the data compression processing unit 160b sets the previous character c0 = ε, and substitutes the first character of the XML data D (XML data 150a) for the current character c1 (step S111).

  The data compression processing unit 160b initializes the counting table T and sets the in-tag flag f = 0 (step S112). The data compression processing unit 160b determines whether or not the next character exists in the XML data D (step S113). When the next character does not exist in the XML data D (step S114, No), the data compression processing unit 160b outputs the counting table T (step S115).

  On the other hand, when the next character exists in the XML data D (Yes at Step S114), the data compression processing unit 160b sets c0 = c1, and substitutes the next character for c1 (Step S116). The data compression processing unit 160b determines whether c1 is a tag start symbol (<) or a tag end symbol (>) (step S117).

  If c1 is not a tag start symbol and not a tag end symbol (step S118, No), the data compression processing unit 160b proceeds to step S121. On the other hand, when c1 is a tag start symbol or a tag end symbol (Yes in step S118), the data compression processing unit 160b sets f = 1 if c1 is a tag start symbol, and sets f = 0 if c1 is a tag end symbol. (Step S119).

  If f = 1 (No at Step S120), the process proceeds to Step S113. When f = 0 (step S120, Yes), the data compression processing unit 160b registers a character string of length 2 connecting c0 and c1 in the counting table T. If the character string is not registered, the data compression processing unit 160b increases the number of appearances of the character string by 1 (step S121), and proceeds to step S113.

  Next, the processing procedure of the dictionary construction / character string replacement processing shown in step S102 of FIG. 33 will be described. FIG. 35 is a flowchart showing a processing procedure of dictionary construction / character string replacement processing. As shown in FIG. 35, the data compression processing unit 160b determines whether or not there is a character in the unused character set U (step S131).

  If there is no character in the unused character set U (No in step S132), the data compression processing unit 160b outputs the XML data D as the character string compressed data C, and the compression dictionary data Dic (compression dictionary data) 150e) is output (step S133).

  On the other hand, when there is a character in the unused character set U (Yes in step S132), the data compression processing unit 160b has the highest frequency (number of appearances) among character strings not registered in the compression dictionary data 150e. The largest character string s is searched from the counting table T (step S134).

  The data compression processing unit 160b determines whether or not the number of appearances of the character string s is equal to or greater than the count threshold α (step S135). When the number of appearances of the character string s is less than the count threshold α (No at Step S136), the data compression processing unit 160b proceeds to Step S133.

  When the number of occurrences of the character string s is equal to or greater than the count threshold α (step S136, Yes), the data compression processing unit 160b compresses the set (s, a) for the character a in the unused character set U. In addition to the data Dic, the character a is deleted from U (step S137).

  The data compression processing unit 160b replaces all the character strings s in the XML data D with the characters a (step S138), executes character string counting processing (step S139), and proceeds to step S131. The character string counting process shown in step S139 of FIG. 35 is the same as the character string counting process shown in FIG.

  Next, the processing procedure of the tag name replacement process shown in step S103 of FIG. 33 will be described. FIG. 36 is a flowchart illustrating a processing procedure of tag name replacement processing. As shown in FIG. 36, the data compression processing unit 160b scans the XML data D and registers all tag names appearing in D in the correspondence table T (correspondence table 150c) (step S141).

  The data compression processing unit 160b assigns codes to all tag names registered in the correspondence table T (Step S142). The data compression processing unit 160b converts all tag names of the XML data D into codes assigned to the correspondence table T (step S143), and outputs encoded data B (encoded data 150b) (step S144). .

  Next, the construction process executed by the AC machine construction unit 160c, the collation processing unit 160d, and the cutout processing unit 160e will be described. FIG. 37 is a flowchart illustrating the processing procedure of the construction processing. As shown in FIG. 37, the AC machine construction unit 160c executes an extraction condition reception process (step S201).

  Then, the AC machine construction unit 160c executes an AC machine construction process (Step S202), and the collation processing unit 160d and the cut-out processing unit 160e execute a cut-out position calculation process (Step S203).

  Here, the processing procedure of the extraction condition reception processing shown in step S201 of FIG. 37 will be described. FIG. 38 is a flowchart illustrating the processing procedure of the extraction condition reception processing. As shown in FIG. 38, the AC machine construction unit 160c receives the extraction condition Q (extraction condition 150g) (step S211).

  The AC machine construction unit 160c rewrites the item name in the extraction condition Q using the correspondence table 150c (step S212), and sets the rewritten extraction condition Q as the extraction condition Q '(step S213).

  Subsequently, the processing procedure of the AC machine construction processing shown in step S202 of FIG. 37 will be described. FIG. 39 is a flowchart showing the processing procedure of the AC machine construction processing. As shown in FIG. 39, the AC machine construction unit 160c constructs a trie T (step S221), and adds a failer transition to the trie (step S222).

  The AC machine construction unit 160c adds a skip transition to the AC machine AC (Π) (step S223) and outputs the compressed data AC machine AC_b (Π, dic) (step S224).

  Here, the process for constructing the cocoon trie T shown in step S221 of FIG. 39 will be described. FIG. 40 is a flowchart illustrating a processing procedure of processing for constructing a cocoon tri-T. As illustrated in FIG. 40, the AC machine construction unit 160c creates an initial state (id = 0) and sets a trie (Π) to a trie that includes only the initial state (step S231).

  The AC machine construction unit 160c sets all normal transition destinations in the initial state to the initial state (step S232), and determines whether there is a pattern in the bag (step S233). When the pattern does not exist in the bag (step S234, No), the AC machine construction unit 160c outputs the bag trie T (Π) (step S235).

  When the pattern does not exist in the bag (step S234, Yes), the AC machine construction unit 160c extracts one pattern from the bag and sets the extracted pattern to p (step S236). The AC machine construction unit 160c executes pattern registration processing (step S237), and proceeds to step S233.

  Next, the process procedure of the pattern registration process shown in step S237 of FIG. 40 will be described. FIG. 41 is a flowchart showing a processing procedure of pattern registration processing. As shown in FIG. 41, the AC machine construction unit 160c sets the current state s to the initial state of the trie T, and sets the state r before s to an empty state (step S241).

  The AC machine construction unit 160c determines whether or not the next character exists in the pattern p (step S242). When the next character does not exist in the pattern p (No in step S243), the AC machine construction unit 160c performs the pattern list Plist [code (a of r for the last character (a) and the state r of the pattern p. ]] Is substituted for p (step S244), and a trie T is output (step S245).

  On the other hand, when the next character exists in the pattern p (Yes in step S243), the AC machine construction unit 160c sets the next character as a and sets the ascii code of a to code (a) (step S246). . The AC machine construction unit 160c determines whether or not the normal transition g [code (a)] = Null of s (step S247).

  If the normal transition g [code (a)] = Null (step S248, Yes), the AC machine construction unit 160c proceeds to step S250. On the other hand, if g [code (a)] = Null is not satisfied (No at Step S248), the AC machine construction unit 160c creates a new state n and sets g [code (a)] = n (Step S248). S249). The AC machine construction unit 160c substitutes the state s for the state r, substitutes g [code (a)] for the state s (step S250), and proceeds to step S242.

  Next, processing for adding a failer transition to the trie (Π) shown in step S222 of FIG. 39 will be described. FIG. 42 and FIG. 43 are flowcharts showing a processing procedure of processing for adding a failer transition to a trie.

  As shown in FIG. 42, the AC machine construction unit 160c substitutes the initial state for all of the states s that can be normally shifted from the initial state, and registers the current state s in the queue ( Step S251).

  The AC machine construction unit 160c determines whether or not a state exists in the state list queue (step S252). When no state exists in the state list queue (step S253, No), the AC machine construction unit 160c outputs the current trie T as the AC machine α (step S254).

  When there is a state in the state list queue (Yes in step S253), the AC machine construction unit 160c sets s as the head state of the state list queue and removes s from the state list queue (step S255).

  The AC machine construction unit 160c determines whether all the normal transitions of the state s are Null (step S256). When all the normal transitions of the state s are Null (step S257, Yes), the AC machine construction unit 160c proceeds to step S252.

  On the other hand, when all the normal transitions of the state s are not Null (step S257, No), the AC machine construction unit 160c determines a set of all the characters a that satisfy g [code (a)] ≠ Null in the state s. A is set to A (step S258).

  The AC machine construction unit 160c determines whether or not a character exists in the set A (step S259). The AC machine construction unit 160c proceeds to step S252 when no character exists in the set A (step S260, No).

  On the other hand, if there is a character in set A (Yes in step S260), AC machine construction unit 160c moves to FIG. 43, extracts one character from set A, and sets the extracted character to a (step). S261).

  The AC machine construction unit 160c adds the common transition destination next = g [code (a)] of the state s to the tail of the state list queue (step S262), repeats the failer transition from the state s, and normally The initial state in which the transition destination is not Null is set to f (step S263).

  The AC machine construction unit 160c determines the pointer fnext = g [code (a)] to the normal transition destination of the character a for the state f (step S264), and sets the next transition destination of the state next to fnext = g [code (a )] (Step S265). The AC machine construction unit 160c adds the transition pattern list that gives the state fnext to the transition pattern list that gives the state next (step S266), and proceeds to step S259 in FIG.

  Next, a processing procedure for adding a skip transition to the AC machine AC (ＡＣ) shown in step S223 of FIG. 39 will be described. FIG. 44 is a flowchart of a process procedure for adding a skip transition to the AC machine AC (Π). As shown in FIG. 44, the AC machine construction unit 160c sets n = 1 (step S271), and determines whether or not the nth row exists in the compressed dictionary data dic (step S272).

  The AC machine construction unit 160c outputs the current AC machine AC (Π) as the compressed data AC machine AC_b (Π, dic) when the nth row does not exist in the compressed dictionary data dic (No in step S273). (Step S274).

  On the other hand, when the nth line exists in the compression dictionary data dic (Yes in step S273), the AC machine construction unit 160c sets the pair character string as (a, b) in the nth line of the compression dictionary data dic. The character string to be replaced is set to X (step S275).

  The AC machine construction unit 160c executes a first skip transition creation process (step S276), and executes a second skip transition creation process (step S277). The AC machine construction unit 160c sets n = n + 1 (step S278), and proceeds to step S272.

  Subsequently, the processing procedure of the first skip transition creation processing shown in step S276 of FIG. 44 will be described. FIG. 45 is a flowchart of a process procedure of the first skip transition creation process. As shown in FIG. 45, the AC machine construction unit 160c sets a set of all paths of the AC machine AC (Π) as P (step S281), and determines whether there is a path in P (step S282). .

  The AC machine construction unit 160c outputs the current AC machine AC (Π) when there is no path in P (step S283, No) (step S284). On the other hand, when a path exists in P (step S283, Yes), AC machine construction unit 160c extracts an arbitrary path from P (step S285).

  The AC machine construction unit 160c scans the path from the initial state toward the leaf. And if sg [code (a)] = t and tg [code (a)] = u hold for states s and t on the path, create a skip transition sg [code (X)] = u (Step S286). The AC machine construction unit 160c deletes the path from P (step S287), and proceeds to step S282.

  Subsequently, the processing procedure of the second skip transition creation processing shown in step S277 of FIG. 44 will be described. FIG. 46 is a flowchart showing the processing procedure of the second skip transition creation processing.

  The AC machine construction unit 160c searches all states of the AC machine AC (Π) (step S291), and for a certain state s, sg [code (a)] = t and tg [code (b)] = NULL Are removed from the set F (step S292).

  The AC machine construction unit 160c determines whether or not the state t exists in the set F (step S294). When the state t does not exist in the set F (step S293, No), the AC machine construction unit 160c ends the second skip transition creation process.

  On the other hand, when the state t exists in the set F (step S294, Yes), the AC machine construction unit 160c selects an arbitrary state t from the set F (step S295). If the AC machine construction unit 160c sets t.fail.g [code (X)] = u and sg [code (a)] = t to be s, the skip transition sg [code (X)] = u is created (step S296). The AC machine construction unit 160c deletes the selected state t (step S297), and proceeds to step S293.

  Next, the processing procedure of the cutout position calculation process shown in step S203 of FIG. 37 will be described. By executing the cut-out position calculation process, the cut-out processing unit 160e calculates the position of the encoded data 150b corresponding to the extraction condition 150g, and extracts the calculated position information as compressed data. FIG. 47 is a flowchart showing the processing procedure of the extraction position calculation process. As shown in FIG. 47, the cutout processing unit 160e scans the first character of the encoded data B (step S301), and determines whether the scanned character is a character string portion (step S302).

  When the scanned character is a character string portion (Yes in step S303), the cutout processing unit 160e causes the compressed data AC machine 150f to transition by one character in accordance with the character, and k = 1 is set (step S304), and the process proceeds to step S311.

  On the other hand, when the scanned character is not a character string portion (No at Step S303), the cutout processing unit 160e determines whether the scanned character is a start tag or an end tag (Step S305).

  When the scanned character is an end tag (No in step S306), the cutout processing unit 160e registers the current position in end and sets the start of the last element of the stack S when f = 1 and k = 1. And (start, end) is registered in R (step S307). Here, the current position to be registered in end corresponds to the number up to “>” of the end tag code counted from the first character of the encoded data 150b. The clipping processing unit 160e pops (takes out) the tag code from the stack S, sets k = 0 (step S308), and proceeds to step S310.

  On the other hand, when the scanned character is a start tag (step S306, Yes), the cutout processing unit 160e pushes (stores) the tag code and the current position start on the stack S (step S309). Here, the current position start corresponds to the number up to “<” of the start tag code counted from the first character of the encoded data 150b.

  The cutout processing unit 160e sets f = 1 when the value of the stack S matches the item part of the post-conversion extraction condition Q ′, sets f = 0 when the value does not match (step S310), and scans all characters. It is determined whether or not (step S311).

  If all the characters have not been scanned (No at Step S312), the cutout processing unit 160e scans the next character (Step S313), and proceeds to Step S302. On the other hand, when all the characters have been scanned (step S312, Yes), the cutout processing unit 160e outputs R (step S314).

  As described above, in the search device 100 according to the first embodiment, when the data compression processing unit 160b compresses the XML data 150a, the character string included in the XML data 150a and the delimiter included in the XML data 150a. Encoded data 150b encoded by distinguishing characters is generated. Since the encoded data 150b is encoded by distinguishing the character string and the delimiter, even if the extraction processing unit 160e extracts the compressed data from the encoded data 150b as it is based on the extraction condition, an accurate search result is obtained. Can be obtained.

  By the way, among the processes described in the present embodiment, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or a part can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-described document and drawings can be arbitrarily changed unless otherwise specified.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. Further, all or any part of each processing function performed in each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

  FIG. 48 is a diagram illustrating a hardware configuration of a computer configuring the search device 100 according to the embodiment. As shown in FIG. 48, the computer (search device) 200 includes an input device 201, a monitor 202, a RAM (Random Access Memory) 203, a ROM (Read Only Memory) 204, and other devices (for example, a terminal device). A communication device 205 that transmits and receives data between them, a medium reading device 206 that reads data from a storage medium, a CPU (Central Processing Unit) 207, and an HDD (Hard Disk Drive) 208 are connected by a bus 209.

  The HDD 208 stores a search program 208b that exhibits the same function as that of the search device 100 described above. The search process 207a is activated when the CPU 207 reads and executes the search program 208b. Here, the search process 207a corresponds to the data management unit 160a, the data compression processing unit 160b, the AC machine construction unit 160c, the collation processing unit 160d, and the extraction processing unit 160e shown in FIG.

  In addition, the HDD 208 stores various data 208 a corresponding to the data stored in the storage unit 150. The CPU 207 reads various data 208a stored in the HDD 208, stores it in the RAM 203, compresses the XML data using the various data 203a stored in the RAM 203, and maintains the data corresponding to the extraction condition as compressed data. Extract.

  Incidentally, the search program 208b shown in FIG. 48 is not necessarily stored in the HDD 208 from the beginning. For example, a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card inserted into a computer, or a hard disk drive (HDD) provided inside or outside the computer. The search program 208b is stored in the “fixed physical medium”, and “another computer (or server)” connected to the computer via a public line, the Internet, a LAN, a WAN, or the like. However, the search program 208b may be read from these and executed.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Supplementary Note 1) A search system that executes a search process on a set of compressed data,
A data set in which each data is divided by the delimiter information indicating the attribute of the data is acquired, replaced with encoded data based on the data included in the data set, and further encoded based only on the delimiter information indicating the attribute of the data A search system comprising: a compression processing unit that compresses the data set by substituting with data.

(Supplementary note 2) The search system according to supplementary note 1,
Based on the relationship between the compressed data set delimiter information and the delimiter information before replacement when the search condition having the attribute and data to be searched is acquired for the data set compressed by the compression processing unit A replacement processing unit for replacing the attribute of the search condition with encoded data;
Based on the search condition in which the attribute of the search condition is replaced with encoded data and the compressed data set, the area on the compressed data set specified in the search condition is specified, and the specified area A search processing unit for outputting data included in the search system.

(Supplementary note 3) The search system according to supplementary note 2,
The search processing unit reads the compressed data set from the beginning, extracts encoded data corresponding to data surrounded by position delimiter information corresponding to the attribute of the search condition, and extracts the extracted encoded data And determining whether or not the data corresponding to the data of the search condition has the data of the search condition, and specifying a region on the compressed data set specified by the search condition based on the determination result system.

(Appendix 4)
A data set having a hierarchical structure in which each data is delimited by delimiter information indicating data attributes is obtained, replaced with encoded data based on the data included in the data set, and delimiter information indicating the data attributes A storage medium storing a program for realizing a compression processing function of compressing the data set by replacing with encoded data based on the above.

(Supplementary note 5) The storage medium according to supplementary note 4,
Based on the relationship between the delimiter information of the compressed data set and the delimiter information before replacement when the search condition having the attribute and data to be searched is acquired for the data set compressed by the compression processing function A replacement processing function for replacing the attribute of the search condition with encoded data;
Based on the search condition in which the attribute of the search condition is replaced with encoded data and the compressed data set, the area on the compressed data set specified in the search condition is specified, and the specified area A storage medium storing a program for realizing a search processing function for outputting data included in the.

(Supplementary note 6) The storage medium according to supplementary note 5,
The search processing function reads the compressed data set from the beginning, extracts encoded data corresponding to data surrounded by position delimiter information corresponding to the attribute of the search condition, and extracts the extracted encoded data And determining whether or not the data corresponding to the data of the search condition has data of the search condition, and specifying an area on the compressed data set specified by the search condition based on the determination result Medium.

(Appendix 7) The compression device is
Obtaining a data set having a hierarchical structure in which each piece of data is divided by division information indicating data attributes;
Replacing the encoded data based on the data included in the data set, and further compressing the data set by replacing the encoded data based only on the delimiter information indicating the attribute of the data. Characteristic compression method.

(Appendix 8) The search device is
The relationship between the compressed data set delimiter information and the delimiter information before replacement when the search condition having the attribute and data to be searched is acquired for the data set compressed by the compression device according to appendix 7 Replacing the attribute of the search condition with encoded data based on:
Based on the search condition in which the attribute of the search condition is replaced with encoded data and the compressed data set, the area on the compressed data set specified in the search condition is specified, and the specified area And a step of outputting data included in the search method.

(Supplementary Note 9) In the step of outputting the data, the compressed data set is read from the head, and encoded data corresponding to the data surrounded by the delimiter information of the position corresponding to the attribute of the search condition is extracted. Determining whether or not the data corresponding to the extracted encoded data has the data of the search condition, and specifying an area on the compressed data set specified by the search condition based on the determination result The search method according to appendix 8, which is characterized by that.

110 Input unit 120 Output unit 130 Communication control IF unit 140 Input / output control IF unit 150 Storage unit 150a XML data 150b Encoded data 150c Correspondence table 150d AC machine 150e Compression dictionary data 150f Compression data AC machine 150g Extraction condition 150h Extraction after conversion Condition 160 Control unit 160a Data management unit 160b Data compression processing unit 160c AC machine construction unit 160d Collation processing unit 160e Extraction processing unit 200 Computer (search device)
201 Input device 202 Monitor 203 RAM
203a, 208a Various data 204 ROM
205 Communication Control Device 206 Medium Reading Device 207 CPU
207a Search process 208 HDD
208b Search program 209 Bus

Claims (3)

Of the data including the first data and the second data indicating the attribute of the first data, at least the second data is replaced with encoded data having a size smaller than the second data. A search device for searching specified data from the replacement data,
When a search condition including data indicating the attribute of the designated data is received, the storage unit storing a relationship before and after the replacement of the second data is referred to, and the data indicating the attribute of the designated data is the code A replacement processing unit for replacing the data with
Using the search condition in which the data indicating the attribute of the designated data is replaced with the encoded data, data included in an area on the replacement data specified by the search condition is acquired from the replacement data And a search processing unit. Of the data including the first data and the second data indicating the attribute of the first data, at least the second data is replaced with encoded data having a size smaller than the second data. A search method of a search device for searching for specified data from the replacement data,
When a search condition including data indicating the attribute of the designated data is received, the storage unit storing a relationship before and after the replacement of the second data is referred to, and the data indicating the attribute of the designated data is the code Replaced with data
Using the search condition in which the data indicating the attribute of the designated data is replaced with the encoded data, data included in an area on the replacement data specified by the search condition is acquired from the replacement data A search method characterized by executing processing. Of the data including the first data and the second data indicating the attribute of the first data, at least the second data is replaced with encoded data having a size smaller than the second data. To the search device that searches the specified data from the replacement data
When a search condition including data indicating the attribute of the designated data is received, the storage unit storing a relationship before and after the replacement of the second data is referred to, and the data indicating the attribute of the designated data is encoded Replaced with data
Using the search condition in which the data indicating the attribute of the designated data is replaced with the encoded data, data included in an area on the replacement data specified by the search condition is acquired from the replacement data A search program characterized by causing processing to be executed.

Images