JP2011221845A - Text processing apparatus, text processing method and text processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently create a phrase tree allowed to be used for encoding text data at a high compression ratio.SOLUTION: A frequency counting means 1c counts up the appearance frequency of a selected character given to a node of a decision position. When the appearance frequency of the selected character given to the node of the decision position reaches a predetermined threshold, a node addition means 1b adds a slave node corresponding to the selected character for the node of the decision position to a phrase tree 2a. When there exists the slave node corresponding to the selected character for the node of the decision position, the node addition means 1b changes the decision position to the slave node corresponding to the selected character for the node of the decision position. When the slave node corresponding to the selected character does not exist and the appearance frequency of the selected character given to the node of the decision position does not reach the predetermined threshold, a decision position change means 1d changes the decision position to a slave node of a node of a route.

Description

  The present invention relates to a text processing device, a text processing method, and a text processing program for processing text data.

In a computer system, information processing may be performed on text data as a processing target. For example, information retrieval for text data is performed.
In recent years, with the expansion of applications of information communication technology, the amount of text data to be processed tends to increase. Therefore, various techniques for improving the efficiency of information processing for processing text data have been considered. For example, a data sort processing method that performs data sort processing at high speed using a small-capacity memory is considered. In addition, a data set dividing method that can efficiently divide a data set along a designated item has been considered.

  By the way, when searching text data, the processing speed can be increased by efficiently detecting a character string that matches the search keyword from the stored text data. For example, there is a compression pattern matching technique. In compressed pattern matching, text data is compressed and stored, and pattern matching is performed without decompressing the compressed text.

  In the compression pattern matching, the partial character string included in the input text is encoded into a code that can uniquely identify the partial character string so that the text data can be searched while being compressed. By encoding the partial character string included in the text data, the data amount can be reduced as compared with the original text data.

  When a specific character string in a text is encoded, if a partial character string having a high appearance frequency or a long partial character string is encoded, the compression rate is improved. As a technique for improving the compression rate, a STVF (Suffix Tree based Variable-length-to-Fixed-length code) coding technique is considered.

  In the STVF encoding technique, VF (Variable-length-to-Fixed-length code) encoding is performed using a suffix tree trimmed based on frequency information as a phrase tree.

JP 2007-11784 A JP 2007-11548 A

Takuya Kida, "Suffix Tree Based VF-Coding for Compressed Pattern Matching" Hokkaido University, TCS Research Reports, TCS-TR-A-08-36, Division of Computer Science, 18 November 2008 Takuya Kida, "Suffix Tree Based VF-Coding for Compressed Pattern Matching," dcc, Proceedings of the 2009 Data Compression Conference, 16-18 March 2009, pp.449

  However, there is a problem that it takes time to create a phrase tree by STVF encoding. That is, the creation of a suffix tree in STVF encoding takes time for processing because the entire text data is analyzed. Moreover, the creation of a suffix tree takes more time as the amount of text data increases.

  The present invention has been made in view of the above points, and is a text processing device, a text processing method, and a text processing method capable of efficiently creating a phrase tree that can be used for encoding text data at a high compression rate. An object is to provide a text processing program.

  In order to solve the above-described problem, a text processing apparatus having a character selection unit, a node addition unit, and a frequency count unit is provided. A character selection means selects a character in order from the character string in text data. The node adding means associates a plurality of nodes corresponding to characters that can appear in the text data in advance in a tree structure as children of the root node, and child nodes corresponding to characters that can appear in the text data for each node A phrase tree memory that stores a phrase tree in which a node identifier and the number of appearances of each character that appears next when each node is determined are associated with each node. Refer to the means. Then, the node adding means starts the determination position from the root node, and when there is a child node corresponding to the character selected by the character selection means for the determination position node, the determination position is set to the child node. Moving. In addition, the node adding means has no child node corresponding to the character selected for the node at the determination position, and the number of appearances of the selected character assigned to the node at the determination position has reached a predetermined threshold. If not, the determination position is moved to the child node corresponding to the selected character for the root node. Further, the node adding means has no child node corresponding to the character selected for the node at the determination position, and the number of appearances of the selected character assigned to the node at the determination position has reached a predetermined threshold. If it is, the child node corresponding to the selected character to which a new identifier is assigned is added to the node at the determination position, and the determination position is moved to the added child node. The frequency counting means refers to the phrase tree storage means, and if there is no node corresponding to the character selected for the node at the judgment position, counts the number of appearances of the selected character to which the node at the judgment position is given. Up.

  Also provided is a text processing method in which a computer executes a process similar to the process executed by the text processing apparatus. Furthermore, a text processing program for causing a computer to execute processing similar to the processing executed by the text processing device is provided.

  According to the above text processing device, text processing method, and text processing program, it is possible to efficiently create a phrase tree that can be used for encoding text data at a high compression rate.

It is a figure which shows the outline | summary of embodiment. It is a figure which shows an example of a system configuration. It is a figure which shows one structural example of the hardware of the server used for this Embodiment. It is a figure which shows an example of the node structure body of a summary trie. It is a figure which shows the example of a summary try. It is a block diagram which shows the function of a server. It is a block diagram which shows the detailed function of a text encoding part. It is a flowchart which shows the procedure of a text encoding process. It is a flowchart which shows the procedure of summary trie creation and text compression processing. It is a flowchart which shows the procedure of summary trie creation and text compression processing. It is a figure which shows the example of the summary trie of an initial state, and the compressed text. It is a figure which shows the example of the summary trie and the compressed text after processing the 1st character. It is a figure which shows the example of the summary trie and the compressed text after processing the 2nd character. It is a figure which shows the example of the summary trie and the compressed text after processing the 3rd character. It is a figure which shows the example of the summary trie and the compressed text after processing the 4th character. It is a figure which shows the example of the summary trie and the compressed text after processing the 5th character. It is a figure which shows the example of the summary trie and the compressed text after processing the 6th character. It is a figure which shows the example of the summary trie and the compressed text after processing the 7th character. It is a figure which shows the example of the summary trie and the compressed text after processing the 8th character. It is a figure which shows the example of the summary trie and the compressed text after processing the 9th character. It is a figure which shows the example of the summary trie and the compressed text after processing the 10th character. It is a figure which shows the example of the summary trie after the value of position "i" becomes larger than the number of characters "n", and the compressed text. It is a figure which shows the function of the server which concerns on 3rd Embodiment. It is a block diagram which shows the detailed function of the text encoding part which concerns on 3rd Embodiment. It is a flowchart which shows the procedure of the text encoding process which concerns on 3rd Embodiment. It is a flowchart which shows the detailed procedure of a compression process. It is a block diagram which shows the function of the text encoding part which concerns on 4th Embodiment. It is a figure which shows an example of the data structure of a threshold value table memory | storage part. It is a flowchart which shows the procedure of the text encoding process which concerns on 4th Embodiment.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram showing an outline of the embodiment. The text processing apparatus 1 includes a character selection unit 1a, a node addition unit 1b, a frequency count unit 1c, and an identifier output unit 1d.

  The phrase tree storage means 2 stores a phrase tree 2a. In the phrase tree 2a, a plurality of nodes corresponding to characters that can appear in the text data 3 are associated in advance in a tree structure as children of the root node. The phrase tree 2a can add a child node corresponding to a character that can appear in the text data to each node. Further, in association with each node of the phrase tree 2a, an identifier of the node and the number of appearances of each character that appears next when each node is set as the determination position are given.

  In the example of FIG. 1, each node is represented by a circle, and the identifier of the corresponding node is indicated in the circle indicating the node. The phrase tree 2a indicates a relationship between nodes in the tree structure by arrows connecting the nodes. Of the two connected nodes, a node closer to the root node is a parent node, and a node far from the root node is a child node. The characters corresponding to the child nodes are shown next to the arrows connecting the nodes.

  The root node is provided with a child node corresponding to each character that can be included in the text data 3 from the initial state, for example. In the example of FIG. 1, it is assumed that the text data 3 can include four characters “a, b, c, d”, and the nodes of the root of the identifier “0” have identifiers “1” to “4”. Four child nodes are provided.

  The character selection unit 1 a selects characters in order from the character string in the text data 3. In the example of FIG. 1, the text data 3 includes a character string “abcabcdabc”. For example, the character selection unit 1a selects characters one by one in order from the left of “abcabbcdabc”.

  The node addition unit 1b refers to the phrase tree storage unit 2. Then, the node adding unit 1b starts the determination position from the root node. When there is a child node corresponding to the character selected by the character selection unit 1a with respect to the node at the determination position, the node addition unit 1b moves the determination position to the child node.

  Further, the node adding means 1b, when there is no child node corresponding to the character selected for the node at the determination position, the number of appearances of the selected character given to the node at the determination position reaches a predetermined threshold. Judge whether or not. When the number of appearances of the selected character given to the node at the determination position does not reach the predetermined threshold, the node adding means 1b moves the determination position to the child node corresponding to the selected character for the root node. To do. On the other hand, when the number of appearances of the selected character assigned to the node at the determination position has reached a predetermined threshold, the node adding means 1b is selected by adding a new identifier to the node at the determination position. Add a child node corresponding to the specified character. At this time, the node adding means 1b moves the determination position to the added child node.

  For example, the node adding unit 1b adds child nodes until the number of nodes in the phrase tree 2a reaches a predetermined number. In other words, the node adding means 1b can prevent the child node from being added when the number of nodes exceeds a predetermined number.

  Note that the node added to the phrase tree 2a corresponds to a partial character string (phrase) indicated by an array of characters corresponding to each node on the route when the node is traced from the root node to the corresponding node. That is, the identifier of the added node is a code representing the character string corresponding to the node.

  The frequency counting unit 1 c refers to the phrase tree storage unit 2. When there is no node corresponding to the character selected for the node at the determination position, the frequency counting unit 1c counts up the number of appearances of the selected character to which the node at the determination position is assigned. For example, the frequency counting unit 1c increases the number of appearances of the selected character given to the node at the determination position by “1”. Note that the frequency counting unit 1c counts up the number of appearances of the selected character to which the node at the determination position is added even when there is a node corresponding to the character selected with respect to the node at the determination position. May be.

  The frequency counting means 1c can count up the number of appearances prior to the processing by the node adding means 1b when there is no node corresponding to the character selected for the node at the determination position. In this case, the node addition unit 1b determines whether or not the number of appearances after counting up has reached a threshold value.

  The identifier output unit 1d refers to the phrase tree storage unit 2. When there is no child node corresponding to the character selected for the node at the determination position, the identifier output unit 1d outputs the identifier given to the node at the determination position. The identifier output means 1d outputs the identifier assigned to the node at the determined position after the movement when the determined position is moved by selecting the last character of the character string in the text data 3. Also good. Here, each identifier output by the identifier output means 1 d becomes a code for each partial character string in the text data 3. An identifier string output from the identifier output unit 1 d corresponding to all character strings in the text data 3 becomes a code word string 4 obtained by encoding the text data 3. For example, the identifier output unit 1d can store the codeword string 4 in the storage device. The identifier output means 1d can also send the code word string 4 sequentially from the generated code via the network.

  According to such a text processing apparatus 1, the character selection means 1a selects the character strings in the text data 3 in order. Then, the frequency count means 1c counts up the number of appearances of the selected character given to the node at the determination position.

  When there is a child node corresponding to the character selected for the node at the determination position, the node addition unit 1b changes the determination position to the child node corresponding to the selected character at the node at the determination position. Be changed. As a result, as long as there is a child node corresponding to the selected character, the node at the determination position transitions from the root of the phrase tree toward the leaf.

  If there is no child node corresponding to the character selected for the node at the determination position, and the number of appearances of the selected character assigned to the node at the determination position reaches the threshold value, the node adding means 1b Thus, the node is added to the phrase tree 2a. That is, the node adding unit 1b adds a child node corresponding to the selected character for the node at the determination position to the phrase tree 2a. In this case, the node adding unit 1b moves to the child node to which the determination position is added.

  If there is no child node corresponding to the character selected for the node at the determination position, and the number of appearances of the selected character given to the node at the determination position does not reach the threshold value, the node addition means The determination position is moved by 1b. In other words, the node addition means 1b moves the determination position to the child node corresponding to the selected character for the root node of the phrase tree 2a. As a result, if there is no child node corresponding to the selected character, the process returns to the root node and collation with the phrase tree is performed by the partial character string starting from the selected character.

  Further, when there is no child node corresponding to the character selected for the node at the determination position, the identifier output means 1d outputs the identifier assigned to the node at the determination position. For this reason, when all the characters in the text data 3 are selected, the identifier output means 1d outputs a code word string 4 obtained by encoding the entire text data 3. Thereby, the text data 3 can be encoded in parallel with the construction process of the phrase tree 2a.

  Thus, by selecting characters in the text data 3, a phrase tree 2a composed of nodes corresponding to character strings whose appearance frequency is a predetermined value or more is generated. The text data 3 is encoded using the phrase tree 2a as an encoding dictionary, and a code word string 4 is output.

Since the generated phrase tree 2a is composed of nodes corresponding to partial character strings having a high appearance frequency, encoding at a high compression rate is possible by encoding using the phrase tree 2a.
Further, the construction of the phrase tree 2a by the text processing apparatus 1 can be performed in a short time because the construction of the suffix tree from the entire text data 3 is not required as compared with STVF encoding. Moreover, since the text data 3 can be encoded in parallel with the construction of the phrase tree 2a, the text data 3 can also be encoded efficiently. For example, the encoding by the text processing apparatus 1 can be performed about 20 times faster than the STVF encoding technique (the time required for encoding is 1/20).

  Further, the encoding of the text data 3 by the text processing device 1 can be sequentially performed unlike the STVF encoding. That is, when STVF encoding is used, after constructing a suffix tree based on the text data 3, the suffix tree is trimmed (a process for leaving only nodes corresponding to high-frequency character strings). Is generated (first pass processing). In STVF encoding, text data 3 is encoded using the generated phrase tree (second pass process). Thus, since STVF encoding is a two-pass algorithm, encoding of text data cannot be started until the processing of the first pass is completed, and sequential processing is difficult.

  On the other hand, the text processing device 1 can select characters in order from the beginning of the text data 3, and can construct the phrase tree 2a and encode the text data 3 from the characters. Then, each character in the text data 3 may be selected as a processing target once. Therefore, sequential processing for each character becomes possible. Since sequential processing is possible, for example, text data 3 transferred over a network is generated by sequentially encoding partial character strings in the text data 3 while minimizing communication delay. Can be transferred at any time.

  When priority is given to improving the compression ratio over sequential processing, after the phrase tree 2a is constructed, the phrase tree 2a may be used to select again from the first character of the text data 3 in order and perform the encoding process. . Thereby, the compression rate of the text data 3 can be increased.

  The phrase tree 2a has an advantageous property for realizing high-speed compressed pattern matching of the codeword string 4. Compressed pattern matching is a technique for performing pattern matching on a compressed text without decompression. Properties of compression techniques that are advantageous for high-speed compression pattern matching include VF encoding, static encoding dictionaries, and higher compression rates. As described above, the phrase tree 2a has a high compression rate. The phrase tree 2a generated by the text processing device 1 can be VF-encoded by setting the number of identifier bits of each node used as a code to a fixed length. If the code has a fixed length, the delimiter between the codes is clear, and the processing load for collating the character string to be collated with the code word string 4 can be reduced. Further, when the addition of the node to the phrase tree 2a ends when the number of nodes for the phrase tree 2a reaches a predetermined number, the subsequent phrase tree 2a can be used as a static coding dictionary.

Thus, the text processing apparatus 1 can create a phrase tree 2a suitable for high-speed compressed pattern matching.
[Second Embodiment]
In the second embodiment, the text encoding technique according to the first embodiment is applied to a system that performs a search by compressed pattern matching.

  FIG. 2 is a diagram illustrating an example of a system configuration. In the example of FIG. 2, a plurality of clients 201 and 202 are connected to the server 100 via the network 10. The server 100 compresses and stores the text document input from the clients 201 and 202 by encoding. Further, in response to the text search request from the clients 201 and 202, the server 100 searches for a character that matches the search request (search query) from the compressed and held text document. Further, in response to the decompression request from the clients 201 and 202, the server 100 decompresses the compressed text document to the original state.

  Clients 201 and 202 are computers used by users. The user can operate the clients 201 and 202 to transmit a text document from the clients 201 and 202 to the server 100. The user can operate the clients 201 and 202 to transmit a search request and a decompression request from the clients 201 and 202 to the server 100.

  FIG. 3 is a diagram illustrating a configuration example of server hardware used in the present embodiment. The server 100 is entirely controlled by a CPU (Central Processing Unit) 101. A RAM (Random Access Memory) 102 and a plurality of peripheral devices are connected to the CPU 101 via a bus 108.

  The RAM 102 is used as a main storage device of the server 100. The RAM 102 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the CPU 101. The RAM 102 stores various data necessary for processing by the CPU 101.

  Peripheral devices connected to the bus 108 include a hard disk drive (HDD) 103, a graphic processing device 104, an input interface 105, an optical drive device 106, and a communication interface 107.

  The HDD 103 magnetically writes and reads data to and from the built-in disk. The HDD 103 is used as a secondary storage device of the server 100. The HDD 103 stores an OS program, application programs, and various data. Note that a semiconductor storage device such as a flash memory can also be used as the secondary storage device.

  A monitor 11 is connected to the graphic processing device 104. The graphic processing device 104 displays an image on the screen of the monitor 11 in accordance with a command from the CPU 101. Examples of the monitor 11 include a display device using a CRT (Cathode Ray Tube) and a liquid crystal display device.

  A keyboard 12 and a mouse 13 are connected to the input interface 105. The input interface 105 transmits a signal sent from the keyboard 12 or the mouse 13 to the CPU 101. The mouse 13 is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.

  The optical drive device 106 reads data recorded on the optical disk 14 using laser light or the like. The optical disk 14 is a portable recording medium on which data is recorded so that it can be read by reflection of light. The optical disk 14 includes a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (ReWritable), and the like.

  The communication interface 107 is connected to the network 10. The communication interface 107 transmits and receives data to and from other computers or communication devices via the network 10.

  With the hardware configuration as described above, the processing functions of the present embodiment can be realized. Although FIG. 3 shows the hardware configuration of the server 100, the clients 201 and 202 can also be realized with the same hardware configuration.

  Next, the data structure of the summary trie will be described. A summary trie has an ordered tree structure called a trie. In the summary trie, node structures representing nodes (nodes) are connected by a pointer.

  FIG. 4 is a diagram illustrating an example of the node structure of the summary trie. The node structure 20 is provided with a node ID 21, a label 22, a counter list 23 of child nodes, and a pointer list 24 to the child nodes.

  The node ID 21 is an identifier for identifying the node structure 20. An integer of 1 or more incremented by 1 is assigned to the node ID 21 in the order in which the node structure 20 is created. The node ID 21 is used as a compression code.

  The label 22 is a character corresponding to the node structure 20. When the node structure 20 of the child node is connected to the node structure of the parent node in a predetermined array corresponding to the corresponding character, the character corresponding to each node structure is specified by the order on the array it can. In this case, the node structure 20 may not have the label 22.

  The child node counter list 23 is provided with counters corresponding to the types of characters that may exist in the text data to be processed. For example, if the characters that can exist in the text data are alphabets, counters corresponding to the number of characters of the alphabet are prepared. In addition, when one-byte characters having a predetermined code system can exist in the text data, 256 counters are prepared. Each counter included in the child node counter list 23 is associated with a child node. In the counter corresponding to the child node, the number of times that the character corresponding to the child node appears next to the character string of each node from the root node to the node of the node structure 20 is set. The initial value of each counter is “0”.

  In the pointer list 24 to the child node, pointers corresponding to the types of characters that can exist in the text data to be processed are provided. For example, if the characters that can exist in the text data are alphabets, pointers corresponding to the number of characters of the alphabet are prepared. In addition, when 1-byte characters with a predetermined code system can exist in text data, 256 pointers are prepared. Each pointer included in the child node pointer list 24 is associated with a child node. Information for uniquely specifying the node structure of the child node is set in the pointer corresponding to the child node. The initial value of each pointer is “NULL”.

By connecting the nodes of such a node structure, a summary trie that is an ordered tree structure (trie) is created.
FIG. 5 is a diagram illustrating an example of a summary trie. In the example of FIG. 5, the characters that can exist in the text data are “a, b, c, d”, and a set Σ (Σ = {a, b, c, d}) including these characters is defined. . In this case, the number of elements (| Σ |) included in Σ is “4”.

  The summary trie 30 is composed of a plurality of nodes 31 to 37. Each of the nodes 31 to 37 has a node structure as shown in FIG. Counters corresponding to a, b, c, and d are provided in the counter list of the child nodes in the nodes 31 to 37, respectively. In the pointer list to the child nodes in each of the nodes 31 to 37, pointers corresponding to a, b, c, and d are provided.

  The node 31 is a root node, and the node ID is “0”. The label of the node 31 that is the root node is set to “root”. The node 32 is a node corresponding to the character “a”, and “a” is set in the label. The node ID of the node 32 is “1”. The node 33 is a node corresponding to the character “b”, and “b” is set in the label. The node ID of the node 33 is “2”. The node 34 is a node corresponding to the character “c”, and “c” is set in the label. The node ID of the node 34 is “3”. The node 35 is a node corresponding to the character “d”, and “d” is set in the label. The node ID of the node 35 is “4”. The node 36 is a node corresponding to the character “b”, and “b” is set in the label. The node ID of the node 36 is “5”. The node 37 is a node corresponding to the character “c”, and “c” is set in the label. The node ID of the node 37 is “6”.

  The values of the counters in the counter list of the child node provided in the node 31 are all “0”. In the pointer list to child nodes provided in the node 31, pointers to four nodes 32 to 35 which are child nodes are set.

In FIG. 5, the pointer and the node pointed to by the pointer are connected by an arrow. The “NULL” pointer is indicated by “•” in the figure.
Of the counters in the counter list of the child node provided in the node 32, “2” is set only for the counter corresponding to the character “b”, and the values of the other counters are “0”. In the example of FIG. 5, when the counter value reaches “2”, a child node of a character corresponding to the counter is assumed to be created. Therefore, a node 36 corresponding to the character “b” is created as a child node of the node 32. The pointer corresponding to the character “b” of the node 32 points to the node 36.

  Since the nodes 33 to 35 do not have a counter whose value has reached “2”, no child nodes are created. In the node 36, the counter corresponding to the character “c” is “2”, and the node 37 is created as a child node corresponding to the character “c”.

  A character string can be encoded using such a summary trie 30. For example, the character string “a, b, c” corresponds to the route from the node 31 to the node 37 of the route. Therefore, the character string “a, b, c” can be encoded into the node ID “6” of the node 37.

  In the second embodiment, characters of text data are read one character at a time, and a summary trie 30 is created and a character string is encoded sequentially. As described above, stream processing for creating a summary trie and encoding a character string is possible, thereby improving the efficiency of the text data encoding process.

  FIG. 6 is a block diagram illustrating functions of the server. The server 100 includes an encoding dictionary storage unit 110, a compressed text storage unit 120, a text encoding unit 130, a search unit 140, and a decompression unit 150. In the example of FIG. 6, it is assumed that the uncompressed text 40 is input from the client 201 and the search query 41 or the decompression request 43 is input from the client 202.

  The encoding dictionary storage unit 110 stores a summary trie used as an encoding dictionary. For example, a part of the storage area of the RAM 102 or the HDD 103 is used as the encoding dictionary storage unit 110.

  The compressed text storage unit 120 stores text data (compressed text) whose data amount is compressed by encoding. For example, a part of the storage area of the RAM 102 or the HDD 103 is used as the compressed text storage unit 120.

  The text encoding unit 130 encodes the character string in the uncompressed text 40 input from the client 201 and compresses the data amount. Note that the text encoding unit 130 creates a summary trie format encoding dictionary when encoding a character string. The text encoding unit 130 stores the created summary trie in the encoding dictionary storage unit 110. In addition, the text encoding unit 130 stores the compressed text whose data amount is compressed in the compressed text storage unit 120.

  The search unit 140 responds to the search query input from the client 202, refers to the compressed text in the compressed text storage unit 120, and stores the uncompressed text 40 of a character string that matches the character string specified by the search query. Locating above. The search for the compressed text can use, for example, a pattern matching technique of a VF code encoded by a VF encoding technique for encoding a variable length information source to a fixed length. In pattern matching for compressed text, pattern matching can be performed without decompressing the compressed text. For example, pattern matching using a SIGMA search algorithm, a KMP (Knuth-Morris-Pratt) algorithm, an AC (Aho-Corasick) algorithm, or the like is possible. The search unit 140 transmits the search result 42 to the client 202. The search result 42 indicates, for example, the position of the character string hit in the search in the uncompressed text 40 (from what character to what character).

  In response to the decompression request input from the client 202, the decompression unit 150 decompresses the compressed text in the compressed text storage unit 120. Then, the decompressing unit 150 transmits the decompressed text 44 that has been decompressed to the client 202.

Next, details of the text encoding unit 130 will be described.
FIG. 7 is a block diagram showing detailed functions of the text encoding unit. The text encoding unit 130 includes a frequency threshold storage unit 131, a maximum code number storage unit 132, a character selection unit 133, a node creation unit 134, a frequency count unit 135, and a code output unit 136.

  The frequency threshold storage unit 131 stores a threshold for the appearance frequency of a character string. For example, a part of the storage area of the RAM 102 or HDD 103 is used as the frequency threshold storage unit 131. For character strings whose appearance frequency exceeds the threshold, a corresponding node is created.

  The maximum code number storage unit 132 stores a maximum code number indicating the maximum number of codes included in the summary trie. For example, a part of the storage area of the RAM 102 or HDD 103 is used as the maximum code number storage unit 132. When the code number of the summary trie reaches the maximum code number, the reservation trie is not updated thereafter.

  The character selection unit 133 selects characters one by one from the top in the input uncompressed text 40. The character selection unit 133 passes the selected character to the node creation unit 134, the frequency count unit 135, and the code output unit 136. Note that the character selection unit 133 can temporarily store the input uncompressed text 40 in, for example, the RAM 102 and select characters one by one from the RAM 102.

  Each time the node creation unit 134 acquires a character from the character selection unit 133, the node creation unit 134 determines whether it is necessary to create a node for the summary trie. Note that the node creation unit 134 manages information (node pointer) indicating a node that is a determination position among nodes included in the summary trie. Then, the node creation unit 134 determines whether it is necessary to create a node by comparing the node structure of the node at the determination position with the acquired character. When it is determined that the node needs to be created, the node creation unit 134 creates a new node structure and adds the node structure to the summary trie stored in the encoding dictionary storage unit 110.

  When the frequency count unit 135 acquires a character from the character selection unit 133, the frequency count unit 135 updates a counter provided in the node at the determination position in the summary trie. For example, when the node corresponding to the non-encoded partial character string up to the character acquired from the character selection unit 133 has not yet been created, the frequency counting unit 135 counts up the counter value corresponding to the character. .

  Each time the code output unit 136 acquires a character from the character selection unit 133, the code output unit 136 determines whether it is necessary to output a code corresponding to a partial character string that is not encoded. When outputting the code, the code output unit 136 stores the node ID of the node in the summary trie as a code in the compressed text storage unit 120. For example, if the node that is the determination position is a leaf of the summary trie, and the node corresponding to the character selected by the character selection unit 133 is not newly created as a child node of the node that is the determination position, the code output unit 136 Is output. In this case, the code output unit 136 encodes the non-encoded partial character string up to the character immediately before the character acquired from the character selection unit 133 into the node ID of the node serving as the determination position. The code output unit 136 stores the code in the compressed text storage unit 120.

Next, the procedure of the text encoding process will be described in detail. The input of the text encoding process is the uncompressed text 40, the threshold value, and the maximum number of codes.
Here, the number of characters included in the uncompressed text 40 is “n” (n is an integer of 1 or more). Then, a character string in the uncompressed text 40 is defined as T = T [1],..., T [n] using an array.

  The threshold value is read from the frequency threshold storage unit 131 and set to the variable “α”. Further, the value of the maximum code number is read from the maximum code number storage unit 132 and set to the variable “K”.

  The output of the text encoding process is a summary trie and compressed text. Here, it is assumed that the summary trie is “D”. The structure of the summary trie “D” is obtained by associating a plurality of node structures with pointers as shown in FIG. Further, the compressed text is “C = Compress (T)”. The compressed text “C” is information in which codes generated by encoding are arranged in the order of generation.

  In the text encoding process, a node pointer “P” indicating the node at the determination position in the summary trie “D” is used. For example, the node pointer “P” is set with the node ID of the node at the determination position.

  Also, the current size of the summary trie is “k”. The size “k” of the summary trie “D” is the number of codes included in the summary trie “D”. If the node ID of the root node of the summary trie is “0” and the node IDs are assigned in order from “1” to the other nodes as in the example of FIG. 5, the codes included in the summary trie “D” The number is equal to the maximum value of the node ID.

  The position of the current character to be processed in the text data is “i”. The position i can take an integer of 0 or more. The position of the character is represented by the order of the corresponding character from the beginning in the text. By setting the position “i” indicating the order of the characters to be processed to the index of the array “T”, the characters to be processed can be extracted.

FIG. 8 is a flowchart showing the procedure of the text encoding process. In the following, the process illustrated in FIG. 8 will be described in order of step number.
[Step S11] Each element in the text encoding unit 130 initializes information. For example, the character selection unit 133 initializes the value of the character position “i” to “0”.

  The node creation unit 134 initializes the summary trie “D” in the coding dictionary storage unit 110. The initialized summary trie “D” represents a state in which all alphabets are registered with a count “0”. For example, the node creation unit 134 creates a node structure of the root node, and connects a node structure of a child node corresponding to each element included in the set Σ under the node structure of the root node. At this time, the node creation unit 134 sets all the values of the counters included in each node to “0”. The node creation unit 134 sets the value of the pointer of each node other than the root node to “NULL”. The number of codes in the initial state is the number of elements in the set Σ, and the number of elements is set to the summary trie size “k”.

  Further, the node creation unit 134 initializes the node pointer “P”. The initial value of the node pointer “P” indicates the root node of the summary trie “D”, and is expressed as “P = root (D)”.

  The code output unit 136 initializes the compressed text in the compressed text storage unit 120. A null character string is set in the initialized compressed text, and is expressed as “C = ε”. ε means an empty character string.

[Step S12] The character selection unit 133 increments the value of the position “i” (i = i + 1). As a result, the first character in the uncompressed text 40 is processed.
[Step S13] The node creation unit 134 determines whether the value of the position “i” is larger than the number of characters “n”. If the value of position “i” is larger than the number of characters “n”, the process proceeds to step S20. If the value of position “i” is less than or equal to the number of characters “n”, the process proceeds to step S14.

  [Step S14] If the value of the position “i” is equal to or less than the number of characters “n”, the node creation unit 134, the frequency count unit 135, and the code output unit 136 cooperate to execute summary trie creation and text compression processing. Details of this processing will be described later (see FIG. 9).

[Step S15] The character selection unit 133 increments the value of the position “i” (i = i + 1). As a result, the next character in the uncompressed text 40 becomes the processing target.
[Step S16] The node creation unit 134 determines whether the size “k” of the summary trie “D” is less than the maximum code number “K”. If the size “k” of the summary trie “D” is less than the maximum code number “K”, the process proceeds to step S13. If the size “k” of the summary trie “D” is equal to or greater than the maximum code number “K”, the process proceeds to step S17.

  [Step S17] The node creation unit 134 determines whether the value of the position “i” is greater than the number of characters “n”. If the value of position “i” is larger than the number of characters “n”, the process proceeds to step S20. If the value of position “i” is equal to or less than the number of characters “n”, the process proceeds to step S18.

[Step S18] If the value of the position “i” is equal to or less than the number of characters “n”, the code output unit 136 executes text compression processing. Details of this processing will be described later (see FIG. 10).
[Step S19] The character selection unit 133 increments the value of the position “i” (i = i + 1). As a result, the next character in the uncompressed text 40 becomes the processing target. Thereafter, the process proceeds to step S17.

  [Step S20] When the value of the position “i” becomes larger than the number of characters “n”, the code output unit 136 writes the code of the node indicated by the node pointer “P” at the end of the compressed text “C”. Thereafter, the process ends.

Next, details of the summary trie creation and text compression processing (step S14) will be described.
FIG. 9 is a flowchart showing a procedure of summary trie creation and text compression processing. In the following, the process illustrated in FIG. 9 will be described in order of step number.

  [Step S21] The node creation unit 134 determines whether or not a child node corresponding to the processing target character “T [i]” exists among the child nodes of the node indicated by the node pointer “P”. . Specifically, the node creation unit 134 acquires a character corresponding to T [i] from the character selection unit 133. Next, the node creation unit 134 refers to the node structure of the node indicated by the node pointer “P”, and confirms the content of the pointer corresponding to the acquired character. If the corresponding pointer is a valid value other than “NULL”, a child node corresponding to the acquired character exists.

If a child node exists, the process proceeds to step S22. If no child node exists, the process proceeds to step S23.
[Step S22] When a child node exists, the node creation unit 134 newly sets a child node corresponding to the character “T [i]” of the node designated by the current node pointer “P” as a node pointer “ The designated destination of “P”. Thereafter, the summary trie creation and text compression processing ends, and the process proceeds to step S15 in FIG.

  [Step S23] When there is no child node, the frequency counting unit 135 increments the value of the counter corresponding to the acquired character in the node specified by the current node pointer “P” by one. A counter value indicating the appearance frequency of the character “T [i]” at the node indicated by the node pointer “P” can be expressed as “count (P, T [i])”.

  [Step S24] The node creation unit 134 sets the value of the counter (count (P, T [i])) corresponding to the acquired character in the node specified by the current node pointer “P” to the threshold “ It is determined whether it is equal to “α”. If the value of the counter is equal to the threshold “α”, the process proceeds to step S25. If not equal, the process proceeds to Step S28.

  [Step S25] When the value of the counter is equal to the threshold value “α”, the node creation unit 134 subordinates the node specified by the current node pointer “P” to the child corresponding to the character “T [i]”. Create a node. Specifically, the node creation unit 134 generates a node structure to which a new node ID is assigned. For example, the node creation unit 134 stores the node ID of the last created node, and sets the value obtained by adding 1 to the node ID as the node ID of the created node. The label of the created node is the letter “T [i]”. In addition, initial values are set in each counter and each pointer in the created node structure. Then, the node creation unit 134 sets a value indicating the newly created node to the pointer corresponding to the character “T [i]” in the node designated by the current node pointer “P”.

[Step S26] The node creation unit 134 designates the node pointer “P” as the child node newly created in Step S25.
[Step S27] The node creation unit 134 increments the value of the size “k” of the summary trie “D”. Thereafter, the summary trie creation and text compression processing ends, and the process proceeds to step S15 in FIG.

  [Step S28] When it is determined in step S24 that the value of the counter is not equal to the threshold value “α”, the code output unit 136 converts the code of the node designated by the node pointer “P” to the compressed text “C”. Write at the end of.

  [Step S29] The node creation unit 134 designates the node pointer “P” as a child node of the root node corresponding to the character “T [i]” among the child nodes of the root node. Thereafter, the summary trie creation and text compression processing ends, and the process proceeds to step S15 in FIG.

Next, details of the text compression process (step S18) will be described.
FIG. 10 is a flowchart showing a procedure of summary trie creation and text compression processing. In the following, the process illustrated in FIG. 10 will be described in order of step number.

  [Step S31] The code output unit 136 determines whether or not a child node corresponding to the processing target character “T [i]” exists among the child nodes of the node indicated by the node pointer “P”. . If a child node exists, the process proceeds to step S32. If no child node exists, the process proceeds to step S33.

  [Step S32] When there is a child node, the code output unit 136 newly sets a child node corresponding to the character “T [i]” of the node designated by the current node pointer “P” as a node pointer “ The designated destination of “P”. Thereafter, the text compression process ends, and the process proceeds to step S19 in FIG.

[Step S33] If no child node exists, the code output unit 136 writes the code of the node designated by the node pointer “P” at the end of the compressed text “C”.
[Step S34] The code output unit 136 sets the designation destination of the node pointer “P” as a child node corresponding to the character “T [i]” among the child nodes of the root node. Thereafter, the summary trie creation and text compression processing ends, and the process proceeds to step S19 in FIG.

  In this way, the summary trie “D” and the compressed text “C” are created. Here, since a child node is added to the summary trie “D” only when the value of the counter reaches the threshold value “α”, only a node corresponding to a character string having a high appearance frequency can be created.

  Moreover, when the size “k” of the summary trie “D” becomes equal to or greater than the maximum code number “K”, the summary trie “D” is not updated and only the creation of the compressed text “C” by the text compression process is continued. That is, if the size of the summary trie “D” exceeds the maximum number of codes, a new node is not created even if the node counter of the summary trie “D” exceeds the threshold “α”. Thereby, the size of the summary trie “D” is suppressed to a predetermined value or less.

Next, a specific example of generating a summary trie and compressed text will be described.
FIG. 11 is a diagram illustrating an example of an initial summary trie and compressed text. In the example of FIG. 11, it is assumed that the characters that can be included in the uncompressed text 40 are four characters “a, b, c, d”. That is, the set Σ = {a, b, c, d} and the alphabet size (| Σ |) = 4. The character string in the input uncompressed text 40 is assumed to be “abcabcdabc”. That is, the array T = “abcabbcdabc”. Further, it is assumed that the threshold value “α = 2” and the maximum code number “K = 256”. That the maximum code number is “256 (2 to the 8th power)” indicates that the code has a fixed length of 1 byte.

  In the initialization phase, an initial summary trie 50 is created. The created summary trie 50 has five nodes 51 to 55. The node 51 is a root node. The nodes 52 to 55 are child nodes of the node 51 corresponding to the characters “a, b, c, d”, respectively. In the node 51, pointers to the nodes 52 to 55 which are child nodes are set.

  In the initial state, the values of the counters in the nodes 51 to 55 are all “0”. In the initial state, the node pointer “P” points to the node 51 (P = root (D)). In the example of FIG. 11, a flag figure in which the letter “P” is written is assigned to the node indicated by the node pointer “P”.

An empty character string (ε) is set in the compressed text 60 by the initialization process. In addition, “0” is set in the position “i” of the character to be processed.
The position “i” is incremented from the state shown in FIG. 11 so that i = 1. Then, the first character “a” in the uncompressed text 40 is selected as a processing target.

  FIG. 12 is a diagram illustrating an example of the summary trie and the compressed text after the first character is processed. In the node 51 of the route designated by the node pointer “P” in the initial state, there are child nodes corresponding to all characters. Therefore, the designation destination of the node pointer “P” is changed to the node 52 corresponding to “a” among the child nodes of the node 51 in accordance with the first character “a”. At this time, no code is written to the compressed text 60.

The position “i” is incremented from the state shown in FIG. 12, and i = 2. Then, the second character “b” in the uncompressed text 40 is selected as a processing target.
FIG. 13 is a diagram illustrating an example of the summary trie and the compressed text after the second character is processed. The node 52 has no child node corresponding to the character “b”. Therefore, the counter corresponding to the character “b” in the node 52 is counted up to “1” (count (1, b) = 1). At this time, count (1, b) <α. Therefore, the code “1” corresponding to the node 52 is written at the end of the compressed text 60 ”. As a result, the compressed text 60 becomes “C = 1”. Then, the designation destination of the node pointer “P” is changed to the node 53 corresponding to “b” among the child nodes of the node 51.

The position “i” is incremented from the state shown in FIG. 13 so that i = 3. Then, the third character “c” in the uncompressed text 40 is selected as a processing target.
FIG. 14 is a diagram illustrating an example of the summary trie and the compressed text after the third character is processed. The node 53 has no child node corresponding to the character “c”. Therefore, the counter corresponding to the character “c” in the node 53 is counted up to “1” (count (2, c) = 1). At this time, count (2, c) <α. Therefore, the code “2” corresponding to the node 53 is written at the end of the compressed text 60. As a result, the compressed text 60 becomes “C = 12.” Then, the designation destination of the node pointer “P” is changed to the node 54 corresponding to “c” among the child nodes of the node 51.

The position “i” is incremented from the state shown in FIG. 14, and i = 4. Then, the fourth character “a” in the uncompressed text 40 is selected as a processing target.
FIG. 15 is a diagram illustrating an example of the summary trie and the compressed text after the fourth character is processed. The node 54 has no child node corresponding to the character “a”. Therefore, the counter corresponding to the character “a” in the node 54 is counted up to “1” (count (3, a) = 1). At this time, count (3, a) <α. Therefore, the code “3” corresponding to the node 54 is written at the end of the compressed text 60. As a result, the compressed text 60 becomes “C = 123”. Then, the designation destination of the node pointer “P” is changed to the node 52 corresponding to “a” among the child nodes of the node 51.

The position “i” is incremented from the state shown in FIG. 15, and i = 5. Then, the fifth character “b” in the uncompressed text 40 is selected as a processing target.
FIG. 16 is a diagram illustrating an example of the summary trie and the compressed text after processing the fifth character. The node 52 has no child node corresponding to the character “b”. Therefore, the counter corresponding to the character “b” in the node 52 is counted up to “2” (count (1, b) = 2). At this time, count (1, b) = α. Therefore, a node 56 is created as a child node corresponding to the character “b” of the node 52. At this time, a value indicating the node 56 is set in the pointer corresponding to the character “b” of the node 52. A node ID “5” is assigned to the newly created node 56 and a label “b” is set. Initial values are set in the counter and pointer in the node 56. Then, the designation destination of the node pointer “P” is changed to the newly created node 56. At this time, no code is written to the compressed text 60.

The position “i” is incremented from the state shown in FIG. 16, and i = 6. Then, the sixth character “c” in the uncompressed text 40 is selected as a processing target.
FIG. 17 is a diagram illustrating an example of the summary trie and the compressed text after processing the sixth character. The node 56 has no child node corresponding to the character “c”. Therefore, the counter corresponding to the character “c” in the node 56 is counted up to “1” (count (5, c) = 1). At this time, count (5, c) <α. Therefore, the code “5” corresponding to the node 56 is written at the end of the compressed text 60. As a result, the compressed text 60 becomes “C = 1235”. Then, the designation destination of the node pointer “P” is changed to the node 54 corresponding to “c” among the child nodes of the node 51.

The position “i” is incremented from the state shown in FIG. 17 so that i = 7. Then, the seventh character “d” in the uncompressed text 40 is selected as a processing target.
FIG. 18 is a diagram illustrating an example of the summary trie and the compressed text after processing the seventh character. In the state of FIG. 17, the node 54 has no child node corresponding to the character “d”. Therefore, the counter corresponding to the character “d” in the node 54 is counted up to “1” (count (3, d) = 1). At this time, count (3, d) <α. Therefore, the code “3” corresponding to the node 54 is written at the end of the compressed text 60. As a result, the compressed text 60 becomes “C = 1353”. Then, the designation destination of the node pointer “P” is changed to the node 55 corresponding to “d” among the child nodes of the node 51.

The position “i” is incremented from the state shown in FIG. 18, and i = 8. Then, the eighth character “a” in the uncompressed text 40 is selected as a processing target.
FIG. 19 is a diagram illustrating an example of the summary trie and the compressed text after processing the eighth character. The node 55 has no child node corresponding to the character “a”. Therefore, the counter corresponding to the character “a” in the node 55 is counted up to “1” (count (4, a) = 1). At this time, count (4, a) <α. Therefore, the code “4” corresponding to the node 55 is written at the end of the compressed text 60. As a result, the compressed text 60 becomes “C = 123534”. Then, the designation destination of the node pointer “P” is changed to the node 52 corresponding to “a” among the child nodes of the node 51.

The position “i” is incremented from the state shown in FIG. 19, and i = 9. Then, the ninth character “b” in the uncompressed text 40 is selected as a processing target.
FIG. 20 is a diagram illustrating an example of the summary trie and the compressed text after the ninth character has been processed. The node 52 has a child node corresponding to the character “b”. Therefore, the designation destination of the node pointer “P” is changed to the node 56 corresponding to “b” among the child nodes of the node 52 in accordance with the ninth character “b”. At this time, no code is written to the compressed text 60.

The position “i” is incremented from the state shown in FIG. 20 so that i = 10. Then, the tenth character “c” in the uncompressed text 40 is selected as a processing target.
FIG. 21 is a diagram illustrating an example of the summary trie and the compressed text after the 10th character is processed. In the state of FIG. 20, the node 56 has no child node corresponding to the character “c”. Therefore, the counter corresponding to the character “c” in the node 56 is counted up to “2” (count (5, c) = 2). At this time, count (5, c) = α. Therefore, a node 57 is created as a child node corresponding to the character “c” of the node 52. At this time, a value indicating the node 57 is set in the pointer corresponding to the character “c” of the node 56. The newly created node 57 is given a node ID “6” and a label “c” is set. Initial values are set in the counter and pointer in the node 57. Then, the designation destination of the node pointer “P” is changed to the newly created node 57. At this time, no code is written to the compressed text 60.

  The position “i” is incremented from the state shown in FIG. 21 so that i = 11. The eleventh character does not exist in the text. Therefore, the value of the position “i” is larger than the number of characters “n”.

  FIG. 22 is a diagram illustrating an example of the summary trie and the compressed text after the value of the position “i” is larger than the number of characters “n”. When the value of the position “i” becomes larger than the number of characters “n”, the construction process of the summary trie 50 ends. Then, the code of the node 57 indicated by the node pointer “P” is written in the compressed text 60. As a result, the compressed text 60 becomes “C = 1235346”.

  In this way, the summary trie 50 and the compressed text 60 are created based on the uncompressed text data. When the text encoding technique shown in the second embodiment is used, it becomes unnecessary to create a suffix tree like STVF. Since the suffix tree is created by reading all text data, it takes time to construct. Therefore, the text encoding technique shown in the second embodiment can perform encoding about 20 times faster than STVF by eliminating the need to create a suffix tree.

  Moreover, in the text encoding technique shown in the second embodiment, characters in the text can be read one by one from the beginning and sequentially processed (stream processing). If sequential processing can be performed, for example, it can be used for encoding transmission data in communication between computers. That is, when a suffix tree is constructed from all text data as in STVF, and text data is encoded after creating a phrase tree from the suffix tree, the encoded data is read until the entire transmission data is read. Cannot be created. Therefore, stream processing becomes difficult. On the other hand, in the text encoding technique shown in the second embodiment, encoding can be performed sequentially from the top of transmission data, and encoded data can be transmitted sequentially. Thereby, the communication delay by encoding can be suppressed to the minimum.

[Third Embodiment]
In the third embodiment, a plurality of uncompressed texts can be compressed and used as a character search target.

  FIG. 23 is a diagram illustrating functions of a server according to the third embodiment. The server 100a according to the third embodiment includes an encoding dictionary storage unit 110a, a compressed text storage unit 120a, a text encoding unit 130a, a search unit 140a, and a decompression unit 150a.

The encoding dictionary storage unit 110a and the compressed text storage unit 120a have the same functions as the elements of the same name in the second embodiment shown in FIG.
The text encoding unit 130a has compressed control symbols for distinguishing a plurality of uncompressed texts 71, 72, and 73 in addition to the functions of the text encoding unit 130 of the second embodiment shown in FIG. It has a function of inserting into the text 80. As the control symbol, a symbol that does not overlap with the code assigned in the summary trie is used. For example, in the example of FIG. 23, “$” is inserted in the compressed text 80 as a control symbol indicating a delimiter between the uncompressed texts 71 to 73.

  The search unit 140a has a function of including information indicating uncompressed text including the hit character string in the search result, in addition to the function of the search unit 140 of the second embodiment shown in FIG. For example, the uncompressed text including the hit character string is indicated by the order in the compressed text 80. For example, the search unit 140a counts the number of control symbols “$” preceding the position of the character string hit in the search, and adds the number obtained by adding 1 to the order of the uncompressed text including the corresponding character string. Are included in the search result 42.

  When the decompression unit 150a receives the decompression request 43 specifying the order of uncompressed text, in addition to the functions of the decompression unit 150 of the second embodiment shown in FIG. 6, the decompression unit 150a is designated in the compressed text 80. It has a function of decompressing codeword strings in the specified order. For example, the decompressing unit 150a divides the compressed text 80 using the control symbols as boundaries, and generates a plurality of codeword strings. Then, the decompressing unit 150 a counts the order of the code word strings from the beginning of the compressed text 80, decompresses the code word strings in the requested order, and creates the decompressed text 44.

  FIG. 24 is a block diagram illustrating detailed functions of the text encoding unit according to the third embodiment. The text encoding unit 130a includes a frequency threshold storage unit 131a, a maximum code number storage unit 132a, a character selection unit 133a, a node creation unit 134a, a frequency count unit 135a, a code output unit 136a, and a control symbol output unit 137. The frequency threshold storage unit 131a, the maximum code number storage unit 132a, the character selection unit 133a, the node creation unit 134a, the frequency count unit 135a, and the code output unit 136a are elements having the same names in the second embodiment shown in FIG. Has the same function.

  The control symbol output unit 137 indicates that the code output unit 136a has output the last code generated from one uncompressed text (the code stored in step S20 in FIG. 8) to the compressed text storage unit 120a. To detect. Then, the control symbol output unit 137 stores the control symbol “$” at the end (after the last code) of one uncompressed text.

  Next, the procedure of the text encoding process will be described in detail. Note that the input of the text encoding process is the uncompressed text 71, 72, 73 having the number of texts M (M is an integer of 1 or more), the threshold value, and the maximum number of codes. Here, text data strings included in the M uncompressed texts 71, 72, 73,... Are T1,. Also, the text number for identifying the uncompressed text that is currently processed is “m” (m is an integer of 1 or more). A control symbol indicating a boundary between texts is “$”. The meanings of the other symbols are the same as in the second embodiment.

FIG. 25 is a flowchart illustrating the procedure of the text encoding process according to the third embodiment. In the following, the process illustrated in FIG. 25 will be described in order of step number.
[Step S41] Each element in the server 100a initializes information. For example, the character selection unit 133a initializes the value of the character position “i” to “0”. In addition, the character selection unit 133a initializes the value of the text number “m” to “1”.

  The node creation unit 134a initializes the summary trie “D” in the coding dictionary storage unit 110a. Furthermore, the node creation unit 134a initializes the node pointer “P”. The code output unit 136a initializes the compressed text in the compressed text storage unit 120a. A null character string is set in the initialized compressed text, and is expressed as “C = ε”. ε means an empty character string.

  [Step S42] The character selection unit 133a determines whether or not the text number “m” is larger than the text number “M”. If the text number is greater than the number of texts, the process ends. If the text number is less than or equal to the number of texts, the process proceeds to step S43.

[Step S43] The character selection unit 133a acquires the “m” -th text data “Tm”.
[Step S44] The character selection unit 133a sets “1” as the value of the position “i” (i = 1). As a result, the first character of the “m” th text data “Tm” becomes the processing target.

[Step S45] The character selection unit 133a substitutes the length (number of characters) of the text data “Tm” for the number of characters “n”.
[Step S46] The compression processing is performed by the cooperation of the character selection unit 133a, the node creation unit 134a, the frequency count unit 135a, and the code output unit 136. Details of this processing will be described later (see FIG. 26).

[Step S47] The control symbol output unit 137 writes the control symbol “$” at the end of the compressed text “C”.
[Step S48] The node creation unit 134a initializes the node pointer “P” (P = root (D)).

[Step S49] The character selection unit 133a adds 1 to the text number “m”. Thereafter, the process proceeds to step S42.
FIG. 26 is a flowchart showing a detailed procedure of the compression process. In the following, the process illustrated in FIG. 26 will be described in order of step number.

  [Step S51] The node creation unit 134a determines whether the value of the position “i” is larger than the number of characters “n”. If the value of position “i” is larger than the number of characters “n”, the process proceeds to step S58. If the value of position “i” is equal to or less than the number of characters “n”, the process proceeds to step S52.

  [Step S52] If the value of the position “i” is equal to or less than the number of characters “n”, the node creation unit 134a, the frequency count unit 135a, and the code output unit 136a cooperate to execute summary trie creation and text compression processing. The details of this processing are the same as the summary trie creation and text compression processing in the second embodiment shown in FIG.

  [Step S53] The character selection unit 133a increments the value of the position “i” (i = i + 1). As a result, the next character in the uncompressed text with the text number “m” becomes the processing target.

  [Step S54] The node creation unit 134a determines whether the size “k” of the summary trie “D” is less than the maximum code number “K”. If the size “k” of the summary trie “D” is less than the maximum code number “K”, the process proceeds to step S51. If the size “k” of the summary trie “D” is equal to or greater than the maximum code number “K”, the process proceeds to step S55.

  [Step S55] The node creation unit 134a determines whether the value of the position “i” is larger than the number of characters “n”. If the value of position “i” is larger than the number of characters “n”, the process proceeds to step S58. If the value of position “i” is equal to or less than the number of characters “n”, the process proceeds to step S56.

  [Step S56] If the value of the position “i” is equal to or less than the number of characters “n”, the code output unit 136a executes text compression processing. Details of this processing are the same as the text compression processing in the second embodiment shown in FIG.

  [Step S57] The character selection unit 133a increments the value of the position “i” (i = i + 1). As a result, the next character in the uncompressed text with the text number “m” becomes the processing target. Thereafter, the process proceeds to step S55.

  [Step S58] When the value of the position “i” becomes larger than the number of characters “n”, the code output unit 136 writes the code of the node indicated by the node pointer “P” at the end of the compressed text “C”. Thereafter, the compression process ends, and the process proceeds to step S47 in FIG.

  In this way, control symbols can be inserted between encoded individual text data. With this control symbol, the code word string in the compressed text can be divided for each uncompressed text from which the code is generated. And the uncompressed text which became the production | generation origin of each codeword sequence can be discriminate | determined by the order of the codeword sequence divided | segmented in the compressed text.

[Fourth Embodiment]
In the fourth embodiment, the threshold value “α” of a node included in the summary trie can be dynamically changed. In encoding using summary trie, the compression rate varies greatly depending on how the threshold value “α” is set. If the threshold “α” is too large, the summary trie does not grow easily, and a code cannot be assigned to a long character string. Conversely, if the threshold “α” is too small, the summary trie grows too quickly and exceeds the maximum size before reading back data. Therefore, in the fourth embodiment, using the threshold value table, the threshold value “α” is initially kept small, and the threshold value “α” is gradually increased as the amount of data to be read increases.

  FIG. 27 is a block diagram illustrating functions of the text encoding unit according to the fourth embodiment. The text encoding unit 130b encodes the uncompressed text 40 and generates a summary trie and a compressed text. The summary trie is stored in the encoding dictionary storage unit 110b as an encoding dictionary. The compressed text is stored in the compressed text storage unit 120b. The encoding dictionary storage unit 110b stores a summary trie format encoding dictionary. The compressed text storage unit 120b stores the compressed text.

  The text encoding unit 130b includes a frequency threshold storage unit 131b, a maximum code number storage unit 132b, a character selection unit 133b, a node creation unit 134b, a frequency count unit 135b, a code output unit 136b, a threshold table storage unit 138, and a threshold setting unit. 139. The frequency threshold storage unit 131b, the maximum code number storage unit 132b, the character selection unit 133b, the node creation unit 134b, the frequency count unit 135b, and the code output unit 136b are elements having the same names in the second embodiment shown in FIG. Has the same function.

  The threshold value table storage unit 138 stores a threshold value table indicating the correspondence between the number of encoded characters and the threshold value. For example, a part of the storage area of the RAM 102 or the HDD 103 is used as the threshold table storage unit 138.

  The threshold setting unit 139 sets the threshold in the frequency threshold storage unit 131b according to the number of characters selected by the character selection unit 133b and passed to the node creation unit 134b. At this time, the threshold setting unit 139 refers to the threshold table storage unit 138 and determines a threshold to be set.

  FIG. 28 is a diagram illustrating an example of a data structure of the threshold table storage unit. The threshold value table storage unit 138 stores a threshold value table 138a. The threshold value table 138a has columns for the number of characters and the threshold value. Information arranged in the horizontal direction in the threshold value table 138a is associated with each other.

  A numerical value range indicating the number of encoded characters is set in the number of characters column. In the example of FIG. 28, the number of characters is indicated by the number of bytes indicating the amount of character data. For example, if it is a 1-byte character, the numerical value of the number of bytes becomes the number of characters as it is. If it is a 2-byte character, the number of characters is half the number of bytes.

  In the threshold value column, a threshold value when characters within the range of the corresponding number of characters are encoded is set. For example, if the data amount of the encoded character is within the range of 1 byte to 1 Kbyte, 10 is set as the threshold “α”. If the encoded character data amount is within the range of 1 Kbytes to 100 Kbytes, 100 is set as the threshold “α”. When the character data amount reaches the value “1K” at the boundary between “1 to 1K” and “1K to 100K”, for example, a threshold having a larger value (“100” in this example) is set.

FIG. 29 is a flowchart showing the procedure of the text encoding process according to the fourth embodiment. In the following, the process illustrated in FIG. 29 will be described in order of step number.
[Step S61] Each element in the text encoding unit 130b initializes information. For example, the character selection unit 133b initializes the value of the character position “i” to “0”. The node creation unit 134b initializes the summary trie “D” in the coding dictionary storage unit 110b. Further, the node creation unit 134b initializes the node pointer “P”. The code output unit 136b initializes the compressed text in the compressed text storage unit 120b.

[Step S62] The character selection unit 133b increments the value of the position “i” (i = i + 1). As a result, the first character in the uncompressed text 40 is processed.
[Step S63] The node creation unit 134b determines whether the value of the position “i” is larger than the number of characters “n”. If the value of position “i” is larger than the number of characters “n”, the process proceeds to step S71. If the value of position “i” is equal to or less than the number of characters “n”, the process proceeds to step S64.

  [Step S64] The threshold value setting unit 139 obtains a threshold value “α” based on the threshold value table 138a and the position “i”. For example, if the character included in the uncompressed text 40 is a 1-byte character, the threshold setting unit 139 acquires the value of the position “i” as the number of bytes corresponding to the number of encoded characters. Next, the threshold setting unit 139 selects a range including the acquired number of bytes from the number of characters column of the threshold table 138a. Further, the threshold setting unit 139 acquires a threshold corresponding to the selected range from the threshold table 138a. Then, the threshold setting unit 139 stores the acquired threshold in the frequency threshold storage unit 131b.

  [Step S65] If the value of the position “i” is equal to or less than the number of characters “n”, the node creation unit 134b, the frequency count unit 135b, and the code output unit 136b cooperate to execute summary trie creation and text compression processing. The details of this process are the same as those of the second embodiment shown in FIG.

[Step S66] The character selection unit 133b increments the value of the position “i” (i = i + 1). As a result, the next character in the uncompressed text 40 becomes the processing target.
[Step S67] The node creation unit 134b determines whether the size “k” of the summary trie “D” is less than the maximum code number “K”. If the size “k” of the summary trie “D” is less than the maximum code number “K”, the process proceeds to step S63. If the size “k” of the summary trie “D” is equal to or greater than the maximum code number “K”, the process proceeds to step S68.

  [Step S68] The node creation unit 134b determines whether the value of the position “i” is greater than the number of characters “n”. If the value of position “i” is larger than the number of characters “n”, the process proceeds to step S71. If the value of position “i” is equal to or less than the number of characters “n”, the process proceeds to step S69.

  [Step S69] If the value of the position “i” is equal to or less than the number of characters “n”, the code output unit 136b executes text compression processing. The details of this process are the same as those of the second embodiment shown in FIG.

  [Step S70] The character selection unit 133b increments the value of the position “i” (i = i + 1). As a result, the next character in the uncompressed text 40 becomes the processing target. Thereafter, the process proceeds to step S68.

  [Step S71] When the value of the position “i” becomes larger than the number of characters “n”, the code output unit 136b writes the code of the node indicated by the node pointer “P” at the end of the compressed text “C”. Thereafter, the process ends.

  In this way, the threshold “α” can be dynamically changed. Thereby, even if the number of encoded characters gradually increases as the system is operated, an appropriate threshold “α” corresponding to the number of encoded characters is set. For example, by gradually increasing the value of the threshold “α” as the number of encoded characters increases, it is possible to suppress a situation where the summary “try” does not grow easily due to the threshold “α” being too large. . In addition, since the threshold “α” is too small, it is possible to suppress a situation in which the summary trie grows too quickly and the number of nodes exceeds the maximum number of codes before reading back data.

[Other application examples]
The above processing functions can be realized by a computer. In that case, a program describing the processing contents of the functions that the server should have is provided. By executing the program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic storage device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Examples of the magnetic storage device include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape. Optical discs include DVD, DVD-RAM, CD-ROM / RW, and the like. Magneto-optical recording media include MO (Magneto-Optical disc).

  When distributing the program, for example, a portable recording medium such as a DVD or a CD-ROM in which the program is recorded is sold. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, each time the program is transferred from the server computer, the computer can sequentially execute processing according to the received program.

  In addition, at least a part of the above processing functions can be realized by an electronic circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic Device).

  As mentioned above, although embodiment was illustrated, the structure of each part shown by embodiment can be substituted by the other thing which has the same function. Moreover, other arbitrary structures and processes may be added. Further, any two or more configurations (features) of the above-described embodiments may be combined.

The techniques disclosed in the above embodiments include the techniques shown in the following supplementary notes.
(Supplementary Note 1) Character selection means for selecting characters in order from a character string in text data;
A plurality of nodes corresponding to characters that can appear in the text data are associated in advance in a tree structure as children of the root node, and child nodes corresponding to characters that can appear in the text data can be added to each node. Refers to the phrase tree storage means for storing the phrase tree associated with each node and assigned with the node identifier and the number of times each character appears next when each node is determined. The determination position is started from the node of the root, and when there is a child node corresponding to the character selected by the character selection unit with respect to the node of the determination position, the determination position is included in the child node. , The child node corresponding to the selected character does not exist with respect to the node at the determination position, and the appearance number of the selected character given to the node at the determination position is Is not reached the predetermined threshold, the determination position is moved to a child node corresponding to the selected character for the root node, and the selected character is corresponding to the determination position node. If there is no child node and the number of appearances of the selected character given to the node at the judgment position has reached a predetermined threshold, a new identifier is given to the node at the judgment position Node adding means for adding a child node corresponding to the selected character and moving the determination position to the added child node;
If the node corresponding to the selected character does not exist with respect to the node at the determination position with reference to the phrase tree storage unit, the number of appearances of the selected character to which the node at the determination position is added is counted. Frequency counting means to increase,
A text processing apparatus comprising:

  (Supplementary Note 2) With reference to the phrase tree storage means, if there is no child node corresponding to the selected character with respect to the node at the determination position, an identifier assigned to the node at the determination position is output. The text processing apparatus according to appendix 1, further comprising identifier output means.

  (Supplementary Note 3) When the determination position is moved by selecting the last character of the character string in the text data, the identifier output means displays the identifier assigned to the node of the determination position after the movement. The text processing device according to attachment 2, wherein the text processing device outputs the text processing device.

  (Supplementary note 4) The text processing apparatus according to any one of supplementary notes 1 to 3, wherein the node addition unit performs addition of a child node until the number of nodes of the phrase tree reaches a predetermined number.

(Supplementary note 5) The text processing device according to any one of supplementary notes 1 to 4, further comprising threshold value changing means for changing the threshold value according to the total amount of the selected characters.
(Additional remark 6) The said threshold value change means enlarges the value of the said threshold value, so that the total amount of the said selected character increases, The text processing apparatus of Additional remark 5 characterized by the above-mentioned.

  (Additional remark 7) The control symbol output means which outputs the control symbol which shows that it is the last of the code | cord | chord corresponding to the character string in the said text data after outputting the identifier provided to the node of the said judgment position after movement is further provided The text processing device according to attachment 3, wherein the text processing device is provided.

(Appendix 8) The computer
Select characters in order from the character string in the text data,
A plurality of nodes corresponding to characters that can appear in the text data are associated in advance in a tree structure as children of the root node, and child nodes corresponding to characters that can appear in the text data can be added to each node. Refers to the phrase tree storage means for storing the phrase tree associated with each node and assigned with the node identifier and the number of times each character appears next when each node is determined. And
Start the decision position from the node of the route;
If there is a child node corresponding to the selected character with respect to the node at the determination position, the determination position is moved to the child node;
There is no child node corresponding to the selected character for the node at the determination position, and the number of appearances of the selected character assigned to the node at the determination position does not reach a predetermined threshold value. The determination position is moved to a child node corresponding to the selected character for the root node, and
There is no child node corresponding to the selected character with respect to the node at the determination position, and the number of appearances of the selected character given to the node at the determination position has reached a predetermined threshold value. In this case, a child node corresponding to the selected character with a new identifier is added to the node at the determination position, and the determination position is moved to the added child node.
When there is no node corresponding to the selected character with respect to the node at the determination position, the number of appearances of the selected character to which the node at the determination position is given is counted up.
A text processing method characterized by the above.

(Supplementary Note 9) The computer further includes:
The phrase tree storage means is referred to, and when there is no child node corresponding to the selected character for the node at the determination position, an identifier given to the node at the determination position is output. The text processing method according to appendix 8.

(Supplementary Note 10) The computer further includes:
The supplementary note 9, wherein when the determination position is moved by selecting the last character of the character string in the text data, an identifier given to the node of the determination position after the movement is output. Text processing method.

(Supplementary note 11) The text processing method according to any one of Supplementary notes 8 to 10, characterized in that the addition of child nodes is performed until the number of nodes in the phrase tree reaches a predetermined number.
(Supplementary Note 12) The computer further includes:
12. The text processing method according to any one of appendices 8 to 11, wherein the threshold value is changed according to the total amount of the selected characters.

(Additional remark 13) When changing the said threshold value, the value of the said threshold value is enlarged, so that the total amount of the said selected character increases, The text processing method of Additional remark 12 characterized by the above-mentioned.
(Supplementary Note 14) The computer further includes:
11. The text according to claim 10, wherein after the identifier assigned to the node at the determined position after movement is output, a control symbol indicating the end of the code corresponding to the character string in the text data is output. Processing method.

(Supplementary note 15)
Select characters in order from the character string in the text data,
A plurality of nodes corresponding to characters that can appear in the text data are associated in advance in a tree structure as children of the root node, and child nodes corresponding to characters that can appear in the text data can be added to each node. Refers to the phrase tree storage means for storing the phrase tree associated with each node and assigned with the node identifier and the number of times each character appears next when each node is determined. And
Start the decision position from the node of the route;
If there is a child node corresponding to the selected character with respect to the node at the determination position, the determination position is moved to the child node;
There is no child node corresponding to the selected character for the node at the determination position, and the number of appearances of the selected character assigned to the node at the determination position does not reach a predetermined threshold value. The determination position is moved to a child node corresponding to the selected character for the root node, and
There is no child node corresponding to the selected character with respect to the node at the determination position, and the number of appearances of the selected character given to the node at the determination position has reached a predetermined threshold value. In this case, a child node corresponding to the selected character with a new identifier is added to the node at the determination position, and the determination position is moved to the added child node.
When there is no node corresponding to the selected character with respect to the node at the determination position, the number of appearances of the selected character to which the node at the determination position is given is counted up.
A text processing program for executing a process.

(Supplementary Note 16) The computer further includes:
The phrase tree storage means is referred to, and when there is no child node corresponding to the selected character for the node at the determination position, an identifier given to the node at the determination position is output. The text processing program according to appendix 15.

(Supplementary Note 17) The computer further includes:
The supplementary note 16, wherein when the determination position is moved by selecting the last character of the character string in the text data, an identifier given to the node of the determination position after the movement is output. Text processing program.

(Supplementary note 18) The text processing program according to any one of supplementary notes 15 to 17, wherein the child nodes are added until the number of nodes of the phrase tree reaches a predetermined number.
(Supplementary note 19) The computer further includes:
The text processing program according to any one of appendices 15 to 18, wherein the threshold value is changed according to the total amount of the selected characters.

(Supplementary note 20) The text processing program according to supplementary note 19, wherein when the threshold value is changed, the threshold value is increased as the total amount of the selected characters increases.
(Supplementary Note 21) The computer further includes:
18. The text according to claim 17, wherein after the identifier assigned to the node at the determined position after movement is output, a control symbol indicating the end of the code corresponding to the character string in the text data is output. Processing program.

DESCRIPTION OF SYMBOLS 1 Text processing apparatus 1a Character selection means 1b Node addition means 1c Frequency count means 1d Identifier output means 2 Phrase tree storage means 2a Phrase tree 3 Text data 4 Code word string

Claims (7)

A character selection means for selecting characters in order from the character string in the text data;
A plurality of nodes corresponding to characters that can appear in the text data are associated in advance in a tree structure as children of the root node, and child nodes corresponding to characters that can appear in the text data can be added to each node. Refers to the phrase tree storage means for storing the phrase tree associated with each node and assigned with the node identifier and the number of times each character appears next when each node is determined. The determination position is started from the node of the root, and when there is a child node corresponding to the character selected by the character selection unit with respect to the node of the determination position, the determination position is included in the child node. , The child node corresponding to the selected character does not exist with respect to the node at the determination position, and the appearance number of the selected character given to the node at the determination position is Is not reached the predetermined threshold, the determination position is moved to a child node corresponding to the selected character for the root node, and the selected character is corresponding to the determination position node. If there is no child node and the number of appearances of the selected character given to the node at the judgment position has reached a predetermined threshold, a new identifier is given to the node at the judgment position Node adding means for adding a child node corresponding to the selected character and moving the determination position to the added child node;
If the node corresponding to the selected character does not exist with respect to the node at the determination position with reference to the phrase tree storage unit, the number of appearances of the selected character to which the node at the determination position is added is counted. Frequency counting means to increase,
A text processing apparatus comprising:   Referring to the phrase tree storage means, and when there is no child node corresponding to the selected character for the node at the determination position, identifier output means for outputting an identifier assigned to the node at the determination position The text processing apparatus according to claim 1, further comprising:   The identifier output means outputs the identifier assigned to the node at the determined position after the movement when the determined position is moved by selecting the last character of the character string in the text data. The text processing apparatus according to claim 2, wherein   The text processing apparatus according to claim 1, wherein the node adding unit adds a child node until the number of nodes in the phrase tree reaches a predetermined number.   The text processing apparatus according to claim 1, further comprising a threshold value changing unit that changes the threshold value according to the total amount of the selected characters. Computer
Select characters in order from the character string in the text data,
A plurality of nodes corresponding to characters that can appear in the text data are associated in advance in a tree structure as children of the root node, and child nodes corresponding to characters that can appear in the text data can be added to each node. Refers to the phrase tree storage means for storing the phrase tree associated with each node and assigned with the node identifier and the number of times each character appears next when each node is determined. And
Start the decision position from the node of the route;
If there is a child node corresponding to the selected character with respect to the node at the determination position, the determination position is moved to the child node;
There is no child node corresponding to the selected character for the node at the determination position, and the number of appearances of the selected character assigned to the node at the determination position does not reach a predetermined threshold value. The determination position is moved to a child node corresponding to the selected character for the root node, and
There is no child node corresponding to the selected character with respect to the node at the determination position, and the number of appearances of the selected character given to the node at the determination position has reached a predetermined threshold value. In this case, a child node corresponding to the selected character with a new identifier is added to the node at the determination position, and the determination position is moved to the added child node.
When there is no node corresponding to the selected character with respect to the node at the determination position, the number of appearances of the selected character to which the node at the determination position is given is counted up.
A text processing method characterized by the above. On the computer,
Select characters in order from the character string in the text data,
A plurality of nodes corresponding to characters that can appear in the text data are associated in advance in a tree structure as children of the root node, and child nodes corresponding to characters that can appear in the text data can be added to each node. Refers to the phrase tree storage means for storing the phrase tree associated with each node and assigned with the node identifier and the number of times each character appears next when each node is determined. And
Start the decision position from the node of the route;
If there is a child node corresponding to the selected character with respect to the node at the determination position, the determination position is moved to the child node;
There is no child node corresponding to the selected character for the node at the determination position, and the number of appearances of the selected character assigned to the node at the determination position does not reach a predetermined threshold value. The determination position is moved to a child node corresponding to the selected character for the root node, and
There is no child node corresponding to the selected character with respect to the node at the determination position, and the number of appearances of the selected character given to the node at the determination position has reached a predetermined threshold value. In this case, a child node corresponding to the selected character with a new identifier is added to the node at the determination position, and the determination position is moved to the added child node.
When there is no node corresponding to the selected character with respect to the node at the determination position, the number of appearances of the selected character to which the node at the determination position is given is counted up.
A text processing program for executing a process.

Images