JP5317418B2 - Program and inverted index storage method - Google Patents

Description

  The present invention relates to a transposed index compression method for high-speed full-text search and sequential addition of document data for a large-scale document set.

  Today, electronic documents are indispensable in every scene of economic activity, and a huge amount of electronic documents are created every day. The expansion of the Internet has also caused an explosion in the amount of electronic documents. In order to make the best use of these electronic documents, a document retrieval technique for retrieving desired documents in a short time is essential.

  A typical style of document retrieval is to output a document including a specified word from a given document set in as short a time as possible. For this purpose, a data structure called an inverted index is used. FIG. 1 is a schematic diagram of a document set 101 to be searched and a transposed index 102 constructed based on the set. When an index word 103 appears in a document 104, the information consisting of the identification number 105 of the document and the number of appearances 106 of the index word in the document is called posting 107, and all postings related to the index word 103 A list obtained by collecting is called a transposed list 109. In FIG. 1, 108 is a pointer to the transposed list 109 assigned for each index word. A data structure that stores an inverted list of all index words of the search target document is an inverted index 102.

  Unlike languages in which words such as English, French and Spanish are separated by spaces, it is difficult to accurately divide a document into words in Japanese, Korean and Chinese. Therefore, a transposed index may be constructed by using an arbitrary partial character string in a document consisting of consecutive n characters called n-gram instead of a word, and is known to be useful in practice. (Information Retrieval Algorithm, Kenji Kita et al., Kyoritsu Publishing). n is an integer of 1 to 10 mainly. In this specification, a word and a partial character string of length n are collectively referred to as an index word.

  As the number or size of documents to be searched increases, the transposed index becomes huge. Therefore, it is important to keep the size compact. A technology that compresses posting at a high compression rate while achieving high-speed access has also been proposed (IHWitten, A. Moffat, and TC Bell, Managing Gigabytes 2nd Ed .: Compressing and Indexing Documents and Images, Morgan Kaufmann Publishers Inc. ., San Francisco, CA, USA, 1998, and F. Scholer, HE Williams, J. Yiannis, and J. Zobel. Compression of Inverted Indexes for Fast QueryEvaluation, In Proceedings of the 25th Annual International ACM SIGIR Conference on Researchand Development in Information Retrieval, pp222-229, 2002).

  One of the important issues in document retrieval is a differential update technique that, when a new document is given, updates the transposed index to reflect the new document and makes it possible to retrieve the new document. As for differential update technology, technologies mainly for updating inverted indexes on disk have been developed (J. Zobel and A. Moffat, Inverted files for text search engines, ACM Comput. Surv., 38 (2 ), 2006). However, in recent years, the main memory installed in computers has increased, and depending on the search target document set, it is possible to distribute inverted indexes to multiple computers and handle large-scale inverted indexes on-memory. It was. Even if an inverted index is placed on the disk, the updated part is recorded in the main memory as long as the capacity permits, and writing to the disk is delayed until the memory runs out, thereby reducing the number of disk accesses. In many cases, the time required for updating the index is reduced. Therefore, development of an on-memory inverted index difference update technique is extremely important in providing a document search system with a quick response. In the differential update technique for the inverted index on the disk, storing the inverted list in a highly continuous area is a major issue in order to shorten the seek time during search. On the other hand, when an on-memory inverted index is targeted, continuity may be sacrificed to some extent. However, since the memory has a smaller capacity than the disk, it is necessary to suppress the amount of additional data such as a pointer added to the permutation list itself.

  When a static document set that does not need to be updated is a search target, a two-pass method in which the original document set is read twice is effective as an index construction method that suppresses the size of the inverted index to the minimum necessary level. . In the two-pass method, when the document set is first read, the size of an area necessary for storing the transposed list of each index word is calculated. Based on this result, after securing a necessary memory area, the document set is read again to actually create a transposed list. In the two-pass method, an inverted list that does not waste memory can be constructed, but the cost of reading a document set twice is high. Moreover, the document set needs to be given in advance, and addition of a new document is impossible unless the entire index is reconstructed.

  As shown in FIG. 2, the most basic data structure for realizing the difference update of the transposed index on-memory is a list structure in which a pointer 201 indicating the position of the next posting is given to the posting 107. By adopting a list structure, the transposed list of each index word can be expanded to the maximum length of memory whenever necessary. However, if both the integer value and the pointer are expressed by the same number of bits, if the posting is composed of two integers of the document identification number and the appearance frequency of the index word in the document, the pointer occupying the entire data structure The ratio will be 33% or more. Therefore, the amount of memory consumed in the list structure reaches 1.5 times the original data amount, and the overhead due to storing the pointer is very large.

  If the transposed list is arranged in a continuous memory area, a pointer for connecting postings becomes unnecessary. At this time, a technique is known in which a large amount of memory is allocated so that an empty area remains at the end of each transposed list, and a new posting can be added. There has also been proposed a method for predicting the degree of expansion of the future permutation list from the past update status and acquiring more necessary memory (Non-Patent Document 1).

  On the other hand, Buttcher et al. Proposed a method of reducing pointer overhead based on a transposed list having a list structure (Non-Patent Document 2). In the method of Buttcher et al., A plurality of postings are stored in a continuous memory area, and these areas are connected by pointers, thereby reducing overhead caused by pointers per posting. Further, the index word transposed list to which postings are frequently added reflects the fact that there will be many additions in the future, and the size of the newly acquired memory area is exponentially increased. However, in consideration of the possibility that a wasteful area increases if a too large memory is secured, an upper limit is set on the size of the memory area to be acquired. By using these techniques together, it has been experimentally confirmed that the Buttcher et al. Method can suppress an increase in data amount to 5% or less compared to the 2-pass method.

W.-Y. Shieh, C.-P. Chung, A statistics-based approach to incrementally update inverted files, Inf.Processing and Management, 41 (2): 275-288, 2005. S. Buttcher and C.L.A.Clark, Memory management strategies for single-pass index construction in text retrieval systems, University of Waterloo Technical Report CS-2005032 / Wumpus Technical Report 2005-02, 2005.

  The data structure of the inverted index must be able to be updated in a short time no matter what document is given while minimizing the amount of additional data excluding the inverted list itself. In the method of Non-Patent Document 1, if the number of newly added documents is large, eventually the acquired memory is insufficient and it is necessary to secure a new memory. Therefore, the timing for updating the data structure can be delayed by acquiring more memory, but a data structure that can expand the transposed list to an arbitrary length as long as the memory capacity permits is eventually necessary. In the method of Non-Patent Document 2, despite the relatively simple data structure, an inverted index with sufficiently low overhead is realized compared to the two-pass method. However, since the top part of the inverted list is divided into a large number of memory areas having different sizes, the speed at the time of retrieval is reduced, and the access method to the inverted list is complicated, resulting in an increase in processing time. There is also room to reduce the memory required for the pointer.

  An object of the present invention is to provide a data structure of a transposed index that is compact and can handle any new document addition request for high-speed document search and transposed index update.

The present invention prepares three types of memory areas for storing an inverted list, and uses them differently for each length of the inverted list, thereby reducing the memory overhead caused by pointers and preventing segmentation of the memory area. The data structure is provided (FIG. 3). These three types of memory areas are referred to as type 1, type 2, and type 3 in this specification. In this specification, unless otherwise specified, the size of the area is expressed in bytes. Less than,
b is a user parameter that is an integer representing the size of a fixed-size memory area allocated for each index word;
B is a user parameter that is an integer representing the size of the smallest memory area that is additionally allocated for each index word,
r is a user parameter that is an integer that adjusts the size of the newly allocated memory area when the additional allocated memory area is insufficient,
M is a user parameter that is an integer representing the size of the maximum memory area that is additionally allocated for each index word and guarantees continuity,
W is the size of the area required to represent the location of the acquired memory area,
W ′ is the size of the area necessary to represent the bit length of the data in the acquired memory area,
N is the smallest integer such that B * r ^N ≧ M,
[x] is the smallest integer not less than x,
Let c (x) = [(x + 7) / 8]. Note that c (x) is the size in bytes of an area necessary for storing a bit string of length x.

  In this specification, the base of the logarithm log is 2. Furthermore, b <B <M is assumed to ensure that the size of the newly allocated memory area is larger than the original memory area.

  The memory areas of types 1 to 3 are selectively used as follows depending on the length λ of the permutation list to be stored in bit units.

(1) Type 1 memory area
Used to store a transposed list such that c (λ) ≦ W. In the prior art, the transposed list is stored in an area pointed to by a pointer prepared for each index word. In the present invention, a type 1 memory area is secured in the area 301 for storing the pointer. This allows a short transposed list to be stored without using any pointers. The type 1 memory area 301 is useful for very rare index terms that appear only once or twice in the document set. This is because if the pointer is used in the transposed list of c (λ) ≦ W, the pointer occupies more than 50% by using only one pointer.

(2) Type 2 memory area
Used to store an inverted list of W <c (λ) ≦ B * r ^N−1 . The type 2 memory area 302 is secured in an area pointed to by a pointer prepared for each index word. For the transposed list of length λ, a memory area having a size of B * r ^n-1 is secured and the transposed list is recorded. Here, n is a minimum integer satisfying c (λ) ≦ B * r ⁿ⁻¹ .

(3) Type 3 memory area
Used to store a transposed list of c (λ)> B * r ^N-1 . The type 3 memory area has a list structure in which a plurality of memory areas 303 of size M (hereinafter referred to as segments) are connected by pointers. The i-th segment in this list is denoted as A (i) below. Let M (i) be the size of the area available for recording the transposed list in A (i), and when k ≧ 1, M ′ (k) = M (1) + ... + M (k ). Further, it is defined that M ′ (0) = 0. With this definition, M ′ (k) is the number of bytes of the longest transposed list that can be recorded in a type 3 memory area consisting of k segments. The specific value of M (i) depends on the implementation of the method of the present invention, an example of which will be described later. In order to record pointers and other information for configuring the list structure, the number of bytes of the transposed list that can be recorded in each segment is smaller than M, and M (i) <M.

  The number of segments reserved for the transposed list of length λ is the smallest integer k that satisfies c (λ) ≦ M ′ (k). The type 3 memory area has a list structure and requires a pointer, but if M is made large and M (i) is made sufficiently large, the percentage of the pointer will occupy compared to the case where a pointer is prepared for each posting. Since the frequency of pointer access when reading the transposed list is relatively small, the access time and memory consumption caused by the pointer can be suppressed. Moreover, since the size of the secured segment is constant at M, mounting is easy.

  If r is an integer and M is a multiple of B, the size of each successive memory area is always an integer multiple of B in the type 2-3 memory areas. Therefore, by acquiring an array containing a large number of B-byte memory areas in advance and using an integer indicating the number of the area in place of the pointer, the number of bits can be reduced compared to using the pointer directly. The location of the memory area can be specified, and W may be reduced.

  According to the present invention, there is provided a data structure of an inverted index in which posting is recorded in a highly continuous memory area with little memory overhead caused by a pointer and a new document can be added.

  An embodiment of an apparatus for creating and updating an inverted index provided by the present invention will be described. An overview of the entire device is shown in FIG. A memory (main memory) 402 is connected to the central processing unit (CPU) 401. An auxiliary storage 403, a rewritable removable medium 404 such as a DVD-RAM, a network 405, and a user terminal 406 are connected as necessary. The apparatus according to the present embodiment includes an index word extraction unit S501, an index word appearance number acquisition unit S502, a document identification number assignment unit S503, a posting compression unit S504, and a transposed index update unit S505 as programs on the memory 402 executed by the CPU 401. Is provided.

  The flow of data and processing in this apparatus is shown in FIG. First, a new search target document 104 is acquired as input from the memory 402, auxiliary storage 403, removable media 404, or network 405 as necessary. Two types of processing are performed on the document 104. First, the index word extraction means S501 cuts out the index word 103 in the document 104. As described above, a known morphological analysis method or n-gram can be used as the extraction means (information search algorithm, Kitakenji et al., Kyoritsu Shuppan). Further, the number of times each index word appears in the document 104 is obtained by the appearance number obtaining means S502. To obtain the number of appearances, simply prepare one integer variable for each index word and initialize it to 0, then increment it by 1 each time the occurrence of the index word is detected in the document 104. That's fine. Further, the document identification number assigning unit S503 assigns a document identification number to the document 104.

  One preferred embodiment of this means keeps one integer variable initialized to 1, assigning the value of this variable when a new document identification number is needed, and immediately following that variable's value of 1 It is to increase. Through S501, S502, and S503, a posting 107 including the document identification number 105 and the index word appearance count 106 is obtained for each index word. This posting is preferably compressed using the posting compression means S504. The compressed posting 506 is added to the inverted list corresponding to the index word in the inverted index 407 constructed in the memory 402. The position of the transposed list corresponding to the index word is recorded in the pointer for each index word. If necessary, the transposed index constructed on the memory 402 may be transferred to the auxiliary storage 403, the removable medium 404, and the network 405 for storage. This process is repeated each time a new search target document 104 is given.

  An example of the inverted index according to the method of the present invention will be described with reference to FIG. This data structure includes a basic area 601, an extended area 602, and a free area list recording area 603. The basic area is a variable length array that can store a plurality of memory areas of size b. Similarly, the extension area is a variable length array capable of storing a plurality of memory areas of size B. An example of an embodiment of a variable length array will be described later. The empty area list recording area 603 is a fixed-length array composed of N elements 605 having a size W. The nth element empty [n] 605 in this area is initialized to a constant NULL. The definition of NULL will be described later. The individual memory areas of size b in the basic area 601 are hereinafter referred to as basic slots. Similarly, each memory area 604 of size B in the expansion area 602 is hereinafter referred to as a slot. Let k be the slot number of the kth slot in the extension area. When a plurality of consecutive slots are used as a continuous memory area, the slot number of the first slot is set as the slot number of this memory area.

In order to prevent the newly acquired memory area from being an integral multiple of the original and from rapidly increasing in size, r = 2 is assumed. The size of all the memory area secured in the extended region 602 is B * r ⁿ so that it can (n is an integer) representation, to secure the M to M = B * r ^N. Furthermore, the longest transposed list that can be stored in the type 2 memory area is longer than the type 1 memory area, and the longest transposed list that can always be stored in the type 3 memory area is longer than the type 2 memory area. In order to guarantee, it is assumed that parameters satisfying b ≦ B ≦ B * r ^N−1 ≦ M (1) are given.

From the basic area, one basic slot is assigned to each index word. The information recorded in the basic slot is as follows.
(1) Which of the types 1 to 3 is the memory area in which the transposed list of the index word is recorded.
(2) -1 For Type 1, the bit length λ of the inverted list and the inverted list itself.
(2) -2 For type 2, the bit length λ of the transposed list and the pointer to the memory area.
(2) -3 For Type 3, pointer to the first element of the memory area list.

  The data structure of the basic slot will be described with reference to FIG. The basic slot 701 includes three partial regions R 702, L 703, and C 704.

● Partial area R 702
Record the integer n. The meaning of this integer n is as follows.
n = 0: This transposed list of index words is stored in the type 1 memory area.
1 ≦ n ≦ N: This index word transposition list is stored in a type 2 memory area.
The size of the memory area is B * r ^n-1 .
n = N + 1: This transposed list of index words is stored in a type 3 memory area.
● Partial area L 703
When the type 1 or 2 memory area is used, the bit length of the transposed list is recorded. In the case of type 3, the partial area L has no meaning.
● Partial area C 704
When a type 1 memory area is used, the transposition list itself is recorded. In the case of type 2, the slot number of the memory area in which the transposed list is recorded is recorded. In the case of type 3, the slot number of the first segment A (1) in the list of the memory area where the transposed list is recorded is recorded.

  Hereinafter, the partial area y in the memory area x is denoted as x {y}.

Next, the data structure of the memory area acquired on the slot of the extension area 602 will be described.
(2) Type 2 memory area (801) The entire memory area is used to store a transposed list (FIG. 8).
(2) Type 3 memory area (901, 902 or 903) The data structure will be described with reference to FIG.

● Partial area P3 904
The slot number of the next segment in the list structure shown in FIG. 9 is recorded. However, if the segment including the partial area P3 is the last segment in the list structure, a special numerical value NULL indicating the end is recorded. Two preferred examples are shown for selecting the special numeric value NULL.
1. Slot numbers are counted in order from 1, and NULL is set to 0. The slot number is counted from 0, and the slot 0 is not used, so that NULL may be set to 0.
2. The maximum numeric value that can be recorded in W bytes is NULL.
In FIGS. 6, 9, and 11, the constant NULL is indicated by a hatched line 606.
● Partial area L3 905
The bit length of the transposed list recorded in the segment including the partial area L3 is recorded.
● Partial area P'3 906
Only the first segment A (1) 901 of the list structure is provided with a partial region P′3 906. In this partial area, the slot number of the last segment of the list structure is recorded. With this information, the end of the inverted list can be accessed within a certain time regardless of the length of the inverted list, and a new element can be quickly added to the inverted list.
● Partial area C3 907
Stores the transpose list.

  In the above data structure, the sizes of the partial areas P3, L3, and P'3 are preferably W, W ', and W, respectively. In this case, M (1) = M− (2W + W ′) and M (i) = M− (W + W ′) (when i ≧ 2).

The selection criteria for the value of each parameter and the size of each partial area will be described. First, W may be the number of bytes necessary to represent the slot number. For 64-bit architectures, the maximum is W = 8 bytes, but if you know that the slot number is n at most, including NULL, W = c (log (n) +1) bytes is sufficient. So that we can build the inverted index of megabytes class at a minimum, n is preferably made larger than 2 ^20. Therefore, W is preferably at least 3 bytes. On the other hand, W ′ may be 8 bytes, but the size of the memory areas and segments of type 1 and 2 can be suppressed by M, so W ′ = c (log (8 * M) +1) bytes is sufficient. . Even if W <= 1 byte, if M <32 bytes, the length of the transpose list in the segment can be expressed, but in this case, W + W 'is at least 4 bytes and exceeds 10% of M. This is insufficient as a solution to the original problem of suppressing the additional data amount other than the transposed list. If W ′ = 2, M <2 ¹³ = less than 8192 suffices, so M can be increased to reduce the proportion of additional data. As described above, r = 2 should be satisfied. Since the basic slot 701 includes a partial area C 704 and the size of the partial area C 704 is greater than or equal to W, b ≧ W ≧ 3. In order to use shift and logical operations for address calculation and to speed up the calculation, the basic slot size b is preferably a power of 2, but this is possible because the basic slot is assigned to all index words. It needs to be as small as possible.

Accordingly, the value of b is preferably 4 or more in consideration of W being 3 or more. B is also preferably a power of 2 like b, and in order to make the type 2 memory area larger than type 1, b <B, and the type 3 memory area is made larger than the type 2 memory area. Therefore, it is preferable to satisfy B <M, respectively. From b <B, B is preferably 4 * 2 = 8 or more. On the other hand, since M = B * r ^N and r = 2, M is also a power of 2. Therefore, it is preferable that M ≦ 4096 = 8192/2. For this reason, it is preferable that B ≦ 2048 = 4096/2. Further, from M> B, it is preferable that M ≧ 16 = 8 * 2. Since the value to be recorded in the partial area R 703 is log (M / B) ≦ log (8192/8) = 10, an integer equal to or less than 10 + 1 = 11 may be used. The length is sufficient if it is 4 bits that can represent an integer from 0 to 11, and a smaller number of bits may be used depending on the selection of M and B. The partial area L 704 may be [log | W | +1] bits or more.

  Next, a method for creating a transposed list using this data structure will be described in detail.

1. When adding a new posting having a bit length λ at the beginning of an index word, one new basic slot a 701 is acquired from the top of the unused portion of the basic area. Based on λ, the type of the memory area is selected from types 1 to 3.
1-1. If c (λ) ≦ W, type 1 is selected as described above. In this case, in the basic slot a, 0 is recorded in a {R}, λ is recorded in a {L}, and the posting is recorded in a {C}.
1-2. If W <c (λ) ≦ B * r ^N−1 , type 2 is selected as described above. In this case, for the smallest integer n satisfying B * r ^n-1 ≧ c (λ), r ^n-1 consecutive slots are acquired from the extension region. The posting is recorded in the r ^n-1 consecutive slots. Then, n is recorded in a {R}, λ is recorded in a {L}, and the first slot number of the continuous slot is recorded in a {C}.
1-3. If B * r ^N-1 <c (λ), type 3 is selected as described above. In this case, first, r ^N consecutive slots are acquired from the extension area and used as A (1). The size of this area is M = B * r ^N bytes. Record the slot number of N + 1 in a {R} and the slot number of A (1) in a {C}. Further, any of the following 1-3-1 or 1-3-2 is performed by λ.

1-3-1. When c (λ) ≤ M (1)
Record NULL in A (1) {P3}, λ in A (1) {L3}, NULL in A (1) {P'3}, and pad the A (1) {C3} from the beginning. Record.

1-3-2. When c (λ)> M (1)
Let k be the smallest integer that satisfies c (λ) ≦ M ′ (k). The variable i is set to 2 and the following processes 1-32-1 to 4 are performed while i ≦ k.
1-3-2-1. Acquire new r ^N consecutive slots from the extension area, and call it A (i).
1-3-2-2. A (i-1) {P3} is the slot number of A (i), A (i-1) {L3} is M (i-1) * 8, A (i -1) Record from the M ′ (i−2) +1 byte to the M ′ (i−1) byte of the posting in {C3}.
1-3-2-3. Add 1 to i.
1-3-2-4. If i ≦ k, repeat the process from 1-3-2-1.
When the processing of 1-3-2-1 to 4 is completed, record NULL in A (k) {P3} and integer λ-M '(k-1) * 8 in A (k) {L3} Further, A (k−1) {C3} is recorded with the M ′ (k−1) +1 byte and subsequent bytes of the posting packed from the beginning. Then, the slot number of A (k) is recorded in A (1) {P'3}.

  Next, a method for adding a new posting to an existing transposed list of an index word will be described. Assume that the bit length of the existing transposed list is λ, and the bit length of the new posting is λ ′.

2. Refer to the basic slot a ′ corresponding to the index word, and acquire the type of the memory area that stores the existing transposed list based on a ′ {R}.
2-1. When a ′ {R} = 0, that is, when the existing inverted list is type 1, the length λ of the existing inverted list is acquired from a ′ {L}.
2-1-1. When c (λ + λ ′) ≦ W, the updated transposed list is also stored in the type 1 memory area. Simply add a new posting to the end of the existing transpose list and replace a '{L} with λ + λ'.
2-1-2. When W <c (λ + λ ′) ≦ B * r ^N−1 , the updated transposed list is stored in the type 2 memory area. First, rn ^-1 consecutive slots are acquired from the extension area. However, n is the smallest integer that satisfies B * r ⁿ⁻¹ ≧ c (λ + λ ′). The existing transposed list is copied into the r ^n-1 consecutive slots from the head, and a new posting is recorded immediately after that. Then, n is recorded in a ′ {R}, λ + λ ′ is recorded in a ′ {L}, and the slot number of the newly acquired memory area is recorded in a ′ {C}.
2-1-3. When B * r ^N-1 <c (λ + λ ′), the updated transposed list is stored in a type 3 memory area. First, “new posting with bit length λ” is read as “existing transposed list with bit length λ”, and “a” is read as “a ′”, and processing 1-3 is executed. At this time, let A (k) be the last acquired segment. Hereinafter, the integer k is used for convenience of explanation, but it is sufficient that the slot number of the memory area A (k) is known even when the value of k is unknown in the implementation. Next, the following process 2-1-3 ′ is executed.
2-1-3 '. If A (k) {L3} + λ'≤M (k), add a new posting at the end of A (k) {C3} and add λ to A (k) {L3}. Adding 'ends the addition process. When A (k) {L3} + λ ′> M (k), first, M (k) * 8−A (k) {L3} is set to the variable D. At this time, D is the bit length of the transposed list that can be added to A (k). Next, the portion from the beginning of the new posting to D bits is added to A (k) {C3}, and M (k) * 8 is recorded to A (k) {L3}. Further, while D <λ ′ is satisfied, the following processes 2-1-3-1 to 7 are repeated.
2-1-3-1. R ^N consecutive slots are newly acquired from the extension area and set to A (k + 1).
2-1-3-2. Record the slot number of A (k + 1) in A (k) {P3}.
2-1-3-3. Set the variable d to the smaller of D + M (i) * 8 and λ ′.
2-1-3-4. Record the D + 1th to dth bits of the new posting in A (k + 1) {C3} from the beginning.
2-1-3-5. Record dD in A (k + 1) {L3}.
2-1-3-6. Replace k with k + 1 and D with d.
2-1-3-7. If D <λ ′ is satisfied, the processing from 2-1-3-1 is performed again.

Let A (k ') be the last acquired segment. NULL is recorded in A (k ′) {P3}, and the slot number of A (k ′) is recorded in A (1) {P′3}. The slot number of A (1) can be obtained by referring to a ′ {C}.
2-2. If 1 ≦ a ′ {R} ≦ N, that is, if the existing transposed list is recorded in the type 2 memory area, B * r ⁿ for the integer n written in a ′ {R} ^{If -1} ≧ c (λ + λ '), add a new posting immediately after the existing transposed list in the memory area where the existing transposed list is written, and add λ' to a '{L} The process ends. If B * r ^n-1 <c (λ + λ ′), first, the same processing as in 2-1 is performed to record the existing transposed list and the new posting in a new memory area. Then, the memory area in which the existing transpose list has been recorded is discarded and made reusable. A method for discarding and reusing the memory area will be described later.
2-3. If a '{R}> B * r ^N-1 , that is, if an existing transposed list is recorded in the type 3 memory area, it will be at the slot position written in a' {C}. For a segment A (1), refer to A (1) {P3}. If A (1) {P3} is NULL, A (1) is the last segment in the list structure. If A (1) {P3} is not NULL, the segment pointed to by the slot number written in A (1) {P3} is the last segment. For convenience of explanation, assume that this last segment is kth in the list. For implementation, it is only necessary to know the slot number of the last segment, and there is no need to obtain the value of k. At this stage, if the process 2-1-3 ′ is executed, a new posting can be added.

If a new posting is added many times according to the above procedure, the first stored posting is copied many times from the type 1 or 2 memory area to the type 2 or 3 memory area. However, the average number of copies m of each byte included in one transposition list is limited as follows. However, n is the maximum integer n such that c (λ)> B * r ^{n−1 with} respect to the length λ of the transposed list.

m ≦ (b + B + B * r + B * r ² + B * r ³ +… + B * r ^n-1 ) / c (λ)
≦ (b + B + B * r + B * r ² + B * r ³ +… + B * r ^n-1 ) / B * r ^n-1
= B / (B * r ^n-1 ) + (r- ^(n-1) + r- ^(n-2) + ... + r ^-2 + r ^-1 +1)
≦ 1 + r / (r-1) = 2 + 1 / (r-1) ≦ 3
Therefore, the time cost of the copy process is not large.

A method of discarding the memory area and making it reusable will be described. In the method described here, the discarded memory area is managed by the list structure shown in FIG. N lists are prepared for each size of the memory area to be discarded, and the nth list manages an empty area of size B * r ^n-1 . At the time of reuse, a memory area is allocated from the top of the list having an appropriate size. The empty area list recording area 603 includes N elements W 605 as described above. In the nth (n = 1, 2,..., N) element 605, the slot number of the first free area in the list connecting the free areas of size B * r ⁿ⁻¹ is recorded. Each element 605 is initialized to NULL at the start of processing. When a memory area A having a size B * r ^n-1 is discarded, the value written in the nth element 605 so far is copied to the first W byte portion 1102 and the nth element 605 is copied. In the field, the slot number of the newly discarded area A is recorded. The value copied at this time is NULL if A is the first memory area in the list. On the other hand, when adding a posting, if a memory area of r ^n-1 slots is required, first try to get from the nth free area list, and if it fails, it will be a new unused one Assign a slot. More specifically, it is as described in (1) and (2) below.

(1) When the n-th element 605 of the free area list recording area 603 is not NULL, the memory A having the slot number written in the n-th element 605 is allocated, and the allocation is made to the n-th element 605. The slot number written in the first W byte portion 1102 of the memory A is copied.

(2) When the n-th element 605 of the free area list recording area 603 is NULL, continuous r ^n-1 slots are allocated from the head of the unused part of the extension area.

  In addition to this method, any known garbage collection method may be used as a method for reusing discarded memory.

  In addition, as a form of implementation of the variable length array, when there is insufficient capacity in the acquired memory, a new memory area having a size obtained by multiplying the size of the existing area by the ratio P is acquired. A common method is to copy all the elements of an array. However, in the method of the present invention, the entire variable length array need not be a continuous memory area. Therefore, the variable length array is divided into a plurality of blocks 1002, and pointers to the respective blocks 1002 are stored in another variable length array 1001 and accessed through these pointers (FIG. 10). If new continuous slots are required, allocation is attempted from the existing block. However, there are cases where the designated number of continuous slots cannot be secured due to lack of empty slots or slot fragmentation. In this case, a new block is added to the variable length array 1003, and a designated number of consecutive slots are allocated from the new block. The variable length array 1001 of pointers is preferably secured in a continuous memory area so that any element of the variable length array can be accessed in a fixed time regardless of the length of the variable length array 1003. For this reason, it is preferable to implement the pointer array 1001 by copying all the elements of the existing array to a new memory area.

  The sizes of the regions P, C, P3, L3, and P′3 are W or W ′ bytes, but they do not have to be aligned with byte boundaries, and may be truncated in units of bits as necessary.

  In addition, as a compression method for posting, a variable byte method in which individual posting boundaries are aligned with byte boundaries (F. Scholer, HE Williams, J. Zobel, Compression of Inverted Indexes for Fast Query Evaluation, Proc. 25th Ann. Int'l ACM When using the method of SIGIR Conf. Research and Development in Information Retrieval, pp. 222-229, 2002), the length of the transposed list can be expressed in bytes instead of bits. Sometimes it can be shortened.

  The data structure of the present invention can also be applied when managing a plurality of variable length byte sequences, such as a hash table.

  According to the present invention, a data structure of an inverted index capable of sequentially adding search target documents is provided. Since this data structure has a small number of pointers, the memory overhead is small, and a short transposition list of a certain length or less is placed in a continuous memory area, so that it can be accessed at high speed and is easy to implement.

FIG. 3 is a schematic diagram of a document set to be searched and a transposed index constructed based on the set. Explanatory drawing of the transposed list comprised by connecting posting by list structure. The figure which shows three types of memory areas allocated for the transposition list. 1 is a schematic diagram showing one form of an apparatus for carrying out the present invention. The figure which shows the example of the flow of the data by this invention, and a process. Schematic which shows the example of the data structure of the whole transposed index. The figure which shows the data structure of a basic slot. The figure which shows the data structure of the memory area of type 2. The figure which shows the data structure of the memory area of type 3. Schematic of variable length sequence. Explanatory drawing of the list structure which manages an empty slot.

Explanation of symbols

101: Set of search target documents
102: Inverted index
103: Index terms
104: Search target document
105: Document identification number
106: Number of times an index word appears in a document
107: Posting
108: Pointer to the inverted list assigned to each index word
109: Transpose list
301: Type 1 memory area
302: Type 2 memory area
303: One element of a type 3 memory area, that is, a segment
401: Central processing unit (CPU)
402: Memory (main storage)
403: Auxiliary storage device
404: Removable media
405: Network
406: User terminal
407: Inverted index that can add documents sequentially
601: Basic area in the data structure of the inverted index of the present invention
602: Extended area in data structure of inverted index of the present invention
603: Free area list recording area in the data structure of the inverted index of the present invention
604: Slot in extended area
605: One element of free area list recording area
606: Diagonal line representing the constant NULL
701: Basic slot
702: Area for storing the type of memory area and the size of the memory area acquired in the case of type 2 in the basic slot
703: Area for storing the length of the area occupied by the transposed list in the type 1 or type 2 memory area in the basic slot
703: Record the transposition list when using a type 1 memory area in the area corresponding to one index word in the basic area, and the location of the memory area when using a type 2 or 3 memory area Area to do
801: Type 2 memory area for storing transposition list
802: Basic slot corresponding to type 2 memory area
901: The first segment of the list structure that constitutes a type 3 memory area
902: A segment other than the first and last segments of the list structure that constitutes a type 3 memory area
903: Last segment of list structure that constitutes a type 3 memory area
908: Basic slot corresponding to type 3 memory area
1001: An array that stores pointers to individual blocks in a variable-length array
1002: One of the blocks to store the actual data in the variable length array
1003: One form of variable length sequence used in the present invention
1101: Memory area discarded and unused
1102: An area of size W that stores the slot number of the next memory area in the list structure that connects the memory areas that are discarded and become unused areas

Claims (8)

For all index words of a document set consisting of a plurality of electronic documents each assigned an identification number, the identification number of the electronic document in which the index word appears for each index word and the appearance of the index word in the electronic document In a program for causing a computer to execute a process of storing an inverted index storing an inverted list in which postings that are sets of times are stored in a computer-readable recording medium,
Using a first type of memory area and a second type of memory area to store the transposition list;
In the case of an inverted list with a data size of W bytes or less, storing the inverted list in the first type memory area as a continuous memory area;
When the data size is larger than W bytes, the transposed list is stored in the second type memory area in which one or more continuous memory areas are connected by a list structure, and is combined by the list structure. Recording the position of the last memory area in the first memory area, and further storing a pointer to the first area of the second type memory area in the first type memory area;
A program that causes a computer to execute. For all index words of a document set consisting of a plurality of electronic documents each assigned an identification number, the identification number of the electronic document in which the index word appears for each index word and the appearance of the index word in the electronic document In a program for causing a computer to execute a process of storing an inverted index storing an inverted list in which postings that are sets of times are stored in a computer-readable recording medium,
Using a first type of memory area, a second type of memory area, and a third type of memory area to store the transposition list;
In the case of an inverted list with a data size of W bytes or less, storing the inverted list in the first type memory area as a continuous memory area;
When the data size is larger than W bytes and equal to or less than M bytes corresponding to the value M specified by the user, the transposition list is stored in the second type memory area as a continuous memory area, and the first Storing a pointer to the first area of the two types of memory areas in the first type of memory area;
When the data size is larger than M bytes, the transposed list is stored in the third type memory area in which one or more continuous memory areas are connected by a list structure, and is combined by the list structure. Recording the position of the last memory area in the first memory area, and further storing a pointer to the first area of the third type memory area in the first type memory area;
A program that causes a computer to execute. 3. The program according to claim 2 , wherein the size of the second type memory area and the size of each memory element of the third type memory area are B bytes corresponding to a specified integer B. A program characterized by securing a new memory area by acquiring one or more elements from a variable-length array consisting of memory areas of size B, which is an integer multiple. The program according to claim 2 or 3, wherein the value of M is a power of 2 and is not less than 16 and not more than 4096. The program according to claim 3, wherein the value of B is a power of 2 and is not less than 8 and not more than 2048. The program according to claim 5, wherein the value of M is a power of 2 and is not less than 16 and not more than 4096. For all index words of a document set consisting of a plurality of electronic documents each assigned an identification number, the identification number of the electronic document in which the index word appears for each index word and the appearance of the index word in the electronic document In a method of storing an inverted index storing an inverted list in which postings that are sets of times are stored in a computer-readable recording medium through information processing by a computer,
The computer uses a first type of memory area and a second type of memory area to store the transposition list,
When the data size is a transposed list of W bytes or less, the computer stores the transposed list in the first type memory area as a continuous memory area;
When the data size is larger than W bytes, the computer stores the transposed list in the second type memory area in which one or more continuous memory areas are connected by a list structure, and is combined in the list structure. Recording the position of the last memory area in the first memory area, and storing a pointer to the first area of the second type memory area in the first type memory area;
A transposed index storage method characterized by comprising: For all index words of a document set consisting of a plurality of electronic documents each assigned an identification number, the identification number of the electronic document in which the index word appears for each index word and the appearance of the index word in the electronic document In a method of storing an inverted index storing an inverted list in which postings that are sets of times are stored in a computer-readable recording medium through information processing by a computer,
The computer uses a first type of memory area, a second type of memory area, and a third type of memory area to store the transposition list,
When the data size is a transposed list of W bytes or less, the computer stores the transposed list in the first type memory area as a continuous memory area;
When the data size is larger than W bytes and equal to or less than M bytes corresponding to the value M specified by the user, the computer stores the transposed list in the second type memory area as a continuous memory area, and Storing a pointer to the first area of the second type memory area in the first type memory area;
When the data size is larger than M bytes, the computer stores the transposed list in the third type memory area in which one or more consecutive memory areas are connected by a list structure, and is combined in the list structure. Recording the position of the last memory area in the first memory area, and further storing a pointer to the first area of the third type memory area in the first type memory area;
A transposed index storage method characterized by comprising:

Images