NO329298B1 - Full-text searchable index for effective history search in frequently changing document collections - Google Patents

Abstract

A text index includes a hierarchy of inverted subindexes and allows time-based searching in a document collection.

Description

FIELD OF THE INVENTION

The invention relates to an index for full-text searches that enables efficient history searches in document collections that frequently change. In particular, the present invention relates to an index with a time-based structure so that a document collection can be searched at any particular time in its history as given by a time granularity.

PRIOR ART

Time-based databases have been the subject of research in the field of information science and technology for several years, with the main emphasis on indexing and querying of structured data.

"Snapshof isolation is used in databases to allow the execution of search queries in a limited and recent history of updates to the database. International patent application No. PCT/NO2008/000445 shows, among other things, snapshot isolation methods and their implementation for unstructured text indexing and text searching.

An article by Klaus Berberich, Srikanta Bedathur, Thomas Neumann, Gerhard Weikum, "A time machine for text search", Proceedings of the 30th annual international ACM SIGR conference on research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands r http:// Dortal. acc. org/citation. cfm?id= 1277741. 1277831 & coll=ACM& dl=AC M& CFID=14908098& CFTOKEN=513613851

shows a method for indexing and searching a text collection for the purpose of reducing the granularity of time specification in search, so that the size of the index can be reduced. Unfortunately, the reduction in time granularity also reduces the precision of the search queries.

An article by K. Nørvåg and A. O. N. Nybø. "DyST: Dynamic and Scalable Temporal Text Indexing" shows another attempt to implement text indexing in the time domain

r http:// ieeexplore. eeee. org/ xpls/ abs all. isp?arnumber=1636000]

A regular, inverted index can perform lookups for a search term in two disk accesses (one to read the lexicon entry; one to read the entry list). Time-based indexes have a much larger administration lookup time. At lookup time, existing time-based index structures either have to perform many disk accesses for each search term or (in the case of document collections that change frequently) process very large volumes of data due to all the changes in the time interval being queried.

In both cases, an unwanted large administration can arise.

OBJECTS OF THE INVENTION

In light of the shortcomings of the current time-based indexes, it is a main purpose of the present invention to provide an index that increases the access rate for search queries and at the same time limits the volume of data that must be processed for each search query.

Another purpose is to provide an index that allows text searching in a document collection as it exists at any particular time.

Finally, one purpose is to allow efficient history searching in document collections that change frequently.

SUMMARY OF THE INVENTION

The above purposes as well as further features and advantages are realized according to the present inventions with an index comprising a hierarchy of inverted indexes with varying time granularity, each level of the hierarchy comprising a number of partitions identified by a level number and time stamp after the last addition of a document, where each partition is itself an augmented, inverted index containing all additions and all deletions of documents in the interval from the end of the preceding partition until markup, and where the partition contains delta lists for each symbol of a document.

Further features and benefits will appear from the attached independent requirements.

The invention will be better understood by reading the detailed discussion of the construction and use of the index according to the present invention and read in conjunction with the attached drawing, in which fig. 1 shows the layout and structure of an inverted index as known in the art,

fig. 2 is a general representation of the index structure according to the present invention, and

fig. 3 the indexing and encoding of a list of symbols for deleted documents.

DEFINITIONS AND BACKGROUND

Some concepts that are central to the present invention will now be defined.

A document is the unit that is indexed. A document can be a text file, a database record, or any logical unit of data.

A document identifier (docid) is an integer that uniquely identifies the document.

A document consists of a sequence of symbols. Symbols are typically words or numbers, but can also be URLs, ISBN numbers, postcodes, phrases, names, etc.

An inverted index is a well-known index structure used for full-text searching. It consists of two main parts: a posting file and the encyclopedia. The posting file consists of many posting lists, one for each symbol indexed by the index. The posting list is logically a list of occurrences of the symbol. The list is normally sorted by docid. A document level list has an entry for each document in which the symbol occurs. The entry in such a list normally contains docid and the number of times the symbol occurs in the document. A symbol level list includes an entry for each instance of the symbol. The entry in this list contains the docid and the position of the symbol within the document. If there are multiple entries for a document, they are sorted in ascending order. The present invention supports both of these types, but also other variations that contain additional information in each posting entry.

The lexicon is a mapping from the symbol to the offset in the record file for the symbol. It can be implemented by a B-tree or similar index structure.

An inverted symbol level index is shown in fig. 1. The words "the" and "them" are indexed. The word "the" has 45 occurrences and the posting list starts at offset 76416 in the posting file and has a size of 103 bytes. The first occurrence in document 3 is at position 17 and the last in document 231 at position 3. The posting list for the word "them" begins immediately afterwards with its first occurrence in document 4 at position 48.

It is assumed that documents will be added as time goes on. Documents are numbered with a growing document identifier (docid) and each document is identified with a time stamp. The time stamp corresponds to some real time such as the number of days, hours or seconds since the system was installed. It is possible to convert a time stamp directly to and from a real time.

The granularity of the timestamp defines the temporal granularity of the query in the data. If the time granularity is 1 hour, it is only possible to query the index for the state of the document collection at the start or end of an hourly interval.

The deletion of a document is also assigned a time stamp. Document updates are supported by first deleting the document and then inserting the document and assigning it a new identifier.

DETAILED DISCUSSION

Now the structure, layout and function of the index according to the invention will be discussed. It is assumed that documents will be added as time goes on. The documents are numbered with an increasing identifier, and each document is identified with a time stamp. The time stamp corresponds to some real time such as the number of days, hours or seconds since the system was installed. The granularity of the timestamp specifies the temporal granularity of the query in the data. A document deletion is also assigned a timestamp. Document updates are supported by first deleting the document and then inserting the document and assigning it an identifier. The solution proposed according to this invention is to form a hierarchy of inverted indices with varying time granularity. The structure of the index according to the invention is shown in fig. 2, which reproduces three levels in the hierarchy (levels 1 to 3). Level 1 is at the lowest level and the level number is increased by 1 for each level up in the hierarchy. The number of levels will grow as time passes and more documents are inserted. The time is given by time markings and starts at 1 on the left in docid/map. On the right is the current time 41. At each level there is a number of index partitions P numbered as n:m. n is the level and m is the timestamp of the last addition to the partition. A partition contains all document additions and deletions in the interval from the end of the preceding partition until m.

Each partition is a boosted, inverted index. Instead of a simple

posting list for each symbol in the index there is a list of posting lists. Each such sublist contains entries for all additions and all deletions within the time granularity of the partition for that symbol. Such a sublist is called a delta list. In the figure, a delta list is named i-j, where / is the level and j is the time stamp at the start of the documents in the delta list.

An entry in the delta list contains the same values as for a regular posting list. For a document level index, the entry pair contains (docid, position). If the position is 0, the document in which the symbol appeared has been deleted. If there is an entry with position 0, this must be the only entry for this docid in the delta list. When the document is deleted, such a deletion entry must be created for all symbols that occur in the original document. To be able to delete a document, it is therefore necessary to have the original document or to have a list of symbols that appeared in the document.

An alternative way of handling deletions that does not require the original document is to have a special symbol DT that only registers deleted documents. Whenever a document is deleted, the deletion entry is inserted into the posting list for that symbol instead of all the other symbols. This will have a number of advantages:

• Does not require the original document

• A deletion is only registered once (saves space)

• The insertion in all posting lists except the DT posting list will take place on growing document identifiers and make it easier to later merge multiple delta lists. Merging is done during the construction of higher-level partitions and during query processing.

The main disadvantage is that the DT posting list is likely to be large and must always be processed during query processing. For a simple search query with a simple search term that rarely occurs, the posting list for the terms themselves will be small, but it must be possible to compare it to the DT posting list, which will be large. The DT posting list should therefore be cached in memory and indexed for quick look-up. The DT list will only need to record the docid and other fields can be ignored. Both B-tree and skip-lists [Pugh, William (June 1990). "Skip lists: a probabilistic alternative to balanced trees". Communications of the ACM 33 (6): 668-676)] can be used. A simple solution is to register every nth entry in a separate table. If compression is used, the main DT list should have the compression reset for these entries. Fig. 3 illustrates this. In the figure, n = 4. on the left is the original list of deleted documents. On the right, the list is the delta code (an entry is saved as the difference from the previous entry) and every fourth entry (numbers 0, 4, 8 and 12) is registered in the ship index. As shown in the figure, the delta encoding is reset for these entries. A value (eg 601) can be accessed directly by performing a binary or sequential search in the skip index and finding the entry that is less than or equal to that value. The pointer can be followed into the DT list which can be sequentially decoded and checked until the value is greater than or equal to the value being looked for.

An entry list in the augmented inverted index is encoded as a sequence of delta lists. Such coding can be as shown in table 1.

NDELTAS is the number of delta lists in this posting list. DELTA-/' is the /'th delta list. A delta list for a symbol level list can be coded as shown in Table 2.

TIMESTAMP is the time stamp for document insertions and deletions in this delta list. SIZE is an optional field that gives the size of this delta list in bytes. It makes it easier to skip this delta list if the timestamp is outside the time interval of a query. NENTRIES is the number of entries in the delta list, DOCID-/ and POS-/ are the docid and position of the entry in the delta list. NDELETES is the number of deletion entries in this delta list and can be used to calculate more correct relevance figures (necessary for TF/IDF calculations).

Given the format of the posting lists as stated above, the inverted indices discussed here can be constructed by any method described in the literature and will not be discussed further here. This includes techniques for both incremental and batch indexing.

The total index consists of multiple partitions as described above. A partition is either open or closed. An open partition is still being updated, while a closed partition is complete and will no longer change. There is only one open partition on each level.

In fig. 2, only the rightmost partition on each level is open. The partitions 3:1, 2:37 and 1:41 are open, and all the other partitions are closed. New deltas will be added at level 1 as documents are added or deleted. Every time a partition at level L reaches a certain size (number of documents, number of delta lists, number of bytes or time interval), the partition is closed, and a new open partition is created. The size threshold should be different for each level, as higher level partitions should be larger than lower level ones. One scheme could be to let the partitions at one level in the hierarchy be 100 times longer (in time) than in the level below.

When a partition is closed, it is also scanned and its delta lists merged into a single delta list which is then added to the open partition at level L+l. If there are no open partitions at level L+l, a new partition is created at this level. When a partition is closed, it also records which partition the previously merged partition was added to and which partition it will be merged into. These partitions are called F' partition and S partition respectively. In fig. 2 is the P partition of partitions 1:25, 2:13, 1:29 2:25 and 2:25 3:1. The partitions 1:1, 2:1 and 3:1 have no P partitions. The S partition of both partitions 1:25 and 1:29 is 2:25.

During merging, the entries for the document that was first inserted and then deleted within the same partition are removed. Fig. 2 shows how partition 1:13 is merged into delta list 2-13 and 2:13 is merged into delta list 3-13.

A partition will never change after it has been closed and can be frozen. If hierarchical constructions for the already inverted indexes are used, the subpartitions can be merged into a single partition before it is frozen.

docid map is an additional structure needed to effectively support time-based queries. In fig. 2 it is shown as a matrix with one column for each delta list. For each delta list, the timestamp of the first document in the delta list is added to the docid for that document, and the partition at level 1 where the delta list occurs is recorded. As the docids are allocated sequentially, they will increase with increasing timestamps. The docid image can be registered in a B-tree by using a timestamp as a key and will therefore be sorted by increasing timestamp. The entry in the B-tree will be:

The docid mapping is easy to construct, as new entries are only added at the end of the B-tree as new delta lists are formed.

The new index is able to find symbols and obtain a list of ranked documents added in the time interval from S to T and containing these symbols. A common search query is of the form "find the documents where the symbol F occurs at time T'. In this case, the start time S is set as the start time (e.g. 0). For the search query "find documents inserted after S that contain the symbol V" is the time T equal to the time stamp for the last inserted document.

Polling is done by merging delta lists from a partition at each level. The following steps are followed: • Look up the docid mapping and find the entries with the largest timestamp less than or equal to the timestamp S and the largest timestamp less than or equal to the timestamp T.

A partition at each level is selected for the time stamp T:

o The level 1 partition is the partition in the docid image entry for

T.

o The partition at level / (/ > 1) is the P-partition at level i-l.

A partition at each level is selected for the time stamp S:

o The partition at level 1 is the partition the docid image entry for

S.

o The partition at level i (z > 1) is the S-partition at level i-l.

If S is the beginning of time (there is no lower bound), this step can be ignored.

Find the symbol or symbols in each of the selected partitions.

For each symbol:

o Take the delta lists in each partition selected above and select the delta lists that are entirely contained in the time interval between S and T. Depending on application semantics, the level 1 partitions that include the timestamp S and/or T may be included.

o Take the selected delta lists and accumulate the total number of occurrences

of the symbol. This number can be used for relevance calculations.

o Take the selected delta lists and merge them together. The delta lists are sorted by increasing docids and can be merged using a priority queue. If there is a delete entry for a document, that document should be completely ignored.

The merged documents for each symbol can now be used to combine the entire search query including Boolean operators such as AND/OR as well as phrase matching.

If a special symbol is used for deleted documents, the DT symbol may be part of the search query and the documents in the merged list for this must be eliminated from the result.

Fig. 2 illustrates a search query between S=8 and T=30. The partitions selected are 1:5, 1:29, 2:1, 2:25 and 3:1. They include all operations between the time marker S=8 and T=30.1 1:5 delta list 1-8 is selected, in 1:29 delta lists 1-29 and 1-30, in 2:1 delta list 2-9, in 2:25 delta list 2 -25 and in 3:1 the delta list 3-13 is selected. The delta lists 1-8, 1-29, 1-30, 2-9, 2-25 and 3-13 are then merged together.

By forming a hierarchy of changes in the time domain in the index according to the present invention, a document collection that grows by updates can be searched at any date unit in the history.

As mentioned, the inverted indices described herein can be constructed by any method described in the literature and known to those skilled in the art.

The present invention makes a compromise between the number of disk accesses and at the same time limits the volume of data that must be processed for each query. The present invention is therefore particularly advantageous if it is used for document collections that frequently change and where the documents are inserted and deleted with great frequency.

Claims (4)

1. Index for full-text search that enables efficient history searching in document collections that change frequently, characterized in that the index comprises a hierarchy of inverted indexes with varying time granularity, where each level of the hierarchy comprises a number of partitions identified by a level number and time stamp after the last addition of a document, where each partition is itself an augmented inverted index containing all additions and all deletions of documents in the interval from the end of the preceding partition until the timestamp, and where the partition contains delta lists for each symbol of a document. 2. Index according to claim 1, characterized in that each delta list is a list of all insertions and deletions of documents that took place in a time interval given by a minimum time granularity for the search queries. 3. Index according to claim 1, characterized in that it includes a special symbol for registering deleted documents. 4. Index according to claim 1, characterized in that it includes a directly accessed index to obtain a faster response to a list of deleted documents.