NO327318B1 - Steps to improve the efficiency of a search engine - Google Patents

Abstract

In a method of improving the efficiency of a search engine in accessing, searching and retrieving information in the form of documents stored in document or content magazines, the search engine comprises a group of search nodes placed on one or more servers. An index of the stored documents is created. The search engine processes a search query from a user and returns a result set of documents corresponding to the search query. The search engine index is configured based on one or more document properties and is partitioned, reproduced and distributed across the group of search nodes. The search queries are processed on the basis of the distributed index. The method implements a structure for distributing the index of a search engine across a series of hosts in a computer cluster and is based on three orthogonal mechanisms for index distribution, namely index partitioning, index reproduction and assignment of the replicas to the hosts. In this way, different ways of configuring the index of a search engine are obtained, and a greatly improved resource utilization is obtained combined with any desired degree of fault tolerance.

Description

The invention relates to a method for improving the efficiency of a search engine when accessing, searching and retrieving information in the form of documents stored in document or content magazines, where an indexing system in the search engine collects the stored documents and generates an index for them, where the application of a user's search query on the index will return to the user a result set with at least some documents corresponding to the search query, and where the search engine comprises a group of search nodes located on one or more servers.

In particular, the invention shows how a new structure for index distribution on a search engine, and even more particularly on a business search engine, can be formed.

Building a search engine presents challenges for several reasons:

• Performance. The latency to calculate a search query response must be very low and the enterprise search engine must support a high query rate. • Scalability. Performance must scale with the number of documents and the arrival rate of search queries. • Fault tolerance. The search engine must maintain high availability and high performance even during hardware failures.

To satisfy the above three requirements, search engines use sophisticated methods to distribute their indexes over a potentially large cluster of hosts.

Known technique

An overview and discussion of known technology relevant to the present invention will now be given. All literature references are identified by abbreviations in brackets at the appropriate place in the following. A full bibliography is provided in an appendix at the end of the description.

In order to improve the efficiency of search systems, much research has recently been conducted on the distribution of search engine indexes. Earlier work dealt with how posting lists should be distributed and investigated the trade-off between distribution of posting lists based on index terms (here also called keywords) and documents [BadOl, MMROO, RNB98, TGM93, CKE<+>90, MWZ06]. The present invention is based on the insight that a global choice between these two alternatives is suboptimal because the statistical properties of keywords and documents vary in a typical search environment. This can be illustrated by the phrase "one size does not fit all" for two examples: • For a keyword k whose posting list can be accommodated on a single memory page, the distribution of k's posting lists over several hosts will in effect increase the response time for search queries involving k because many hosts would be involved in retrieving the posting list even though a single host would be able to retrieve the posting list with a single memory access. For a keyword k<1> if the posting list does not fit on a single memory page, however, the distribution of k' posting lists over the set of hosts will reduce the response time and different parts of the posting list can be found in parallel. • For an unpopular keyword k that only appears in a few queries, the reproduction of its posting list is a waste of resources, as there is little opportunity for parallelism when executing the queries and thus not many queries will ever read k's posting lists in parallel from different hosts. The posting list for an occurring popular keyword k' will, however, be accessed by many search queries and should thus be reproduced to enable parallelism.

In order to better understand the prior art, a brief discussion of a search engine architecture as known in the art and currently used shall be given, with reference to fig. 1 showing a block diagram of a search engine as it will be known to those skilled in the art, its main subsystems and its interfaces respectively to a content domain, i.e. the document store which can be subjected to a search, and a client domain comprising all users submitting search queries to the search engine for retrieval of question-matching documents from the content domain.

The search engine 100 according to the present invention and as known in the art, comprises different subsystems 101-107. The search engine can access document or content magazines located in a content domain or space, from which the content can either be actively pushed into the search engine or, with the use of a data coupler, pulled into the search engine. Typical warehouses include databases, sources available via ETL (Extract-Transform-Load) tools such as Informatica, any XML-formatted warehouse, files from file servers, files from web servers, document management systems, content management systems, e-mail systems, communication systems, collaboration systems and rich media, such as audio, images and video. The recovered documents are delivered to the search engine 100 via a content API (Application Programming Interface) 102. The documents are then analyzed in a content analysis step 103, which is also referred to as a content pre-processing subsystem to prepare the content for improved search and discovery operations. Typically, output from this content analysis step 103 can be an XML representation of the input document. Output from the content analysis is used to feed the core search engine 101. The core search engine 101 can typically be located on a server farm in a decentralized manner to allow processing of large document volumes and high query loads. The core search engine 101 accepts user requests and produces lists of corresponding documents. The document arrangement is usually determined according to a relevance model that measures the likely importance of a given document relative to the search query. In addition, the core search engine 101 can produce additional metadata about the result set such as summary information on document attributes. The core search engine 101 itself comprises additional subsystems, namely an indexing subsystem 101a for crawling and indexing content documents and a search subsystem 101b for performing the actual search and retrieval. Alternatively, the output from the content analysis step 103 can be fed into an optional alert engine 104. The alert engine 104 will have stored a set of search queries and can determine which search queries would have been satisfied by the given document input. A search engine can be accessed from a number of different clients and applications which can typically be mobile or computer-based client applications. Other clients include PDAs and gaming devices. These clients, located in a client space or domain, deliver requests to a search query or client API 107 in the search engine. The search engine 100 will typically have a further subsystem in the form of a search question analysis step 105 to analyze and refine the search question with a view to constructing a derived search question that can extract more meaningful information. Finally, output from the core search engine 101 is typically further analyzed in another subsystem, namely a result analysis step 106 to produce information or visualizations that are used by the clients. - Both steps 105 and 106 are connected between the core search engine 101 and the client API 107, and in case the notification engine 104 is present, it is connected in parallel with the core search engine 101 and between the content analysis step 103 and the query and result analysis step 105;106.

To improve the search speed of a search engine, international published patent application WO00/68834 proposes a search engine with a two-dimensional, linearly scalable parallel architecture for searching a collection of text documents D, where the documents can be divided into a number of partitions di, d2,... d„, where the collection of documents D is preprocessed in a text filtering system, so that a preprocessed document collection Dp and corresponding preprocessed partitions dp], dp2, ..., dpn are obtained, where an index / can be generated from the document collection D such that for each previously preprocessed partition dp], dp2, ..., dpn, a corresponding index i}, i2,..., i„ is obtained, where searching in a partition d of the document collection D takes place with a partition-dependent data set dp k, where 1< k < n , and where the search engine comprises data processing units that form sets of nodes connected in a network. A first set of nodes comprises sending nodes Na, a second set of nodes comprises searching nodes Np, a third set of nodes comprises indexing nodes NY. The search nodes Np are grouped in columns which are connected via network in parallel between the sending nodes Na and an indexing node Nr. The sending nodes Na are arranged to process search questions and search answers, the searching nodes Np are arranged to accommodate search software, and the indexing nodes are arranged to generate indexes / for the search software. Optionally, collection nodes Ng can be arranged in a fourth node set and arranged to process search queries so that the sending nodes can be freed from this task. The two-dimensional scaling takes place respectively with a scaling of data volume and scaling of the search engine performance with a respective adaptation of the architecture.

The schematic arrangement of this scalable search engine architecture is shown in fig. 2 which illustrates the principle of two-dimensional scaling. An important advantage of this architecture is that the query response time is mainly independent of directory size, as each search query is executed in parallel on all search nodes Np. Furthermore, the architecture is inherently fault tolerant so that failures in the individual nodes will not result in a system crash, only a temporary reduction in performance.

Although the architecture shown in fig. 2 provides a multi-level data and function parallelism so that large volumes of data can be searched efficiently and very quickly by a large number of users at the same time, it is fraught with certain disadvantages and is therefore far from optimal. This is due to the fact that the row and column architecture is based on a mechanical and rigid partition scheme that does not take into account modalities in the keyword distribution and user behaviour, as expressed by frequency distributions of search terms or keywords, and access patterns.

Furthermore, it is known from US Patent No. 7293016 B1 (Shakib & al., assigned to Microsoft Corporation) to arrange indexed documents in an index according to a static ranking and partitioned according to this ranking. The index partition is progressively searched starting from a partition containing those documents with the highest static rank to locate documents containing a search term, and a score is calculated based on the existing set of documents found so far in the search and based on the range of static ranks in the next partition to be scanned. The next partition is scanned to locate documents containing a search term when the calculated score is above a target score. A search can be stopped when no more relevant results will be found in the next partition.

US Published Patent Application No. 2008/033943 Al (Richards & al., assigned to BEA Systems, Inc.) relates to a decentralized search system with a central queue of document-based data records, where a group of nodes is assigned to different partitions, indexes for a group of documents are stored in each partition, and the nodes in the same partition independently process document-based data records from the central queue to form the indexes.

The prior art does not provide a construction structure based on general perceptions of the distributional properties of keywords and search queries and thus does not achieve the flexibility of an implementation that does, with resulting performance improvements and reduction in resource requirements.

In particular, the growth of indexes has been a concern, and a number of specific methods that elegantly handle direct-linked index construction have been developed [BCL06]. These techniques are orthogonal to the structure that emerges by using the method according to the present invention, as will be apparent from a detailed description thereof.

The present invention does not take into account specific ranking algorithms, as it is assumed that the user always wants all answers to the search question. However, these can be straightforwardly extended to some of the recently developed ranking algorithms [RPB06, AM06, LLQ<+>07] and algorithms for new search query models [CPD06, LT1T07, ZS07, DEFS06, TKT06, JRMG06, YJ06, KCMK06]. Algorithms for finding the best corresponding answer to the search query when response functions are combined have also been the focus of much research [PZSD96, Fag99, MYL02]. However, these methods are orthogonal to an index distribution structure as realized by the method of the present invention, and they can also be easily incorporated.

The techniques used by the present invention for query processing with partitioned posting lists are based on basic ideas derived from parallel database systems [DGG<+>86]; however, parallel database systems were developed for database management systems that create structured data, while the focus of the present invention is corporate and Internet searching, where the search query is executed over collections of often unstructured or semi-structured documents.

There is also known technique regarding text query processing in peer-to-peer systems, where the goal is to coordinate loosely connected hosts with an emphasis on finding search results without broadcasting a search query to all hosts in the network [RV03, LLH<+>03, ODODg02, SMwW<+>03 ,CAN02, KRo02, SL02, TXM03, TXD03, BJR03, TD04]. The main assumption in these known publications concerns the degree of coupling between the hosts and is different from the initial basis for the present invention which assumes that all hosts are tightly coupled and under the control of a single quantity, e.g. in a cluster in a corporate data center which is the dominant architecture today. The conceptual structure upon which the present invention is built maps directly onto this architecture by assuming a tightly coupled set of hosts.

In light of the shortcomings and disadvantages of the above-mentioned known technique, it is a main purpose of the present invention to provide a method which substantially improves the performance of a search engine.

Another object of the present invention is to configure the index of a search engine and especially a business search engine on the basis of realizing that keywords and documents will differ with respect to intrinsic as well as extrinsic characteristics, e.g. such as provided by modalities in search and access patterns.

Finally, it is a purpose of the present invention to optimize an index configuration with respect to inherent features of the search system itself as well as its operating environment.

The above purposes as well as further features and advantages are realized according to the present invention with a method characterized by configuring the search engine's index on the basis of one or more document properties and at least one among a fault tolerance level, a desired search performance, document meta-properties and an optimal utilization of resources; to partition the index, to reproduce the index, to distribute the thus partitioned and reproduced index over the group of search nodes, so that the index partitions of and the replicas thereof are assigned to said one or more servers on which the group of search nodes is located, and to process search queries on the basis of the distributed index.

Further features and advantages of the present invention will be apparent from the appended, independent claims.

The present invention will be better understood with reference to the subsequent detailed discussion of the general background and relevant embodiments read in conjunction with the attached drawing, in which fig. 1 shows a simplified block diagram of a search engine as known in the art and discussed above;

fig. 2 a diagram and a scalable search engine architecture as used in the well-known search service AllTheWeb and discussed in the above,

fig. 3 the concept of mapping function,

fig. 4 the concept of host assignment,

fig. 5 the concept of mapping functions for rows and columns, and fig. 6 the concept of classification of keywords.

In order to describe the present invention in its entirety, some assumptions and initial considerations must now be discussed. Next, the new structure for index distribution made possible by the method according to the present invention will be described.

For the present invention, a simplified model of a search engine is introduced. The notation used is summarized in table 1.

Table 1: Notation used in this patent application

There is a set of keywords K = { kj, ..., k„} and a set of documents D = { di, ..., dm}. Each document d is a list of keywords and is identified by a unique identifier called a URL. An instance is a tuple ( k, u) that indicates that the document associated with URL u contains the keyword k. A document record is a tuple ( u, date) that indicates that the document associated with URL u was produced on a given date.

In practice, an instance contains other data, such as the position of the keyword in the document or data useful in determining the ranking of the document obtained on the basis of a search query. In addition, a document also has other associated metadata next to the document record, e.g. an access control list. None of these conditions are important for features of the index which are the subject of the following discussion.

The index of a search engine consists of a set of occurrences and a set of document records. There is a set of occurrences for each keyword k, hereafter referred to as the posting set for the keyword k. The posting set for the keyword k contains all occurrences of the keyword k, and it contains only occurrences of the keyword k. To be consistent with the prior art, posting sets are assumed to be arranged in a fixed order (for example, lexicographically by URL), and the ordered set of postings of a keyword x will be denoted as the posting list PL(k) of the keyword k\ ' the following representation. The set of document records contains only one document record for each document, and it contains only documents.

Now search questions and question processing will be discussed in some detail. Users issue search queries, and a search query q consists of a set of keywords q = { k})..., ki} c K. The present invention assumes a model for a search query in which a user would prefer to find any document that contains all keywords in the search query. It can be assumed that the arrival time for each query q follows an exponential distribution and can thus be characterized by a single parameter Xq, the average arrival rate for the query q. Note that this probabilistic model of search queries implies that the search queries are independent. A question load co is a function that associates each search question q e 2 with an arrival rate X-ojfø). From a question load, the arrival rate \ m( K) can be calculated for each keyword By summing over all search questions containing k, formally

The following simplified way to logically process a query q =

{/Ci,..., Ki} shall be assumed. For each keyword k, - its posting list PL(K-j) for / e {1,...,^} is found, and their average in the URL fields is calculated. Formally, the following rational algebraic expression calculates query results QueryResultfø) for each query q = {*•/,..., Ki) :

QueryResultfa) = 7IurlPL(ki) fl...fl 7IurlPL(k/)

There are more sophisticated ways to define QueryResultfø); for example, the user may wish to see only subset of QueryResultfø), and may also wish to see this subset in ranked order.

Physical execution

The present invention assumes that a cluster of workstations is modeled as a set of hosts H= { hh0} [ACPtNt95]. Furthermore, each host is assumed to have a single disk memory with a fixed amount of storage space in DiskSize units. Note that for ease of presentation, the translation of the abstract unit of storage into a concrete unit of bytes is omitted in this model. Each instance is assumed to have a fixed size of 1 unit. For a keyword k and its posting list PL(k), the size of the posting list |PL(k)|, defined as the number of occurrences in PL(k).

Each host h is assumed to be able to deliver connected total performance that allows it to retrieve buc(/z) units of storage within latencyBound milliseconds; this number is an aggregate unit that includes CPU speed, the amount of main memory available, and the latency and transfer rate from the host's disk memory. Furthermore, in the following, all hosts are assumed to have identical performance, and thus the dependency buc(/z) or h can be omitted and reference is made only to bue as the number of units that any host can recover within latencyBound milliseconds.

A structure for index distribution will now be discussed. Specifically, the structure or architecture as realized by the method according to the present invention includes three aspects, namely partitioning, reproduction (replication) and host assignment, as indicated below.

Partitioning

For each keyword, its posting list is partitioned into one or more components. This partitioning of posting lists into components is performed to be able to distribute the posting list over multiple hosts, so that all components can be retrieved in parallel.

Reproduction (replication)

For each cue, each of its components is reproduced a certain number of times in multiple component replicas for each component. Component replicas are formed for several reasons. The first reason for reproduction is fault tolerance; in the event that a host storing a component fails, the component can be read from another host. The second reason for replication is improved performance because queries can retrieve a component from any of the hosts on which the component is replicated, thus load balancing.

Host assignment

After partitioning and reproduction, each component replica of a posting list is assigned to a host, but upon assignment subject to the restriction that two component replicas of the same component and the same partition must not be assigned to the same host. The host assignment enables the placement of the components to be optimized globally over the keywords. Components of keywords that usually occur together in questions could, for example, be placed together to reduce the cost of question processing.

Now the corresponding three parts of the structure of the index distribution according to the method of the present invention must be introduced.

1. Partitioning the posting lists into components.

2. Reproduction of the components.

3. Mapping the components to hosts.

For the first part, a function numPartitions(-) is selected which uses a keyword k as input and returns the number of components into which the posting list PL(k) is partitioned; the resulting components are C01(k-),C02(x-),..., COnum<p>artmons()C)(>). In addition, a function occLoc(-) is selected which uses an instance as input and outputs the number of components in which this instance is located. Thus if occLoc((ac, w)) = U then { k, u) € C0i(K"). Note that if ( k u) e PL(k), then 1 < occLoc((ac w)) < numPartitions(Ac).

For the second part, a function numReplicas(-) is selected which uses a key word k as input and returns the number of component replicas of the partitions of the posting list for k. The original component is included in the number of component replicas. For a keyword k, there are thus numReplicas(x) • numPartitions(jr) component replicas. If the correct numPartitions(x) components are combined, then together they will comprise PL(k-); and for any component CO^k) there can be numReplicas(jt) identical component replicas. If the keyword k in a workload co has an arrival rate ÅJ^ k), and load is evenly balanced between

numReplicas(K-) component replicas, in particular the arrival rate for this keyword for each of the component replicas will be

For the third part, a function hostAssign ( k, i, j ) is selected which takes as input a keyword k, a replica number i and component number j and returns the host that stores component replica / of component j of the posting list PL(k). Note that two identical component replicas (that is, replicas of each other) must be mapped to different hosts. Formally, hostAssign ( K, i\, j) hostAssign ( K> h, j) must hold for je {l,...numPartitions(/c)} and i\,

ii e {l,...numPartitions(/r)} with iyt i2.

Figures 3 and 4 show an exemplary instance of the structure according to the present invention for a keyword k with a posting list of eight instances: A, B, C, D, E, F, G and H. In the example, numPartitions(K) has = 4, (ie the posting list for a: is partitioned into four components) and a numReplicas(x-) = 3 (ie there are three component replicas). Five hosts h\, h2, h3, h4, and h5 are given. The function hostAssign(x, 1, 2) = h\, hostAssign(/c, 2, 2) = h2, hostAssign(x, 3, 1) = ^5.

An instantiation of the three functions numPartitions(-) numReplicas(-) and hostAssign(x, i, j) shall be called an index configuration for a search engine.

Given the structure shown above, the physical model for processing a query q can now be introduced. Processing a search query q involves three steps. 1. For each keyword k e q a set of hosts is identified such that if the union of the component replicas is stored on the hosts comprising PL(k) . numRepIicas(/r) > 1, then there is more than one such quantity, and different quantities can be chosen based on other characteristics, e.g. the load on a host. 2. For each keyword k e q, the selected component replica must be found for all selected hosts. 3. It is necessary to calculate Query Resultfø), which requires an average to be formed between the different posting lists.

Now these three steps are to be processed in turn.

In the first step, note that the function hostAssign(/c, i, j) for each keyword k encodes the set of hosts where all component replicas of the posting list for k are stored.

In the second step, each host involved in processing the query q (as selected in the first step) retrieves all its local component replicas for all keywords involved in the query.

In the third step, each host will first form sections with the local component replica of all keywords. The results of the local cuts are then processed further to complete the calculation of Query Resultfø).

Now the problem of the index construction can be defined as follows: A set of hosts that have connected storage space DiskSize and performance arch is given. Also given is a set of keywords with posting lists PL(x-i),...PL(xrOT) having sizes I PL(k-() PL(xrm) |, as well as a query load co.

For the index construction problem, there must now be functions numPartitions(-), numReplicas(-), and hostAssign so that the expected latency to answer a query q is below latencyBound, where the expectation is over the amount of all possible query sequences.

In the following, a discussion of some embodiments will be given with the help of specific and exemplary instances thereof.

1. AllTheWeb Rows and Columns

The AllTheWeb architecture Rows and Columns (as a tribute to the search system AllTheWeb as described in the above introduction) is a trivial instantiation of the structure, cf. fig. 5 which reproduces the mapping functions for host mapping. In this architecture, there is an array of hosts consisting of r rows and c columns. The matrix can be visualized as follows:

By using a spread function on URLs that are independent of the keyword, the postings of any keyword are approximately evenly partitioned into c components. Each component is then reproduced within the column; one component replica for each row, resulting in r component replicas. To reconstruct the posting list of a keyword, one host from each column must be accessed, but it is not necessary to select all these hosts from the same row, and this flexibility facilitates query load balancing between hosts and improves fault tolerance.

In order to form the connection to the notation for the structure as realized by the method according to the present invention, the above three functions must be instantiated. Due to the row and column scheme, it follows that for all keywords k e K, that numPartitions(v) = c, and numRepIicas(Ar) = r holds, and for all URLs u and kx, k2 e K the following must hold:

occLoc((k"i, «)) = occLoc((k"2, u)),

i.e. for all URL u, the function occLoc((/c, u)) is independent of the keyword k. The hostAssign function is also very simple. Let hostAssign(*:, i,j) = (i,j), where i is the row of the host, and gy denotes the column of the host in the r x c matrix. Note that if the number of columns c is chosen appropriately, then all component replicas of any single keyword k will be read in parallel within latencyBound. The smallest number c is the following:

When query processing is performed in AllTheWeb's row and column architecture as shown in fig. 2, only one host in each column needs to be involved, even for multi-keyword queries, as the function occLoc((k; u)) is independent of k.

However, AllTheWeb's Rows and Columns have a number of drawbacks. First, the number of hosts accessed for a keyword x: is independent of the length of h! s posting list; c hosts must always be accessed even for keywords with very short posting lists. Second, AllTheWeb's Rows and Columns do not take into account the popularity of a keyword in the query load; each component reproduces itself r times even if the associated keyword is accessed relatively rarely. Third, the changes in the physical layout for AllTheWeb's Rows and Columns are limited to the addition of hosts in multiples of c and r at once, resulting in an additional row or an additional column in the architecture.

Adding a new (c host) row is relatively straightforward; however, adding a new (c host) column is non-trivial. To illustrate this point, consider an instance of AllTheWeb Rows and Columns with r rows and c columns and which uses a connected function occLocc(-) with a set of values {1,c}. When another row is added, a new function oecLoc'() with a set of values {1, c+1} must be selected and generally have:

occLoc((at, u)) * occLoc' (( k, u)),

so that all posting lists need to be repartitioned according to occLoc'( ), which basically results in a rebuilding of the entire index.

2. Fully Adaptive Rows and Columns

Now a solution according to the present invention will be described which takes into account both the difference in the size of posting lists and the difference in the popularity of the key words in the search query. The essence of this new solution is that AllTheWeb Rows and Columns are instantiated differently for each keyword: Each keyword can have a different number of rows and columns. In other words, by using the method according to the present invention, a solution with fully adaptive rows and columns is obtained.

Consider a keyword k. Start with an instantiation of numPartitions(zc). Since each host can only retrieve arc units and satisfy the global query latency requirement latencyBound, PL(*r) is partitioned into components. Thus, each component is sized so that it can be read within the query latency requirement from a single host. Note that for a keyword that has very short posting lists, one (or very few) component(s) are formed, while for keywords that have long posting lists, many components are formed.

The question is now how many component replicas should be created for a keyword k. It should be remembered that component replicas are created with fault tolerance in mind and to distribute the query load across the hosts. To tolerate/unreachable hosts, force numReplicas(xr) >/. To balance the question load, posting lists for popular keywords (in the question load) are replicated more frequently than posting lists for rare keywords. So the number of replicas becomes inversely proportional to the arrival rate of the keyword in the load.

By making numPartitions(/c) and numReplicas(K) different for each keyword k, a number of rows and columns specific to each keyword is obtained. The number of columns still indicates the number of partitions and the number of rows the number of replicas for each partition. However, keywords with long posting lists have many columns, and keywords with short posting lists have few columns. Popular keywords have many rows, unpopular keywords have few columns. Compared to AllTheWeb Rows and Columns, Fully Adaptive Rows and Columns results in less imbalance in the size of the components for different keywords. It is thus achieved that each component replica is now normalized in the sense that each component replica has approximately the same size (up to a difference in arc) and has approximately the same arrival rate.

A number of different ways are provided to assign component replicas to hosts. Given o hosts, one possibility would be to spread each keyword /cr over the numbers from 1 to o and then assign the components sequentially (mod 6) to hosts. Conceptually, this backs up a key word k k! s specific matrix with numPartitions(x) columns and numRepIicas(/c) rows sequentially in o hosts. Formally, the assignment function has the following form. Let keywhash(-) be a function up to {1, o) with the property that

for i e K to {1, ..., o} and x- e Å". Then the submatrix for the keyword k i i/ can be set up row by row as follows: hostAssign(*; i, j) = (keywHash(K) + { i - 1) • numPartitions(/c) + ( j - 1)) mod o,

where i e {1, ..., numReplicas(x)} andj e {1, ..., numPartitions(xr)}.

With this instantiation of hostAssign, the question now is how many component replicas will be assigned to a host. With Fully Adaptive Rows and Columns, the following simple theorem shows that there will not be a large imbalance between two hosts with respect to the number of component replicas.

Theorem 1

Let s be the total number of component replicas formed over all keywords k, formally

Let o be the number of hosts and assume that hostAssign is defined as in the previous section and assume that s = Q(ø). Then the maximum number of component replicas on each host will h e //be, i.e. the maximum number of component replicas assigned to a host will be in the order of magnitude of the average number of component replicas assigned to a host.

Proof

Follows from "bounds on balls into bins" (bounds on balls into bins) [MR95].

To retrieve the posting list for a keyword tc\ ed processing a search query, a host is selected from each of the numPartitions(x) "virtual columns" for k! s matrix; thus the number of different possibilities for choosing this quantity will be numPartitions(/c)<n>umReplica<s>('<f>).

Processing multi-keyword queries in Fully Adaptive Rows and Columns is much more expensive than in AllTheWeb Rows and Columns. For example, a keyword q = { kuk2} can be considered where

numPartitions(x:i) ^ numPartitions(/c2). Since keywords K\ and k2 are partitioned differently, the posting list for e.g. Kj is repartitioned to correspond to the partitioning of k2, which is an expensive operation. In addition, there is no guarantee that any components of K\ and tc2 are co-located on the same host.

3. Two-Class Rows and Columns (Two-Class Rows and Columns)

A third instantiation is the structure realized by the method according to the present invention is the special case of Fully Adaptive Rows and Columns which results in a much simpler (and cheaper) query processing. As in AllTheWeb Rows and Columns, it is assumed that r - c hosts are arranged in the usual array of hosts.

For Two-Class Rows and Columns, the keyword is classified along two axes. The first axis is the size of the posting list, where keywords are partitioned into short and long keywords based on the size of their posting lists. The second axis is the arrival rate of the keywords in the question load where keywords are partitioned into popular and unpopular keywords based on their arrival rate. This results in four different classes and keywords. • Short, unpopular (SU) keywords. The posting list of a SU keyword k is not partitioned and the minimum number of component replicas is formed to achieve the desired fault tolerance level. For an SU keyword, numPartitions(K) = 1, and numReplicas(/c) =f are thus set. • Long, unpopular (LU) keywords. The posting list of an LU keyword k is partitioned into c components and/component replicas are created for each component to achieve fault tolerance. Thus, for an LU cue xrnumPartitions(x) = c, and numReplicas(K") =/ • Short, popular (SP) cues. The posting list of an SP cue is not partitioned and r component replicas of k!'s posting lists are formed to distribute the ids arrival rate over hosts. For an SP keyword k, numPartitions(x) = 1, and numReplicas(Ac) = r are set respectively. • Long, popular (LP) keywords. The posting list of an LP keyword is partitioned into c components and each component is reproduced r times.For an LP keyword k, numPartitions(K) = c, and numReplicas(K) = r are set respectively.

The instantiations of the two functions numPartitions(-) and numReplicas(-) from the available structures for the different classes of keywords are shown in Table 2. Note that compared to Fully Adaptive Rows and Columns, in Two-Class Rows and Columns there are only four different types of matrices as shown in fig. 6, where you have the error tolerance level/= 2.

Table 2

Classification of keywords: Functions (numPartitions(/c), numReplicas(*:))

Let keywHash(-) be as defined and shown in connection with the discussion of Fully Adaptive Rows and Columns as above. Let rowHash(v) be a function from Å" x {l,.../} to {\,..., r} such that rowHash(ftr, i\) * rowHash(*:, i2) for i\ , i2 e {l,...,r} How the component replicas of a cue are assigned to the hosts depends on the class of the cue.

• For a SU keyword, kSu is defined

hostAssign(/f5ty,/,l) = (rowHash( x; z'),(keywHash( mod c) + 1)), forie {l,...j}.

• For an LU keyword klu, is defined

hostAssign(x-z,(/,y) = (rowHash(x:,/),y')s

for / e andye {l,...,c}.

• For an SP keyword ksp is defined

hostAssign(x"5p,z',l) = (/, (keywHash(x:5(/) mod c) + 1),

for i e { l,..., r}.

• For an LP keyword klp is defined

hostAssign(Ki,/»,zV) <=> (i,j),

for / e { l,..., r} and; e {l,...,c}.

Similar to the previous section, the following theorem can be proved.

Theorem 2

Let s be the total number of component replicas formed over all keywords k, (regardless of the class of k), formally

Let o be the number of hosts and assume that hostAssign is defined as in that

previous paragraph. Then the maximum number of component replicas at each host will h e //be, i.e. the maximum number of component replicas assigned to a host is of the order of magnitude of the average number of component replicas assigned to a host.

Proof Follows from "bounds on balls into bins" (bounds on balls into bins) [MR95].

Given the function hostAssign(*r,v) for the different classes and keywords are

query processing in Two-Class Rows and Columns is not very complicated, and there are usually many possibilities for how a search query can be processed. Table 3 describes how a search query q = { k\, k2} can be processed with two keywords; query processing for queries with more than two keywords is analogous.

The amount of possible amounts of hosts to select for

question processing is straightforward, given this discussion.

Based on the existing structure obtained by the method according to the present invention and discussed above, a few practical considerations shall be outlined in the following.

Firstly, use of the method according to the present invention allows an expansion of the search system's structures which means that each word can have more than a single row and column occurrence. This shall be described immediately in the following.

In the above embodiments it has so far been assumed that for each keyword k there are two functions numPartitions(/c) and numReplicas(/r). However, for performance reasons, a keyword can be partitioned in more than one way and possibly have different numbers of replicas for the different partitions. For example, in Two-Class Rows and Columns, the posting list for an SP keyword ksp is reproduced over a column. In addition to the present reproduction, the posting list of KSp can be partitioned over one row because KSp often occurs together with another LP cue klp which is partitioned over all rows.

This extension can be characterized by connecting sets of functions from the resulting structure with each keyword and using the method according to the present invention; e.g. could a keyword xrha two sets of functions {numPartitionsi(x-), numReplicasi(x)} and {numPartitions2(Kr), numReplicas2(K)}. The number of quantities could be keyword dependent. This greatly increases the options for search query processing. However, this extension shall not be introduced formally here, as it is conceptually straightforward.

Second, those skilled in the art will appreciate that using the method according to the present invention to form structures for real search systems, including enterprise search systems, may allow various optimizations thereof. Such optimizations shall take the method according to the present invention as their starting point, but their reduction to practice is assumed to lie outside the scope of the present invention, and they shall consequently not be further discussed here.

The method according to the present invention realizes a structure for distributing the index of a search engine over several hosts in a data processing cluster. The structures shown distinguish between three orthogonal mechanisms for distributing a search index: index partitioning, index reproduction, and assignment of replicas to hosts. Instantiations of these mechanisms provide different ways to allocate the index to a search engine, including popular methods known from the literature and new methods that completely outperform the prior art in terms of resource usage and performance, while achieving the same level of fault tolerance.

Furthermore, the method according to the present invention recognizes for the first time that different keywords and different documents in a search engine can have different properties (such as length or access frequency). The structure realized by using the method according to the present invention forms a configuration of the index on a search engine according to these characteristics. The structure also serves to outline how queries are processed for the configuration space enabled by realizations of the structure.In addition, instantiations of this structure lead to existing index configurations known in the art as well as to new index configurations that would not be possible with the prior art.

Claims (10)

1. Method for improving the efficiency of a search engine in accessing, searching and retrieving information in the form of documents stored in document or content magazines, where an indexing system in the search engine collects the stored documents and generates an index for these, where the application of a user's search query on the index will return to the user a result set with at least some documents corresponding to the search query, where the search engine comprises a group of search nodes located on one or more servers, and where the method is characterized by to configure the search engine's index on the basis of one or more document properties and at least one among a fault tolerance level, a desired search performance, document meta-properties and an optimal resource utilization;, to partition the index, to reproduce the index, to distribute the thus partitioned and reproduced index over the group of search nodes, so that the index partitions of and the replicas thereof are assigned to said one or more servers on which the group of search nodes is located, and to process search queries on the basis of the distributed index. 2. Procedure according to claim 1, character ti sert by distributing the index so that a search query latency is below a user-specified latency limit. 3. Procedure according to claim 1, characterized by processing a search query with posting lists that have different classifications. 4. Procedure according to claim 3, characterized by classifying posting lists for search queries on the basis of length and popularity, the latter being determined by the user access pattern. 5. Method according to claim 4, where the group of search nodes comprises r rows and c columns, characterized by using a number of partitions different from the number of replicas for each query keyword, so that the number of rows and columns is different for each query keyword, and to distribute the index taking into account either the size differences of the posting lists or the popularity differences of the posting lists, or both. 6. Method according to claim 4, where the group of search nodes comprises r rows and c columns, characterized by to classify a search query in two dimensions, a first dimension being a posting list size and a second dimension the arrival rate for each query keyword, so that a query keyword is partitioned in the first dimension as respectively short and long and in the second dimension as respectively popular and unpopular, and distributing the index taking into account at least one of the posting lists' size differences, the posting lists' popularity differences, and the cost of processing a search query. 7. Procedure according to claim 3, characterized by dividing the query posting lists into components to balance the query processing load between the query nodes. 8. Procedure according to claim 7, characterized by reproducing posting list components so that, in order to increase the fault tolerance level, identical replicas of these are formed. 9. Procedure according to claim 8, characterized by assigning the component replicas to the search nodes to balance the processing load for search queries between them. 10. Procedure according to claim 1, characterized by distributing the index of the search engine to a two-dimensional, linearly scalable group of search nodes, where the scaling itself is used to handle variations either in the data volume or in the frequency of search queries, or both.