JP6050503B2 - Mail indexing and retrieval using a hierarchical cache - Google Patents

Description

CROSS-REFERENCE of other application claims the priority of September 21, Chinese Patent Application No. 201210357269.6, filed 2012 "METHOD AND SYSTEM FOR ESTABLISHING MAIL INDICES AND METHOD AND SYSTEM FOR SEARCHING MAIL ". This application is incorporated herein by reference for all purposes.

  The present application relates to the field of network data processing, and in particular, to a method and system for establishing a mail index and performing a mail search.

  As Internet communication has become increasingly widespread and more and more users communicate via email (especially email or email), mailbox search has become an important search technology among data searches. Mailbox searches are usually based on a mailbox index. That is, all user mail is typically retrieved using the mailbox index.

  One existing method for establishing a mail index is as follows: In general, a mailbox index is established in the form of an inverted index. For example, it is assumed that there are three mail files named doc_id1, doc_id2, and doc_id3, and all include the phrase “hello my world”. Thus, the transposed index record that stores the keyword and mail file associations is as follows:

  hello-> doc_id1, doc_id2, doc_id3

my-> doc_id1, doc_id2, doc_id3

  world-> doc_id1, doc_id2, doc_id3;

  The above-mentioned inverted index record is stored in the inverted index file. The offset position and length of each inverted index record in the inverted index file is recorded, and the offset position is written to the dictionary file as follows:

  {“Hello”: {“file_path”: “/ xxx / inverted_index_file”, “offset”: 0}};

  Assuming that the user searches for mail containing “hello”, all mail containing this keyword in the dictionary file can be found. That is, the address “/ xxx / inverted_index_file” is found. This inverted index file is then opened and the position of offset “0” is fetched, so three emails {doc_id1, doc_id2, doc_id3} can be fetched.

  However, when new mail is added, the inverted index file needs to be updated to ensure the integrity of the search results. For example, it is assumed that a new mail (doc_id4) is added. This mail also includes a total of three keywords “hello my world”. Therefore, at this point, the inverted index record needs to be updated as follows:

  hello-> doc_id1, doc_id2, doc_id3, doc_id4

  my-> doc_id1, doc_id2, doc_id3, doc_id4

  world-> doc_id1, doc_id2, doc_id3, doc_id4;

  When the updated inverted index record is stored in the inverted index file, the original of the two inverted index records “my-> doc_id1, doc_id2, doc_id3, doc_id4” and “world-> doc_id1, doc_id2, doc_id3, doc_id4” The storage location needs to be changed in the inverted index file. At the same time, the corresponding offset value in the dictionary file needs to be modified.

  Therefore, in the above-described method, when a new mail is added, other related data contents of the inverted index file need to be shifted.

  As described above, an existing mailbox search using a mail index typically requires a keyword search of the entire inverted index file. As the size of mail data grows, a mailbox server can have hundreds of millions of subscribers and billions of individual mail messages. In order to store such a large amount of data, a large amount of hard disk IO resources are required, and it becomes difficult or impossible to quickly index a mailbox. Furthermore, the storage cost of a large amount of mail is very high for the mail server. As a result, a large amount of storage resources can be constrained.

  Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

  In order to more clearly describe the technical proposals of the embodiments of the present application or the prior art, a brief description of the drawings that are necessary to describe the embodiments or the prior art is given below. Apparently, the drawings in the following description are only a few of the embodiments described herein. Those skilled in the art can obtain other drawings based on these drawings without spending creative efforts.

1 is a block diagram illustrating one embodiment of a mail indexing system that utilizes a multi-level cache. FIG.

6 is a flowchart illustrating an embodiment of a process for establishing a mail index.

9 is a flowchart illustrating one embodiment of a process for transferring a level 2 inverted index record to a level 3 cache.

6 is a flowchart illustrating another embodiment of a process for establishing a mail index.

6 is a flowchart illustrating another embodiment of a process for establishing a mail index.

The flowchart which shows one Embodiment of mail search processing.

1 is a block diagram illustrating one embodiment of a system configured to establish a mail index. FIG.

The block diagram which shows one Embodiment of a transfer unit.

1 is a block diagram of a system configured to establish a mail index.

FIG. 3 is a block diagram illustrating another embodiment of a system configured to establish a mail index.

The block diagram which shows one Embodiment of a mail search system.

  The present invention is a process, apparatus, system, composition of matter, computer program product embodied on a computer readable storage medium, and / or processor (stored in and / or stored in a memory connected to a processor). A processor configured to execute the provided instructions) and can be implemented in various forms. In this specification, these implementations or any other form that the invention may take may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or memory that is described as being configured to perform a task is a general component that is temporarily configured to perform a task at a certain time, or It may be implemented as a particular component that is manufactured to perform a task. As used herein, the term “processor” is intended to refer to a processing core configured to process one or more devices, circuits, and / or data such as computer program instructions.

  The following provides a detailed description of one or more embodiments of the invention with reference to the drawings illustrating the principles of the invention. Although the invention has been described in connection with such embodiments, it is not limited to any embodiment. The scope of the invention is limited only by the claims and includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of illustration, and the present invention may be practiced according to the claims without some or all of these specific details. For the purpose of simplicity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.

  The techniques described herein may be utilized in many general purpose or special purpose computer system environments or configurations. Examples of these are personal computers, servers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, Including a mainframe computer, a distributed computing environment with any of the above systems or devices.

  This application may be described in the general context of computer-executable commands (such as program modules) being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., for performing particular tasks or implementing particular abstract data types. The present application may be executed in a distributed computing environment. In such a distributed computer environment, a task is executed by a remote processing device connected via a communication network. In a distributed computing environment, program modules may be stored on a local or remote computer storage medium that includes a storage device.

  FIG. 1A is a block diagram illustrating one embodiment of a mail indexing system that utilizes a multi-level cache. In this example, the system 150 includes one or more mailbox servers 152 and hierarchical caches 154 to 158. Mailbox server 152 indexes email messages and stores the index information in a hierarchically structured cache. Although three cache levels are shown for illustrative purposes, a different number of caches may be used in other embodiments. Level 1 cache is implemented using low latency memory, such as random access memory (RAM) of server 152. Level 2 and level 3 caches are implemented using one or more components that have a latency greater than the low latency memory used to implement level 1 caches, such as hard disks or other storage devices. In some embodiments, the inverted index record established for the mail is first stored in a level 1 cache. When the data stored in the level 1 cache reaches a first predetermined threshold, all level 1 inverted index records in the level 1 cache are all level 2 cache files to make room for future inverted index records. Forwarded to Level 2 cache stores cache files. When the amount of data in the level 2 cache reaches a second predetermined threshold, the level 2 data is transferred to the level 3 cache that stores the inverted index file. This process for establishing a mail index can improve hard disk input / output (I / O) performance because it reduces the frequency of hard disk reads and writes.

  FIG. 1B is a flowchart illustrating one embodiment of a process for establishing a mail index. Process 100 may be performed on a system (such as 150).

  At step 101, the email message is processed to obtain keywords associated with the email message.

  In some embodiments, word segmentation is performed on an email message for which an index is established to obtain keywords for the email message. In some embodiments, an index is established only for email text, so word splitting is applied only to email text. In some embodiments, the message subject also needs to be the basis for establishing the index, so word segmentation is applied to both the email text and the email subject to obtain email keywords. The One skilled in the art can implement this process by selecting a word segmentation tool according to the specific requirements of the system.

  It can be seen that each time a user receives a new mail, the mail index establishment method disclosed in the embodiment of the present application may be performed on the new mail.

  At step 102, the acquired keyword is used as a basis for updating the level 1 transposed index record stored in the level 1 cache.

  After the current mail undergoes word splitting, a transposed index record is created that establishes the correspondence between the keyword and the mail message. For example, assume that the identifier of the current mail is “doc5”, and this current mail has two keywords “keyword1” and “keyword5”. In this example, the inverted index record is generated as shown below:

  keyword 1: doc5

  keyword5: doc5

  In some embodiments, the level 1 cache is configured in low latency memory to store the initial inverted index record. That is, the inverted index record generated first for the current mail is initially stored only in the level 1 cache in memory. An inverted index record contains key / value pairs. The “key” of the inverted index record in the level 1 cache refers to various keywords, and the “value” is an identifier of the mail message. Assume that the initial inverted index record of the level 1 buffer of this embodiment is as follows:

  keyword 1: doc1, doc2, doc3;

  keyword2: doc1, doc4

  Therefore, the newly generated inverted index record from the current mail is merged with the existing inverted index record to update the inverted index record in the level 1 cache, and the updated inverted index record is as follows: become:

  keyword 1: doc1, doc2, doc3, doc5;

  keyword 2: doc1, doc4;

  keyword5: doc5

  At step 103, it is determined whether the size of the level 1 inverted index record in the level 1 cache exceeds a first predetermined threshold. If so, control proceeds to step 104, otherwise the process ends.

  In some embodiments, at least two cache levels are configured to implement I / O operations. Level 1 cache is organized in low latency memory and data is recorded as memory mode inverted index records. Level 2 caches are configured using higher latency memory or storage components (such as hard disks). As new mail is processed to obtain an inverted index record, the record is written to the level 1 cache. When the level 1 cache in memory reaches a first predetermined threshold (eg, 2 MB), all transposed index records in the level 1 cache are written to a buffer file in the level 2 cache.

  If the size of the level 1 cache has not reached the first threshold, the inverted index record of the current mail message is written to the level 1 cache. If the current mail matches the search criteria when the user performs the next search, the current mail is found in the level 1 cache. This ends the index establishment process for the current mail.

  At step 104, all level 1 inverted index records in the level 1 cache are transferred to the level 2 cache and stored in the level 2 cache file. In some embodiments, the transfer is implemented as deleting data from the old location after copying the data to the new location. The inverted index record is merged with the existing inverted index record as described in step 102.

  At step 105, it is determined whether the current level 2 cache file size exceeds a second predetermined threshold. If so, control proceeds to step 106, otherwise the process ends.

  In the embodiment of the present application, the threshold size (second predetermined threshold) for the level 2 cache file is such that the current level 2 cache file in which the level 1 inverted index record in the level 1 cache is written reaches the predetermined threshold. When configured, all level 2 inverted index records in the level 2 cache file are configured to be written to the level 3 inverted index file.

  In this step, if the size of the level 2 cache file does not reach the first predetermined threshold, the current mail transposition index record is written to the level 2 cache. That is, the current mail transposition index record is stored in the level 2 cache file. When the user performs the next search, if the current mail matches the search criteria, the current mail is found in the level 2 cache file. This ends the index establishment process for the current mail.

  A level 2 cache file serves as a buffer that is used to avoid the following phenomenon: after the level 1 buffer in memory is full, the transposed index record in the level 1 buffer belongs to various keywords. When writing directly to the index file, it is necessary to write multiple level 3 transposed index files simultaneously. For example, assume that there are 10 inverted index records in the level 1 cache in memory, and that 10 keywords in these 10 inverted index records belong separately to 5 level 3 inverted index files. At this point, if the inverted index record in the level 1 cache is written directly to the level 3 inverted index file, the write operation must be performed simultaneously on the five level 3 inverted index files. When the level 2 cache file functions as a buffer file, these 10 inverted index records need only be written to one level 2 cache file. With the addition of other inverted index records (some of which can eventually be written to the same five level 3 inverted index files), the level 2 cache file grows to a certain threshold (eg 4 MB) One-time classification is performed on the level 2 transposed index in the two cache files. Ten inverted index records, along with many other inverted index records in the buffer file, are written to the level 3 inverted index file to which the keyword belongs. This greatly reduces the frequency of writing files to the hard disk.

  At step 106, all level 2 inverted index records in the level 2 cache file are transferred to the level 3 inverted index file according to the path file. This path file stores the mapping relationship that establishes the correspondence between the keyword and the level 3 inverted index file. In some embodiments, a path file may be generated to record the mapping relationship between keywords and level 3 inverted index files (ie, which keywords are stored in which level 3 inverted index files). The path file records the mapping relationship from the keyword to the level 3 inverted index file. An example of a record in a path file is as follows:

  keyword1 to keyword3: / xx / level_3_inverted_index_file_name_1

  keyword2-. keywordx: / xx / level_3_inverted_index_file_name_2

  ...

  keywordn to keyword: / xx / level_3_inverted_index_file_name_n

  The keywords may be stored in alphabetical order in different deposit files. In one example, keyword 1 is “abacus”, keyword 3 is “azure”, keyword 2 is “back”, keyword x is “butter”, and so on. Other configurations are possible. After the path file is constructed, it is possible to find the level 3 inverted index file where the keyword is located from the mapping relationship recorded in the path file in the subsequent mailbox search. The system can then determine which mail contains the keyword after fetching the level 3 inverted index file. Thus, the path file defines the path from the keyword to the level 3 inverted index file.

  FIG. 2 is a flowchart illustrating one embodiment of a process for transferring a level 2 inverted index record to a level 3 cache. In some embodiments, process 200 performs step 106 of process 100.

  At step 201, all level 2 transposed index records in the level 2 cache file are transferred to memory (eg, read from the disk storage location to memory).

  At step 202, a level 3 inverted index file corresponding to a keyword in the level 2 inverted index record is determined based at least in part on the path file. As described above, the path file includes keywords and paths associated with each level 3 transposed index file. The path is used to determine the level 3 inverted index file corresponding to the keyword in the level 2 inverted index record.

  At step 203, the level 2 transposed index record is written to the determined level 3 transposed index file according to the keyword.

  In some embodiments, the write speed is increased by using the append mode to perform a write operation on the level 3 transposed index file. Thus, in some embodiments, when a level 2 inverted index record is written to a level 3 inverted index file, all the inverted index records in the level 2 cache file are appended to the determined level 3 inverted index file according to the keyword. Fetched in mode. The append mode is a standard mode used for file editing. In append mode, new data is added directly to the end of the file.

  When the threshold is exceeded, the system fetches the path file. At this point, the system scans the transposed records in the buffer file and fetches the keyword (eg keyword 1) of each record. Next, the path file information is used to learn in which transposed file the keyword1 record should be placed, and then the record is added to the file.

  Embodiments herein use an approach that includes a level 1 cache, a level 2 cache file, and a level 3 transposed index file. The inverted index record established for the mail message is first stored in the level 1 cache; when the level 1 cache reaches the first predetermined threshold, all level 1 inverted index records in the level 1 cache are: Transferred to level 2 cache files; when the level 2 cache files reach a second predetermined threshold, the level 2 transposed index records in those files are transferred to the level 3 transposed index file. This approach allows the system to avoid an excessive number of write operations to the hard disk due to having a large number of inverted index records for mail. As a result, the inverted index record writing speed in the process of establishing the mail index increases. This not only increases the speed of establishing the mail index, but also reduces the impact from an excessive number of write operations to the disk and improves disk IO performance.

  In addition, a portion of the mail index in the present embodiment is stored in a level 1 cache implemented using low latency memory. As a result, the hard disk does not store the entire mail index. Thus, if new mail is processed frequently, the buffer in memory prevents the mail from having an excessive impact on disk writes. In addition, real-time search objectives can be achieved by increasing the speed for the newest data.

  FIG. 3 is a flowchart illustrating another embodiment of a process for establishing a mail index. In some embodiments, process 300 is performed after process 100 is complete.

  In this example, a third predetermined threshold is also established for the level 3 inverted index file; when the size of the level 3 inverted index file reaches the third predetermined threshold file, the file may be multiple (eg, Divided into two) inverted index subfiles, and these inverted index subfiles must be smaller than a third predetermined threshold. In this way, each level 3 transposed index file can be guaranteed not to become excessively large, thus ensuring access speed to the mail index.

  In step 301, it is determined whether the size of the level 3 inverted index file exceeds a third predetermined threshold. If so, control proceeds to step 302. The threshold for the level 3 inverted index file may be the same as or different from the second predetermined threshold for the level 2 cache file.

  In this example, the threshold size configuration for the level 1 cache, level 2 cache file, and level 3 inverted index file is determined by the number of disk read / write operations within a specific time period and the user searching for mail. Considering factors such as how much time is allocated to return user search results when The smaller the file, the faster the read / write speed. However, if the cache size is very small, the number of files becomes excessive and the read / write speed becomes slow. Therefore, the magnitude of the threshold is adjusted empirically according to actual conditions.

  In the current process, if the size of the level 3 inverted index file does not reach the third predetermined threshold (eg, 4 MB), the subsequent process is not executed.

  At step 302, the level 3 inverted index file is divided into multiple (eg, two) inverted index subfiles.

  If the level 3 inverted index file is too large, the level 3 inverted index file can be split based on keywords. The file is divided according to the keyword granularity. That is, the inverted index record corresponding to one keyword becomes only one inverted index subfile. Thus, when a query is subsequently made regarding mail associated with one keyword, only one level 3 inverted index file is fetched. Furthermore, this level 3 inverted index file does not exceed a threshold size (eg, 4 MB), thus ensuring that the mail index is fetched quickly. An example is as follows:

  keyword 1: doc1, doc2, doc3. . .

  keyword 2: doc1, doc3, doc4. . .

  keyword3: doc1, doc6. . .

  ...

  keywordn: doc1, dock. . .

  If this file is too large, it is divided into multiple (for example, two) inverted index subfiles at the record entry boundaries.

  In some embodiments, when a level 3 inverted index file is split into two inverted index subfiles, two factors need to be considered. The first is a keyword granularity for ensuring that an inverted index record corresponding to one keyword exists in one inverted index subfile. The second is the need to take into account the fact that the two inverted index subfiles are preferably as close as possible to each other. In this way, the frequency of subsequent subfile division is minimized.

  At step 303, the path file is updated according to the split inverted index subfile.

  Since the level 3 inverted index file exceeding the third predetermined threshold is divided into two inverted index subfiles each having its own identifier, the inverted index corresponding to various keywords in the original level 3 inverted index file. Records change. Therefore, the path file needs to be updated according to the divided inverted index subfile. After the path file is updated, the original level 3 inverted index file may be deleted.

  FIG. 4 is a flowchart illustrating another embodiment of a process for establishing a mail index. In some embodiments, process 400 is performed after process 100 is complete. In this example, the level 3 inverted index file is configured to have two parts: an initial static compressed file (eg, a zip file) and an incremental file, where the initial static compressed file is a compressed inverted index record. Incremental files store uncompressed inverted index records. In this embodiment, when inverted index records are generated for new mail, these newly generated inverted index records may be written to an incremental file. As a result, mail indexes can be established more quickly and the impact of excessive frequent disk writes can be reduced. Disk IO performance can be improved and storage resources for level 3 inverted index files can be further saved.

  As described above, at the end of the process 100, the level 2 inverted index record is an incremental file determined from among the level 3 inverted index file according to the path file specifying the mapping relationship information of the keyword and the corresponding level 3 inverted index file. Forwarded to Following the process 100, at step 409, it is determined whether the incremental file exceeds the incremental threshold. If so, control proceeds to step 410.

  In this embodiment, a threshold is configured for the incremental file. For example, the incremental file is preferably below the 4MB threshold. When the incremental file reaches a predetermined threshold, the content stored in the incremental file is compressed to reduce the size of the level 3 inverted index file and the amount of hard disk space it occupies. In this step, if the incremental file does not exceed the incremental threshold, the subsequent steps are not performed.

  At step 410, the initial static compressed file is decompressed to obtain a decompressed initial static file. In some embodiments, when the size of the incremental file exceeds the incremental threshold, the contents of the initial static compressed file are first fetched and decompressed to obtain an uncompressed initial static file.

  At step 411, the decompressed initial static file and incremental file are merged to obtain a merge file.

  At step 412, the merge file is compressed using any suitable technique to produce a current static compressed file (eg, a zip file). In some embodiments, the incremental file is released from its original contents when the merge file is compressed to produce the current static compressed file. The purpose of all this is to use compression techniques to lower the storage cost for level 3 inverted index files, but at the same time reduce the computational and disk writing costs that result from compression and decompression that accompany frequent data writes. It is to be.

  FIG. 5 is a flowchart illustrating an embodiment of the mail search process. The process 500 is executed after the mail index is established using the process 100 or the like.

  When a user searches for mail, the first step is to receive the searched keyword from the user. In step 501, the search keyword sent by the user is obtained.

  At step 502, a level 3 transposed index file corresponding to the keyword is determined according to the path file. As described above, this path file stores the mapping relationship that establishes the correspondence between the keyword and the level 3 inverted index file.

  At step 503, a first mail set that includes the keywords is determined based on the level 3 inverted index file.

  The level 3 inverted index file includes keywords that are determined and searched according to the path file. In some embodiments, the level 3 inverted index file is then loaded into a binary tree data structure in memory to facilitate searching. The binary tree data structure is searched to find mail messages that contain the searched keyword. This set of mail messages is considered the first mail set.

  At step 504, a second mail set that includes the keyword is determined based on the level 1 cache.

  In some examples, a level 1 cache may have several emails that contain keywords. If these mails in the cache have not yet reached the first predetermined threshold, they have not been fetched into the level 2 cache file. Therefore, it is necessary to search in the level 1 cache to find these mails, which form a second mail set.

  At step 505, a third mail set that includes the keyword is determined based on the level 2 cache file.

  In some examples, a level 2 cache file may have several emails that contain keywords. If this mail in the level 2 cache file has not yet reached the second predetermined threshold, the mail has not been fetched into the level 3 inverted index file. Therefore, it is necessary to search the level 2 cache file to find these mails, which form a third mail set.

  At step 506, the first mail set, the second mail set, and the third mail set are merged to obtain search results.

  Finally, the sum of the first mail set, second mail set, and third mail set merged after being found separately is the search result for the user's current keyword. The steps may be arranged in other orders, for example, as an alternative, the first, second, and third mail sets are from Level 1 cache, Level 2 cache, and Level 3 cache, respectively. Note that it may correspond to an email message.

  During the mail search in this embodiment, the mail is searched according to the user search keyword in the level 1 cache, the level 2 cache file, and the level 3 inverted index file. Furthermore, the sizes of the level 1 cache, the level 2 cache file, and the level 3 inverted index file do not exceed the respective predetermined thresholds. Therefore, by executing separate searches for these three, it is possible to reduce the read / write frequency of the hard disk, increase the mail search speed, and improve the hard disk IO performance.

  FIG. 6 is a block diagram illustrating one embodiment of a system configured to establish a mail index. The system is configured to perform a process, such as process 100, and includes:

  A word segmentation unit 601 configured to perform word segmentation on an email and obtain keywords for the email.

  A level 1 cache update unit 602 configured to use the current mail keyword as the basis for updating the level 1 inverted index record stored in the level 1 cache.

  A first determination unit 603 configured to determine whether the size of a level 1 inverted index record in the level 1 cache exceeds a first predetermined threshold.

  A first transfer unit 604 configured to transfer all level 1 transposed index records in the level 1 cache to a level 2 cache file if the result of the first determination unit is positive.

  A second determination unit 605 configured to determine whether the current level 2 cache file size exceeds a second predetermined threshold.

  If the result of the second determination unit is affirmative, all the level 2 inverted index records in the level 2 cache file are stored in the level 3 inverted index according to the keyword and the path file storing the mapping relation information of the level 3 inverted index file. A second transfer unit 606 configured to transfer to a file.

  FIG. 7 is a block diagram illustrating an embodiment of a transfer unit. In this example, the system 700 is used to implement the second transfer unit 606 of FIG. 6 and comprises:

  A first fetch module 701 configured to fetch (ie, transfer) all level 2 inverted index records in a level 2 cache file into memory.

  A determination module 702 configured to use a path file as a basis for determining a level 3 inverted index file corresponding to a keyword in a level 2 inverted index record.

  A second fetch module 703 configured to fetch (transfer) level 2 transposed index records to the determined level 3 transposed index file according to the keywords. In some embodiments, the second fetch module 703 is specifically configured to fetch all of the inverted index records in the level 2 cache file to the determined level 3 inverted index file in append mode according to the keyword.

  FIG. 8 is a block diagram of a system configured to establish a mail index. In this example, system 800 comprises the following in addition to an index establishment unit similar to 600 in FIG.

  A third determination unit 801 configured to determine whether the size of the level 3 inverted index file exceeds a third predetermined threshold.

  A split unit 802 configured to split a level 3 inverted index file into two inverted index subfiles if the result of the third determination unit is positive.

  A path file update unit 803 configured to update the path file according to the two split inverted index subfiles.

FIG. 9 is a block diagram illustrating another embodiment of a system configured to establish a mail index. System 900 comprises the following:

  A word splitting unit 601 configured to perform word splitting on a mail message and obtain keywords for the mail message.

  A level 1 cache update unit 602 configured to use mail message keywords as a basis for updating level 1 inverted index records stored in the level 1 cache.

  A first determination unit 603 configured to determine whether the size of a level 1 inverted index record in the level 1 cache exceeds a first predetermined threshold.

  A first transfer unit 604 configured to transfer all level 1 transposed index records in the level 1 cache to a level 2 cache file if the result of the first determination unit is positive.

  A second determination unit 605 configured to determine whether the current level 2 cache file size exceeds a second predetermined threshold.

  A first fetch module 701 configured to fetch all level 2 inverted index records in a level 2 cache file into memory.

  A decision module 702 that uses a path file to determine a level 3 inverted index file corresponding to a keyword in a level 2 inverted index record. In some embodiments, the level 3 inverted index file includes an initial static compressed file and an incremental file.

  A second fetch module 703 configured to fetch level 2 transposed index records into an incremental file according to keywords.

  A fourth determination unit 901 configured to determine whether the incremental file exceeds an incremental threshold;

  A decompression unit 902 configured to decompress an initial static compressed file to obtain a decompressed initial static file if the result of the fourth determination unit is positive.

  A merge unit 903 configured to merge the initial static file and the incremental file to obtain a merge file.

  A compression unit 904 configured to compress the merge file to generate a current static compressed file.

  In this embodiment, the level 3 inverted index file is classified as a compressed initial static file or as an incremental file. Thus, the mail index can be established more quickly and the impact of excessive frequent disk writes can be reduced. Disk IO performance can be improved and storage resources for level 3 inverted index files can be further saved.

  FIG. 10 is a block diagram illustrating an embodiment of a mail search system. The mail search system 1000 cooperates with the mail index establishment system described above and includes the following:

  A keyword acquisition unit 1001 configured to acquire a search keyword transmitted by a user.

  A decision unit 1002 configured to determine a level 3 inverted index file corresponding to the keyword according to a path file that stores a mapping relationship that establishes a correspondence between the keyword and the level 3 inverted index file.

  A first mail set determination unit 1003 configured to determine a first mail set in which a keyword is located from a level 3 inverted index file.

  A second mail set determination unit 1004 configured to determine a second mail set in which the keyword is located from the level 1 cache.

  A third mail set determination unit 1005 configured to determine a third mail set in which the keyword is located from the level 2 cache file.

  A search result acquisition unit 1006 configured to merge the first mail set, the second mail set, and the third mail set to obtain search results.

  During the mail search in this embodiment, the mail is searched according to the user search keyword in the level 1 cache, the level 2 cache file, and the level 3 inverted index file. Furthermore, the sizes of the level 1 cache, the level 2 cache file, and the level 3 inverted index file do not exceed the respective predetermined thresholds. Therefore, by executing separate searches for these three, it is possible to reduce the read / write frequency of the hard disk, increase the mail search speed, and improve the hard disk IO performance.

  The modules / units described above may be implemented as hardware, such as programmable logic devices and / or application specific integrated circuits designed to perform specific functions as software components running on one or more processors, or Can be implemented as a combination of them. In some embodiments, the module / unit may be a non-volatile storage medium (optical It may be embodied in the form of a software product that can be stored on a disk, flash storage device, portable hard disk, etc. Modules / units may be implemented on a single device or distributed across multiple devices. The functions of the modules / units may be integrated with each other or further divided into a plurality of submodules / subunits.

  Although the embodiments described above have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not intended to be limiting.

[Application Example 1]
  A method for processing email messages, comprising:
  Obtaining a set of keywords related to the email message;
  Using one or more computer processors to update a set of transposed index records stored in a level 1 cache based at least in part on the set of keywords;
  Determining whether the size of the set of inverted index records stored in the level 1 cache exceeds a first predetermined threshold;
  Transferring the set of inverted index records in the level 1 cache to a level 2 cache when the first predetermined threshold is exceeded;
  Determining whether the size of the level 2 cache file exceeds a second predetermined threshold;
  When the second predetermined threshold is exceeded, the level 3 cache that stores a set of inverted index files according to a path file that stores mapping relationship information between the keyword and the corresponding level 3 inverted index file Transferring the inverted index record in the level 2 cache file;
A method comprising:
[Application Example 2]
  The method according to application example 1, further comprising:
  Determining whether the size of the level 3 inverted index file exceeds a third predetermined threshold;
  Dividing the level 3 inverted index file into two inverted index subfiles when the size of the level 3 inverted index file exceeds the third predetermined threshold;
  Updating the path file according to the two split inverted index subfiles;
A method comprising:
[Application Example 3]
  In the method described in Application Example 1, the step of transferring the inverted index record in the level 2 cache file to the level 3 inverted index file includes:
  Fetching the level 2 inverted index record in the level 2 cache file into memory;
  Determining the level 3 inverted index file corresponding to the keyword in the level 2 inverted index record based on the path file;
  Transferring the level 2 inverted index record to the determined level 3 inverted index file according to the keyword;
Including a method.
[Application Example 4]
  The method according to application example 3, wherein the step of transferring the inverted index record in the level 2 cache file to the determined level 3 inverted index file according to the keyword is performed in an append mode. .
[Application Example 5]
  The method of application example 3, wherein one level 3 inverted index file comprises an initial static compressed file and an incremental file.
[Application Example 6]
  The method according to application example 5, wherein the step of transferring the inverted index record in the level 2 cache file to the level 3 inverted index file according to the keyword includes
  Transferring the level 2 inverted index record to the incremental file according to the keyword;
  Determining whether the incremental file exceeds an incremental threshold;
  Decompressing the initial static compressed file to obtain a decompressed initial static file when the incremental threshold is exceeded;
  Merging the unzipped initial static file and the incremental file to obtain a merge file;
  Compressing the merge file to generate a current static compression;
Including a method.
[Application Example 7]
  The method according to application example 1, further comprising:
  Obtaining a set of one or more search keywords sent by a user;
  Determining a level 3 inverted index file corresponding to the search keyword sent by the user according to the path file;
  Based on the level 3 inverted index file, a first mail set including the search keyword is converted to a second mail set including the search keyword based on the level 1 cache, and then the level 2 cache file is stored. Based on, determining a third mail set that includes the search keyword;
  Merging the first mail set, the second mail set, and the third mail set to obtain a search result;
A method comprising:
[Application Example 8]
  A system for processing email messages,
  One or more processors,
    Retrieve a set of keywords related to an email message,
    Updating a set of inverted index records stored in a level 1 cache based at least in part on the set of keywords;
    Determining whether the size of the set of transposed index records stored in the level 1 cache exceeds a first predetermined threshold;
    Transferring the set of inverted index records in the level 1 cache to a level 2 cache when the first predetermined threshold is exceeded;
    Determine whether the size of the level 2 cache file exceeds a second predetermined threshold;
    When the second predetermined threshold is exceeded, the level 3 cache that stores a set of inverted index files according to a path file that stores mapping relationship information between the keyword and the corresponding level 3 inverted index file One or more processors configured to transfer an inverted index record in a level 2 cache file;
  One or more memories connected to the one or more processors and configured to provide instructions to the one or more processors;
A system comprising:
[Application Example 9]
  The system according to application example 8, wherein the one or more processors further include:
  Determine whether the size of the level 3 inverted index file exceeds a third predetermined threshold;
  Dividing the level 3 inverted index file into two inverted index subfiles when the size of the level 3 inverted index file exceeds the third predetermined threshold;
  A system configured to update the path file according to the two divided inverted index subfiles.
[Application Example 10]
  The system according to application example 8, wherein the transfer index record in the level 2 cache file is transferred to the level 3 transfer index file.
  Fetching the level 2 inverted index record in the level 2 cache file into memory;
  Determining the level 3 inverted index file corresponding to the keyword in the level 2 inverted index record based on the path file;
  Transferring the level 2 inverted index record to the determined level 3 inverted index file according to the keyword.
[Application Example 11]
  The system according to application example 10, wherein the transfer of the inverted index record in the level 2 cache file to the determined level 3 inverted index file according to the keyword is executed in an append mode. .
[Application Example 12]
  The system of application example 10, wherein one level 3 inverted index file comprises an initial static compressed file and an incremental file.
[Application Example 13]
  The system according to Application Example 12, wherein the transfer index record in the level 2 cache file is transferred to the level 3 transfer index file according to the keyword.
  Transferring the level 2 inverted index record to the incremental file according to the keyword;
  Determining whether the incremental file exceeds an incremental threshold;
  If the incremental threshold is exceeded, decompress the initial static compressed file to obtain a decompressed initial static file;
  Merge the unzipped initial static file and the incremental file to obtain a merge file,
  Compressing the merge file to generate a current static compression.
[Application Example 14]
  The system according to application example 8, wherein the one or more processors further include:
  Get a set of one or more search keywords sent by the user,
  Determining a level 3 inverted index file corresponding to the search keyword sent by the user according to the path file;
  Based on the level 3 inverted index file, a first mail set including the search keyword is converted to a second mail set including the search keyword based on the level 1 cache, and then the level 2 cache file is stored. And determining a third mail set that includes the search keyword,
  A system configured to merge the first mail set, the second mail set, and the third mail set to obtain a search result.
[Application Example 15]
  A computer program product for processing e-mail messages, embodied in a tangible computer readable storage medium,
  Computer instructions for obtaining a set of keywords associated with the email message;
  Computer instructions for updating a set of inverted index records stored in a level 1 cache based at least in part on the set of keywords;
  Computer instructions for determining whether the size of the set of transposed index records stored in the level 1 cache exceeds a first predetermined threshold;
  Computer instructions for transferring the set of inverted index records in the level 1 cache to a level 2 cache when the first predetermined threshold is exceeded;
  Computer instructions for determining whether the size of the level 2 cache file exceeds a second predetermined threshold;
  When the second predetermined threshold is exceeded, the level 3 cache that stores a set of inverted index files according to a path file that stores mapping relationship information between the keyword and the corresponding level 3 inverted index file Computer instructions for transferring the inverted index record in the level 2 cache file;
A computer program product comprising:

Claims (15)

A method for processing email messages, comprising:
Obtaining a set of keywords related to the email message;
Using one or more computer processors to update a set of transposed index records stored in a level 1 cache based at least in part on the set of keywords;
Determining whether the size of the set of inverted index records stored in the level 1 cache exceeds a first predetermined threshold;
Transferring the set of inverted index records in the level 1 cache to a level 2 cache when the first predetermined threshold is exceeded;
Determining whether the size of the level 2 cache file exceeds a second predetermined threshold;
When the second predetermined threshold is exceeded, the level 3 cache that stores a set of inverted index files according to a path file that stores mapping relationship information between the keyword and the corresponding level 3 inverted index file Transferring the inverted index record in the level 2 cache file;
A method comprising: The method of claim 1, further comprising:
Determining whether the size of the level 3 inverted index file exceeds a third predetermined threshold;
Dividing the level 3 inverted index file into two inverted index subfiles when the size of the level 3 inverted index file exceeds the third predetermined threshold;
Updating the path file according to the two split inverted index subfiles;
A method comprising: The method of claim 1, wherein transferring the inverted index record in the level 2 cache file to a level 3 inverted index file comprises:
A step of fetching the level 2 inverted index records of the level 2 cache file in memory,
Determining the level 3 inverted index file corresponding to the keyword in the level 2 inverted index record based on the path file;
Transferring the level 2 inverted index record to the determined level 3 inverted index file according to the keyword;
Including the method.   4. The method of claim 3, wherein the step of transferring the inverted index record in the level 2 cache file to the determined level 3 inverted index file according to the keyword is performed in append mode. .   4. The method of claim 3, wherein one level 3 inverted index file comprises an initial static compressed file and an incremental file. 6. The method of claim 5, wherein the step of transferring the inverted index record in the level 2 cache file to the level 3 inverted index file according to the keyword comprises:
Transferring the level 2 inverted index record to the incremental file according to the keyword;
Determining whether the incremental file exceeds an incremental threshold;
Decompressing the initial static compressed file to obtain a decompressed initial static file when the incremental threshold is exceeded;
Merging the unzipped initial static file and the incremental file to obtain a merge file;
Compressing the merge file to generate a current static compression;
Including the method. The method of claim 1, further comprising:
Obtaining a set of one or more search keywords sent by a user;
Determining a level 3 inverted index file corresponding to the search keyword sent by the user according to the path file;
Based on the level 3 inverted index file, a first mail set including the search keyword is converted to a second mail set including the search keyword based on the level 1 cache, and then the level 2 cache file is stored. Based on, determining a third mail set that includes the search keyword;
Merging the first mail set, the second mail set, and the third mail set to obtain a search result;
A method comprising: A system for processing email messages,
One or more processors,
Retrieve a set of keywords related to an email message,
Updating a set of inverted index records stored in a level 1 cache based at least in part on the set of keywords;
Determining whether the size of the set of transposed index records stored in the level 1 cache exceeds a first predetermined threshold;
Transferring the set of inverted index records in the level 1 cache to a level 2 cache when the first predetermined threshold is exceeded;
Determine whether the size of the level 2 cache file exceeds a second predetermined threshold;
When the second predetermined threshold is exceeded, the level 3 cache that stores a set of inverted index files according to a path file that stores mapping relationship information between the keyword and the corresponding level 3 inverted index file One or more processors configured to transfer an inverted index record in a level 2 cache file;
One or more memories connected to the one or more processors and configured to provide instructions to the one or more processors;
A system comprising: 9. The system of claim 8, wherein the one or more processors are further
Determine whether the size of the level 3 inverted index file exceeds a third predetermined threshold;
Dividing the level 3 inverted index file into two inverted index subfiles when the size of the level 3 inverted index file exceeds the third predetermined threshold;
A system configured to update the path file according to the two divided inverted index subfiles. 9. The system of claim 8, wherein transferring the inverted index record in the level 2 cache file to a level 3 inverted index file is:
The level 2 inverted index records of the level 2 cache file fetched in the memory,
Determining the level 3 inverted index file corresponding to the keyword in the level 2 inverted index record based on the path file;
Transferring the level 2 inverted index record to the determined level 3 inverted index file according to the keyword.   11. The system of claim 10, wherein transferring the inverted index record in the level 2 cache file to the determined level 3 inverted index file according to the keyword is performed in append mode. .   12. The system of claim 10, wherein one level 3 inverted index file comprises an initial static compressed file and an incremental file. 13. The system of claim 12, wherein transferring the inverted index record in the level 2 cache file to the level 3 inverted index file according to the keyword is:
Transferring the level 2 inverted index record to the incremental file according to the keyword;
Determining whether the incremental file exceeds an incremental threshold;
If the incremental threshold is exceeded, decompress the initial static compressed file to obtain a decompressed initial static file;
Merge the unzipped initial static file and the incremental file to obtain a merge file,
Compressing the merge file to generate a current static compression. 9. The system of claim 8, wherein the one or more processors are further
Get a set of one or more search keywords sent by the user,
Determining a level 3 inverted index file corresponding to the search keyword sent by the user according to the path file;
Based on the level 3 inverted index file, a first mail set including the search keyword is converted to a second mail set including the search keyword based on the level 1 cache, and then the level 2 cache file is stored. And determining a third mail set that includes the search keyword,
A system configured to merge the first mail set, the second mail set, and the third mail set to obtain a search result. A computer program for processing email messages,
A function to retrieve a set of keywords related to an email message;
A function for updating a set of inverted index records stored in a level 1 cache based at least in part on the set of keywords;
A function for determining whether the size of the one set of inverted index records stored in the level 1 cache exceeds a first predetermined threshold;
A function for transferring the set of inverted index records in the level 1 cache to a level 2 cache when the first predetermined threshold is exceeded;
A function for determining whether the size of the level 2 cache file has exceeded a second predetermined threshold;
When the second predetermined threshold is exceeded, the level 3 cache that stores a set of inverted index files according to a path file that stores mapping relationship information between the keyword and the corresponding level 3 inverted index file A function for transferring inverted index records in level 2 cache files;
A computer program that causes a computer to realize

Images