JP5657498B2 - File search system - Google Patents

Description

  The present invention relates to a technique for efficiently generating, updating, and managing a search index for a large group of files.

  With the recent diversification of applications and lower storage costs, the amount of data stored in storage has increased explosively. Along with this, the amount of document data handled in the company has become enormous. For this reason, the importance of a search system for effectively utilizing a large amount of data is increasing.

  Usually, when the number of documents to be searched is enormous, search performance is improved by generating search indexes (index data) in advance. In addition, methods such as installing the same search index on multiple search servers to distribute the load and distributing search processing by dividing search indexes on multiple search servers to improve search performance Generally used as a method.

  In such a technical background, various techniques have been proposed as a search index generation method. For example, Patent Document 1 discloses a technique for reducing the size deviation of the divided search indexes as much as possible.

JP 2011-70257 A

Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Webhttp: //www.akamai.com/dl/technical_publications/ConsistenHashingandRandomTreesDistributedCachingprotocolsforrelievingHotSpotsontheworldwideweb.pdf

  However, in consideration of the current IT situation, the amount of data to be searched is considered to increase further in the future. In addition, the number of search servers and the number of divided indexes are easily expected to become enormous. Therefore, the inventors think that a mechanism capable of generating a split index at high speed will be required in the future.

  Therefore, the inventors have a first processing function unit that generates a list of file paths to be generated for the split index to be assigned to each search server for the purpose of high-speed generation of the split index among the above-described problems, and the list A processing function between the second processing function unit that generates the split index based on the third processing function unit that places the generated split index in the search server, and the first to third processing function units. A file search system having a fourth processing function unit realized by pipeline processing is proposed.

  According to the present invention, a split index can be generated at high speed. Problems, configurations, and effects other than those described above will become apparent from the following description of embodiments.

The figure which shows the conceptual structure of the search system which concerns on embodiment. The figure which shows the function structural example of a search server. The figure which shows the function structural example of a distributed processing server. The figure which shows the function structural example of a management server. The figure which shows the example of a data structure of an index ID table. The figure which shows the data structure example of a search server management table. The figure which shows the data structure example of a file management table. The figure which shows the initialization flow of a system. The figure which shows the initialization flow of an index ID table. The figure explaining the example of index ID table which initialization was complete | finished. The figure which shows the production | generation flow of the index list by a scanner module. The figure which shows the production | generation flow of the division | segmentation index by an index production | generation module. The figure which shows the arrangement | positioning flow of the division | segmentation index to a search server. The figure which shows the processing flow at the time of the addition of a search server. The figure which shows the processing flow at the time of deletion of a search server.

  In the following embodiment, processing functions necessary for realizing the search system according to the embodiment will be described by being divided into a plurality of sections. In the following embodiments, when referring to the number of elements, etc. (including the number, numerical value, quantity, range, etc.), unless otherwise specified and in principle limited to a specific number in principle, It is not limited to the specific number, and may be more or less than the specific number. In the following embodiments, the constituent elements (including element steps and the like) are not necessarily indispensable unless otherwise specified or apparently essential in principle.

  Further, in the following embodiments, each configuration, function, processing unit, processing unit, and the like may be partially or entirely realized as, for example, an integrated circuit or other hardware. Further, each configuration, function, and the like described above may be realized by a processor interpreting and executing a program that realizes each function. That is, it may be realized as software. Information such as programs, tables, and files for realizing each function can be stored in a memory, a hard disk, a storage device such as an SSD (Solid State Drive), or a storage medium such as an IC card, an SD card, or a DVD.

  Control lines and information lines indicate what is considered necessary for the description, and do not represent all control lines and information lines necessary for the product. In practice, it can be considered that almost all components are connected to each other.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same or related reference symbols throughout the drawings for describing the embodiments, and the repetitive description thereof is omitted. In the following embodiments, the description of the same or similar parts will not be repeated in principle unless particularly necessary.

[Entire configuration of search system]
FIG. 1 shows a configuration example of a search system according to this embodiment. The search system according to this embodiment includes a search client 100, a search server 101, a file server 102, a distributed processing server 103, and a management server 104, which are connected to each other through a network 105. The network 105 can be realized using a network generally known as a local area network (LAN), a wide area network (WAN), or the like. The network 105 may be a wired network or a wireless network. Further, the search system does not need to be constructed in one region / country, and may be constructed across a plurality of regions / countries.

Search client configuration
The search client 100 is a computer in which an environment capable of operating a Web browser is installed, and is not limited to a stationary type, but includes a portable computer, a portable information terminal, and a mobile phone terminal. The search client 100 uses a HTTP (Hypertext Transfer Protocol) or the like to transmit a search query to the search server 101, a function to acquire a search result from the search server 101, and the acquired search result to the user. A display function. A plurality of search clients 100 exist on the search system.

Search server configuration
FIG. 2 shows an internal configuration example of the search server 101. The search server 101 is a server that receives a search query from the search client 100, executes a search process, and returns a search result. A plurality of search servers 101 exist in the search system, and each holds a local storage 201. In the local storage 201, a search split index 202 generated based on a file group stored in the file server 102 is stored.

  In the search server 101, an index management module 203 and a search module 204 are installed. The index management module 203 is a program for managing / updating the split index 202. The search module 204 is a program that executes a search process using a search split index. Incidentally, the index management module 203 and the search module 204 are installed in each of the search servers 101.

  The split index 202 is a search index generated by the index generation module 403 and the distributed processing server 103 on the management server 104 based on a file group stored on the file server 102. As will be described later, the divided index 202 is an index divided for each index ID based on the consistent hash method. Note that a divided index 202 is associated with the index ID, and the divided index 202 is arranged on the search server 101 through this association. The number of divided indexes 202 (number of divided indexes) to be arranged on the search server 101 is determined in advance by the administrator.

  The index management module 203 is a module that arranges and manages the split index 202 in the search server 101. When the split index 202 is newly generated, the index management module 203 downloads the split index 202 to the local disk 201 of the search server 101 and stores it.

  When the split index 202 already exists in the search server 101 and the update operation of the split index 202 is executed, the index management module 203 merges the newly generated split index with the existing split index 202. Generate the latest split index.

  When the number of split indexes held in the entire system increases due to the addition of the search server 101, the index management module 203 has a function of further splitting the existing split index 202 stored in each search server 101. Have. The newly added index management module 203 of the search server 101 has a function of aggregating indexes newly divided by other search servers 101 and generating one divided index 202.

  The index management module 203 in the search server 101 to be deleted has a function of dividing the divided index 202 held in its own server again according to the index ID and allocating it to the divided indexes 202 of other search servers 101.

  The search module 204 is a search engine that has a function of generating a search result for a search query received from the search client 100 using the split index 202 arranged in the search server 101 and returning the search result to the search client 100. is there. The search module 204 also has a function of executing search processing in a distributed manner in cooperation with each search module 204 installed in another search server group.

[File Server Configuration]
The file server 102 is a server that stores a large amount of document data created in a company or the like. A plurality of file servers 102 exist in the search system. Each file server 102 is connected to the distributed processing server 103 and the management server 104 through a protocol such as NFS (Network File System) or CIFS (Common Internet File System). Thereby, each module on the distributed processing server 103 and the management server 104 can access a file existing on the file server 102 and acquire file information.

[Configuration of distributed processing server]
FIG. 3 shows an internal configuration example of the distributed processing server 103. A plurality of distributed processing servers 103 exist in the search system. The plurality of distributed processing servers 103 is a server group having a function of processing one processing instruction in a distributed manner in cooperation with other servers.

  A distributed file system 302 and a distributed processing module 303 are installed in the distributed processing server 103. The distributed processing server 103 is provided with a local storage 301. The distributed file system 302 is a module that uses the local storage 301 and makes a common file system available from all the distributed processing servers 103. The distributed processing module 303 is a module having a function of generating the split index 202 in a distributed manner in cooperation with other distributed processing servers 103 when receiving an instruction from the index generation module 402 of the management server 104.

[Management server configuration]
FIG. 4 shows an internal configuration example of the management server 104. The management server 104 is a server having a server management function such as a search server 101, a file server 102, and a distributed processing server 103 that constitute the search system. The local storage 401 of the management server 104 includes a scanner module 402, an index generation module 403, a pipeline control module 404, a system management module 405, an index ID table 406, a search server management table 407, for controlling the generation of divided indexes. A file management table 408 is installed. These modules may exist in addition to the management server 104. For example, all or some of these modules may be directly operable on the distributed processing server 103.

  The scanner module 402 scans a file / directory on the file server 102 to acquire a list of file / folder path names and their attribute information, and the file / folder is newly created / updated / deleted. It is a module having a function of determining whether or not the file is in the state and generating an index list in which the file path that is the target of the index is described.

  The functions of the scanner module 402 can be realized by executing the following processing functions. For example, by using a Linux Find command, a list of file / directory paths on the file server 102 and their attribute information are acquired. Thereafter, a hash value of the acquired file attribute information is calculated. Next, the hash value 702 (FIG. 7) of the file attribute information stored in the file management table 408 (described later) acquired at an arbitrary timing is compared with the calculated hash value. It is determined whether or not it is an index target.

  If the hash values are the same, the scanner module 402 determines that the corresponding file / directory has not been updated, and excludes it from the indexing target. If the hash values are different, the scanner module 402 determines that the file / directory has been updated, and sets it as an indexing target.

  If the file management table 408 has a file / folder path 701 but cannot be obtained by the Find command, the scanner module 402 writes information to the index list so that the file path indicates file deletion.

  The index list is a text file that describes the file path and processing status of the index processing target. The file path and the processing status described in the index list are temporary files generated by the scanner module 402 extracted from the file management table 408, and are used by the index generation module 403 described later.

  The scanner module 402 does not scan from the root to the deepest part of the file system on each file server 102 at once, but outputs an index list for each folder hierarchy or for each arbitrary folder hierarchy, and the index generation module 403 and the index generation module 403 Pipeline processing is executed between the management modules 203. As a result, before the scanner module 402 completes scanning of the file server 102, the index generation module 403 and the index management module 203 can start index generation / update processing, thereby increasing the index generation speed. It can be realized.

  The index generation module 403 is a module having a function of causing the distributed processing server 103 to generate indexes in a distributed manner based on the index list output by the scanner module 402. The index generation module 403 calculates a hash value corresponding to the file path based on the consistent hash method, and obtains a corresponding index ID from the hash value. Also, the index generation module 403 generates a divided index for each index ID.

  The processing of the index generation module 403 is divided into processing units called tasks and distributed to a plurality of distributed processing servers 103. Note that the task is executed on the distributed processing server 103 separately into the first distributed processing and the second distributed processing. These processes can also be realized by using MapReduce, which is known as a large-scale distributed processing technique. In that case, the first distributed processing is realized as Map processing, and the second distributed processing is realized as Reduce processing. Detailed operation will be described later.

  The pipeline control module 404 is a module for controlling the processes of the scanner module 402, the index generation module 403, and the index management module 203 in a multiplexed manner in order to speed up the index generation. Detailed operations regarding pipeline control of each module will be described later.

  The system management module 405 includes a function for managing a group of servers existing on the search system and initializing various tables, and a function for providing a user interface for an administrator to input parameters relating to system initialization. It is a module which has.

  An example of the index ID table 406 is shown in FIG. The index ID table 406 stores a virtual index ID 501 and an index ID 502, and is used to acquire an index ID from a file path. The index ID table 406 is used as a means for realizing the consistent hash method.

  The following describes the consistent hash method. The consistent hash method has an integer scale of 0 to 2 ^ 128-1 (2 ^ 128 is a value based on the MD5 hash method. MD5 is an example and any hash algorithm can be used). A hash value of the index ID is obtained and arranged on the circled circle, and the range on the circle is divided. Note that acquiring the hash value of the index ID means applying a hash function such as MD5 using the index ID as a character string.

  In order to obtain the index ID from the file path, the hash value is obtained from the file path by using the same hash function (MD5 in this example), arranged on the circumference, and rotated counterclockwise from that position first. The index ID corresponding to the encountered hash value is the index ID associated with the file path. The above is the basic concept of consistent hash. However, in the simple consistent hash method, the number of files assigned to each index ID depends on the interval divided on the circumference.

  For this reason, if only the hash value of the index ID is divided, when the index ID is added / deleted, the number of files allocated to each index ID is biased. This means that the index size is biased among the respective split indexes, and the search performance is deteriorated. For this reason, it is necessary to level the index size.

  In order to perform leveling, it is necessary to shorten the interval between the points corresponding to the index IDs arranged on the circumference. Therefore, a virtual index ID corresponding to the virtual node of the consistent hash method is generated. The virtual index ID is a hash value linked to the index ID, and n virtual index IDs are generated for each index ID, and the size is leveled between the respective divided indexes existing on the system. A method for generating and using the virtual index ID will be described later.

  An example of the search server management table 407 is shown in FIG. The search server management table 407 includes an index ID 601, an arrangement destination search server name 602 in which a divided index associated with the index ID is arranged, a path 603 for storing the divided index, and a storage destination for the deletion index list. It is a table in which a path 604 is stored. The deletion index list 604 is a temporary file generated by the index generation module 403, and is a text file in which the file path to be deleted is written for each line in the divided index 202 already arranged on the search server 101. is there.

  An example of the file management table 408 is shown in FIG. The file management table 408 is a table for storing and managing a list of file / folder path names 701 existing on the file server 102, their attribute information, and a hash value 702 generated from the attribute information. The hash value 702 stored in the table is compared with the hash value 702 generated from the attribute information of the file acquired when the scanner module 402 executes the scan, and the file update state (processing status) 703 is checked.

[Search server management table initialization flow]
FIG. 8 shows an initialization flow of the search server management table 407. Here, it is assumed that there are two search servers 101 and two divided indexes 202 are arranged on each search server 101. That is, it is assumed that the number of index divisions in the entire search system is 4 (= 2 × 2). Further, it is assumed that the names of the two search servers are “Search 1” and “Search 2”.

  First, in order to initialize the search server management table 407, the administrator sets the number of index divisions based on the number of search servers 101 and the number of split indexes 202 arranged on each search server 101 (S801). ).

  As described above, in this description, two divided indexes 202 are arranged on each of the two search servers 101. For this reason, the total number of index divisions is four. When this information is input to the system management module 405, the system management module 405 determines an index ID to be allocated to each divided index 202 (S802). In this specification, the index ID is an ascending number starting from 0. That is, the system management module 405 allocates index IDs in the order of “0”, “1”, “2”, “3”.

  Next, the system management module 405 associates each index ID with the search server 101 (S803), and stores the result in the search server management table 407 (S804). In this embodiment, the system management module 405 automatically associates the index ID with the search server, but may be set manually by the administrator.

  For example, in this embodiment, the entries of the search server management table 407 include “index ID = 0, placement destination search server name = Search 1”, “index ID = 1, placement destination search server name = Search 1”, “index ID = 2, “location destination search server name = Search 2” and “index ID = 3, placement destination search server name = Search 2”. Note that the split index storage destination path 603 and the deletion index list storage destination path 604 are blank at the stage after initialization. Thus, the initialization of the search server management table 407 is completed.

[Index ID table initialization flow]
FIG. 9 shows an initialization flow of the index ID table 406. The initialization of the index ID table 406 is also executed at the same timing as the initialization of the search server management table 407.

  First, the administrator sets the number of index divisions based on the number of search servers 101 and the number of split indexes arranged on each search server 101 (S901), and determines an index ID (S902).

  Also here, it is assumed that there are four index IDs of “0”, “1”, “2”, and “3”. Note that the number of virtual index IDs is 2 for one index ID. The number of virtual index IDs is an arbitrary fixed value determined so that the number of files finally linked to the index IDs is leveled.

  Next, the system management module 405 generates an arbitrary virtual index ID for one index ID (S903). For example, the virtual index IDs associated with the index ID “0” are “0-0” and “0-1”, and the virtual index IDs associated with the index ID “1” are “1-0”, “1-1”, and the index. The virtual index IDs associated with the ID “2” are “2-0” and “2-1”, and the virtual index IDs associated with the index ID “3” are “3-0” and “3-1”.

  Subsequently, the system management module 405 acquires a hash value from the character string of the virtual index ID (S904). Thereafter, the system management module 405 stores the acquired hash value in the column of the virtual index ID 501 of the index ID table 406, and stores the index ID associated with this virtual index ID in the column of the index ID 502 of the entry. (S905).

  FIG. 10 shows an example of the index ID table 406 that has been initialized. By using this table, when a file path is given, it is possible to know which index ID the file path is associated with. For example, when the hash value of the file path “/FileServer1/test.txt” is obtained and found to be “29999999999”, this hash value is arranged between the points of item number 3 and item number 4, Hit the entry point (when turning to the left on the consistent hash circumference). Since the index ID of item number 3 is “3”, it can be seen that the index ID of the file path “/FileServer1/test.tx” is “3”.

  This table is an implementation method of the consistent hash method. When an index ID is obtained from a file path based on this table and a split index is generated for each index ID, the size of each split index or the number of files to be linked Leveling is realized.

[Index list generation flow]
FIG. 11 shows an index list generation flow by the scanner module 402. First, the pipeline control module 404 instructs the scanner module 402 to start generating an index list in the first layer of the folder tree (S1101). As described above, the generation of the index list is not limited to one layer, but may be executed for any number of layers.

  Next, the scanner module 402 accesses the file management table 408 and checks whether a file group of the designated hierarchy exists (S1102). If there is an entry in the file path of the designated hierarchy, the scanner module 402 sets “−1” indicating deletion in the processing status column (S1103). If there is no entry in the file path of the specified hierarchy, the scanner module 402 skips S1103.

  After that, the scanner module 402 executes a Find command with a file search hierarchy designation option (S1104). This can be implemented by setting maxdepth = 1 (when the hierarchical depth is 1) in the Find command on an actual Linux OS.

  When the file / folder path of the designated hierarchy and its attribute information are acquired, the scanner module 402 acquires a hash value based on each attribute information (S1105).

  Subsequently, the scanner module 402 uses the file path acquired by Find as a key, and inquires of the file management table 408 whether the file path exists (S1106).

  If the file path does not exist in the file management table 408 (No in S1106), it means that the file is newly created. Therefore, in this case, the scanner module 402 newly creates an entry with the file path 701 as a key in the file management table 408, and adds “1” indicating new creation to the file hash 702 and the processing status 703 (S1107). ).

  On the other hand, if the file path 701 exists in the file management table 408 (Yes in S1106), it means that the file is already registered in the file management table 408. In this case, the scanner module 402 executes a hash value check (S1108). Specifically, the scanner module 402 acquires the file hash 702 of the entry with the matching file path 701 from the file management table 408 and compares it with the hash value acquired by the Find command.

  If the hash values match (Yes in S1108), it means that there was no file update. Accordingly, in this case, the scanner module 402 sets “0” to the processing status of the entry with the matching file path (S1109).

If the hash values do not match (No in S1108), it means that the file has been updated. Therefore, in this case, the scanner module 402 overwrites the file hash 702 with a new hash value, and overwrites “2” indicating that the file has been updated in the processing status 703 (S1110).

  With the above processing, the file processing of the designated hierarchy (“0” = no processing, “1” = new index generation, “2” = index update, “−1” = deletion from the index) is confirmed.

  Next, the scanner module 402 accesses the file management table 408, acquires an entry whose processing status 703 is “1”, “2”, or “−1” in the entry of the designated folder hierarchy, and indexes it. The data is written in the list and stored on the distributed file system 302 (S1111). That is, only files that have changed are extracted. The index list is a text file that describes the file path and processing status of the index processing target.

  Thereafter, the scanner module 402 notifies the pipeline control module 404 of the storage destination path of the index list and the end of generation (S1112).

  Thereafter, the scanner module 402 acquires the directory entry designated by the pipeline control module 404 from the file management table 408, and generates an index list while increasing the folder depth to 2, 3,.

[Flow of split index generation]
FIG. 12 shows a generation flow of the split index 202 by the index generation module 403. The index generation module 403 generates the split index 202 based on the index list given from the scanner module 402. The processing of the index generation module 403 is divided into a plurality of processing units called tasks for one index list, and is distributedly processed on a plurality of distributed processing servers 103. The task generation and processing on the distributed processing server will be described below.

  First, the scanner module 402 notifies the pipeline control module 404 of the end of index list generation (S1201). At this time, the pipeline control module 404 checks whether or not index generation can be started on the distributed processing server 103 (S1202).

  If index generation can be started on the distributed processing server 103 (Yes in S1202), the pipeline control module 404 notifies the index generation module 403 of the generation start of the split index and the storage path of the index list. (S1203). If index generation cannot be started (No in S1202), the pipeline control module 404 returns to the determination process in S1202 again after a predetermined waiting time (S12021).

  Receiving the previous notification, the index generation module 403 acquires an index list from the distributed file system 302 (S1204). The index generation module 403 performs the following processing from S1205 to S1207 as the first distributed processing.

  First, the index generation module 403 divides the index list into an arbitrary number (S1205). The number here is a number determined from the number of distributed processing servers 103 and the processing performance. An index list is a text file that describes the file path and processing status of the index processing target. When this file is divided, multiple index lists are generated by simply dividing the file according to the number of divisions. Will be.

  Each of the divided index lists is processed as a plurality of tasks on the distributed processing server 103. Each task process in the first distributed process acquires the file path described in the divided index list (S1206), inquires the index ID table, and acquires the index ID (S1207).

  When all of the first task processing is completed, the index generation module 403 performs grouping based on the index ID on the distributed processing server 103, and generates an index list using the index ID as a key (S1208).

  Next, as a second distributed process, the index generation module 403 performs the following processes from S1209 to S1212.

  First, the index generation module 403 starts processing as a plurality of tasks on the distributed processing server 103 for an index list (there is a list corresponding to the index ID) using the index ID as a key.

  In the task processing in the second distributed processing, the file path and the processing status are acquired from the index list using the index ID as a key (S1209).

  Next, the task processing checks the processing status (S1210). If the processing status is “1” (= new file creation) or “2” (= file update), each task creates a split index after downloading the file from the file server 102 (S1211). . Note that the split index generated at this time is temporarily generated on the local storage 301 of the distributed processing server 103.

  On the other hand, when the processing status is “−1” (= deleted from the index), each task outputs the file path as a deletion index list as a deletion index list (S1212). The deletion index list is a text file in which a file path to be deleted from a split index already arranged on the search server 101 is written for each line.

  Thereafter, the index generation module 403 uploads the divided index and the deletion index list generated by the second task processing as a set to the distributed file server 103 (S1213).

  After that, the index generation module 403 stores the uploaded storage destination in the split index storage destination path 603 and the deletion index list storage destination path 604 (S1214), and notifies the pipeline control module 404 of the completion of split index generation. (S1215).

  As described above, on the distributed processing server 103, the first distributed processing and the second distributed processing are executed, and the task processing is executed simultaneously and in parallel. Thereby, the generation speed of the split index is improved. Note that the number of distributed processes can be further adjusted by executing the second task process using the virtual index ID in the consistent hash method.

  Further, the scanner module 402 and the index generation module 403 operate asynchronously. For this reason, when a plurality of index list generations by the scanner module 402 are completed, the processing of S1201 to S1213 can be multiplexed, and the generation speed of the split index is improved.

[Flow of split index allocation to search servers]
FIG. 13 shows the split index generated by the index generation module 403 (at this time, the split index is stored on the distributed file system 302 instead of the search server 101), and the index management module 203 stores the search server. 101 is a flow to be arranged in 101.

  The flow shown in FIG. 13 starts when the pipeline control module 404 receives a generation end notification of the split index 202 from the index generation module 403 (S1301). The pipeline control module 404 that has received this notification makes an inquiry to the search server management table 407, and acquires the location search server name 602 using the index ID as a key (S1302).

  Next, the pipeline control module 404 makes an inquiry to the index management module 203 on the specified search server 101 as to whether or not index processing is possible (S1303). If the index process is possible (Yes in S1303), the pipeline control module 404 instructs the index management module 203 to start the index process (S1304). If the index management module 203 is executing another process, the pipeline control module 404 waits for a certain time (S1305).

  Next, the index management module 203 checks whether or not a split index already exists (S1306). If the split index 202 already exists on the same search server 101 (Yes in S1306), the index management module 203 downloads the split index 202 and the deletion index list corresponding to the index ID from the distributed file system 302 (S1307). ).

  The index management module 203 deletes the index from the existing split index 202 existing on the search server 101 based on the delete index list (S1308). Next, the index management module 203 merges the downloaded divided index 202 with the existing divided index 202 to generate the latest divided index 202 (S1309).

  On the other hand, if the split index 202 does not exist on the same search server 101 (No in S1306), the index management module 203 downloads the split index 202 corresponding to the index ID from the distributed file system 302 (S1310). .

  Subsequently, the index management module 203 requests the search module 204 to mount the split index 202 (S1311). As a result, the split index is mounted on the search module 204, and the search can be executed.

  Finally, the index management module 203 notifies the pipeline control module 404 of the end of the allocation of the split index and completes the processing (S1312).

[Search server addition flow]
FIG. 14 shows a processing flow executed when the search server 101 is added to the search system.

  This processing flow starts when the administrator inputs addition of the search server 101 to the system management module 405 (S1401).

  Upon accepting that the search server 101 has been added, the system management module 405 assigns a new index ID to the newly added search server 101 (S1402). For example, when one search server 101 is newly added to a search system in which two search servers 101 are arranged, and when two divided indexes 202 are arranged in one search server 101, Index IDs 4 and 5 are assigned to the newly added search server 101.

  Next, the system management module 405 creates an entry for the newly generated index ID 601 in the search server management table 407, and sets the location search server name 602 in the entry (S1403). That is, the search server management table 407 is initialized.

  Thereafter, the system management module 405 stores the hash value of the virtual index ID 501 associated with the newly generated index ID 502 in the index ID table 406 (S1404). That is, the index ID table is initialized.

  When a new virtual index ID is added to the index ID table 406, the pipeline control module 404 starts to relocate the divided indexes to the index management modules 203 on all the search servers 101 that are relocation targets. A command is issued (S1405). That is, the existing search server 101 related to the rearrangement is instructed to rearrange the split index.

  The index management module 203 of the search server 101 that has received the rearrangement command sequentially acquires the file path from the top of the existing split index 202 (S1406).

  Next, the index management module 203 calculates a hash value from the file path, queries the index ID table 406, and acquires the index ID (S1407).

  Next, the index management module 203 determines whether or not the acquired index ID is a newly added index ID (S1408). If the index ID is not new (No in S1408), the index management module 203 performs no processing for the file path. If the index ID is new (Yes in S1408), the index management module 203 extracts the entry from the split index, and generates and adds a split index associated with the new index ID (S1409).

  Note that the operations of S1406 to S1409 are processed for all the file paths registered in the split index 202. Further, it is assumed that the split index is temporarily generated on the local storage 201 of the search server 101.

  Thereafter, the newly created split index is uploaded to the distributed file system 302 (S1410), and the pipeline control module 404 is notified of the end of splitting (S1411).

  The pipeline control module 404 instructs the index management module 203 of the newly added search server 101 to start index allocation processing in the order in which the end notification is received from the index management module 203 on each search server 101. (S1412).

  The newly added index management module 203 of the search server 101 downloads the split index from the distributed file system 302 and repeats the split index merge process (S1413). As described above, the distributed index 202 can be generated on the newly added search server 101.

[Search server deletion flow]
FIG. 15 shows a processing flow when the search server 101 is reduced from the search system. This processing flow starts when the administrator inputs the reduction of the search server 101 to the system management module 405 (S1501).

  Upon accepting that the search server 101 has been deleted, the system management module 405 acquires from the search server management table 407 the index ID 601 of the entry whose search server 101 that is the reduction target is the destination file server name. A virtual index ID associated with the index ID is calculated and acquired (S1502).

  Thereafter, the system management module 405 deletes the virtual index ID acquired in S1502 from the index ID table 406 (S1503).

  Next, the system management module 405 instructs the pipeline control module 404 to delete the index to the index management module 203 of the search server 101 to be reduced (S1504).

  Upon receiving the instruction, the index management module 203 obtains the file paths registered in the split index 202 in order from the beginning to the end (S1505).

  Next, the index management module 203 calculates a hash value from the acquired file path, and inquires the index ID table for the index ID corresponding to the calculated hash value (S1506).

  After that, the index management module 203 extracts the index ID of the file path entry from the split index 202, generates a new split index linked to the acquired index ID, or adds the index data to the split index. . Thereafter, the index management module 203 generates a split index for merging with the relocation destination (S1507). When this division processing is completed, there are a plurality of division indexes for each index ID in the local storage 201 of the search server 101 that is the deletion target.

  Next, the index management module 203 uploads the split index for each index ID generated in S1507 onto the distributed file system 302 (S1508).

  Subsequently, the index management module 203 notifies the system management module 405 of (1) completion of generation of the split index for each index ID and (2) storage destination information on the distributed file system 302 (S1509).

  Upon receiving this notification, the system management module 405 issues an instruction to the pipeline control module 404, and issues an instruction for merging indexes to the index management modules 203 on all the search servers 101 that are the targets of relocation. (S1510).

  Receiving the instruction, each index management module 203 performs download and merge processing of the split index, and generates the latest split index 202 (S1511).

  After completion of the above, the index management module 203 deletes the deletion target search server 101 from the system (S1512).

[Summary]
According to the present embodiment, by setting a virtual node (virtual index ID) in a consistent hash space that maps a hash value corresponding to a search index, leveling of the size of the split index and suppression of bias are simultaneously performed. Can be realized. Thereby, the improvement of search performance is realizable.

  Further, according to the present embodiment, addition or deletion of a divided index accompanying physical addition or deletion of the search server 101 can be flexibly handled by rearrangement of virtual nodes (virtual index IDs). As a result, it is possible to simplify the management of the plurality of divided indexes 202 associated with each search server 101.

  Further, according to the present embodiment, the generation of the split index by the pipeline processing can be distributed to the plurality of distributed processing servers 103 and executed. Thereby, the production | generation speed | rate of a division | segmentation index can be improved. Furthermore, by generating the split index on the distributed processing server 103 for each index ID, it is possible to reduce unnecessary network traffic and disk I / O between the distributed processing servers when the split index is generated. Thereby, the generation of the split index can be more efficiently and speeded up.

  Further, according to the present embodiment, by generating an index list generation process for giving a file path to be generated as a split list for each arbitrary number of hierarchies in the folder tree in the file server, split index generation is performed. Can be made more efficient and faster.

DESCRIPTION OF SYMBOLS 100 ... Search client 101 ... Search server 102 ... File server 103 ... Distributed processing server 104 ... Management server 105 ... Network 201 ... Local storage 202 ... Split index 203 ... Index management module 204 ... Search module 301 ... Local storage 302 ... Distributed file system 303 ... Distributed processing module 401 ... Local storage 402 ... Scanner module 403 ... Index generation module 404 ... Pipeline control module 405 ... System management module 406 ... Index ID table 407 ... Search server management table 408 ... File management table

Claims (6)

In a file search system that searches large file systems,
A first processing function unit for scanning a file / directory of the file system to generate a list of file paths;
A process of assigning an index ID to each of the divided indexes assigned to each search server, a process of associating each index ID with the search server that is the placement destination, the number of search servers, and the number of divided indexes to be placed on each search server Based on the number, a process of associating n virtual index IDs with each index ID, and a hash value (first hash value) in the consistent hash space is obtained from the character string of the virtual index ID. Processing, processing for generating a table indicating a correspondence relationship between the hash value (first hash value) and the index ID, and a hash value (second second) in the consistent hash space from the file path read from the list (Hash value) and the hash value (second hash value) Referring to Buru, a second processing function unit that executes a process of determining the index ID to be associated with each file path, and a process of generating a split index for each of the index ID,
A third processing function unit for placing the generated split index in the search server ;
A file search system comprising: The file search system according to claim 1 ,
The file processing system, wherein the second processing function unit divides the list into an arbitrary number, and distributes the generation processing of the divided index for each divided list to a plurality of distributed processing systems. The file search system according to claim 2 ,
The file processing system, wherein the second processing function unit divides the list for each index ID. The file search system according to claim 1,
The first processing function unit generates a list of the file paths for each arbitrary folder hierarchy in the file system, each of the product, to give the generated list to the second processing function unit Feature file search system. The file search system according to claim 1,
When adding a search server, the correspondence between the hash value (first hash value) of the virtual index ID set in the consistent hash space that maps the hash value uniquely calculated from the file path and the index ID is updated. A fourth processing function unit,
A file processing system comprising: a fifth processing function unit that executes rearrangement of divided indexes using the correspondence relationship after update. The file search system according to claim 1,
When deleting a search server, the corresponding index ID is calculated from the file path registered in the split index assigned to the search server to be deleted, and merged for each relocation destination search server corresponding to each index ID And a sixth processing function unit for generating a divided index for use.

Images