WO2012164735A1

WO2012164735A1 - Information processing device, data management method, and data management program

Info

Publication number: WO2012164735A1
Application number: PCT/JP2011/062759
Authority: WO
Inventors: 美穂村田; 裕一槌本
Original assignee: 富士通株式会社
Priority date: 2011-06-03
Filing date: 2011-06-03
Publication date: 2012-12-06
Also published as: JPWO2012164735A1; JP5633646B2

Abstract

In order to prevent the concentration of a data management load, an information processing device (10) is used in an information processing system wherein a set of keys is divided into multiple segments, and data which is stored in association with a key is managed in units of segments. A storage unit (11) stores segment information indicating the relationship between the key and the segment for at least segments #1 and #2 of the multiple segments. A control unit (12) references the segment information and determines whether to divide segment #1 into two or more segments on the basis of the amount of data stored in association with the key belonging to segment #1 and on the basis of a threshold value #1. The threshold value #1 takes a different value than a threshold value #2 which is used to determine whether to divide segment #2.

Description

Information processing apparatus, data management method, and data management program

The present invention relates to an information processing apparatus, a data management method, and a data management program.

As one of data management methods in the information processing system, a method in which a key and data are associated with each other and stored in one or more storage devices can be considered. Software such as application software performs data operations such as data reading (reading) and writing (writing) by specifying a key, for example. When data is distributed and stored in a plurality of storage devices, for example, the storage device of the data storage destination is determined using a key and a hash function, and the storage device storing the desired data is searched. A method is conceivable.

Also, a method of dividing a set of keys into a plurality of segments and managing the data in units of segments can be considered so that the data can be managed efficiently. For example, it is conceivable to determine a storage device for storing data on a segment basis. As a method of dividing a set of keys into a plurality of segments and assigning the keys to the segments, for example, a method using a binary tree called PHT (Prefix Hash Tree) has been proposed.

Each node of PHT is given a binary number with a length corresponding to the depth from the root node as a label. The root node has node 0 and node 1 as child nodes. Node 0 has node 00 and node 01 as child nodes. Each key is assigned to one leaf node (a node that has no child nodes) by comparing the prefix (a few bits from the beginning of the key) with the label when the key is expressed in binary number. . Keys and data are managed in units of segments corresponding to leaf nodes. The depth of PHT (the degree of segment division) is determined according to the amount of data to be managed, for example.

By the way, as the information processing system is operated, data is written to the storage device, and the amount of data to be managed may increase. Therefore, when managing data in segment units, it is conceivable to dynamically divide a segment having a large data amount into a plurality of segments. However, depending on the standard setting method for determining whether or not to divide a segment, there is a problem that division of a plurality of segments may occur at the same time. If segment segmentation is concentrated at the same time, the load on the information processing system will increase due to processing such as data movement and management information updates, resulting in performance degradation such as a delay in response to data operations requested at that time. May occur.

In one aspect, an object of the present invention is to provide an information processing apparatus, a data management method, and a data management program capable of suppressing concentration of data management load.

In one embodiment, there is provided an information processing apparatus used in an information processing system that divides a set of keys into a plurality of segments and manages data stored in a storage device in association with the keys in units of segments. The information processing apparatus includes a storage unit and a control unit. A memory | storage part memorize | stores the segment information which shows the relationship between a key and a segment about at least the 1st and 2nd segment among several segments. The control unit refers to the segment information, and divides the first segment into two or more segments based on the amount of data stored in association with the key belonging to the first segment and the first threshold value. To determine. The first threshold value is different from the second threshold value used for determining whether to divide the second segment.

In one embodiment, there is provided a data management method performed by an information processing system that divides a set of keys into a plurality of segments and manages data stored in a storage device in association with the keys in units of segments. In the data management method, the amount of data stored in the storage device is calculated in association with the key belonging to the first segment among the plurality of segments. Whether to divide the first segment into two or more segments is determined based on the calculated amount of data and the first threshold value corresponding to the first segment. The first threshold value is different from the second threshold value used for determining whether to divide the second segment of the plurality of segments.

In one embodiment, there is provided a data management program that is executed by a computer used in an information processing system that divides a set of keys into a plurality of segments and manages data stored in a storage device in association with the keys in units of segments. Provided. The data management program causes the computer to execute the following processing. The amount of data stored in the storage device is calculated in association with the key belonging to the first segment among the plurality of segments. Whether to divide the first segment into two or more segments is determined based on the calculated amount of data and the first threshold value corresponding to the first segment. The first threshold value is different from the second threshold value used for determining whether to divide the second segment of the plurality of segments.

According to one embodiment, the concentration of data management load can be suppressed.
These and other objects, features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings which illustrate preferred embodiments by way of example of the present invention.

It is a figure which shows the information processing apparatus of 1st Embodiment. It is a figure which shows the information processing system of 2nd Embodiment. It is a block diagram which shows the hardware example of a server apparatus. It is a figure which shows the example of a request process. It is a figure which shows the example of a segment management tree. It is a figure which shows the example of the arrangement | positioning method of the segment to a server apparatus. It is a block diagram which shows the example of software of a server apparatus. It is a flowchart which shows a request process. It is a flowchart which shows segment division. It is a figure which shows the example of the threshold value of a segment. It is a sequence diagram which shows request processing and segment division.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating the information processing apparatus according to the first embodiment. The information processing apparatus 10 is used in an information processing system that divides a set of keys into a plurality of segments and manages data stored in the

storage devices

21 and 22 in association with the keys in units of segments. One or both of the

storage devices

21 and 22 may be a storage device included in the information processing apparatus 10. The information processing apparatus 10 includes a storage unit 11 and a control unit 12.

The storage unit 11 stores segment information. The segment information indicates the relationship between the key and the segment for at least segments # 1 and # 2 among the plurality of segments. The segment information may represent a segment by a tree structure called a trie tree or a prefix tree. The storage unit 11 may be a storage device such as a RAM (Random Access Memory) or an HDD (Hard Dirk Drive).

The control unit 12 refers to the segment information in the storage unit 11, and divides the segment # 1 into two or more segments based on the amount of data stored in association with the key belonging to the segment # 1 and the threshold value # 1. Judge whether to do. For example, the data amount is compared with the threshold value # 1, and it is determined that the segment # 1 is divided when the data amount exceeds the threshold value # 1. The amount of data can be confirmed by accessing the storage device in which segment # 1 is arranged. The threshold value # 1 is different from the threshold value # 2 used for determining whether to divide the segment # 2. For example, the threshold value # 1 is calculated from the identification information of the segment # 1 according to a predetermined calculation formula, and the threshold value # 2 is calculated from the identification information of the segment # 2 according to a predetermined calculation formula. The control unit 12 may be realized as a program executed using a CPU (Central Processing Unit) and a RAM.

In addition, when the segment information is a plurality of segments expressed in a tree structure, the control unit 12 may calculate a threshold value so that at least the segments at the same hierarchical level have different values. In addition, when a storage device that stores data is selected from the

storage devices

21 and 22 in units of segments, the control unit 12 changes the arrangement of segments in the

storage devices

21 and 22 by dividing the segment # 1. Also good. For example, when segment # 1 is arranged in storage device 21, at least one of two or more segments divided from segment # 1 may be changed from storage device 21 to storage device 22.

According to the first information processing apparatus 10, the amount of data stored in one of the

storage devices

21 and 22 in association with the key belonging to the segment # 1 among the plurality of segments is calculated. Based on the calculated amount of data and the threshold value # 1 corresponding to the segment # 1, it is determined whether the segment # 1 is divided into two or more segments. The threshold value # 1 takes a value different from the threshold value # 2 used to determine whether the segment # 2 is divided among a plurality of segments.

Thus, the segment whose data amount has been increased by writing data in the

storage devices

21 and 22 can be further dynamically divided into two or more segments. For example, since the segment can be divided according to the actual increase amount of data, it is not necessary to know in advance how the data amount of each segment increases (data bias). Therefore, segment management such as segment change in the

storage devices

21 and 22 is facilitated. In addition, since the necessity of division is determined using different threshold values for segment # 1 and segment # 2, the probability of division of segments # 1 and # 2 occurring at the same time can be reduced. Therefore, it is possible to suppress an increase in the load on the information processing system due to concentrated processing such as segment arrangement change and segment information update, and it is possible to suppress a decrease in performance of the information processing system.

[Second Embodiment]
FIG. 2 illustrates an information processing system according to the second embodiment. The information processing system according to the second embodiment includes

server apparatuses

100, 100a, 100b, and 100c, a management apparatus 200, and a client apparatus 300.

Server apparatuses

100, 100 a, 100 b, 100 c and management apparatus 200 are connected to a network 41. The client device 300 is connected to the network 42. The information processing system according to the second embodiment may be realized as a so-called cloud system.

The

server apparatuses

100, 100a, 100b, and 100c are server computers that manage data in a distributed manner. The

server devices

100, 100a, 100b, and 100c store a key for identifying the data and the data in association with each other, and hold segment information for searching for the server device storing the data from the key. ing. Any server device can accept a data processing request from the client device 300. The server device that has received the request including the key searches the server device in which the processing target data is stored based on the segment information, and requests the searched server device for data processing.

The management apparatus 200 is a computer used by a user (for example, an information processing system administrator). The management device 200 manages the

server devices

100, 100a, 100b, and 100c based on user operations. For example, when a server device that stores data is added to the information processing system, the management device 200 transmits information on the added server device to the

server devices

100, 100a, 100b, and 100c. Also, setting information for adjusting the distributed arrangement of data is transmitted to the

server apparatuses

100, 100a, 100b, and 100c.

The client device 300 is a computer used by a user (for example, a user of a cloud service). In the client device 300, for example, application software that handles data stored in the

server devices

100, 100a, 100b, and 100c is executed. The client device 300 transmits a data processing request to an arbitrary server device among the

server devices

100, 100a, 100b, and 100c via the

networks

41 and 42. In the request, one key or a range of keys is specified. Data processing includes data read (read) and write (write).

Regardless of which server device receives the request from the client device 300, the requested data processing is executed by transmitting a message between the

server devices

100, 100a, 100b, and 100c. As described above, the information processing system according to the second embodiment can be designed so that a device that becomes a bottleneck for data processing does not occur as much as possible, and can improve availability, response performance, and the like. .

FIG. 3 is a block diagram illustrating a hardware example of the server device. The server apparatus 100 includes a CPU 101, a RAM 102, an HDD 103, an image signal processing unit 104, an input signal processing unit 105, a disk drive 106, and a communication unit 107. The unit is connected to a bus in the server device 100. The

server devices

100a, 100b, and 100c, the management device 200, and the client device 300 can also be realized by the same hardware as the server device 100.

The CPU 101 is an arithmetic device that controls information processing in the server device 100. The CPU 101 reads out at least a part of the programs and data stored in the HDD 103, expands them in the RAM 102, and executes the programs. The server device 100 may include a plurality of arithmetic devices and execute information processing in a distributed manner.

The RAM 102 is a volatile memory that temporarily stores programs and data handled by the CPU 101. Note that the server device 100 may include a type of memory other than the RAM, or may include a plurality of memories.

The HDD 103 is a non-volatile storage device that stores programs such as an OS (Operating System) program and an application program, and data used for information processing. The HDD 103 reads / writes data from / into the built-in magnetic disk according to instructions from the CPU 101. The server device 100 may include a nonvolatile storage device (for example, SSD (Solid State Drive)) other than the HDD, or may include a plurality of storage devices.

The image signal processing unit 104 outputs an image to the display 31 connected to the server apparatus 100 in accordance with an instruction from the CPU 101. As the display 31, for example, a CRT (Cathode Ray Tube) display or a liquid crystal display can be used.

The input signal processing unit 105 acquires an input signal from the input device 32 connected to the server apparatus 100 and outputs it to the CPU 101. As the input device 32, for example, a pointing device such as a mouse or a touch panel, a keyboard, or the like can be used.

The disk drive 106 is a drive device that reads a program and data recorded on the recording medium 33. As the recording medium 33, for example, a magnetic disk such as a flexible disk (FD: Flexible Disk) or HDD, an optical disk such as a CD (Compact Disk) or a DVD (Digital Versatile Disk), a magneto-optical disk (MO: Magneto-Optical disk). Can be used. For example, the disk drive 106 stores the program and data read from the recording medium 33 in the RAM 102 or the HDD 103 in accordance with a command from the CPU 101.

The communication unit 107 is a communication interface that communicates by connecting to the network 41. The connection method to the network 41 may be wired or wireless. That is, the communication unit 107 may be a wired communication interface or a wireless communication interface.

FIG. 4 is a diagram illustrating an example of request processing. In FIG. 4, examples of requests include (A) a read request designating a single key, (B) a read request designating a key range, and (C) a write request. The server apparatus 100a stores (2, data2), (3, data3) as a key and data pair (key, value), and the server apparatus 100c stores (4, data4), (5, data5). Suppose you are.

(A) When the server apparatus 100 receives the read request get (2) designating the key 2 from the client apparatus 300, the server apparatus 100 determines that the data corresponding to the key 2 is stored in the server apparatus 100a. Then, a message indicating a read request for the key 2 is transmitted to the server device 100a. The server apparatus 100a transmits data2 to the server apparatus 100. The server apparatus 100 transmits data2 to the client apparatus 300.

(B) When the server apparatus 100 receives the read request range_get (2, 5) designating the key range 2 to 5 from the client apparatus 300, the data corresponding to the

keys

2 and 3 is stored in the server apparatus 100a, and the key It is determined that data corresponding to 4 and 5 is stored in the server device 100c. Then, a message indicating a read request for the

keys

2 and 3 is transmitted to the server apparatus 100a, and a message indicating a read request for the

keys

4 and 5 is transmitted to the server apparatus 100c. The server apparatus 100a transmits

data

2 and 3 to the server apparatus 100. The server apparatus 100 c transmits

data

4 and 5 to the server apparatus 100. The server apparatus 100 transmits data 2 to 5 to the client apparatus 300.

(C) When the server apparatus 100 receives the write request set (1, data1) specifying the key 1 from the client apparatus 300, the server apparatus 100 determines that the data corresponding to the key 1 is stored in the server apparatus 100a. Then, a message indicating a write request for key 1 is transmitted to server device 100a. The server device 100a writes data1 to its own storage device and reports the completion of writing to the server device 100. The server apparatus 100 reports the completion of writing to the client apparatus 300.

As described above, the

server apparatuses

100, 100a, 100b, and 100c hold keys and data in association with each other, and read data from the storage device and write data to the storage device in units of keys. The key to be processed is specified by application software executed on the client device 300, for example. Therefore, the complexity of data processing of the

server apparatuses

100, 100a, 100b, and 100c is suppressed, and the load is reduced.

When there is no data corresponding to the key specified in the write request, the write request means adding data. On the other hand, if data corresponding to the key specified in the write request already exists, the write request means data overwrite. The confirmation of whether or not the data corresponding to the key exists and the determination of whether or not overwriting is possible are performed, for example, under the responsibility of application software executed on the client device 300. The information processing system according to the second embodiment can be used, for example, for managing data (for example, log data) using time as a key.

FIG. 5 is a diagram showing an example of the segment management tree. The

server apparatuses

100, 100a, 100b, and 100c have segment information describing a segment management tree. As the segment management tree, an ordered tree called a trie tree or a prefix tree can be used. Non-Patent Document 1 (YatinYChawathe, Sriram Ramabhadran, Sylvia Ratnasamy, Anthony LaMarca, Scott Shenker and Joseph Hellerstein, "A Case Study in Building Layered DHT Applications") may be used.

In the segment management tree, the key space is hierarchically divided into sections called segments. Each node in the segment management tree corresponds to a segment. Each node is given a label having a length corresponding to the depth of the node. For example, label 0 is assigned to the left child node of the root node, and label 1 is assigned to the right child node. Further, when a label L is given to a certain node, a label L + 0 is given to the left child node, and a label L + 1 is given to the right child node. Each key included in the key space is associated with any one leaf node (a node having no child node) to which a label matching the prefix of the key is assigned.

As an example, consider the case where the key can be represented by a 5-bit binary number. If the key specified from the client device 300 is not expressed in binary, it may be converted into binary. Further, as shown in FIG. 5, a segment management tree including five leaf nodes assigned

labels

000, 001,01, 10, and 11 is considered. In this case, the keys 00000 to 0111 are associated with the node with the label 000. Keys 00100 to 00111 are associated with the node with label 001. Keys 01000 to 01111 are associated with the node of label 01. Keys 10000 to 10111 are associated with leaf nodes of label 10. Keys 11000 to 11111 are associated with leaf nodes of label 11.

Segments corresponding to leaf nodes of the segment management tree are arranged in the

server apparatuses

100, 100a, 100b, and 100c. For example, the segment of the label 11 is arranged on the server device 100. The segments with

labels

000 and 10 are arranged in the server apparatus 100a. The segment with the

label

001, 01 is arranged in the server device 100c. A server device that arranges a segment can be determined based on, for example, a hash value calculated by applying a hash function to the label of the segment.

In this way, by determining the server device that arranges data in segment units, the locality of the keys can be maintained, and the data associated with the keys having similar values are arranged on the same server device as much as possible. become. For this reason, range-designated data processing such as range-designated read requests can be efficiently executed. The depth of the segment management tree (the degree of segment division) is dynamically adjusted according to the amount of data stored in the

server apparatuses

100, 100a, 100b, and 100c. As will be described later, a segment with an increased data amount is further automatically divided into a plurality of segments.

In the above description, a binary tree is used as an example of the segment management tree, but a tree with an arbitrary number of branches may be used. When each node other than the leaf node has b child nodes (b is an integer of 2 or more), a b-ary label may be assigned to each node. For example, when b = 3, the root node has child nodes with

labels

0, 1, and 2. The node with label 0 has child nodes with

labels

00, 01, 02. When associating a key with a segment, it is only necessary to convert the key to b-base and compare the key prefix with the label.

FIG. 6 is a diagram illustrating an example of a method of arranging segments on the server device. As described above, the segment corresponding to the leaf node of the segment management tree is arranged in one of the

server devices

100, 100a, 100b, and 100c according to the label of the segment.

Here, let us consider a hash value space in which the value range of the hash function (values that the hash value can take) is 0 to 2 ⁿ −1 (n is a natural number) and the value range is a loop. A hash function corresponding to each server device is calculated by applying a hash function to identification information (for example, address) of each of the

server devices

100, 100a, 100b, and 100c. Also, a hash function corresponding to the segment is calculated by applying a hash function to the segment label. Then, based on the positional relationship on the loop between the hash value of each server device and the hash value of the segment, the server device in which the segment is arranged is determined. For example, on the loop, the server device having the minimum hash value that precedes the hash value of the segment is selected.

For example, the hash value h (s1) of the server device 100, the hash value h (s2) of the server device 100a, and the hash value h (0) of the label 0 are h (s2)> h (0)> h (s1). If so, the segment with label 0 is placed in the server device 100a. In addition, the hash value h (s3) of the server device 100b, the hash value h (s4) of the server device 100c, and the hash value h (001) of the label 001 are h (s4)> h (001)> h (s3). If so, the segment with the label 001 is placed in the server device 100c.

If the LSB (Least Significant Bit) of the label is 0, the consecutive 0s from the LSB toward the MSB (Most Significant Bit) are deleted and the hash function is applied. However, the MSB 0 is not deleted. For example, label 00,000 is converted to label 0, and label 10 is converted to label 1. Therefore, the hash values of the labels 0.000,000 are the same, and the hash values of the

labels

1, 10 are the same. Accordingly, when a certain segment is divided into two or more segments, at least one of the two or more segments after the division is arranged in the same server device as the original segment.

However, the method of arranging the segments on the

server devices

100, 100a, 100b, 100c is not limited to the above method. For example, a method is conceivable in which a hash value calculated from a segment label is divided by the number of server devices to obtain a remainder (residue), and a server device in which segments are arranged is selected according to the remainder. For example, when the hash value is divided by 4, the server apparatus 100 is selected when the remainder = 0, the server apparatus 100a is selected when the remainder = 1, the server apparatus 100b is selected when the remainder = 2, and the server apparatus 100c is selected when the remainder = 3. To do.

FIG. 7 is a block diagram illustrating a software example of the server device. The server device 100 includes a communication processing unit 110, an event processing unit 120, and a data storage unit 130. The communication processing unit 110 and the event processing unit 120 can be realized as programs executed using the CPU 101 and the RAM 102, for example. The data storage unit 130 can be realized as a storage area on the RAM 102 or the HDD 103, for example. The

server apparatuses

100a, 100b, and 100c can also be realized by the same block configuration as the server apparatus 100.

The communication processing unit 110 includes a request reception unit 111, a segment information storage unit 112, a segment management unit 113, and a message processing unit 114.
The request reception unit 111 receives a request from the client device 300 and requests the segment management unit 113 to determine a server device in which a desired segment is arranged. Further, the request reception unit 111 acquires the read data or the write result from the message processing unit 114 and transmits it to the client device 300 as a response.

The segment information storage unit 112 stores segment information describing a segment management tree as shown in FIG. The segment information is updated as the segments are divided. The segment information storage unit 112 may store information indicating a hash function. The segment information storage unit 112 can be realized as a storage area on the RAM 102 or the HDD 103.

The segment management unit 113 searches for one or more segments corresponding to the key or key range specified in the request based on the segment information stored in the segment information storage unit 112. Then, one or more server devices in which the searched segment is arranged are determined from the segment label and the hash function. The segment management unit 113 requests the event processing unit 120 to perform data processing when the searched segment is in its own server device. If the retrieved segment is in another server device, the message processing unit 114 is requested to send a message to the other server device.

Also, the segment management unit 113 updates the segment information stored in the segment information storage unit 112 when the event processing unit 120 reports that the segment has been divided. Then, the message processing unit 114 is requested to transmit a message indicating the segment update to another server device. Further, the segment management unit 113 updates the segment information stored in the segment information storage unit 112 in response to a request from the message processing unit 114. The segment management unit 113 receives the identification information of the server device in which the segment can be arranged as setting information from the management device 200 and holds it.

The message processing unit 114 transmits and receives messages to and from other server devices (

server devices

100a, 100b, and 100c). When the message processing unit 114 receives a read request or write request message from another server device, the message processing unit 114 requests the event processing unit 120 to perform data processing. Also, in response to a request from the segment management unit 113, a read request or write request message is transmitted to another server device. Further, the message processing unit 114 transmits a segment update message to another server device in response to a request from the segment management unit 113. When a segment update message is received from another server device, the segment management unit 113 is requested to update the segment information.

The event processing unit 120 includes a data processing unit 121, a calculation formula storage unit 122, a threshold value calculation unit 123, and a division determination unit 124.
The data processing unit 121 performs data processing in response to a request from the segment management unit 113 or the message processing unit 114. If the read request specifies a key or a key range, the data processing unit 121 reads data corresponding to the key or a data group corresponding to the key range from the data storage unit 130. If the write request specifies a key, the data processing unit 121 writes data in the data storage unit 130 in association with the key.

In addition, when the data processing unit 121 writes the data to the data storage unit 130, the data processing unit 121 inquires of the division determination unit 124 whether or not the segment is necessary. When the division determination unit 124 determines that division is required, the data processing unit 121 performs division processing. For example, a part of the data stored in the data storage unit 130 is moved to another server device. In addition, the segment management unit 113 is notified that the segment has been divided.

The calculation formula storage unit 122 stores a calculation formula for calculating a threshold for determining whether to divide a segment. As will be described later, the threshold value is calculated based on the label of the segment that determines whether or not division is necessary. The calculation formula of the threshold is described by the user of the management apparatus 200 and is set in the calculation formula storage unit 122 by the management apparatus 200. The calculation formula storage unit 122 can be realized as a storage area on the RAM 102 or the HDD 103.

The threshold value calculation unit 123 calculates the threshold value of the specified segment from the segment label specified by the division determination unit 124 and the threshold value calculation formula stored in the calculation formula storage unit 122. Then, the calculated threshold value is returned to the division determination unit 124.

In response to the inquiry from the data processing unit 121, the division determination unit 124 determines whether or not it is necessary to divide the segment where the data has been written. The division determination unit 124 designates the label of the segment in which writing has been performed, and requests the threshold value calculation unit 123 to calculate the threshold value. For example, the segment label is obtained from the key specified in the write request with reference to the segment information. Further, the division determination unit 124 refers to the data storage unit 130 to obtain the data amount of the segment in which writing has been performed (for example, the total amount of data associated with the key belonging to the segment). The division determination unit 124 determines that the division is necessary when the data amount exceeds the threshold value, and determines that the division is not required when the data amount is equal to or less than the threshold value.

The data storage unit 130 stores keys and data in association with each other. The data storage unit 130 may divide the storage area into segments and store keys and data.
FIG. 8 is a flowchart showing request processing. Here, a case where the server apparatus 100 performs processing is considered. The

server apparatuses

100a, 100b, and 100c perform the same processing as the server apparatus 100. In the following, the process illustrated in FIG. 8 will be described in order of step number.

(Step S11) The request reception unit 111 receives a request from the client device 300. The type of request includes a read request designating one key, a read request designating a key range, and a write request designating one key.

(Step S12) The segment management unit 113 determines whether the request received in step S11 specifies a key range. If it is a range designation request, the process proceeds to step S14. Otherwise, the process proceeds to step S13.

(Step S13) The segment management unit 113 refers to the segment information stored in the segment information storage unit 112 and searches for a segment to which the key specified in the request belongs. When searching for a segment, for example, the prefix and label of a key expressed in base b (b is an integer of 2 or more) are compared from the root node of the segment management tree toward the leaf node, and a label that matches the prefix is searched. Identify the leaf node that it has. A segment corresponding to the identified leaf node is a searched segment.

(Step S14) The segment management unit 113 refers to the segment information and searches for all segments including at least one key within the range specified by the request. In the search for a plurality of segments, for example, the longest prefix common to the maximum value and the minimum value of the keys in the range is obtained, and the node having the prefix as a label is specified. Then, a plurality of leaf nodes below the identified node are identified. A plurality of segments corresponding to the specified plurality of leaf nodes are searched segments.

For example, when the key range is 00010 to 00101, the longest prefix common to the minimum value 00010 and the maximum value 0101 is 00. Therefore, the leaf node of label 000,001 that is lower than the node of label 00 is specified. Then, a segment corresponding to the label 000 and a segment corresponding to the label 001 are searched.

(Step S15) The segment management unit 113 calculates a hash value by applying a hash function to the label of the segment searched in step S13 or step S14. And the server apparatus by which the segment is arrange | positioned from a hash value is specified. The server device is specified by, for example, the method shown in FIG. When a plurality of segments are searched in step S14, a plurality of server devices may be specified. The following steps S16 to S18 are executed for each specified server device or each segment.

(Step S16) The segment management unit 113 determines whether another server device has been identified in step S15 as the server device on which the data processing target segment is arranged. If another server device is specified, the process proceeds to step S17. When the own apparatus (server apparatus 100) is specified, the process proceeds to step S18.

(Step S17) The message processing unit 114 transmits a message indicating a read request or a write request to the other server device specified in Step S15. The message includes the key or key range stored in the destination server device among the keys or key ranges specified in the request from the client device 300. Thereafter, the message processing unit 114 receives a response message from another server device. The response message includes data corresponding to the key or a write completion report.

(Step S18) The data processing unit 121 performs data processing in response to a request from the client device 300. In the case of a read request, data corresponding to the key or key range stored in the own device is read from the data storage unit 130 out of the key or key range specified in the request. In the case of a write request, the key and data included in the request are associated with each other and written to the data storage unit 130.

(Step S19) The request reception unit 111 transmits a response to the request received in step S11 to the client device 300. When the request is a read request, the data read from the own device or another server device in steps S17 and S18 is transmitted as a response. If the request is a write request, a report of the completion of writing confirmed in steps S17 and S18 is transmitted as a response.

FIG. 9 is a flowchart showing segment division. Here, a case where the server apparatus 100 performs processing is considered. The

server apparatuses

100a, 100b, and 100c perform the same processing as the server apparatus 100. In the following, the process illustrated in FIG. 9 will be described in order of step number.

(Step S21) The division determination unit 124 identifies the key of the data written by the data processing unit 121 in the data storage unit 130. Then, based on the segment information stored in the segment information storage unit 112, the segment corresponding to the key is searched.

(Step S22) The threshold value calculation unit 123 substitutes the label of the segment searched in Step S21 for the calculation formula stored in the calculation formula storage unit 122, and calculates the threshold value of the segment. As the threshold calculation formula, for example, the following formula (1) is used. In Equation (1), L is a label, R is a predetermined route threshold, and b is the number of branches in the segment management tree. Further, length (L) indicates the number of bits of the label, and value (L) indicates the value of the decimal number expression represented by the label L.

(Step S23) The division determination unit 124 refers to the data storage unit 130 and calculates the data amount of the segment searched in Step S21 (for example, the total amount of data associated with the key belonging to the segment). Then, it is determined whether the data amount exceeds the threshold calculated in step S22. If the data amount exceeds the threshold value, the process proceeds to step S24. If the data amount is less than or equal to the threshold value, the process ends.

(Step S24) The segment management unit 113 defines b child segments obtained by dividing the segment searched in Step S21. For example, when the segment label is L and b = 2, child segments of labels L + 0 and L + 1 are defined.

(Step S25) The data processing unit 121 selects one key belonging to the segment searched in step S21 from the data storage unit 130.
(Step S26) The data processing unit 121 specifies a child segment having a label that matches the prefix of the key selected in Step S25 from the child segments defined in Step S24. Then, the data processing unit 121 assigns the key selected in step S25 to the specified child segment.

(Step S27) The data processing unit 121 determines whether all keys have been selected in step S25. If all keys have been selected, the process proceeds to step S28. If there is an unselected key, the process proceeds to step S25.

(Step S28) The segment management unit 113 applies a hash function to the label of the child segment defined in step S24, and specifies the server device of the placement destination of each child segment. The data processing unit 121 reads out data of a child segment arranged in another server device (data associated with a key belonging to the child segment) from the data storage unit 130, and transmits the data to the other server device. When the data transmission is completed, the child segment data arranged in another server device is deleted from the data storage unit 130.

(Step S29) The segment management unit 113 updates the segment information stored in the segment information storage unit 112 so that the segment division in step S24 is reflected. The message processing unit 114 notifies all other server devices of the segment update.

FIG. 10 is a diagram illustrating an example of the segment threshold. The threshold value example in FIG. 10 is calculated by applying Equation (1) to the segment management tree in FIG.
A predetermined root threshold value R is set for the segment corresponding to the leftmost node (root node and node with a label of 0.00,000) at each hierarchical level. As R, a data amount such as 100 [MB (megabyte)] is determined in advance. The segment corresponding to the node other than the leftmost node, thresholds 2 ^f multiplying the R is set. f can be expressed as a fraction, the number of bits of the label is used for the calculation of the denominator, and the value represented by the label is used for the numerator. Therefore, f becomes a different value between segments of the same hierarchical level, and the threshold value becomes different.

Since the segment of label 1 is f = 1/2, a threshold value of 1.41R is set. Since f = 1/4 is set for the segment of label 01, a threshold value of 1.19R is set. Since f = 2/4 is set in the segment of the label 10, the threshold value 1.41R is set. Since f = 3/4 for the segment of label 11, a threshold value of 1.68R is set.

Here, consider a case where data is evenly added to the entire key space at a constant speed from a state in which the

server apparatuses

100, 100a, 100b, and 100c do not store data (a data set is empty). . For example, it is assumed that the route threshold is R [MB], and data is added at a rate of S [MB / second] (for example, 1 [MB / second]) with respect to the entire key space.

In this case, as the elapsed time from the start of the data addition, the data amount reaches the threshold value in R / S [seconds] in the root node segment. In the segment with label 0, the data amount reaches the threshold value at 2R / S [seconds]. In the label 1 segment, the data amount reaches the threshold at 2.82 R / S [seconds]. In the segment of label 00, the data amount reaches the threshold value at 4 R / S [seconds]. In the segment of label 01, the data amount reaches the threshold value at 4.76 R / S [seconds]. In the segment of label 10, the data amount reaches the threshold value at 5.64 R / S [seconds]. In the segment of label 11, the data amount reaches the threshold at 6.72 R / S [seconds]. In the segment with label 000, the data amount reaches the threshold value at 8 R / S [seconds]. In the segment with label 001, the data amount reaches the threshold value at 8.72 R / S [seconds].

As described above, when the threshold value of each segment is calculated according to the equation (1), when the data is added to the key space evenly, the division of the upper hierarchical level segment is completed, and then the lower hierarchical level is calculated. Segmentation is performed. For example, after the segment of label 1 is divided, the segment of label 00 is divided. After the segment of label 11 is divided, the segment of label 000 is divided. Further, all segments are divided at different timings. Whether or not to divide a segment can be determined from the label of the segment, and therefore can be determined independently between segments.

In the example of FIG. 10, the root threshold value R is assigned to the leftmost node at each hierarchical level and the threshold value is increased from the leftmost node toward the rightmost node. However, the root threshold value R is assigned to the rightmost node at each hierarchical level. The threshold value may be increased from the right end node toward the left end node. In the above description, the data amount and the threshold are compared to determine whether to divide the segment. However, the data amount and the threshold may be compared to determine whether to integrate the segments. For example, when the total data amount of the segments with the label 000,001 is equal to or less than the threshold value, it may be determined that the two segments are integrated. At this time, the threshold used for determination of division and the threshold used for determination of integration may be calculated from different calculation formulas.

FIG. 11 is a sequence diagram showing request processing and segment division. Consider a case where the client apparatus 300 transmits a data write request to the server apparatus 100 and segmentation occurs due to the data write.

(Step S <b> 31) The client device 300 transmits a request indicating data writing to the server device 100. The request includes a key and data.
(Step S32) The server device 100 searches for the segment to which the key belongs from the segment information of the device itself, and identifies the server device (here, the server device 100a) where the segment is arranged using the label and the hash function. To do. Then, a write request message is transmitted to the server apparatus 100a. The message includes a key and data.

(Step S33) The server apparatus 100a associates the key and the data included in the message received from the server apparatus 100 and writes them in the storage device of the own apparatus. Then, a response indicating the completion of writing is transmitted to the server apparatus 100.

(Step S <b> 34) When the server apparatus 100 confirms that the writing is completed in the server apparatus 100 a, the server apparatus 100 transmits a response to the client apparatus 300.
(Step S35) The server apparatus 100a detects that the data amount of the segment has exceeded the threshold value by writing data. Then, the server apparatus 100a further divides the segment into b (for example, two) segments.

(Step S36) The server apparatus 100a specifies the server apparatus (here, the

server apparatuses

100a and 100b) on which the segment after the division is arranged using the label and the hash function. And the key and data of the segment arrange | positioned at the server apparatus 100b are transmitted to the server apparatus 100b. As described above, when one segment is divided into two child segments, one child segment is placed on the same server device as the original segment, and the other child segment is placed on a server device different from the original segment. You can also make it.

(Step S37) The server apparatus 100b associates the key and data received from the server apparatus 100a with each other and writes them in the storage device of the own apparatus. Then, a response indicating the completion of writing is transmitted to the server device 100a.

(Step S38) The server apparatus 100a deletes the key and data of the segment that has been copied to the server apparatus 100b from the storage device of the own apparatus. Further, the server apparatus 100a updates the segment information that the own apparatus has so that the segment division is reflected. Then, the server apparatus 100a transmits a notification indicating the segment update to all other server apparatuses (

server apparatuses

100, 100b, and 100c).

(Step S39) The

server apparatuses

100, 100b, and 100c update the segment information of the own apparatus based on the notification from the server apparatus 100a. Then, a response indicating that segment information has been updated is transmitted to the server device 100a. Thereby, the update of the segment information by the server apparatus 100a is reflected in the segment information which the

server apparatuses

100, 100b, and 100c have, and the segment information is in a synchronized state.

According to the second embodiment, the

server apparatuses

100, 100a, 100b, and 100c store the keys and data in association with each other, and perform data processing for the key specified by the client apparatus 300. The load can be reduced. Moreover, since data is distributed and stored in a plurality of server devices, and any server device can accept a request from the client device 300, it can be designed so that no device becomes a bottleneck. Increases the availability of the processing system.

In addition, since the key space is divided into a plurality of segments and the data is arranged in the server device in units of segments, there is a high probability that data corresponding to keys having similar values are stored in the same server device. Data processing with a specified range can be executed efficiently. In addition, since the necessity of segmentation can be determined independently for each segment based on the label of the segment, each server device can determine without negotiating with other server devices, and scalability can be improved. In addition, since the threshold for determining whether segment division is necessary is calculated so that it differs between segments, it is possible to reduce the probability that division of multiple segments concentrates at the same time, and the load on the information processing system Can be reduced.

As described above, the data management of the second embodiment can be realized by causing the

server apparatuses

100, 100a, 100b, and 100c as computers to execute data management programs, respectively. The program can be recorded on a computer-readable recording medium (for example, the recording medium 33). As the recording medium, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like can be used. Magnetic disks include FD and HDD. Optical disks include CD, CD-R (Recordable) / RW (Rewritable), DVD, and DVD-R / RW.

When distributing the program, for example, a portable recording medium on which the program is recorded is provided. It is also possible to store the program in a storage device of another computer and distribute the program via the network 41. The computer stores, for example, a program recorded on a portable recording medium or a program received from another computer in a storage device (for example, HDD 103), and reads and executes the program from the storage device. However, a program read from a portable recording medium may be directly executed, or a program received from another computer via the network 41 may be directly executed.

The above merely shows the principle of the present invention. In addition, many modifications and variations will be apparent to practitioners skilled in this art and the present invention is not limited to the precise configuration and application shown and described above, and all corresponding modifications and equivalents may be And the equivalents thereof are considered to be within the scope of the invention.

DESCRIPTION OF SYMBOLS 10 Information processing apparatus 11 Storage part 12

Control part

21, 22 Storage apparatus

Claims

An information processing apparatus used in an information processing system that divides a set of keys into a plurality of segments and manages data stored in a storage device in association with the keys in units of segments,
A storage unit that stores segment information indicating a relationship between a key and a segment for at least a first segment and a second segment of the plurality of segments;
Whether to divide the first segment into two or more segments based on the amount of data stored in association with the key belonging to the first segment and the first threshold with reference to the segment information A control unit for determining,
The information processing apparatus, wherein the first threshold value is different from a second threshold value used for determining whether to divide the second segment.
The segment information includes identification information of the first and second segments,
The first threshold value is calculated based on the identification information of the first segment, and the second threshold value is calculated based on the identification information of the second segment.
The information processing apparatus according to claim 1.
The segment information represents the plurality of segments in a hierarchical structure,
The threshold value of each of the plurality of segments is calculated so as to be different between segments at least at the same hierarchical level.
The information processing apparatus according to claim 1 or 2.
The information processing system includes a plurality of storage devices, and selects a storage device that stores data from the plurality of storage devices in units of segments.
The control unit controls the storage device for storing data to be changed for at least one of the two or more segments divided from the first segment.
The information processing apparatus according to any one of claims 1 to 3.
A data management method performed by an information processing system that divides a set of keys into a plurality of segments and manages data stored in a storage device in association with keys in units of segments,
Calculating an amount of data stored in the storage device in association with a key belonging to a first segment of the plurality of segments;
Determining whether to divide the first segment into two or more segments based on the calculated amount of data and a first threshold according to the first segment;
The data management method, wherein the first threshold value is different from a second threshold value used for determining whether to divide a second segment of the plurality of segments.
A data management program that is executed by a computer used in an information processing system that divides a set of keys into a plurality of segments and manages data stored in a storage device in association with keys in units of segments,
Calculating an amount of data stored in the storage device in association with a key belonging to a first segment of the plurality of segments;
Determining whether to divide the first segment into two or more segments based on the calculated amount of data and a first threshold corresponding to the first segment;
The data management program, wherein the first threshold value is different from a second threshold value used for determining whether to divide a second segment of the plurality of segments.