JP3468181B2 - Distributed data processing apparatus, data storage method, load distribution method, and recording medium - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a distributed data processing technique, and more particularly, to a data access system and method having a load balancing mechanism in consideration of improvement of data access performance by a disk cache in a non-shared data access environment. And a recording medium.

[0002]

2. Description of the Related Art When large-scale data is arranged in one disk device and data access is concentrated, the access speed of data on the secondary storage device is slower than the processing speed of other parts of the computer. It becomes a bottleneck. Therefore, conventionally, as a method of improving the access performance of large-scale data on a secondary storage device such as a disk device, a data access method of dividing the data into small fragment data and accessing them in parallel has been used.

Of these, the striping method is well known as a typical method. In the striping method, original data is generated as a plurality of pieces of divided data, arranged on different disks, and when the data is accessed, those pieces of divided data are accessed in parallel.

As a platform for parallel data access, for example, RAID (redundant arrays o)
There are those that use a dedicated device such as a f inexpensive disk system, and those that use a non-shared computer cluster in which the disk devices are connected to independent computer nodes. These systems are often used in applications such as parallel databases and image and voice servers.

Among them, in the non-shared type system, the disk device is not shared between the nodes, and therefore when the disk device of a certain node becomes unavailable due to a failure or the like, the data allocated to the disk device is changed. Access becomes impossible.

Therefore, not only the same small fragment data is arranged in one disk device, but a copy thereof is also redundantly arranged in a plurality of other disk devices to cope with inaccessibility of data due to a failure of the disk device. Things are also being done.

By the way, the response performance (response time) of the secondary storage device may differ depending on the node.
This is because when there is a difference in data access requests between nodes, or the amount of data corresponding to a data access request differs between nodes, for example, the response time of a node that needs to read large data is larger than other nodes. It also occurs when the situation becomes longer.

When the response times are different, the average response time of the entire system becomes longer because it waits for the worst node. Therefore, load balancing technology is implemented to make the response times of all nodes uniform.

In a non-shared cluster, it is desirable to evenly share the processing load among the nodes in order to realize the maximum processing performance.

However, it is not always easy to allocate processing so that the load on each node becomes equal. For example, when the access from the database concentrates on specific data, the load of the node having the data becomes higher than the load of other nodes.

In some search processes, the processing load increases as the number of data items that satisfy the search conditions increases. In this case, even if all nodes are instructed to perform the same search processing, a different load is applied to each node according to the number of relevant data items found in each node.

In a process in which the load applied to each node is known only at the time of execution, it is possible to realize even load distribution by changing the allocation of the process based on the load at the time of execution.

Techniques for performing such load distribution are disclosed in, for example, Japanese Patent Application No. 11-136742 and Japanese Patent Application No. 11-12827.
No. 3, Japanese Patent Application No. 11-279544 (all unpublished at the time of filing of the present application) and the like. In the methods described in these prior applications, a copy of the data (original data) placed in each node is created, and a node different from the node in which the original data is placed (hereinafter referred to as "original node") "Copy node"). As a result,
Since the data is stored in at least two or more nodes, if the load on the original node is heavy, distribute the data access request to the copy node, thereby making the load on each computer node uniform. You can

[0014]

The information processing apparatus is equipped with a disk cache in order to improve the access efficiency to the secondary storage device, and the operating system is also equipped with a disk cache access function. The response time in a system equipped with a disk cache is particularly short when the cache is hit as compared with the case where a secondary storage device including a disk device is accessed.

On the other hand, if there are many processes that are out of the disk cache (miss hits), the data access performance of the entire cluster system will deteriorate.

In the above-mentioned cluster type distributed data access dynamic load balancing system, no special consideration is given to the effect of the disk cache.

Normally, the data access request is distributed to the original node, and when it is determined that the load on the original node is high, the access request is distributed to the copy node. In an environment where there is no disk cache, the load balancing method of the related art is effective.

However, in the system configuration including the disk cache, even if the original node hits the disk cache (hits the cache) and the access request can be processed at high speed, the original node occasionally has a heavy load. The access request is distributed to the copy node that is accessed only, and in this case, the copy node does not hit the cache, so that the data is read from the disk device and the processing often takes a long time.

Further, since the access does not hit the disk cache, the data access efficiency of the entire cluster is reduced. As described above, in the distributed data access of the non-shared cluster system, the performance may be particularly deteriorated due to a miss hit of the disk cache at the access request distribution destination.

Further, since the original data and the copy data are arranged as separate files on the secondary storage device, when the access request to the copy data as a result of the load balancing process is processed, the original data file and the copy data are processed together. Since the file is also accessed, the number of times the secondary storage device is accessed is increased, resulting in deterioration of performance.

Therefore, the present invention has been made in view of the above problems, and an object thereof is a distributed data access device constituted by a non-shared type cluster, in which a variation in read load varies among disk devices. It is an object of the present invention to provide a system, a method, an apparatus, and a recording medium that reduce disk cache misses caused by load balancing processing and improve the read performance of distributed data in a situation that occurs.

Another object of the present invention is to provide a system, method, and method for improving the read performance by making it possible to read both original data and duplicated data in one data access process.
An object is to provide an apparatus and a recording medium. Other objects, features, advantages, etc. of the present invention will be immediately apparent to those skilled in the art from the following description.

[0023]

According to the present invention to achieve the above object, there are provided N data processing devices, secondary storage devices respectively connected to the data processing devices, a control device, and the N data processing devices. In a cluster system including a data processing device and a network device that connects the control device, a distributed data processing device that holds distributed allocation data on each of the secondary storage devices, and the distributed allocation data. Is composed of post-data, original data, and pre-data, the original data is placed immediately after the post-data, the pre-data is placed immediately after the original data, and the N units of data are placed. Among the processing devices, the post data of the distributed arrangement data arranged on the secondary storage device connected to the i-th data processing device is (i-1) mod
Nth (However, (i-1) mod N is the remainder obtained by dividing (i-1) by N)
Is a copy of the first ratio (J_ (i-1)%) of the rear part of the original data of the distributed arrangement data on the secondary storage device connected to the data processing device, and the i-th data processing The previous data of the distributed arrangement data on the secondary storage device connected to the device is (i + 1) mod N of the distributed arrangement data on the secondary storage device connected to the data processing device. The second ratio of the front part of the original data (K_ (i + 1)
%).

In the present invention, the control unit stores a read-out area management table in which the respective areas to be read by each of the data processing apparatuses are stored, and each of the distributions arranged in each of the secondary storage apparatuses. Of the post data of the placement data and the original data corresponding to the post data
A storage unit that stores a copy distribution ratio table that holds a first ratio (J_i%), previous data, and a second ratio (K_i%) of original data corresponding to the previous data, and each of the data processing devices. Load monitoring means for monitoring the load, updating the contents of the read area management table, reducing the area in charge of the data processing apparatus having a heavy load, and expanding the area in charge of the data processing apparatus having a light load; Read request means for requesting the data processing device to read data from the area in charge stored in the read region management table, wherein the data processing device stores the distributed arrangement data arranged in the secondary storage device. A data read-out unit for reading out a part or all of the data, which is designated by the assigned area added to the data read-out request from the read-out requesting unit. There.

[0025]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described. The present invention, in one of its preferred embodiments,
Referring to FIG. 1, N data processing devices (2) each having a local secondary storage device (3-1 to 3-N) are provided.
-1 to 2-N) and the control device (1) are connected via a network (5) to form a cluster system,
In the distributed data processing system that holds the distributed arrangement data (4-1 to 4-N) on the respective secondary storage devices (3-1 to 3-N), the data processing devices (2-1 to 2-
In order to correct the load variation between N), the control device (1) is designed to reduce or expand the read data range in charge of each data processing device (2-1 to 2-N) in fear of the load condition. ) Control.

More specifically, each distributed arrangement data (4-1 to 4-N in FIG. 2) is composed of post-data, original data, and pre-data, and the original data immediately after the post-data. Is placed, and the previous data is placed immediately after the original data, and after the distributed placement data placed on the secondary storage device connected to the i-th data processing device of the N data processing devices. Data is stored on the secondary storage device connected to the (i-1) mod N-th data processing device (where (i-1) mod N is the remainder obtained by dividing (i-1) mod N by N). The first ratio (J_ (i-1)) at the rear of the original data of the distributed data of
%), And the previous data of the distributed arrangement data on the secondary storage device connected to the i-th data processing device is
(i + 1) mod The first part of the original data of the distributed data on the secondary storage device connected to the Nth data processing device .
It is supposed to be a duplicate of the second ratio (K_ (i + 1)%). i th
The first ratio of the original data and the rear data (J_i%)
And the second ratio of the i-th original data to the previous data (K_
The sum of i%) is 100% or more.

When the data processing devices (2-1 to 2-N) access the distributed arrangement data (4-1 to 4-N) in which the post-copy data, the original data and the copy data are arranged continuously, Load monitoring means (13) of the load control device (1)
When a node (data processing device) with a high load is detected by the above, the read area management table (1
Change 2). Further, the read area management table (12) is changed so as to enlarge the read area of the node having the copy of the original data of the node.

[0028]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In order to describe the embodiment of the present invention described above in more detail, an embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing the system configuration of an embodiment of the present invention. Referring to FIG. 1, the first embodiment of the present invention stores a control device 1, N (where N is an integer of 1 or more) data processing devices 2-1 to 2-N, and distributed arrangement data. The secondary storage devices 3-1 to 3-N for recording, the distributed arrangement data 4-1 to 4-N, and the network device 5 connecting the control device 1 and the data processing node are configured.

Secondary storage devices 3-1 to 3-N are connected to the data processing devices 2-1 to 2-N, respectively, and the distributed storage data 4 is connected to the secondary storage devices 3-1 to 3-N. -1 to 4
-N are stored respectively.

The data processing devices 2-1 to 2-N are provided with data reading means 21-1 to 21-N, respectively.
Each of the data processing devices 2-1 to 2-N has a disk cache function including a semiconductor memory that stores read data from the secondary storage devices 3-1 to 3-N.

The control device 1 comprises a read request means 11 and
Read area management table 12 and load monitoring means 13
And a copy distribution ratio table 14.

The read request means 11 is the data read means 21-1 to 21- of the data processing devices 2-1 to 2-N.
The data read request from the secondary storage devices 3-1 to 3-N is distributed to N, and the data read means 21-1 to 21-1.
21-N waits for and receives the data read.

The load monitoring means 13 is a data processing device 2-
i (where i is a node number, 1 ≦ i ≦ N) is recorded, and the read area management table 12 is updated according to the load information.

The data read means 21-1 to 21-N of the data processing devices 2-1 to 2-N are read request means 11 respectively.
In accordance with an instruction from, the distributed arrangement data 4-1 to 4-N are read, and the read data is transmitted to the read requesting means 11.

FIG. 2 is a schematic diagram for explaining distributed arrangement data in one embodiment of the present invention.

Referring to FIG. 2, distributed arrangement data 4-i
(1 ≦ i ≦ N) is composed of three pieces of data: post copy data (i-1) mod N, original data i, pre copy data (i + 1) mod N.

Here, a mod b is a remainder obtained by dividing a by b, that is, a remainder r (0 ≦ r of m = q × n + r).
≦ n−1).

Post-copy data (i-1) mod N
Is placed immediately before the original data i, and the previous copy data (i + 1) mod N is placed immediately after the original data i.

Post-copy data (i-1) mod N
Is the original data (i-1) modN rear part J (i-
1) Data that is a copy of% area.

Previous copy data (i + 1) mod N
Is the front part K (i +) of the original data (i + 1) modN.
1) Data that is a copy of% area.

Here, the sum of the ratio J (i) of the post-copy data of the original data i and the ratio K (i) of the pre-copy data is 10
It is set to 0 or more.

In the example shown in FIG. 2, N> 3, and
For example, the distributed arrangement data 4-1 is the post-copy data N (=
(0 mod N), copy data of a predetermined ratio of the rear area of the original data of the Nth distributed arrangement data 4-N), original data 1, previous copy data 2 (= (2 mod
N) The second distributed arrangement data 4-2 is the copy data of a predetermined area of the front area of the original data of the second distributed arrangement data 4-2.
N) Copy data of a predetermined ratio in the area behind the original data of the first distributed arrangement data 4-1), original data 2, previous copy data 3 (= (3 mod N), third distributed arrangement data 4) -3 of the original data, which is a copy data of a predetermined area in the front area).

FIG. 3 is a diagram showing an example of the configuration of the copy distribution ratio table 14 in one embodiment of the present invention.

Referring to FIG. 3, the copy distribution ratio table 14 is composed of N entries, and each entry is
The node ID field, the original size field, the pre-copy rate field, and the post-copy rate field are provided.

In the original size field, the original data i
The length of is stored.

In the previous copy rate field, the ratio of the length of the previous copy data i to the length of the original data i is stored.

The post-copy ratio field stores the ratio of the length of the post-copy data i to the length of the original data i.

FIG. 4 is a diagram showing an example of the configuration of the read area management table 12 in one embodiment of the present invention.

Referring to FIG. 4, the read area management table 12 is constituted by fields of a node ID, a read start position movement ratio, and a read end position movement ratio for each entry.

In the read start position movement ratio field, the length up to the read start position address and the length of the original data are set to 100 with the starting position of the original data as the origin.
The value represented by the ratio is stored.

The read end position movement ratio field stores a value representing the length up to the read end position when the end position of the original data is the origin and the original data length is 100.

FIG. 5A is a flow chart for explaining the operation of the data reading means of the data processing device in the embodiment of the present invention. Referring to FIGS. 1 and 5 (a),
The operation of the data reading means in the embodiment of the present invention will be described in detail.

The data read means 21-i (1≤i≤N) of the data processing device 2-i receives the data read request from the read request means 11 via the network device 5 (step S1-1).

The data reading means 21-i (1 ≦ i ≦ N) of the data processing device 2-i uses the distributed arrangement data 4-based on the read start position movement ratio and the read end position movement ratio transmitted together with the data read request. The read position on i is determined (step S1-2).

Subsequently, the data is read from the distributed arrangement data 4-i arranged in the secondary storage device 3-i (step S1-3). At this time, if the data area is stored in the disk cache (at cache hit)
To read the data on the disk cache,
Fast response time.

On the other hand, if the data area is not stored in the disk cache (at the time of a mishit), it is read from the secondary storage device and stored in the disk cache.

Then, the data reading means 21-i (1 ≦ i ≦ N) of the data processing device 2-i transmits the read data to the read requesting means 11 of the control device 1 via the network device 5 ( Step S1-4).

FIG. 5B is a flow chart for explaining the operation of the read request means 11 of the control device 1 in the embodiment of the present invention. The operation of the read requesting means 11 in the embodiment of the present invention will be described in detail with reference to FIGS. 1 and 5B.

The read request means 11 of the control device 1 reads the read area management table 12 from the data processing device 2-i.
The read start position movement ratio and the read end movement ratio of (1 ≦ i ≦ N) are obtained for all node IDs (step S2-1).

Next, each data processing device 2-i (1≤i≤
The read request is distributed to N). The read start position movement ratio and the read end movement ratio are added to the distributed read request (step S2-2).

Next, the read requesting means 11 of the control device 1
Receives the data read by each data processing device 2-i (1 ≦ i ≦ N) (step S2-3).

FIG. 6 is a flow chart for explaining the operation of the load monitoring means 13 of the control device 1 in the embodiment of the present invention. The operation of the load monitoring means 13 in one embodiment of the present invention will be described in detail with reference to FIGS. 1 and 6.

The load monitoring means 13 records the load information of all the data processing devices 2-1 to 2-N, and periodically checks the presence of a device with a high load (step S3-1).

When the load monitoring means 13 detects a data processing device 2-i (1≤i≤N) having a load value higher than a certain threshold value (YES branch of step S3-2), the data to be subjected to the load distribution processing. Select the processing device 2-i (step S
3-3).

Next, the load monitoring means 13 instructs the read request means 11 to stop the read request (step S3).
-4).

Next, the read target range of the node having a high load is reduced and the read area management table 12 is updated (step S3-5).

Next, the area out of the read target range of the node having the highest load is allocated to the node having the copy data, and the read area management table 12 is updated (step S3-6).

Next, the read requesting means 11 is instructed to start the read processing, and the load balancing processing is completed (step S3).
-7).

FIG. 7 is a flow chart for explaining the method of selecting the data processing device to which the load balancing process is applied in the embodiment of the present invention.

Referring to FIG. 7, first, a data processing device whose load value is higher than a threshold value is temporarily selected (step S4-
1).

Next, the temporarily selected data processing device 2-i
For (1 ≦ i ≦ N), the data processing device 2
-(I-1), a data processing device having a load value higher than that of the data processing device 2- (i + 1) is selected (step S4-).
2).

As described above, in the embodiment of the present invention, the load balancing process is applied to the selected data processing device by the two-step selection.

In this selection process, a plurality of nodes may be selected. In this case, the load balancing process is applied to all the selected nodes.

FIG. 8 is a flow chart for explaining the processing procedure of the method of determining the data distribution ratio to the data processing device having the copy data in the embodiment of the present invention.

The load monitoring means 13 uses the high load node i (1
Reduction ratio R of the data read range of ≦ i ≦ N (however,
0 ≦ R ≦ 100) is determined. Here, R is the ratio of the data length of the read area after reduction when the data length of the original data P is 100.

A temporary distribution ratio for allocating the area E out of the read range in the original data P to the data processing devices P-1 and P + 1 is determined as the inverse ratio of the load value at each point (step). S5-1).

Data processing device P-1 and data processing device P
If the load values of +1 (however, 0 ≦ P−1, P + 1 ≦ N) are 1 (P−1) and 1 (P + 1), respectively, the data processing device P-1 will give 1 (P + 1) / {l ( P-1) +1
The area of (P + 1)} is obtained as the temporary distribution ratio, and the data processing device P + 1 is l (P-1) / {l (P-1) +.
The area of l (P + 1)} is obtained as the temporary distribution ratio.

If the ratio of the length of the data in the area E to the original record length is LE, the ratio of the length of the area allocated to the data processing devices P-1 and P + 1 to the original record length by the temporary distribution ratio is {LE. × (l (P + 1)
/ {L (P-1) + l (P + 1)}): {LE × l (P
-1) / {l (P-1) + l (P + 1)}} (step S5-2).

Next, it is verified whether or not the node can actually correspond to the read range with respect to the value with the large temporary distribution ratio (step S5-3).

The larger value of the provisionally determined distribution ratio is TR
And

When TR is the node P-1, the previous copy rates of TR and node P are compared (step S5-4).

When TR is the node P + 1, the post-copy ratio of TR and node P is compared (step S5-5).

As a result of the comparison, whether TR is equal to the copy rate,
If it is smaller, the temporary distribution ratio is determined as the main distribution ratio (steps S5-6 and S5-8).

On the other hand, if TR is larger than the copy rate,
TR is made equal to the copy rate, and (TR-copy rate) is moved to the smaller provisional distribution rate to be the main distribution rate (steps S5-7, S5-9).

If the main distribution ratio of the data processing devices P-1 and P + 1 is RR (P-1): RR (P + 1), the read start position movement ratio of the data processing device P is RR (P
-1), the read end position movement ratio is RR (P + 1)
Becomes

The read end position movement ratio of the data processing device P-1 is a value obtained by normalizing RR (P-1) so that the previous copy ratio of the original data P is 100, and the read control device P + 1 starts reading. Position movement ratio is RR
The value obtained by normalizing (P + 1) so that the post-copy ratio of the original data P is 100.

Next, a concrete description will be given. FIG. 9 is a diagram for explaining one embodiment of the present invention.

As shown in FIG. 9, node 1, node 2,
It is assumed that the load values of the nodes 3 are 3, 10 and 5, respectively, and the load monitoring unit 13 applies the load distribution processing to the nodes whose load values exceed 9. The load value is calculated based on the CPU load value of each node (data processing device), the sum of response times to past read requests, the average number of disk accesses per unit time, and the like.

In the example shown in FIG. 9, the load value of node 2 is 1
Since it is 0, load balancing processing is applied.

For example, there is a method of determining the reduction range in which the load value of the node P after the change is the average value of the loads of all the nodes.

In the example shown in FIG. 9, the average load of three nodes is 6, so the load balancing process is applied so that the load value of node 2 becomes 6.

When reading a continuous area from the disk device, since the read amount and the read load are proportional, the node is reduced so as to be in charge of the current 60% read range, and 40% outside the read range. Area is allocated to the nodes 1 and 3.

In the example shown in FIG. 9, the temporary distribution ratio is calculated. The area that is not read by the node 2 is 40% of the current read area, and the range distribution ratio between the node 1 and the node 3 is 5: 3. 1 and the temporary distribution ratio of node 3.

It is verified whether or not a node having a large temporary distribution ratio can be expanded as a read range.

When the pre-copy rate of the node 2 is 60% and the post-copy rate is 50%, the temporary distribution rate is determined as the main distribution rate.

When the pre-copy ratio of node 2 is 20% and the post-copy ratio is 90%, the distribution ratio to node 1 is set to 20%, the post-copy ratio is changed from 15% to 20%, and the main distribution ratio is changed. And

According to the present invention, a part or all of the original data is duplicated and arranged as copy data on a plurality of storage devices, and the distributed arrangement data has a structure in which the original data and the copy data are arranged in series. Are arranged as.

When a load imbalance occurs in the read processing of the distributed data, the load monitoring apparatus reduces the high load data processing apparatus so that a part of the immediately preceding read charge area becomes a new read charge area. The load monitoring device reads the area removed from the area in charge of the high-load data processing device so that the data processing device having the pre-copy data and the post-copy data of the original data of the data processing device will be the area in charge of reading. Expand the range. The data processing device whose read area has been changed by the load monitoring device is
Even after the area is changed, a continuous area is set as a reading area.

Next, a second embodiment of the present invention will be described. In the second embodiment of the present invention, the system configuration is basically the same as the configuration of the above-described embodiment shown in FIG. 1, but the configuration of distributed arrangement data is different.

FIG. 10 shows distributed arrangement data 6-i arranged in the node i (secondary storage device of the i-th data processing device) in the second embodiment of the present invention.
Referring to FIG. 10, m pieces of distributed arrangement data 6-i (m
Is one or more), and each entry is composed of a key and two pieces of element data.

Element data is for each key kp (where p is 1
The original data kp-i, the pre-copy data kp- (i + 1), and the post-copy data kp- (i
-1) and is comprised.

The post-copy data is placed at the beginning of the original data, and the pre-copy data is placed at the end of the original data.

The post-copy data kp- (i-1) in the distributed arrangement data 6-i is the original data Kp- (i-1).
The predetermined ratio J_ (i-1)% of the rear part (second half) is duplicated, and the front copy data kp- (i + 1) is the predetermined ratio K_ (i +) of the front part (first half) of the original data Kp- (i + 1).
1) Data that is a copy of%.

The operation of the second embodiment of the present invention will be described with reference to FIGS.

The read request means 11 issues a read request to the data processing device 2-i, and specifies the read target key together with the read range start position movement ratio and the read end movement ratio.

The data reading means 21-i of the data processing device 2-i reads the assigned range of the element data of the key designated by the read requesting means 11 and sends it to the data reading means 21.

The operation and effect of the second embodiment of the present invention will be described.

In the present embodiment, since the duplicated data and the original data are arranged consecutively for each key even in the data having the array structure, the data distribution range is reduced by the load balancing process performed by the load monitoring means 13. However, since the read range of the element data of each key is included in the read range before the change, it is expected to speed up the read process by the disk cache.

Further, even when the read range is expanded, the read range of the element data of each key is read out from the part continuous with the area before the change, so that the effect of the disk cache can be expected.

The functions of the read request means 11 and the load monitoring means 13 in the control device 1 described above are realized by a program executed on a computer constituting the control device 1. The functions of the data reading units 21-1 to 21-N in the data processing devices 2-1 to 2-N are also realized by the programs executed by the data processing devices 2-1 to 2-N. In this case, a medium recording these programs (for example, semiconductor memory, FD, CD-RO)
M, DVD (digital versatile disk, etc.) or communication medium (which may be either wired or wireless transmission medium), the program is read out through a predetermined interface, and the control device 1 and the data processing device 2- 1-2
The present invention can be implemented by loading and executing the main memory of -N.

[0112]

As described above, according to the present invention,
The following effects are achieved.

The first effect of the present invention is that even if the change of the read area of the distributed data due to the load distribution processing is applied, the read response time by the disk cache can be expected to be shortened.

The reason for this is that in the present invention, since the information processing apparatus controlled by the operating system used in the configuration such as the cluster system is generally equipped with the disk cache, the data before the area change is stored in the disk cache. This is because the data after the read-out area reduction is included in the previous data in the case where it is included. Therefore, even if the read area is reduced, the reduced data processing device can still read the data in the cache. Therefore, it is possible to surely avoid the sharp deterioration of the read response time due to the removal of the disk cache. can do.

According to the present invention, the effect of the disk cache can be expected also in the read area expansion processing.

The first reason is that the area in charge of reading after the change reads out a portion continuous with the area before the change.
This is because if the area before the change is stored in the disk cache, the reading of that portion can be processed at high speed.

The second reason is that since the disk cache is generally recorded in a certain block size unit, if the area before the change is stored in the disk cache,
This is because there is a high possibility that the expanded area will also exist in the disk cache at the same time, and it can be expected that the response time is prevented from being deteriorated due to being out of the disk cache.

Further, in the case of distributing the load for each query, the entire copy is always read when the copy is read, so that the possibility of being out of the cache is large, but the load is large. When distributing the load by changing the read area,
This means that the reading of a copy is performed for a part of the copy, so that the possibility of being missed in the cache and the load when it is missed can be reduced.

The second effect of the present invention is that even if the read area is changed, continuous read processing of the data area is performed, so even if the secondary storage device is a disk device, seek processing does not occur, Therefore, it is possible to prevent the deterioration of the response time due to the seek processing.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of an exemplary embodiment of the present invention.

FIG. 2 is a diagram for explaining original data and distributed arrangement data according to an embodiment of the present invention.

FIG. 3 is a diagram showing an example of a configuration of a copy distribution ratio table in an embodiment of the present invention.

FIG. 4 is a diagram showing an example of a configuration of a read area management table according to an embodiment of the present invention.

FIG. 5A is a flowchart for explaining the operation of the data reading means in the embodiment of the present invention. (B) is a flow chart for explaining the operation of the read requesting means in the embodiment of the present invention.

FIG. 6 is a flowchart for explaining the operation of the load monitoring means in the embodiment of the present invention.

FIG. 7 is a flowchart for explaining an operation of selecting a node to which the load balancing process of the load monitoring means is applied in the embodiment of the present invention.

FIG. 8 is a flowchart for explaining an operation of determining a read range expansion area of a node having copy data of the load monitoring means in the embodiment of the present invention.

FIG. 9 is a diagram for explaining an example of the present invention.

FIG. 10 is a diagram showing a configuration of distributed arrangement data according to another embodiment of the present invention.

[Explanation of symbols]

1 control device 2-1 to 2-N data processing device 3-1 to 3-N secondary storage device 4-1 to 4-N distributed arrangement data 5 Network equipment 6 Distributed arrangement data of the second embodiment 11 Read requesting means 12 Read area management table 13 Load monitoring means 14 Copy distribution ratio table 21 data reading means

Claims (17)

(57) [Claims] 1. N data processing devices, a secondary storage device connected to each of the data processing devices, a control device, and a network device connecting the N data processing devices and the control device. A data storage method of a distributed data processing device, which holds distributed arrangement data on each of the secondary storage devices, in a cluster system comprising: i-th (where 1 ≦ i ≦ N) distribution Placement data is number i
Is configured and a eye after data and the i-th original data and the i-th previous data, the i-th of the original data is located immediately after the data after the i-th, the i-th original data Immediately after the i-th previous data is placed, the i-th subsequent data is (i-1) mod N-th (however,
"(I-1) mod N" is, the original data of the remainder of (i-1) was divided by N)
Data is the latter half of the duplicate, and the latter half of the duplicate (i-1)
mod The ratio to the Nth original data is the ith first
Is the ratio of the i-th previous data, (i + 1) mod N th (where,
"(I + 1) mod N" is, the original data of the (i + 1) remainder was divided by N)
Ri replication der of the first half of the over data, said first half of replication (i + 1)
mod The ratio to the Nth original data is the ith second
The data storage method is characterized by being a ratio of . 2. The i-th first ratio and the i-th first ratio
The sum of the two ratios is 100 % or more in percentage ,
The data storage method according to claim 1, wherein: 3. N data processing devices, a secondary storage device connected to each of the data processing devices, a control device, and a network device connecting the N data processing devices and the control device. In a cluster system comprising: a distributed data processing device that holds distributed allocation data on each of the secondary storage devices, wherein the i-th (where 1 ≦ i ≦ N) distributed allocation data is i number
Is configured and a eye after data and the i-th original data and the i-th previous data, the i-th of the original data is located immediately after the data after the i-th, the i-th original data Immediately after the i-th previous data is arranged, the i-th subsequent data is (i-1) mod N-th (however,
"(I-1) mod N" is, the original data of the remainder of (i-1) was divided by N)
A late replica of chromatography data, the said second half of replication (i-1) to mod N th original data
The ratio to the i-th is the first ratio, and the i-th previous data is (i + 1) mod N-th (however,
"(I + 1) mod N" is, the original data of the (i + 1) remainder was divided by N)
The first half of the data, and the (i + 1) of the first half
mod The ratio to the Nth original data is the ith second
A distributed data processing device, characterized in that: 4. The storage unit for storing a read-out area management table in which the control apparatus records the areas to be read by each of the data processing apparatuses, and the N first ratios and the second ratios. A storage unit for storing a copy distribution ratio table to be stored, a load of each data processing device is monitored, the contents of the read region management table are updated, and a responsible region of the data processing device having a heavy load is stored. And a load monitoring unit for reducing the load of the data processing device having a light load, and a read request for requesting the data processing devices to read data from the charge region stored in the read region management table. Means for distributing distributed data arranged in the secondary storage devices connected to the data processing devices. 4. The distributed data processing device according to claim 3, further comprising data reading means for reading a part or all of the data designated by the assigned area added to the data reading request from the read requesting means. . 5. The i-th first ratio and the i-th first ratio
The distributed data processing device according to claim 3 or 4, wherein the sum of the two ratios is 100 % or more in percentage . 6. In the control device, when executing the load balancing process, the load monitoring means records the load information of the data processing device, periodically checks the existence of a device having a high load, and determines a predetermined value. Detecting a data processing device having a load value higher than the threshold value, selecting a data processing device to be subjected to load balancing processing, and the load monitoring unit instructs the read requesting unit to stop the read request, The read-out area management table is updated by reducing the read-out target range of the data processing apparatus having a high load, and the read-out area is allocated to the data processing apparatus having the copy data, the area outside the read-out target range of the data processing apparatus having a high load. The management table is updated, and the read requesting unit is instructed to start the read processing. Distributed data processing device. 7. The load monitoring means determines the data distribution ratio to the data processing device having copy data, the data read range of a high-load data processing device (P) of the N data processing devices. Reduction ratio R (where R is the ratio of the data length of the read area after reduction when the data length of the original data (P) is 100) is determined, and the original data (P) is read from the read target range. The provisional distribution ratio for allocating the deviated area (E) to the data processing device (P-1) and the data processing device (P + 1) is determined as the inverse ratio of the load value at each time, E) The ratio of the length of the data to the length of the original record and the ratio of the lengths of the areas allocated to the data processing device (P-1) and the data processing device (P + 1) based on the temporary distribution ratio are obtained. If the data processing device (P-1) has a larger value of the temporary distribution ratio, the temporary ratio of the data processing device (P-1) and the pre-copy ratio of the data processing device (P) are larger. When the data processing device (P + 1) is larger, the temporary ratio of the data processing device (P-1) and the post-copy ratio of the data processing device (P) are compared, If the copy ratio is equal to or smaller than the copy ratio, the provisional distribution ratio is determined as the main distribution ratio. On the other hand, if the provisional ratio is larger than the copy ratio, the provisional ratio is set equal to the copy ratio, and the provisional ratio 7. The distributed data processing device according to claim 6, wherein the copy ratio) is moved to a smaller temporary distribution ratio to obtain the main distribution ratio. 8. N data processing devices, a secondary storage device connected to each of the data processing devices, a control device, and a network device connecting the N data processing devices and the control device. in a cluster system comprising comprising the door, a said respective secondary storage device runtime load balancing how put the distributed data processing device holding distributed data respectively on, i-th (where, 1 ≦ i ≦ N ), Each distributed data is i
Is configured and a eye after data and the i-th original data and the i-th previous data, the i-th of the original data is located immediately after the data after the i-th, the i-th original data Immediately after the i-th previous data is arranged, the i-th subsequent data is (i-1) mod N-th (however,
"(I-1) mod N" is, the original data of the remainder of (i-1) was divided by N)
A late replica of chromatography data, the said second half of replication (i-1) to mod N th original data
The ratio to the i-th is the first ratio, and the i-th previous data is (i + 1) mod N-th (however,
"(I + 1) mod N" is, the original data of the (i + 1) remainder was divided by N)
The first half of the data , and the (i + 1) of the first half
mod The ratio to the Nth original data is the ith second
The control device stores and manages a responsible area to be read by each of the data processing devices in a read area management table, monitors the load of each of the data processing devices, and handles the data processing device having a heavy load. The area is reduced, the area in charge of the data processing device having a light load is expanded, and each data processing device is requested to read a part or all of the data specified by the area in charge of the distributed arrangement data.
A runtime load balancing method characterized in that 9. N data processing devices, a secondary storage device connected to each of the data processing devices, a control device, and a network device connecting the N data processing devices and the control device. A cluster system comprising: a data storage method of a distributed data processing device, wherein a plurality of and the same number of distributed arrangement data are respectively stored on each of the secondary storage devices, the i-th (where 1 ≦ i ≦ N) each distributed data is i
Is configured and a eye after data and the i-th original data and the i-th previous data, the i-th of the original data is located immediately after the data after the i-th, the i-th original data Immediately after the i-th previous data is arranged, J (where 1 ≦ J ≦ M on the secondary storage device connected to the i-th (where 1 ≦ i ≦ N) data processing device )
The subsequent data in the th distributed arrangement data is the (i-1) mod N-th data processor (where "(i-1) mod N" is the remainder obtained by dividing (i-1) by N). the J-th original data on the secondary storage device connected to a replica of the rear part of the J-th first ratio, i-th of said data processing device to the connected on the secondary storage device Previous data in the J-th distributed data,
(i + 1) mod Nth (however, "(i + 1) mod N" is the remainder obtained by dividing (i + 1) mod N) data processing device connected to the secondary storage device. is a replica of the J-th front portion of the J-th second ratio of the original data in a distributed arrangement data, data storage wherein the. 10. N data processing devices, a secondary storage device connected to each of the data processing devices, a control device, and a network device connecting the N data processing devices to the control device. A distributed data processing device that holds a plurality and the same number of distributed arrangement data on each of the secondary storage devices in a cluster system that comprises: i-th (where 1 ≦ i ≦ N) Distributed arrangement data is number i
Is configured and a eye after data and the i-th original data and the i-th previous data, the i-th of the original data is located immediately after the data after the i-th, the i-th original data Immediately after the i-th previous data is arranged, J (where 1 ≦ J ≦ M on the secondary storage device connected to the i-th (where 1 ≦ i ≦ N) data processing device )
The subsequent data in the th distributed arrangement data is the (i-1) mod N-th data processor (where "(i-1) mod N" is the remainder obtained by dividing (i-1) by N). the J-th original data on the secondary storage device connected to a replica of the rear part of the J-th first ratio, i-th of said data processing device to the connected on the secondary storage device Previous data in the J-th distributed data,
(i + 1) mod Nth (however, "(i + 1) mod N" is the remainder obtained by dividing (i + 1) mod N) data processing device connected to the secondary storage device. J th is a replica of the front portion of the J-th second ratio of the original data in a distributed arrangement data, distributed data processing apparatus according to claim. Wherein said control device includes a storage unit that stores the read area management table in which the respective data processing device is recorded coverage area to be read, respectively, and the N of said first ratio to said second ratio A storage unit for storing a copy distribution ratio table to be stored, a load of each data processing device is monitored, the contents of the read region management table are updated, and a responsible region of the data processing device having a heavy load is stored. Load monitoring means for reducing the load and expanding the area in charge of the data processing device having a light load, and reading data from the area in charge stored in the read area management table to each data processing device.
Read requesting means for requesting the data processing device,
For distributed data located in the secondary storage device are respectively connected, are designated by the coverage area attached to the data read request from said read request means, data reading means for reading the whole or in part The distributed data processing device according to claim 10, characterized by being provided. 12. A N-data processing device, a secondary storage device connected to each of the data processing devices, a control device, and a network device connecting the N data processing devices and the control device. in a cluster system comprising comprising the door, a run-time load balancing method in a distributed data processing device holding a plurality and the same number of distributed data to the respective secondary storage device on, i-th (where, 1 ≦ i ≤ N), each distributed arrangement data is i
Is configured and a eye after data and the i-th original data and the i-th previous data, the i-th of the original data is located immediately after the data after the i-th, the i-th original data Immediately after the i-th previous data is arranged, J on the secondary storage device connected to the i-th (where 1 ≦ i ≦ N) data processing device (where 1 ≦ J ≦ M) )
The subsequent data in the th distributed arrangement data is the (i-1) mod N-th data processor (where "(i-1) mod N" is the remainder obtained by dividing (i-1) by N). the replication of the rear part of the J-th first ratio of J-th original data on the secondary storage device connected to said secondary being connected to the i-th of said data processing apparatus
The previous data in the J-th distributed arrangement data on the storage device is
(i + 1) mod Nth (However, "(i + 1) mod N" is (i + 1)
Remainder divided by N) before connected to the data processing device
The original data in the Jth distributed arrangement data on the secondary storage device
It is a copy of the J-th second ratio of the front part of the data , wherein the control device stores and manages the area in charge to be read by each of the data processing devices in the read region management table, Monitoring the load, reducing the area in charge of the data processor with a heavy load, expanding the area in charge of the data processor with a light load, and assigning to each of the data processors a part or a part designated by the area in charge of the distributed data Request full read,
A runtime load balancing method characterized in that 13. A predetermined number (N) of data processing devices, each of which has a secondary storage device, and a control device are connected via a network to form a non-shared cluster system, and each of the secondary devices. the distributed data respectively held in the storage device, i-th (where, 1 ≦ i ≦ N) each distributed data of, i th
Is configured and a eye after data and the i-th original data and the i-th previous data, the i-th the original data immediately after the data after the i-th
Is placed immediately after the i-th original data.
Are arranged, and the i-th post data is (i-1) mod N-th (however,
"(I-1) mod N" is the latter half of the original data of (i-1) divided by N) , and the latter half of the (i-1)
mod The ratio to the Nth original data is the ith first
And the i-th previous data is (i + 1) mod N-th (however,
"(I + 1) mod N" is, (i + 1) is the first half of the replica of the original data of remainder) divided by N, the of the first half of replication (i + 1)
mod The ratio to the Nth original data is the ith second
Read area management table in which the control device records the areas in charge to be read by each of the data processing devices.
Comprising a storage unit for storing a storage unit for storing a copy <br/> over distribution ratio table for storing N pieces of the first ratio and the second ratio, the a, the data processing apparatus, wherein Located in secondary storage
Data reading of distributed arrangement data from the control device
Partial or partial, specified by the area of responsibility attached to the request.
Distributed data processing system
System, a program for the control device to execute.
A ram, (a) the monitor the load of each data processing device, and updates the contents of the read area management table, reducing the coverage area of the heavy data processing apparatus load, lightly loaded data processor Load control processing for expanding the area in charge of the control device, and (b) read request processing for requesting data read from the area in charge stored in the read area management table to each of the data processing devices, to the control device. A recording medium on which a program for recording is recorded. 14. A predetermined number (N) of data processing devices, each of which has a secondary storage device, and a control device are connected via a network to form a non-shared cluster system, and each of the secondary devices the distributed data respectively held in the storage device, i-th (where, 1 ≦ i ≦ N) each distributed data of, i th
Is configured and a eye after data and the i-th original data and the i-th previous data, the i-th the original data immediately after the data after the i-th
Is placed immediately after the i-th original data.
Are arranged, and the i-th post data is (i-1) mod N-th (however,
"(I-1) mod N" is, (i-1) is the replication original half after the data of remainder) divided by N, said second half of replication (i-1)
mod The ratio to the Nth original data is the ith first
And the i-th previous data is (i + 1) mod N-th (however,
"(I + 1) mod N" is, (i + 1) is the first half of the replica of the original data of remainder) divided by N, the of the first half of replication (i + 1)
mod The ratio to the Nth original data is the ith second
Read area management table in which the control device records the areas in charge to be read by each of the data processing devices.
A storage unit for storing a storage unit for storing a copy <br/> over distribution ratio table for storing N pieces of the first ratio and the second ratio, the load of each data processing device monitors A load monitoring unit for updating the contents of the read area management table to reduce the area in charge of the data processing apparatus having a heavy load and to expand the area in charge of the data processing apparatus having a light load; A distributed data processing system including read request means for requesting data read from a responsible area stored in the read area management table.
System for executing the data processing device
A program, (a) the distributed data located in said secondary storage device is specified by the added charge area to the data read request from the control device reads data read out part or all processing A recording medium in which a program for causing the control device to execute is recorded. 15. A non-shared cluster system is configured by connecting a predetermined number (N) of data processing devices each having a secondary storage device and a control device via a network, and configuring each of the secondary storage devices. A distributed data processing system for holding distributed arrangement data on a device, wherein the i-th (where 1 ≦ i ≦ N) distributed arrangement data is the i-th
Is configured and a eye after data and the i-th original data and the i-th previous data, the i-th the original data immediately after the data after the i-th
Is placed immediately after the i-th original data.
Are arranged, and the i-th post-data is (i-1) mod N-th (however,
"(I-1) mod N" is, the original data of the remainder of (i-1) was divided by N)
A late replica of chromatography data, the said second half of replication (i-1) to mod N th original data
The ratio to the i-th is the first ratio, and the i-th previous data is (i + 1) mod N-th (however,
"(I + 1) mod N" is, the original data of the (i + 1) remainder was divided by N)
The first half of the data, and the (i + 1) of the first half
mod The ratio to the Nth original data is the ith second
The read area management table in which the responsible areas to be read by each of the data processing devices are recorded, and the number of copies of the N first ratio and the second ratio is stored by the control device. -A storage unit that stores a distribution ratio table, monitors the load of each of the data processing devices, updates the contents of the read region management table, reduces the area in charge of the data processing device that has a heavy load, and reduces the load. Load monitoring means for expanding the area in charge of the light data processing device, and read request means for requesting each of the data processing devices to read data from the area in charge stored in the read area management table, The data processing device designates the distributed arrangement data arranged in the secondary storage device according to the assigned area added to the data read request from the control device. Is the, a data reading means for reading a part or the whole,
A distributed data processing system characterized by the above. 16. The i-th first ratio and the i-th first ratio
The sum of the second ratios is 100 % or more in percentage ,
16. The distributed data processing system according to claim 15, wherein: 17. Each of the distributed arrangement data includes one or a plurality of entries, each entry includes a key and element data, and the element data includes each key kp (where p is 1 or more and m Each of the following), i-th original copy data, i-th post-copy data, and i-th previous copy data are configured, and the i-th post-copy data is the i-th original copy data. are arranged on top the i-th pre-copy data is placed at the end of the i-th of said original data, copying data after the i-th, the first rear portion of the i-th original data the ratio is a replication, distributed data processing system of claim 15, wherein the i-th pre-copy data is a duplicate of a second ratio of the front of the i-th original data, it is characterized.