JP4478000B2 - Data mediation method and data mediation device - Google Patents

Description

  The present invention relates to a data mediating method and a data mediating apparatus.

  A form in which a storage device such as an HDD (Hard Disk Drive) built in a computer or directly connected by a cable has a network connection interface and functions as a single network device is becoming widespread. When a device other than the storage device accesses the storage device, which is a network device, a protocol such as iSCSI (Internet Small Computer System Interface, Non-Patent Document 1) is used.

  Conventionally, when a storage device installed remotely is used, one file is stored in one storage device. However, it is possible to reduce the read / write load on each storage device by distributing one file in a plurality of storage devices. It is also possible to improve processing speed by simultaneously reading and writing to a plurality of storage devices.

When using a storage device installed in a remote network to implement distributed data distribution, it is possible to distribute the load that occurs between the network where the storage device is installed and the backbone network, which occurs during reading and writing. Is possible.
IETF, “Internet Small Computer Systems Interface”, [online], [searched on November 8, 2004], Internet <URL: http://www.ietf.org/rfc/rfc3720.txt>, Chapter 3.2 , 3.4

  In the above technique, one file is divided into a plurality of file parts, and each file part is distributed and recorded in separate storage devices, thereby reducing the load on each storage device and network.

  However, when data is read from a plurality of storage devices at the same time, the data arrives at the client in a state where the data is not aligned, which causes a problem that alignment processing is required.

  When trying to solve the above problems with the prior art, a new device or program, a large-capacity high-speed memory for a buffer used for reading and writing data, and a processor with high computing performance for performing alignment processing Need to be introduced to the client. Therefore, the configuration of each client becomes complicated. As a result, for example, the processing operation is slow, and the size of the housing is increased.

  In addition, when sharing a remote storage device group with a plurality of clients, it is necessary to synchronize information on the distributed arrangement of data among all clients, which causes a problem of overhead generated in the network and clients.

  Since this overhead increases in proportion to the number of clients sharing, as the number of clients increases, the configuration required for the network and clients becomes complicated.

  SUMMARY OF THE INVENTION Accordingly, it is a primary object of the present invention to solve the above problems and to simplify the configuration of a client for a communication system that stores data in a distributed manner.

In order to solve the above-described problem, the present invention provides a client that requests reading / writing of data, a storage device that stores the data requested to be read / written by the client, and stores each data stored in the storage device A network system configured to include a data intermediary device having an index DB including a used area table for associating a location and an unused area table for managing an address and an unused block position on the network of the storage device The data mediating method according to claim 1, wherein the data mediating device comprises a first interface used for communication with the client and a second interface used for communication with the storage device, from the client to the first read request and write requirements via the interface A step of receiving a procedure in which the acquired location information of the read data read request from the index DB on the basis of requests to read data via the second interface to a storage device corresponding to the position information And a fragment of data received via the second interface as a response to the data read request from the storage device, aligned in the read data buffer and stored, and the data stored in the data buffer is the head a step of returning via said first interface to said client in order from the write request to determine the location of the write data from the index DB on the basis of the second to the storage device corresponding to the position information and procedures for requesting writing of data via the interface, the In writing the results received via the second interface in reply to the request for writing of the data from憶device, and executes a a procedure for updating the index DB.

  Thus, when the client accesses the storage devices distributed on the remote network, the data mediating device collectively manages the data and mediates the access to the storage device. Therefore, it is not necessary to introduce a new configuration for the client to directly access the storage device, and the configuration can be simplified. Therefore, regardless of the number of clients, it is possible to reduce the processing operation delay, the increase in the size of the housing, the introduction cost of components, and the like.

  According to the present invention, at least one of the first interface for the data mediating device to communicate with the client or the second interface for communicating with the storage device is an iSCSI (Internet Small Computer). Data is communicated by a data communication protocol in accordance with (System Interface).

  As a result, a client and a storage device compatible with iSCSI, which is an already popular standard, can be used as part of the communication system. Therefore, it is possible for the developer to reduce the cost and period for developing a new client and storage device, and for the system builder, the existing equipment can be utilized, so that the introduction cost can be reduced.

The present invention relates to a client that requests reading / writing of data, a storage device that stores the data that has been requested to be read / written by the client, and a used area table that associates storage locations for each data stored in the storage device And a data mediating device having an index DB including an unused area table that manages the addresses and unused block positions on the network of the storage device, and a data mediating device in a network system comprising: A first interface used for receiving a read request and a write request from the client, a second interface used for communication with the storage device, and obtaining position information of read data from the index DB based on the read request Corresponding to the position information. A read control unit that requests reading of data via the second interface device, the pieces of data received via the second interface in reply to a request of reading of the data from the storage device, read data buffer stores are aligned to, and the reading data constructing unit for returning the data stored in the data buffer from the beginning through the first interface to the client in order, writing from the index DB on the basis of the write request determining the position information of the data, and a write control unit that requests writing of data via the second interface to a storage device corresponding to the position information, the second as a response to the request for writing of the data from said storage device writing received through the interface In result, characterized in that it is constituted by including a write result processing unit that updates the index DB.

  Thus, when the client accesses the storage devices distributed on the remote network, the data mediating device collectively manages the data and mediates the access to the storage device. Therefore, it is not necessary to introduce a new configuration for the client to directly access the storage device, and the configuration can be simplified. Therefore, regardless of the number of clients, it is possible to reduce delays in processing operations, an increase in the size of the housing, the introduction cost of parts, and the like.

  According to the present invention, at least one of the first interface for communicating with the client or the second interface for communicating with the storage device is in accordance with iSCSI (Internet Small Computer System Interface). Data is communicated by a data communication protocol.

  As a result, a client and a storage device compatible with iSCSI, which is an already popular standard, can be used as part of the communication system. Therefore, it is possible for the developer to reduce the cost and period for developing a new client and storage device, and for the system builder, the existing equipment can be utilized, so that the introduction cost can be reduced.

  According to the present invention, when a client accesses a storage device distributed on a remote network, the use of a data mediating device eliminates the need to introduce a new device to the client and simplifies the configuration. it can. Therefore, regardless of the number of clients, it is possible to reduce delays in processing operations, an increase in the size of the housing, the introduction cost of parts, and the like. Since the data mediating apparatus mediates data access from any client connected to the network, it is functionally equivalent compared to a configuration in which each client has a function of directly accessing the storage device.

  FIG. 1 is a configuration diagram illustrating a communication system. In FIG. 1, it is assumed that each network has a communication device based on IP (Internet Protocol) and communication using IP is possible, but a general network may be used instead of IP.

  The communication system in FIG. 1 is a backbone network NC that is an IP network composed of broadband links, an internal IP network NI that is a network to which a client C is directly connected, and a network to which a storage device R is connected. A certain remote IP network NR is connected by an edge router (not shown).

  The internal IP network NI is configured using a LAN (Local Area Network) such as Ethernet (registered trademark), for example, and each interface is a LAN network cable (not shown) and a network hub (not shown). Communication via IP is possible via a relay device such as a router or a router (not shown). Similarly to the internal IP network NI described above, the remote IP network NR is configured using a LAN and can communicate with each other by IP.

  Further, the communication system of FIG. 1 includes a data mediating device 1 connected to the internal IP network NI, a client C that accesses data to the storage device R via the data mediating device 1, and a control from the data mediating device 1. And a storage device R that reads and writes data.

  Note that each device (client C, data mediating device 1, and storage device R) that constitutes the communication system of FIG. 1 includes a memory serving as storage means used when performing arithmetic processing, and an arithmetic processing device that performs the arithmetic processing. The computer is provided with at least. The memory is constituted by a RAM (Random Access Memory) or the like. Arithmetic processing is realized by an arithmetic processing unit configured by a CPU (Central Processing Unit) executing a program on a memory.

  The client C obtains data from the storage device R realized by the prior art connected to the internal IP network NI, and uses the data in the running application or sends data from the running application to the internal IP network NI. Assume a computer having a function of recording in the storage device R connected to the. In some cases, the remote storage device R does not have a function of reading and writing data.

  It is assumed that the storage device R is a storage device that can be directly connected to the network. The storage device R has a function capable of freely reading and writing from an external device in a smaller granularity unit such as a block unit rather than a file unit. In order to perform disk access in units of blocks, the storage device R is implemented with a remote access protocol such as iSCSI, for example.

  The data mediating apparatus 1 includes two types of devices: a reading mediating device 1R (see FIG. 2) and a writing mediating device 1W (see FIG. 3). In these two apparatuses, the contents of the index DB 10 are slightly different. That is, the intermediary device 1W for writing uses an unused area table 12 (see FIG. 4) for managing the positions of unused blocks in the storage device R in the index DB 10 and a used area table 16 (see FIG. 4) for managing the arrangement state of files. However, the reading mediating apparatus 1R has a used area table 16 (see FIG. 5).

  2 includes a client-side interface IC, a storage device-side interface IR, an index DB 10, a read control unit 14, a read data buffer 20, and a read data constructing unit 22. Each component will be specifically described below.

  The client-side interface IC that is an interface for communicating with the client C has two functions regarding the reading process. One is a function for waiting for a data read request from the client C, and the other is a function for returning the requested data to the client C. The client side interface IC can wait for requests from a plurality of clients C at the same time.

  The storage device side interface IR, which is an interface for communicating with the storage device R, has a function of issuing a read command to the storage device R connected to the remote network and acquiring the result regarding the read processing. Basically, there are a plurality of storage device side interfaces IR. However, when the size of the data to be acquired is very small or the storage device R has a free space problem, it is distributed and stored in the plurality of storage devices R. When data that could not be read is read out, it may become a single unit.

  The index DB 10 that associates the storage location of each data stored in the storage device R is a database that manages what data fragment is stored in which part of which storage device R in the group of storage devices R. is there. The index DB 10 has a used area table 16 (see FIG. 5) that manages the arrangement state of files. The used area table 16 manages information on each stored file. Information on the arrangement state of the file is managed, such as the size of each stored file and which range in the file is recorded in which block of which storage device R. Information on the arrangement state of the file is used when reading and updating the file.

  The used area table 16 shown in FIG. 5 includes a file name of a file that is a unit operated by the client C, a data range of the file name (order in the file data), an ID of the storage device R in which the file is stored, It is a correspondence table of the storage block of the storage device R in which the file is stored. For example, the file A is distributed and stored in two storage devices R (ID = A, B).

  The read control unit 14 reads data requested to be read from the client C from the storage device R. For this reason, the reading control unit 14 accesses the used area table 16 to receive a reading condition such as the name of a file to be read and position information in the file at the time of reading, and the corresponding position information is received. Has the function to return. Specifically, the read control unit 14 receives data read conditions from the client-side interface IC, transmits the information to the index DB 10, obtains position information of the requested data fragment, and performs reading. Is determined, and a read request is made to the storage device side interface IR.

  The read data buffer 20 temporarily stores data requested to be read by the client C. Then, the read data construction unit 22 constructs correct data (aligned data) from the data received from the storage device R. In other words, the read data construction unit 22 reads data from the read data buffer 20 and transmits it to the client C when a transmission command is sent to the client side interface IC as soon as preparation for transmission to the client C is completed.

  3 includes a client-side interface IC, a storage device-side interface IR, an index DB 10, a write control unit 18, a write data buffer 24, and a write result processing unit 26. Hereinafter, the description of the same components as those of the read intermediary device 1R illustrated in FIG. 2 will be omitted, and each component of the write intermediary device 1W will be specifically described.

  The client side interface IC is an interface for communicating with the client C. The client side interface IC has two functions regarding the writing process. One is a function for requesting data update or new storage from the client C and receiving the data to be stored, and the other is whether the storage of the data requested by the client C has been completed normally or has failed. This is a function for notifying the client C of the above. The client side interface IC can wait for requests from a plurality of clients C at the same time.

  The storage device side interface IR is an interface when communicating with the storage device R. Then, the storage device side interface IR issues a write command to the storage device R connected to the remote IP network NR with respect to the write processing, and acquires the result. Basically, there are a plurality of storage device side interfaces IR. However, when the size of data to be written is very small or due to a problem with the free space of the storage device R, it cannot be distributed and stored in the plurality of storage devices R. In some cases, it may be a single entity.

  The index DB 10 receives information such as the name of the file to be written, determines whether the data is updated or newly saved, receives the file size, and which storage device R has room to record the data return it. Therefore, in addition to the used area table 16, the index DB 10 further includes an unused area table 12 (see FIG. 4) for managing the positions of unused blocks in the storage device R. In this unused area table 12, information of storage device R groups that the client C regards as one storage device R is managed. Each storage device R is assigned an ID, and the storage device R is uniquely identified by the ID. An IP address is associated with an ID, and the location on the network can be specified by using this information. The capacity and unused block information is used when updating a file or saving a new file.

  The unused area table 12 shown in FIG. 4 includes the storage device R located in the remote IP network NR, the IP address of the storage device R, the ID of the storage device R, the capacity that can be stored in the storage device R, and the storage device. 7 is a correspondence table of R storable unused blocks. For example, in the storage device R (ID = A), there are two blocks, ie, a first block 101 to 120 and a second block 171 to 200 as unused blocks.

  The write control unit 18 writes the data requested to be written by the client C into the storage device R. Specifically, the write control unit 18 transmits the write data condition to the index DB 10 and acquires the location information of the update source file and the availability of the storage device R. In consideration of the conditions obtained from the acquired information, it is determined how to divide the received data and in which storage device R it is appropriate to store, and the storage device R to be written is selected. The selected storage device side interface IR is instructed to save a fragment of data stored in the write data buffer 24.

  The write data buffer 24 temporarily stores data requested to be written by the client C. Therefore, the write data buffer 24 is composed of a large-capacity high-speed memory or the like so as not to cause a delay in data transfer processing from the client C. The write data buffer 24 temporarily stores data received by the client side interface IC.

  The writing result processing unit 26 determines whether all writing has been normally completed. If all are successful, the write result processing unit 26 updates the result in the index DB 10 and instructs the client side interface IC to inform the client C that the storage is successful. If the writing fails for some reason, the writing result processing unit 26 instructs the client C to notify the failure, and performs a process of canceling all the writing operations performed so far.

  FIG. 6 is a flowchart showing data reading processing by the reading mediating apparatus 1R. A communication message in the process of FIG. 6 is shown in FIG.

  First, the read mediating apparatus 1R receives a read request P11 from the client C (S101). Next, the read control unit 14 acquires the position information of the read data from the index DB 10 based on the read request P11 (S102). Then, the storage device side interface IR transmits a message (data fragment request P12) requesting the read data to each storage device R based on the position information of the read data (S103).

  Further, the storage device side interface IR acquires the read data (data fragment P13) from each storage device R as a response to the read data in S103 (S104). Then, the read data construction unit 22 arranges the read data by storing the read data received in S104 in the read data buffer 20 according to the order of the files (S105). Further, the client side interface IC sequentially reads the read data (data P14) arranged in S105 from the read data buffer 20, and transmits it to the client C (S106).

  FIG. 8 is a flowchart showing a data writing process. A communication message in the process of FIG. 8 is shown in FIG.

  First, the mediation device for writing 1W receives a write request and data P21 from the client C (S201). Next, the write controller 18 primarily stores the write data received in S201 in the write data buffer 24 (S202). Then, the storage device side interface IR transmits a message (write request and data P22) requesting storage of the write data to each storage device R (S203).

  Then, the storage device side interface IR acquires the write result P23 from each storage device R as a response to S203 (S204). Note that the write result P23 is, for example, a success flag indicating either write success or write failure. Further, the write result processing unit 26 reflects (updates) the write result of S204 in the index DB 10 (S205). Then, the client-side interface IC transmits the entire file write result P24 to the client C based on the index DB 10 updated in S205 (S206).

  The summary of this embodiment described above is as follows. The present embodiment relates to a method and apparatus for a client C to read / write data from / to storage devices R distributed in a plurality of networks. That is, according to the present embodiment, a lump of data is distributed and stored in a plurality of storage devices R, whereby data writing and reading are processed in parallel. The devices constituting the internal network are the client C, the data mediating device 1 and the storage device R. Note that the client C has a function of reading / writing data from / to a storage device R installed in the same network of the prior art.

  Therefore, when reading and writing data using the storage devices R distributed and installed in the remote network, access is made via the data mediating device 1 installed in the local network. For the client C, the data mediating apparatus 1 operates as if it is a storage device R installed in an internal network realized by the conventional technology, so that no additional investment is required for each client C. As a result, an increase in the load on each client C is suppressed, and only the introduction cost of the data mediating apparatus 1 is required.

  Specifically, the data mediating apparatus 1 once receives a batch of data from the client C and then reads and writes the data. The data mediating apparatus 1 performs a process of disassembling a lump of data and writing it to the storage device R in parallel, reading it from the storage device R in parallel at the time of reading, arranging the order of the pieces of data, and passing them to the client C terminal.

  The means necessary for functioning as the data mediating apparatus 1 receives a read request from the client C, selects one or more remotely located storage devices R in which the requested data is stored, and selects each selected A means for acquiring the entire data or fragment from the remote storage device R, constructing the acquired data into one data and returning it to the client C, and receiving a write request from the client C and suitable for writing the requested data This is means for selecting one or more appropriate remote storage devices R, requesting the selected storage devices R to write the entire target data or fragments, and notifying the client C of success or failure.

  As a result, the data mediating apparatus 1 can operate as a storage device R connected to the network realized in the prior art with respect to the internal network and receive it from the client C. In addition, the remote storage device R has a function of updating the received read / write request. The storage device R can read and write data fragments in accordance with a request from the data mediating device 1. At this time, the client C reads / writes data from / to the set of storage devices R located remotely via the data mediating apparatus 1.

  The present invention described above can be widely modified without departing from the spirit thereof as follows.

  For example, communication is performed from the client C to the data mediating apparatus 1 and from the data mediating apparatus 1 to each remote storage device R. As a communication method, iSCSI may be used. A data access control protocol may be used.

  Further, the data that the data mediating apparatus 1 reads / writes to / from the storage device R may be data itself that is requested to be read / written by the client C, or data that is generated from the requested data. The generated data is, for example, a code for detecting a data error such as a parity code, a code for correcting a data error such as a Hamming code, and the like. Then, the data mediating apparatus 1 may write the same data in only one storage device R, or may copy and write to the plurality of storage devices R. Thereby, since data redundancy can be realized, a system that is resistant to failures (high in reliability) can be constructed.

It is a block diagram which shows the communication system regarding one Embodiment of this invention. It is a block diagram which shows the mediation apparatus for reading regarding one Embodiment of this invention. It is a block diagram which shows the mediation apparatus for writing regarding one Embodiment of this invention. It is a block diagram which shows the unused area | region table regarding one Embodiment of this invention. It is a block diagram which shows the used area | region table regarding one Embodiment of this invention. It is a flowchart which shows the reading process of the data regarding one Embodiment of this invention. It is explanatory drawing which shows the communication in the data reading process regarding one Embodiment of this invention. It is a flowchart which shows the write-in process of the data regarding one Embodiment of this invention. It is explanatory drawing which shows the communication in the data write-in process regarding one Embodiment of this invention.

Explanation of symbols

C client R storage device 1 data mediation device 1R read mediation device 1W write mediation device IC client side interface IR storage device side interface 10 index DB
12 Unused Area Table 14 Read Control Unit 16 Used Area Table 18 Write Control Unit 20 Read Data Buffer 22 Read Data Construction Unit 24 Write Data Buffer 26 Write Result Processing Unit

Claims (4)

Client for requesting reading / writing of data, storage device for storing data requested to read / write by client, used area table for associating storage location for each data stored in storage device, and storage device A data mediating method in a network system configured to include a data mediating device having an index DB including an unused area table that manages addresses and unused block positions on the network,
The data mediating device is
A first interface used for communication with the client and a second interface used for communication with the storage device;
A step of receiving the read and write requests via the first interface from the client,
A step of acquiring position information of read data from the index DB based on the read request, and requesting a storage device corresponding to the position information to read data via the second interface ;
Fragments of data received via the second interface in reply to a request of reading of the data from the storage device, and stores align the read data buffer, in order the data stored in the data buffer from the beginning and procedures to be returned via the first interface to the client,
Determining the position information of the write data from the index DB based on the write request, and requesting the storage device corresponding to the position information to write the data via the second interface ;
Updating the index DB with a write result received via the second interface as a response to the data write request from the storage device;
A data mediating method characterized by executing It said data mediation device, before Symbol first interface or, among the pre-Symbol second interface, at least one interface, the data communication protocol in accordance with the iSCSI (Internet Small Computer System Interface) , for communicating data The data mediation method according to claim 1, wherein: Client for requesting reading / writing of data, storage device for storing data requested to read / write by client, used area table for associating storage location for each data stored in storage device, and storage device A data mediating apparatus in a network system configured to include an index DB including an unused area table that manages addresses and unused block positions on the network,
A first interface for receiving signals read and write requests from said client,
A second interface used for communication with the storage device;
A read control unit that acquires position information of read data from the index DB based on the read request, and requests a storage device corresponding to the position information to read data via a second interface ;
Wherein from the storage device to fragment the data received via the second interface in reply to a request of reading of the data, and stores aligning the read data buffer, in order the data stored in the data buffer from the beginning A read data construction unit that sends a reply to the client via the first interface ;
A write control unit that determines position information of write data from the index DB based on the write request, and requests a storage device corresponding to the position information to write data via a second interface ;
A write result processing unit that updates the index DB with a write result received via the second interface as a response to the data write request from the storage device;
A data intermediary device characterized by comprising. Before SL first interface or, among the pre-Symbol second interface, at least one interface, the data communication protocol in accordance with the iSCSI (Internet Small Computer System Interface) , characterized by communicating data according Item 4. The data mediating apparatus according to Item 3.

Images