KR20200109547A - Method and network attached storage apparatus for sharing files between computers - Google Patents

Abstract

The present invention relates to a method for storing and sharing files without damage even when storing the files through each local file system without cooperation between file systems of computers by allowing computers of a big data processing distributed system such as Hadoop to share network attached disks, and more particularly, to a method for sharing files between computers by providing permission to read to other computers by allocating a disk space exclusively to store files exclusively to each computer so that each computer stores the files only in the disk space allocated to store the files exclusively. Therefore, when using a file sharing method and a network attached storage device of the present invention, files are shared without causing network communication between computers to share the files, thereby having an effect of increasing an effective main memory of each computer and an effect of increasing a CPU effective time to have an effect of increasing the data processing performance of the entire distributed system.

Description

[Method and network attached storage apparatus for sharing files between computers}

The present invention relates to a method for sharing files and a storage device for storing shared files, and more particularly, between computers using network attached disks in a large-scale distributed computing system that processes big data such as a Hadoop system. It relates to a method of sharing files in a network and a network attached storage device for storing files.

Network-attached disks are a type of storage device that is attached to a network and provided to a computer as a disk device, whereas a normal disk mounted on the internal bus of a computer. Initially, it started as a technology that provides storage devices through a Fiber Channel communication protocol network called SAN (Storage Area Network), but afterwards, it is attached to Ethernet, a typical general network, to provide disk space to computers. Technologies have been developed and are widely used.

Network-attached disks attached to Ethernet include iSCSI disks, netdisk disks, AoE disks, and FCoE disks. iSCSI is an acronym for Internet Small Computer Systems Interface. It is a standard protocol for sending and receiving data by providing a disk device to a computer through a network adopted as a standard in 2000 by IBM and Cisco in 1998. Is based on. Patent Publication No. 2002-0059139 is an invention for a disk that is directly attached to a port of a general-purpose network such as Ethernet. Based on the patent, Ximeta, Inc. of the United States developed NetDisk in 2002. Released. FCoE, which was proposed in 2003 and formalized in 2007, is an acronym for Fiber Channel over Ethernet. It is the Fiber Channel protocol that transmits data at high speeds between high-performance computers and storage devices. It is a protocol technology that connects storage devices to a computer using Ethernet. AoE, which was announced in 2004, is an acronym for ATA over Ethernet, and provides computers by connecting devices such as normal hard disks and solid state drives (SSDs) that use AT/ATAPI (AT Attachement Packet Interface) interface standards to Ethernet. It is a protocol to do. Unlike the iSCSI, which connects a small computer system interface (SCSI) disk device to Ethernet, the AoE technology provides a computer with a relatively inexpensive AT/ATAPI disk connected to Ethernet. Patent Registration No. 10-0724028 makes it possible to provide a home server function as a hard disk device to an AV (audio video) device equipped with a hard disk. Patent Registration No. 10-1509183 has a failover disk that is directly attached to the network and implements a failover of the disk, so it is economical compared to the conventional RAID (Redundant Array of Independent Disks) method, which is a storage method of direct network attachment. It relates to the device.

These existing network-attached disk technologies are technologies and devices that connect a disk device to a network to provide a computer with data storage space. However, multiple computers share a network-attached disk and store files on these devices, allowing multiple computers to mutually interact. There is no function to allow file sharing.

Korean Patent Publication No. 10-2002-0059139 Korean Patent Registration Publication No. 10-0724028 Korean Patent Registration Publication No. 10-1509183

Despite the high need to share files among multiple computers in a distributed data processing system such as Hadoop, network-attached disks can not be shared as a file storage device of each computer's local file system. It is not being utilized. In particular, in a big data processing distributed system, the network attached disk is shared among servers, considering that the overhead of transmitting files through the network to share files is a major cause of the performance degradation of the entire system. It is an object of the present invention to provide a technology for directly sharing in a network attached disk without transmitting through a network.

The existing method of sharing files between computers in a distributed system such as the Hadoop system is that each computer reads a file stored in its internal disk into its main memory, then transfers the file to other computers through the network and receives the file. Computers share files between computers by loading files in their main memory and then storing them in their internal disks. In this process, the overhead and overhead of many network communications caused by computers to share files A significant amount of the main memory is occupied by file transfer, resulting in the effect of reducing the main memory to be allocated for processing data. If the data to be processed cannot be loaded into the main memory due to insufficient main memory and a swap phenomenon occurs, the system performance is exponentially degraded, thus increasing the amount of effective main memory and causing network communication. Another object of the present invention is to provide improved performance by reducing the number of files exchanged.

In the present invention, network-attached disks are shared by a plurality of computers and the files are stored in the network-attached disk, so that the files to be shared are not transmitted over the network, and the computers directly access and share the files stored in the network-attached disk. There is another object of the present invention in providing to increase.

As a specific means for achieving the object of the present invention, there is a method of sharing files among a plurality of computers by sharing a network attached disk storing files,

Files are created on the network-attached disk through a local file system, and files are mutually exclusively allocated to the generated files so that each computer can store data exclusively, and each computer is assigned a file to the assigned files. Data is stored by performing write, and the computers share files between computers by allowing them to read files that are not assigned to them,

The disk sectors are mutually exclusively allocated to some of the disk sectors of the network-attached disk so that the computers can independently store data, and each computer writes a file to the allocated sectors to And the computers share files between computers by allowing them to read files stored in sectors not assigned to them,

Mount the partition of the network-attached disk so that one of the computers can exclusively write and read files, and the other computers can only read the corresponding partition of the network-attached disk. Mount and allow computers to share files,

The entire network-attached disk is mounted so that one of the computers can exclusively write and read files, and the other computers mount the network-attached disk so that only reading can be performed. Share files,

The data storage medium of the network-attached disk is a hard disk, solid state drive (SSD), flash memory, multiple array independent disks (RAID, Redundant Array of Inexpensive Disks), and JBOD (Just Bunch of Disks). To share files between multiple computers,

As a network attached storage device that provides disks to a computer by connecting to a network, it transmits blocks of HDFS (Hadoop Distributed File System) stored in the storage device to other storage devices through the network, and transmits the blocks from other storage devices to the network. It composes a network attached storage device equipped with block transmission/reception logic that can receive HDFS blocks through

In addition, as a storage device that connects to a network and provides a disk to a computer, when the computer requests that the computer allocate a file to store data exclusively, the assignment logic that allocates the appropriate file and writes the data to a file that does not have exclusive data storage rights. Compose a network attached storage device equipped with an inspection logic to check whether it is attempting,

In addition, as a storage device that provides disks to the computer by connecting to the network, when the computer requests to allocate disk sectors for exclusively storing data, the allocation logic that allocates the appropriate disk sectors and data to the sectors that do not have exclusive data storage rights Configures a storage device attached to a network equipped with an inspection logic that checks whether writing is attempted,

In addition, as a storage device that provides disks to the computer by connecting to the network, when the computer requests to allocate a disk partition to exclusively store data, the allocation logic that allocates the appropriate disk partition and the partition that does not have exclusive data storage rights The object of the present invention may be achieved by configuring a storage device attached to a network equipped with a check logic that checks whether writing is attempted.

As described above, the use of the file sharing method and the network attached storage device according to the present invention avoids occupying the main memory of the computer in order to transfer file data between computers of an existing distributed system such as Hadoop. Because files are shared without causing network communication, which was necessary for the purpose, there is an excellent effect of improving the data processing performance of the entire distributed system.

1 is a schematic diagram showing the configuration of a Hadoop system.
2 is a conceptual diagram illustrating a process of transmitting and receiving an intermediate result file between a mapper and a reducer of a Hadoop system.
3 is a conceptual diagram showing that a plurality of computers share a file by sharing network attached disks to illustrate the first embodiment.
4 is a system configuration diagram showing computers sharing network attached disks to illustrate the first embodiment.
FIG. 5 is a conceptual diagram illustrating a method of sharing files between computers according to the first embodiment using only their own local file system without cooperation between file systems.
6 is a schematic diagram of a disk partition in a logical block addressing (LBA) method to describe the second embodiment.
7 is a conceptual diagram showing a method of sharing file data between computers using a network attached disk as a simple block storage device according to the second embodiment.
FIG. 8 is a schematic diagram of a process of storing data in a data block in a conventional HDFS (Hadoop Distributed File System, Hadoop Distributed File System) to explain the third embodiment.
FIG. 9 is a schematic diagram illustrating that the HDFS block is copied without occupying the main memory of the HDFS data node by using the network attached storage devices of the present invention, which is a third embodiment.
10 is a block diagram of functional elements of the network attached storage device of the present invention, which is a fourth embodiment.
FIG. 11 is a schematic diagram of a method of allocating an entire disk partition, which is a fifth embodiment, to computers to exclusively have read/write rights.
12 is a block diagram of functional elements of the network-attached storage device of the present invention in which the entire disk partition is allocated according to the sixth embodiment.

The present invention for achieving the above object is that a plurality of computers share network-attached storage devices and create and share files through their own local file system without cooperation with other servers. A method of sharing files without damage and the design of a network attached storage device are the basic features of the technical configuration.

Hereinafter, preferred embodiments of a file sharing method and a network attached storage device according to the present invention will be described with reference to FIGS. 1 to 12. The disk referred to in the embodiments of the present specification refers to a hard disk (HDD) or a solid state drive (SSD), as well as an arbitrary nonvolatile block storage device such as a USB drive or a secure digital (SD) card.

In the specification of the present invention, a Hadoop system has been described as an example. However, the file sharing method and storage device of the present invention include a number of computers in addition to the Hadoop system to share file data between computers of a general distributed system connected to a network. It is obvious that it can be used in any way.

1 is a Hadoop system in which data nodes (1-1, 1-2, 1-3, 1-4), which are thousands of computers, are connected through a network (2) to process large-scale data. It is a schematic diagram showing the composition of. Data nodes (1-1, 1-2, 1-3, 1-4), which are individual computers that process data in the Hadoop system, have mappers (20-1, 20-2), which are functions that process data. , 20-3, 20-4, 20-5) and one or more reducers (21-1, 21-2, 21-3, 21-4, 21-5, 21-6) Large-scale data is divided and processed by multiple data nodes. Mapper is a function that processes data and derives intermediate results. The data to be processed is distributed to several data nodes and stored in data blocks (16-1, 16-2, 16-3), and the data of the data blocks stored in their own data nodes by dividing the entire data by several mappers. The entire data is processed in parallel by processing.

Each mapper that processes data creates intermediate data as files (11-1, 11-2, 11-3, 11-4, 11-5, 11-6) and executes each mapper. Intermediate result files (12-1, 12-2, 12-1, 12-2, etc.) on the disk of the own data node through the local file system (10-1, 10-2, 10-3, 10-4) of the existing data node. 12-3, 12-4, 12-5, 12-6). At this time, each mapper usually creates several intermediate result data files. The intermediate result files (11-1, 11-2, 11-3, 11-4, 11-5, 11-6) generated by the mappers are files (12-1, 12-2, and 11-6) on the local disk. 12-3, 12-4, 12-5, 12-6) are first created in the disk buffer of the operating system of each data node. Files temporarily created in the disk buffer and files stored on the disk have the same data. Invented to distinguish whether a file is the same file but is a file in a disk buffer located in the main memory (3-1, 3-2, 3-3, 3-4) of the data node or a file stored in the disk of the data node In all of the drawings in Fig. 11, intermediate result files in the disk buffer are indicated by number 11, and intermediate result files stored in the disk are indicated by attaching the file number by number 12.

Several intermediate result data files generated by the mapper are transmitted to several reducers (21-1, 21-2, 21-3, 21-4, 21-5, 21-6). Reducer is a function that collects and processes intermediate result data files. Like the mapper, reducers are distributed across multiple data nodes, and multiple reducers for processing a single task can exist for each data node. Mappers send intermediate result files (12-1, 12-2, 12-3, 12-4, 12-5, 12-6) to several reducers to solve the problem, and each reducer sends intermediate results received from several mappers. Collect and process files. To each reducer, intermediate result files that are generated by different mappers and are finally collected and processed are transmitted from several mappers. The result files (14-1, 14-2, 14-3) processed by each reducer in this way are stored on the local disk through the local file system of each data node (15-1, 15-2, 15-3). ), and the entire set of these result files (15-1, 15-2, 15-3) becomes the final processing result data.

The reason for distributing and executing multiple mappers and reducers across data nodes connected to the network is to increase the speed of data processing by distributing large-scale data across multiple data nodes and processing them in parallel.

2 shows a typical process of transmitting an intermediate result file generated by a mapper to a reducer using a hypertext transport protocol (http) communication protocol. For example, the intermediate result file p(11-7) created in the main memory (3-1) by the mapper j(20-1) of data node M(1-1) is transferred to the disk(6-1). -7), and through the HTTP interface program (22-1), the network stack (4-1) of the operating system of the data node M (1-1) and the network interface hardware NIC (Network Interface Card or Network Interface Chip) Transmitted through the network 2 via (5-1), the reducer h(21-1) waiting for the reception of the intermediate result file is sent to the running data node N(1-2). At this time, the intermediate result file p (12-7) stored in the disk (6-1) through the HTTP interface program (22-1) cannot be directly sent through the network (2). The result file p(12-7) must be loaded into the main memory 3-1 as the intermediate result file p(11-8) to be transmitted. The intermediate result file p(11-8) transmitted in this way is the network interface hardware NIC (Network Interface Card or Network Interface Chip)(5) of data node N(1-2) where reducer h(21-1) is running. -2), the intermediate result file p(11-9) in the main memory (3-2) through the network stack (4-2) of the operating system and the HTTP interface program (22-2) and the local file system (10-2). ), and then saved as file p(12-8) on the local disk (6-2), and then reducer h(21-1) is saved on the disk (6-2) again, and the intermediate result file p(12 After accessing -8) and loading the intermediate result file p(11-10) into the main memory (3-2) so that reducer h(21-1) can work, the data of file p(11-10) Process. In FIG. 2, a file p (11-7) created by mapper j (20-1) and a file p stored in the disks 6-1 and 6-2 of the two data nodes 1-1 and 1-2, respectively. (12-7, 12-8) and the files p(11-8, 11-9, 11-10) loaded in the main memory are all files that have the same data, but to distinguish where the files are located, as in FIG. Likewise, the files loaded in the main memory are displayed in number 11, and the files stored in the disk are displayed in number 12.

As such, each data node transmits the intermediate result files generated by the mappers to the reducers of various data nodes. In this process, each data node transmits and receives the intermediate result files, and its main memory (3-1, 3- 2) Due to the overhead of network communication that occupies and transmits/receives, it is not possible to allocate the main memory and the CPU to an effective task that performs data processing, which causes the overall data processing performance to deteriorate.

The present invention relates to a method and a storage device for improving the performance of a large-scale data processing system such as a Hadoop system. More specifically, intermediate result files are stored in a storage device attached to a network so that data nodes, that is, computers, are used as intermediate result files. To share them. When the file sharing method of the present invention is used, the occupancy of the main memory allocated to the disk buffer and the network stack, which was caused in the process of transmitting/receiving through a network to share intermediate result files, is avoided, and data processing by reducing communication overhead. It increases performance.

3 is a conceptual diagram of a system configuration of the present invention in which network attached disks are shared by a plurality of data nodes, that is, computers. In order to avoid occupying the main memory caused when transferring the intermediate result file to the network, in the present invention, the file is shared between computers using the network attached disk 7-1. As described above, disks attached to the network include Internet Small Computer Systems Interface (iSCSI) disks, net disks, and AoE (ATA over Ethernet) disks. In the present invention, the network attachment disk 7-1 is attached to the network 2 to which several data nodes 1-1 and 1-2 are connected, as shown in FIG. 3, so that the data nodes 1-1 and 1- 2) It is used as a local disk.

The intermediate result file h(11-1) generated by mapper j (20-7) of FIG. 3 through its local file system (10-1) is transferred to the network stack (4-1) and the NIC (5-1). Then, it is stored as a file h (12-1) on the network attached disk (7-1). The file h(11-1) created by mapper j(20-7) points to a file in the disk buffer of the operating system of data node M(1-1) on which mapper j(20-7) is running, and contains the same data. The file h(12-1) indicates a file stored on the network attachment disk 7-1. Reducer k(21-7) running on another data node N(1-2) connected via the network(2) saves the intermediate result file h(12-1) stored on the network attached disk(7-1). Through the NIC (5-2), the network stack (4-2), and the local file system (10-2), the data is loaded into its own disk buffer located in its main memory as the intermediate result file h (11-2). Process. Unlike the conventional method shown in FIG. 2, the file sharing method of the present invention of FIG. 3 uses a data node in a file stored on a network attached disk instead of transmitting a file from one data node to another data node through a network to share a file. Approach directly.

4 shows a plurality of network-attached disks 7-1, 7-2, 7-3, 7-4, and 7-5 with a plurality of data nodes 1-1, 1-2, 1-3, that is, This is a system configuration diagram shared by computers. All data nodes connected to the network (2) control the network attached disks (7-1, 7-2, 7-3, 7-4, 7-5) attached to each data node. It is recognized as its own local disk through the device driver software modules 13-1, 13-2, and 13-3. As device drivers (13-1, 13-2, 13-3), in the case of Linux operating systems, software initiators for iSCSI devices have been provided and used since the early 2000s, and various operating systems including Windows and VMware Initiator software, a device driver, has been provided so that network-attached disks can be recognized and used as their own local disks. In the case of NetDisk, device drivers for NetDisk disks have been provided in Linux and Windows systems since the early 2000s. A computer with a disk installed inside the system controls the disk installed inside through the device drive software for internal disk through a local file system to write and read files to the disk. On the other hand, a computer system using a network-attached disk reads and writes files to a network-attached disk through a local file system, but the device driver software for the network-attached disk accesses the network-attached disk through the network and attaches it to the network. Control the disk.

However, simply connecting the network-attached disk to the network as shown in FIG. 4 cannot allow multiple data nodes to independently read or write files while sharing files on the network-attached disk. The reason is that each data node reads and writes files through its own local file system regardless of other data nodes, so each data node sharing a network-attached disk has different data nodes in any sector of the disk. This is because the integrity of the file data is damaged by overwriting different data by different data nodes on the same disk sector of a network-attached disk shared by several data nodes. In other words, since each local file system does not cooperate with each other, each data node maintains different metadata containing information about which file is stored in which sector of the disk, making it impossible to maintain a consistent file system on the shared network attached disk. It becomes. For this reason, network-attached disks such as iSCSI, AoE, FCoE, and NetDisk cannot be shared by network computers to store files.

Figure 5 shows that each computer (1-11, 1-12, 1-15) does not cooperate with each other between other computers and file systems (10-11, 10-12, 10-15). This is a schematic diagram of the file sharing method of the present invention in which files are shared between computers 1-11, 1-12, and 1-13 while using only the file system. The file sharing method of the present invention illustrated in FIG. 5 uses conventional network-attached disks 7-1, 7-2, and 7-3 and a file allocation server 1-15. A disk partition is a contiguous group of sectors of a storage device such as a hard disk or SSD (Solid State Drive). Typically, one physical hard disk or SSD can be divided into several partitions, and individual disk partitions are divided by the operating system. It is recognized as an independent storage device, so only one file system is mounted on one disk partition. Since the file sharing method of all the embodiments of the present invention is applied equally to the entire disk as well as to the disk partition, in the description of all the embodiments of the present invention, the disk and the disk partition may be referred to without distinction.

Each of the computers (1-11, 1-12) and the file allocation server (1-15) in Fig. 5 are used to transfer these network attached disks or partitions (7-1, 7-2, 7-3) to their local files. Mount disks or partitions that have both read and write privileges on the system (10-11, 10-12, 10-15). Mounting a disk or disk partition is a procedure in which the disk or partition is logically attached to a directory in the file system structure. After the mounting procedure, files can be input and output to the mounted disk through the local file system. Mount commands and related software tools are provided for all operating systems. For example, for Linux operating systems, mounting a network-attached disk in the /networkdisk1 directory of the local file system (10-11) of computer 1 (1-11) is as follows: It is executed by using the mount command as shown below. In the example, assuming that any one of the network-attached disks of FIG. 5 is /dev/sda, an example of mounting a disk with a corresponding name is shown. Other disks can of course be mounted in the same way, and the name of a network-attached disk such as /dev/sda is provided through the device driver of the network-attached disk such as the initiator.

In FIG. 5, it is emphasized that each network-attached disk or disk partition (7-1, 7-2, 7-3) is not mounted with write permission on only one specific computer, but is repeatedly mounted with read-write permission on multiple computers. For this, it is indicated by a dotted line (100 ~ 108). In the file sharing method of the present invention, before each computer (1-11, 1-12) accesses the network attached disks or partitions (7-1, 7-2, 7-3) and starts file input/output. The file allocation server (1-15) has files (12-10, 12-11, 12-12, 12-13, 12-10, 12-11, 12-12, 12-13) on the network attached disks or partitions (7-1, 7-2, 7-3) in advance. 12-14, 12-15, 12-16, 12-17). Files created in advance do not have to be the same size. Creating a file of the desired size before saving the actual data can be done using the fallocate command in the case of Linux operating systems. fallocate is a system call and command provided by the Linux operating system, short for file allocate. fallocate creates a file that occupies a certain amount of disk space in advance without actually saving data. The following is an example of creating a 10 MB file with the file name file1.txt using the fallocate command.

The fallocate command secures a disk space of the size that can hold file data of a specified size without actually saving the data. For example, in the example of the fallocate command above, the file1.txt file creates a file by securing 10 MB of disk sectors in advance while actually not storing valid data. The file allocation server (1-5) repeats the fallocate command as many times as necessary, and in advance, the desired number of files (12-10, 12-11, 12-12, 12-13, 12-14, 12- 15, 12-16, 12-17) are created and stored in network attachment disks (7-1, 7-2, 7-3).

After that, when the computers (1-11, 1-12) request the file assignment server (1-15) to allocate a file to which they have write permission, the file assignment server (1-15) creates files (12-10 ~ 12-17), the name of the file including the name and directory path of the disk or disk partition on which the selected file is located is sent to the requesting computer in response to the requesting computer, and the requesting computer sends the data to the file. To be able to save. Each computer stores data in the assigned files in this way, and other computers read the corresponding files containing the stored data, so that each computer does not cooperate with the file systems of other computers, but through its own local file system. You can share computer-generated files. A computer that wants to create a file for the purpose of sharing with other computers does not arbitrarily create a file by itself, but always receives only the file assigned from the file assignment server 1-15 of the present invention and stores the data in the assigned file. Share the file with computers.

When the computer assigned a file from the file assignment server (1-15) saves data in the assigned file, it opens the file in the usual way using the assigned file name and writes the data. Save the data to the file. Below is a simple example of a program that opens the file and saves data using the assigned file name. In the program below, it is assumed that 4000 bytes of data is contained in the array buf and the file file1.txt is stored in the /shared directory.

The fd in line (1) of the program is the file descriptor returned from the operating system as a result of opening the file with read-write permission using the assigned file file1.txt. In order to open a file with both read and write privileges, the O_RDWR flag must be set as the second argument of the open system call. As in the program line (2), the write system call is called to write data to the previously opened file. Line (3) of the above program calls the fsync system call that synchronizes the data of the file, and forces the data stored in the file to be physically stored in the disk partition rather than staying only in the disk buffer of the operating system. In this way, when the file is read by another computer, the actual data can be read. Not only the data of the file, but also the metadata related to the file should be reflected in the disk partition and saved. To do this, synchronize the /shared directory where the file is located as shown in line (5) of the program above. Since the operating system also treats directories as files, an open system call is called for the directory to be synchronized as in the program line (4), and the file descriptor of the directory is returned from the operating system, and the argument of the fsync system call on line (5). ). After synchronizing the files and directories, the use of the files and directories is ended by calling the close system call as shown in lines (6) and (7). After writing data in this way, the file data and related metadata are synchronized so that they are reflected on the disk instead of staying in the disk buffer of the local file system, so that other computers that want to read the file access the file and actually read the data. It should be made readable.

In order for other computers to read and share the files created in this way, pass the directory path and name of the file to computers that want to read the file, and open the file on the computer that wants to read, as in the example of the program above. And read it. However, in the method of the present invention, the computer storing the data in the file creates the file while specifying the size of the file through the fallocate command before saving the actual data. Therefore, when storing the actual data in the file, the fallocate command It is usually different from the size of the file when it is set using. Therefore, the program of the computers that read and share not only the name of the file to be shared, but also the size of the data to be shared, read and share only the data from the beginning of the file to the size of the saved data to the computers that want to read and share. Do not read the file. For example, in the above program example, 4000 bytes of data are stored in the file1.txt file, but file1.txt was created by securing 10 MB of disk space in advance, so the size of file1.txt is 10 MB. By the way, the size of the file1.txt file is 10 MB, but the valid data is stored in 4000 bytes from the front of the file1.txt file. Therefore, for the computer that wants to read the valid data of file1.txt, the effective data size stored in addition to the file name is 4000. Should be passed so that only 4000 bytes can be read, not 10 MB. The following is an example of a program that reads 4000 bytes of data created in the above program example from the file1.txt file. The data_buf in the program below is an array whose size is 4000 bytes and is the name of the array used to read data from the file1.txt file and store the data.

If there is a case where a larger amount of data needs to be stored than the original size of a single assigned file, the computer uses two or more assigned files to store all the data, and then the two or more files storing the data are stored. You just need to combine the files into one file. For example, two files with a size of 1 MB each were assigned, but in order to save 1.5 MB of data to the assigned file, save 1 MB of data to one of the assigned files and save the remaining 0.5 MB to another file. Consolidate files. Consolidating files can be done using the cat command on Linux. The following is an example of creating a single file by appending another file to the end of a file using the cat command of the Linux operating system. The example of merging the files with file2 after file1 and naming the combined file as file1. That is, if a large amount of data is first filled in file1 and then the rest is stored in file2, the previously allocated files can be combined and stored in one file using this file integration method.

In the method of the first embodiment of the present invention, when files are allocated, files may be allocated one at a time whenever necessary, but several files may be allocated from the allocation server 1-15 at a time. In addition, instead of creating and preparing files in advance and assigning them, when the file assignment server (1-15) receives a file assignment request from computers (1-11, 1-12), it dynamically creates a file through the fallocate command. You can also assign it after you have done it. Whether a file is dynamically created and assigned or created in advance, the file assignment computer (1-15) generates the file using the fallocate command and before assigning the file to the computer. By calling a system call, the files and metadata created by the user are synchronized and stored on the network attached disk.

As described above, in the case of the Linux operating system, the fsync system call can be used for synchronization, as well as the data and metadata of files created using the sync command can be reflected on the disk. The following is an example of reflecting data and metadata to disk using the sync command in Linux. In the following example, file1.txt is the name of the file created by the assignment server (1-15).

To summarize the file sharing method of the present invention of FIG. 5, among the files created to occupy only the disk sector without storing actual data yet, the assignment server 1-15 allocates different files to each computer, and each computer The data integrity of files is limited even if each computer writes, reads, and shares files only through its own local file system by limiting data write to the assigned files and preventing other computers from writing to the file. ) To maintain.

The first embodiment of the present invention is a method of storing files on a network attached disk and sharing files with other computers through a local file system of each computer. However, the integrity of file data can be maintained and shared among multiple computers in a similar manner to the first embodiment by simply using a disk or disk partition as a block storage device without using a local file system.

6 shows the disk sectors constituting the disk partition in a linear notation method of the LBA (Logical Block Addressing) method. The physical structure of the hard disk is composed of a cylinder, a head, and a sector, but the disk is indicated in the LBA method, which is a notation method that generally starts from sector 0 and refers to a continuous sector. In other words, a disk or disk partition indicates that disk sectors of a regular size of 512 bytes are contiguous, and if the disk sector is pointed to the LBA method in all disk devices, the corresponding cylinder number, head number, and sector number of the actual hard disk are converted. Since logic is installed, the sector of the disk is referred to as the LBA method to access the desired sector.

As shown in FIG. 6, a disk location may be indicated by a byte address starting from the beginning of the disk partition, and thus the disk location expressed by a byte address is called an offset. For example, if the offset value is 2,000, it points to the 2001-th byte from the front of the disk partition. Since the byte address starts from 0, not 1, the offset 2000 is the 2001 th byte. Since one disk sector is usually 512 bytes, if the offset value is 2000, it indicates the 465th byte of the fourth sector from the front of the partition. That is, since the offset 2000 = 2001th byte, it indicates 465 bytes out of the 4th sector after 3 512-byte sectors. Note that 2001 = 512 * 3 + 465.

FIG. 7 is a schematic diagram of a method of sharing data between multiple computers without mounting a file system on a network-attached disk, but simply using it as a block storage device, and in the description of FIG. 5 of the first embodiment. The concept of sharing device files is similar, except that the entire network-attached disk or partition (7-1) is used as one device file instead of storing and sharing data using files created in the local file system. Do.

Each of the bundles 30-1, 30-2, 30-3, and 30-4 of FIG. 7 is composed of consecutive disk sectors. Herein, the bundle is not a commonly used term, but a term defined herein that refers to a series of disk sectors to describe Embodiment 2 of the present invention. The entire disk partition can be viewed as a whole set of bundles. For example, the disk partition 7-1 of FIG. 7 includes bundles that start from the first bundle 0 (30-1) and continue to the last bundle n (30-4).

Each individual bundle defined in this way is assigned to a specific computer to have write access, and the rest of the computers are limited to read only, so that only computers with write access can store the data files they create in the disk sectors of the bundle. To be. In this way, if different computers are mutually exclusively given exclusive write access to each individual bundle, each computer will store the data file in the bundle to which only it has exclusive write access, but other computers can read the bundle. Therefore, two or more different computers store files in any sector of the disk partition 7-1 and share the file data without overwriting.

For example, by mounting the software module 23 for allocating bundles to the bundle allocation server 1-25 of FIG. 7, each computer (1-21, 1-22) requests a bundle capable of exclusively storing data (201 , 202), the bundle allocation server (1-25) sends the requested computers (1-21, 1-22) to the storage device name where the bundle is located, that is, the disk name or disk partition name, the offset of the start byte, and the disk included in the bundle. The number of sectors is transmitted (203, 204) to allocate a bundle. For example, the name of the disk or disk partition 7-1 in FIG. 7 is /dev/sda1, and the bundle 0 (30-1) consists of 50 sectors starting from the first byte address of the partition 7-1. If the bundle allocation server 1-25 assigns the bundle 0 (30-1) to only computer 1 (1-21) to have exclusive write access, assuming that it is configured, the computer 1 (1-21) of FIG. 7 Bundle information 24-1 includes (/dev/sda1, 0, 50). In this exclusively assigned bundle 0 (30-1), only computer 1 (1-21) can store file data, while other computers k (1-22) can read the stored file data, so computer 1 (1-21) This saved file data is read and shared by another computer k (1-22).

At this time, when the computer to which the bundle has been assigned saves data in the bundle, storing data in the bundle, that is, a contiguous disk sector of the disk device, without going through the local file system, is normally open as shown in the program exemplified in Example 1. It can be implemented using the (open) system call and the write and read system calls. The following is an example program that saves data in an assigned bundle. In the following example program, it is assumed that the name of the partition (7-1) is /dev/sda1. In addition, in the example program, it is assumed that the bundle 0 (30-1) is composed of 50 consecutive sectors starting from byte 0 of the disk partition 7-1.

Program line (1) calls the open system call, passing the O_RDWR flag, which sets the partition name and read/write permission, as arguments. fd is a file descriptor, similar to the case of the program exemplified in Example 1, but in the example program of Example 1, the name of the file created through the file system is passed as an argument, whereas the example program of Example 2 is not a file name. There is a difference in passing the partition name, that is, the name of the storage device as an argument. The operating system treats not only the files created through the file system, but also the storage device itself that inputs and outputs data as one of the files, that is, the device file, and treats it as a file created through the file system, and inputs data into the device file. Allows to use system calls such as write and read that are output. That is, the second embodiment is a method of storing data using a simple storage device without mounting a local file system on the partition 7-1, and sharing file data between the computers 1-21 and 1-22. .

The lseek() system call in the program line (2) is a system call that designates the offset of the opened file. The byte address received as the second argument of lseek() is the point where file input/output starts. In the example program, the value of the offset is 512 * 10, or 5120. SEEK_SET is a whence value to specify that the offset refers to an absolute byte address position starting from the beginning of the disk partition (7-1), not a relative position. Since the address of the first byte of the first sector is 0, not 1, in the example program above, it points to the byte address 5120, so it points to the first byte address of the 11th sector of the disk partition. Thus, line (3) commands 4000 bytes of data in the array buf to be stored starting from the 11th sector of the disk partition. At this time, 4000 = 512 * 7 + 416, so 8 disk sectors are required to store 4,000 bytes of data in a 512 byte sector. In the example program, 4000 bytes spanning 8 sectors starting from the 11th sector and up to the 18th sector Data is saved. 416 bytes of data are stored in the last 18th sector.

Program line (5) is a program that reads 4000 bytes of file data stored in line (3), and the data of the saved file starting from byte address 5210, which is the offset of the file that started writing on line (3). As in program line (4), first call the lseek system call again to return the offset to where it started storing 4000 bytes of data in program line (3), and then read ( Read) This is an example of reading the stored 4000 bytes of data starting from 5210 bytes of the disk partition in line (3) by calling the system call.

Lines (4) and (5) of the example program above may be programs that are executed when the computer 1 (1-21) wants to read the file stored by the other computer k (1-22). Using the example of the above program again, after Computer 1 (1-21) stores the disk partition 5210 bytes to 4000 bytes of data, the partition name, start offset, and data bytes size of the file, i.e. in the example above (/dev /sda1, 5210, 4000) to computer k (1-22), computer k (1-22) can execute the following example program to read the data of the file saved by computer 1 (1-21). have.

For bundle 0 (30-1), computer k (1-22), which has only read permission, must pass O_RDONLY as the second argument of the open system call as in line (1) of the example program above, which is the permission to read the file. It is to designate to be accessed with only.

In the example of the program above, it is shown that files are stored using the byte address 5120 to 4000 bytes of the disk partition /dev/sda1. Of course, computer 1 (1-21) has the byte address 5210 to 4000 bytes of the partition /dev/sda1. It must be located in bundle 0(30-1), which is assigned to have exclusive write access. In the example program, bundle 0(30-1) consists of consecutive sectors spanning 50 sectors from the beginning of disk partition /dev/sda1, that is, byte 0 (512 * 50-1) = 25,599 bytes. The file of the example program is stored in 4000 bytes ranging from address 5210 bytes to address 9209 bytes in bundle 0 (30-1). If the size of the file to be saved is so large that it cannot be saved in one bundle given with write permission, two or more bundles are used to save the file in a similar way to the above program. If the bundles used are not consecutively located in the partition, Note that after setting the offset to the next bundle's starting byte position, the rest of the data is set to the next bundle's starting byte position, and then the lseek system call is called to adjust the offset for file I/O and then continue writing the remaining data of the file. Then, save the file in the same way as the example program above.

In the file sharing method of FIG. 7, the assignment server 1-25 of the present invention grants write permission to another computer to a computer requesting a file write to prevent any computer from damaging the file by writing to a certain sector. By exclusively providing disk sectors that are not assigned, the disk sectors are mutually exclusively distributed between computers, and each computer stores files in disk sectors that it has exclusive write access to, and other computers read and share them. In the method of FIG. 7, all of the bundles do not need to be of a constant size and may be of any suitable size, and when the bundles are allocated, bundles that are not consecutive to each other may be allocated.

Embodiments 1 and 2 show a method of sharing file data in a distributed system such as Hadoop, and this method can be similarly applied to HDFS (Hadoop Distributed File System), a distributed file system defined by Hadoop's system. 8 is a schematic diagram of a method of storing data in a data block of HDFS in a typical case where the replication factor is 3. In HDFS, the name node (8) manages the HDFS blocks (25-1, 25-2, 25-3), which are distributed among several data nodes (1-1, 1-2, 1-3). Specifies which data to store in which block. HDFS blocks are usually composed of 128 MB units to store data, and the copy factor is usually set to 3. That is, if one block is stored in the data node, the primary block (25-1) is additionally copied to two other data nodes with the same content of two duplicate blocks (25-2, 25-3). In a couple of cases, when a failure occurs in the main block 25-1, the duplicate blocks 25-2 and 25-3 are accessed so that data can be processed.

When the client 9 of FIG. 8 requests the name node 8 to create a file for the client 9 to store the data in order to save the data 26 as an HDFS file, the name node 18 Specifies which blocks of the data nodes (1-1, 1-2, 1-3) to divide and store data.

FIG. 8 shows an example in which a client stores one of the main blocks to be stored in the data nodes in a block of the data node. Data node 1 (1-1) in which the main block 25-1 of Fig. 8 is stored transmits the data of the main block 25-1 to data node 2 (1-2) designated by the name node 8. (211) so that data node 2 (1-2) stores the duplicate block 1 (25-2). At this time, data node 1 (1-1) loads the data of the main block (25-1) on the network stack in its main memory and then transmits it through the network interface, thus occupying the main memory in the process of transmitting block data. It consumes its own CPU time. Data node 2 (1-2), which stores the first replica block 1 (25-2), is also sent to data node 3 (1-3) that stores the second replica block 2 (25-3). 2) data is transmitted 212, and at this time, it occupies its own main memory and consumes CPU time in the process of transmitting block data.

HDFS blocks exist as files of each data node's local file system. When a request to create an HDFS file is received from the client 9, the name node 8 informs the client of the file name of each data node to store blocks along with the unique identifier name of the data node to store the data. The given file name refers to the HDFS block (25-1, 25-2, 25-3) of the data node (1-1, 1-2, 1-3).

9 is a diagram of the present invention without occupying the main memory of a data node when duplicating a data block using the storage devices ND1 (40-1), ND2 (40-2), and ND3 (40-3) of the present invention. It is a schematic diagram of the replication between storage devices. All of the HDFS name node 8 and data nodes 1-1, 1-2, and 1-3 are the storage devices ND1 (40-1), ND2 (40-2), and ND3 (40-3) of the present invention. ) Each of the storage devices ND1 (40-1) and ND2 (40-) of the present invention are mounted with read and write privileges on their local file systems (10-5, 10-1, 10-2, 10-3). 2) Access ND3(40-3) as its own local disk. In FIG. 9, the name node 8 and the data nodes 10-1, 10-2, and 10-3 read and write the storage devices 40-1, 40-2, and 40-3 of the present invention, respectively. It is marked with a dotted line to indicate that it was mounted with authority.

In this case, even if data nodes independently write HDFS blocks to network-attached disks without mutual cooperation with other data nodes, the method of maintaining the integrity of the HDFS block is similar to the file sharing method of the first embodiment described in FIG. 5. In the file sharing method of the present invention of FIG. 5, a plurality of computers access a network-attached disk through their local file system, but restricts writing data to only files allocated by the file assignment server. As a method of maintaining integrity, in HDFS block storage, the name node 8 of FIG. 9 HDFS designates a data node to store the block in the same manner instead of the allocation server of FIG. 5, and stores the block in the storage devices of the present invention. .

Specifically, the HDFS name node 8 has HDFS blocks 25-10 and 25-11 in advance in the storage devices ND1 (40-1), ND2 (40-2), and ND3 (40-3) of the present invention. , 25-12, 25-13, 25-14, 25-15, 25-16, 25-17) as separate files. One way to create a block is to use the fallocate command in Linux as described in FIG. 5 to create a HDFS block size, that is, blocks each of 128 MB in size as a file in advance. In order to emphasize that the HDFS blocks 25-10 to 25-17 are actually files of the local file system of the data node where each block is stored, the blocks 25-10 to 25-17 of FIG. 9 The file name is enclosed in parentheses. 9, the storage devices ND1 (40-1), ND2 (40-2), and ND3 (40-3) of the present invention are the name node 8 and all data nodes 1-1, 1-2, 1 It is marked with a dotted line to indicate that it is mounted with read and write permission to -3).

For example, the client 9 of FIG. 9 stores the main block in block 1 (25-10) of the storage device ND 1 (40-1) of the present invention, and the first duplicate block is the storage device ND 2 ( Designated by the name node 8 to store in block q (25-13) of 40-2) and the second duplicate block in block t (25-16) of storage device ND 3 (40-3) of the present invention If it is received, the client 9 transmits data to data node 1 (1-1), and data node 1 (1-1) transmits data to data node 1 (1-1) and block 1 (25) of ND 1 (40-1), as in the case of normal HDFS. Save the main block in -10).

However, the method of storing the duplicate blocks of HDFS in the present invention is different from the data nodes storing the duplicate blocks in the case of a conventional HDFS, the storage devices 40-1, 40-2, and 40- of the present invention without the involvement of the data nodes. 3) It is made directly between. That is, in the storage devices 40-1, 40-2, and 40-3 of the present invention, block transmit/receive logics 50-1, 50-2, 50-3 so that blocks can be directly transmitted/received between the storage devices of the present invention. Is installed to directly exchange and receive duplicate blocks between the storage devices of the present invention without intervening data nodes. For example, the storage device ND 1 (40-1) of the present invention starts to store the main block in its own data block 1 (25-10) and simultaneously transmits the data stored in the block 1 to its own block transmission/reception logic (50-1). Through 220, the data of block 1 (25-10) is transmitted (221) to the block transmission/reception logic (50-2) of ND 2 (40-2) where the first duplicate block is to be stored (221), and ND 2 (40-2) ) Stores the duplicate block in its own block q(25-13). The block transmission/reception logic 50-2 of ND 2 (40-2) receiving the first duplicate block receives data from the block transmission/reception logic 50-1 of ND 1 (40-1) and 13), the duplicate block is stored (222) and data is transmitted (223) to the block transmission/reception logic (50-3) of the ND 3 (40-3), so that the ND 3 (40-3) is transferred to the block t (25- 16), the second duplicate block is stored (224).

That is, the difference between the conventional HDFS block storage and the method of the present invention is that the data nodes do not occupy the main memory of the data nodes to transmit the duplicate block, and the storage devices 40-1, 40-2, and 40-3 of the present invention. ), so that the use of the storage device of the present invention eliminates the network overhead of each data node involved in transmitting the existing HDFS copy block, thereby increasing the speed of performing effective operations. Is to do.

In the description of FIG. 9, it has been described that the main block is stored in the block of the storage device of the present invention by the data node, but the HDFS client 9 also includes the storage devices 40-1, 40-2, and If 40-3) is allowed to be mounted on the client 9 with read and write privileges, the main block is not transmitted to the data node, and the client is placed in the HDFS block previously created in the storage device of the present invention designated by the name node 8. 9) Can be saved directly.

In addition, as described in FIG. 5, files having the same size as the HDFS block size capable of storing blocks in the storage device of the present invention are not created in advance, and the name node 8 writes a new block whenever an HDFS block write request occurs. As in the example in FIG. 5, an HDFS block file may be dynamically generated and used whenever necessary by using the falloate command.

When the storage devices 40-1 to 40-2 of the present invention are used, in addition to the advantage of not occupying the main memory of data nodes when creating duplicate blocks, there is an additional advantage of efficiently managing resources of the Hadoop system. In general, data processing is performed at the data nodes where the data blocks are located. Therefore, the Hadoop system operates processes that manage resources and determines the data processing schedule according to the location of the block to be processed. Affects processing performance. In the case of using the storage device of the present invention, each storage device of the present invention becomes a local disk of all data nodes. Therefore, since any data node can access any block of the storage device of the present invention without needing to distinguish which data block is located at which data node, job scheduling and resource management are simplified, resulting in overall data processing performance. It gets higher.

Embodiment 2 is a method of storing file data on an existing network attached disk using a bundle allocation server. However, the method of having a separate assignment server has a disadvantage that the entire system does not operate if the assignment server becomes a single point of failure and a failure occurs in the assignment server.

Example 4 shown in FIG. 10 uses the storage device 45 of the present invention equipped with the bundle allocation logic 300 and the integrity check logic 301 to each network attached disk, and the allocation server of the second embodiment has a single failure. Overcome the shortcomings that can become points.

For example, when computer 1 (1-31) of FIG. 10 transmits a disk bundle allocation request 160 having exclusive write rights of the desired amount of computer 1 (1-31) to the storage device 45 of the present invention ( 230) The bundle allocation logic 300 mounted in the storage device of the present invention responds to allocating the bundle 0 (30-1) and the bundle 1 (30-2) to the computer 1 (1-11) to have exclusive write permission. Suppose that it is transmitted to computer 1 (1-31) (231). At this time, in the response transmitted from the storage device 45 of the present invention to the computer 1 (1-31), the name of the disk partition 70 in which the bundle to be allocated is located, the disk offset byte address from which the bundle to be allocated starts, and the bundle are configured. The number of consecutive disk sectors is transmitted. Computer 1 (1-31) assigned with bundle 0 (30-1) and bundle 1 (30-2) with exclusive write permission saves the data file to the assigned bundle through the data write command (161-1) ( 232). A method of writing data to an assigned bundle and a method of reading a bundle in which data is stored are the same as those of the example programs shown in Example 2.

In Example 4, the entire partition is used as a single device file instead of using a file of a conventional file system. Therefore, in order for another computer to read data stored by one computer, computer 1 storing data as shown in FIG. 7 ( 1-31) transfers the data starting from which byte address of the disk partition and the size of the data to another computer k (1-32) that wants to read the data (233). The shared file information exchange modules 180-1 and 180-2 of FIG. 10 determine the offset byte address of the disk partition in which data is stored and the size of the data between the data nodes 1-31 and 1-32. It is a send and receive module.

The integrity check logic 301 of the storage device 45 of the present invention of FIG. 10 checks whether the computer that has transmitted the write command is a computer that has write permission to the bundle, Prevents writing.

FIG. 11 is a schematic diagram of a method of allocating a data node, that is, a computer, to have read-write access exclusively to the entire network attached disk or partition. In other words, as in the first embodiment, the entire disk or disk partition is allocated instead of granting and allocating individual files exclusively to the computer.

For example, computer 1 (1-41) of FIG. 11 is the partition /dev/sda1 (70-1) of the network-attached disk 1 (7-11) and the partition /dev/sdb1 (70) of the network-attached disk 2 (7-12). Mount -2) on their local filesystem with read/write privileges, and the rest of the computers mount these partitions on their local filesystem with read privileges only. By doing this, computer 1 (1-41) is assigned to partition /dev/sda1 (70-1) of network-attached disk 1 (7-11) and partition /dev/sdb1 (70-) of network-attached disk 2 (7-12). 2), but other computers (1-42, 1-43, 1-44) have a partition /dev/sda1 (70-1) of network-attached disk 1 (7-11) and a network-attached disk. You cannot save files in partition /dev/sdb1(70-2) of 2(7-12), but you can read the saved files.

Mounting the network-attached disks or partitions to their respective local filesystems as read-only (read-only) is the same as that of mounting the network-attached disks described in Example 1 of FIG. 5 except for specifying the read-only option. Next, read partitions /dev/sda1(70-1) and /dev/sdb1(70-2) from computers (1-42, 1-43, 1-44) except computer 1 (1-41). This is an example of a command to mount exclusively.

Here, ro means read-only permission.

At this time, the computer generating the file creates the file and writes the data to the corresponding disk partition so that other computers can share the data of the file. Data and metadata remaining in the disk buffer of the operating system are actually saved in the disk by calling the fsync system call as described in FIG. 5 of the first embodiment so that all metadata including information indicating whether it is stored is reflected in the disk partition. Synchronize to be saved.

Another method of synchronizing the created intermediate result file to the disk partition is to open the file by specifying the O_SYNC option when opening the file in the first place, and immediately reflect the data and related metadata to the disk partition whenever data is written. There is also. In Linux, for example, if data is written after opening a file by specifying the O_SYNC option as in the following example program, data and related metadata are not temporarily stored in the disk buffer, but directly on the physical disk partition. Is saved.

12 shows whether each individual storage device 46 of the present invention has a write permission assignment logic 310 that allocates a partition and computers that write to the partition have write permission when each computer requests that a partition be allocated. It is a functional element configuration diagram of the storage device of the present invention equipped with the integrity check logic 311 for checking.

For example, if the computer M (1-51) sends a request (162-1) for allocating a partition having exclusive write rights through the network (2) (240), the partition allocation logic of the storage device 46 of the present invention ( 310) responds to the computer requesting that partition 1 (70-10) be assigned a name (241), and computer M (1-51) assigned partition 1 (70-10) is assigned to partition 1 (70-10). A write command 242 is issued to the disk sector belonging to the partition 1 (70-10) by performing file writing, and the write command is transmitted to the storage device 46 of the present invention through the network (2).

The integrity check logic 311 of the storage device of the present invention detects that an attempt is made to write to a partition that does not have write permission and does not allow the write attempt. For example, if the computer N (1-52) transmits the write command 163-2 to the partition 1 (70-10) that does not have write permission (243), the integrity check logic 303 of the storage device 56 of the present invention is It detects and notifies the computer N (1-52) that the write operation has failed without executing the command (244).

1-1: Data node
1-2: data node
2: network
4-1: network stack
5-1: network interface card
6-1: disk
7-1: Network attached disk
8: Name node
9: client
10-1: Local file system
11-1: Interim result file
20-1: Mapper
21-1: reducer

Claims (10)

A method of sharing files among multiple computers by sharing a network-attached disk storing files.
The method of claim 1,
Files are created on the network-attached disk through a local file system, and files are mutually exclusively allocated to the generated files so that each computer can store data exclusively, and each computer is assigned a file to the assigned files. A method of sharing files between the computers by performing write to store data and allowing the computers to read files that are not assigned to them The method of claim 1,
The disk sectors are mutually exclusively allocated to some of the disk sectors of the network-attached disk so that the computers can independently store data, and each computer writes a file to the allocated sectors to And the computers can read files stored in sectors that are not assigned to them, thereby sharing files between the computers The method of claim 1,
Mount the partition of the network-attached disk so that one of the computers can exclusively write and read files, and the other computers can only read the corresponding partition of the network-attached disk. Mounting and how the computers share files The method of claim 1,
The entire network-attached disk is mounted so that one of the computers can exclusively write and read files, and the other computers mount the network-attached disk so that only reading can be performed. How computers share files The method of claim 1,
The data storage medium of the network-attached disk is a hard disk, SSD (Solid State Drive), flash memory, multiple array independent disks (RAID, Redundant Array of Inexpensive Disks), and JBOD (Just Bunch of Disks). To share a file between a plurality of the computers. As a network attached storage device that provides disks to a computer by connecting to a network, it transmits blocks of HDFS (Hadoop Distributed File System) stored in the storage device to other storage devices through the network, and transmits the blocks from other storage devices to the network. Network attached storage device equipped with block transmission/reception logic capable of receiving HDFS blocks through A storage device that connects to a network and provides a disk to the computer, and if the computer asks to allocate a file to store its data exclusively, the assignment logic that allocates the appropriate file and whether it attempts to write data to a file that does not have exclusive data storage rights. Network attached storage device equipped with inspection logic to check A storage device that connects to a network and provides disks to a computer. When the computer requests that the computer allocate disk sectors to store data exclusively, the allocation logic to allocate the appropriate disk sectors and write data to the sectors that do not have exclusive data storage rights. Network attached storage device equipped with inspection logic to check if attempting A storage device that connects to the network and provides disks to the computer, and when the computer requests that a disk partition be allocated exclusively for data storage, the allocation logic to allocate the appropriate disk partition and write data to the partition that does not have exclusive data storage rights. Network attached storage device equipped with inspection logic to check if attempting

Images