JPH0991185A - Distributed computing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a distributed system which permits plural copies of a file, but in which the meaning rule of a file system is a single copy of the file. SOLUTION: Concerning copies of a file to be designated as 'copy file' in the distributed system, plural copies exist in the distributed system and the respective copies are set to be completely equal to the other copies. Namely, access to any copies results in the same as access when one local copy of the file exists and all access processes are executed on the same host. The synchronization of a write operation for the copy of the copy file is realized by transmitting a message designating the same operation to all the other element systems having the copy of the file whenever the element system executes an operation for changing the copy file to the copy of the copy file.

Description

Detailed Description of the Invention

[0001]

FIELD OF THE INVENTION This invention relates to computer systems, and more particularly to synchronizing operations in loosely connected distributed systems.

[0002]

BACKGROUND OF THE INVENTION Originally, computer systems consisted of a single processor and a mass storage device such as a disk drive for storing files. Each user of the processor has a terminal connected to the processor,
I was able to access the file using that processor. In such a system, all access is through the system's single processor,
There was only a single copy of a given file on the system. Such single processor systems are being replaced by distributed systems as the price of processors, memory, and mass storage devices (eg, disk drives) decrease. Distributed systems include
There are a number of local processors and storage devices connected to the network. A simple example of such a distributed system is a file server in which each user has a personal computer or workstation and provides files to the workstations, which workstations are connected to each other and to the file server by a local area network. There was something that was done. Of course, such a distributed system may itself be a component of a larger distributed system, in which case other single processor systems and distributed systems may be connected by a local area network or a wide area network such as a telephone system. Connected.

The advantages of such a distributed system include improved robustness and speed. The improvement in robustness comes from the number of elements that make up the system. For example, the failure of one workstation does not render other workstations on the network useless. In addition, each workstation has its own file system, so there may be multiple copies of a file on the system. If one copy of a file becomes unavailable,
The copy is available from other workstations.
The speed improvement results from the user having local processing power and local storage that is not shared with others.

While local copying of files improves the robustness and speed of distributed systems, local copying of writable files has one major drawback. The drawback is that it is difficult to keep those copies consistent with each other. Ideally, semantics for file operations in distributed systems
Would be the same as in a system where a group of processes share a processor. For example, UNI
X operating system (UNIX is X / OP
(A trademark of the EN Foundation), when a process writes data to a file, no other process can read data from or write data to that file until the first process completes writing . This is true whether the read process has opened the file before or after the write process makes a system call.

The available distributed systems handle the problem of local copies by means of "caches". The server has a single primary copy (master copy) of the file,
The workstation using the file has a cache containing some or all of the file.
Changes in the cached copy are not immediately reflected in the master copy, and vice versa, so cached copies may conflict with each other or with the main copy. As a result, even if a file read by one workstation follows a write by another workstation, the read may not see the result of the write because it uses the cached copy.

The semantics of file operations in systems that use caches reflect a lack of consistency. An example of such a system is Earl. R. Sand
Berg, et al., “Design and Implementation of Sun Network File System (Des
ign and Implementation of the Sun Network File Syst
em) ”, Proceedings of Summer Usenix (6 June 1985)
Mon) Sun Networ, pages 119-130.
k File System (NFS). In NFS, the given file resides on the file server. The workstation may have a cache containing a copy of the given file. The copy in the cache is assumed to be identical to the copy in the file server for 3 seconds after the copy is loaded into the cache. The cached directory data is assumed to be valid for 30 seconds after being fetched.

Another example is M.E. Kaza (M. Kaza
r), "Synchronization and caching issues in the Andrew file system.
in the andrew file system) '', Proceedings of Winter
The Andrew file system described in Usenix (1988). The Andrew file system ensures that after a file system call is complete, the results of that file system call are instantly visible anywhere on the network, with two important exceptions. The first exception is that in the case of a write system call, new data written to the file is not actually written to the main copy until the process writing to the file closes the file. .
The second exception is that the cached copy of the file is
Only when the process opens the file is it checked for consistency with the primary copy. Thus, when two processes share a file, a first process running on one workstation may transfer data written by a second process running on another workstation to the second process. You won't see until after the first process has opened the file after closing it.

[0008]

What is needed is a distributed system that allows multiple copies of a file, but the semantics of the file system are that of a single copy of the file. The object of the present invention is to realize such a distributed system.

[0009]

In the distributed system of the present invention, some files are designated as "duplicate files". A duplicate file is one in which there are multiple copies in the distributed system, each copy being exactly equivalent to the other copy. Exact equality of files means that accessing any copy has the same result as having only one local copy of the file and all access processes running on the same host. Is to be. Thus, the distributed system of the present invention combines the advantages of a local copy of a file with the file system semantics of accessing a single file on the system such that processes run on the same host.

Another feature of the present invention is a technique for synchronizing write operations for copies of duplicate files.
Whenever an element system of a distributed system host performs an operation that modifies a duplicate file on a copy of the duplicate file, a message specifying the same operation will cause all other elements that have that copy of the duplicate file to have a message specifying the same operation. Sent to the system. Since all messages travel over the same channel, the order of operation is the same for all copies. The write operation can be a basic write operation, or it can be the designation of an operation on the data contained in the duplicate file.

Another feature is that the sequential read operations on the duplicate files are synchronized with the write operations, which results in the same semantics as read operations performed on regular files in single processor systems. Will have.

Synchronization of sequential read and write operations is accomplished by a distributed synchronization system that uses read and write tokens for each replicated file and a token server within each element of the distributed system. The element of the system that performs a write operation on the duplicate file must have a write token, and the element that performs a sequential read operation must have a read token or a write token. To get a token, an element's token server requests the token from another element's token server. The token server receives the token only after receiving all messages specifying incomplete write operations from other elements. As a result, write operations are performed in the same order for all local copies, and sequential read operations have the same results for all copies.

Another feature of the synchronization system of the present invention is that it is implemented using seven locks on each local copy of the duplicate file. These locks are in turn implemented using standard operating system shared and exclusive locks.

Yet another feature of the invention is that it can be used to support transactional synchronization of access to replicated files.

An advantage of the present invention is that the operations required to maintain duplicate files can be implemented at the user level of a distributed system. As a result, the present invention does not require special hardware or special operating systems. The preferred embodiment is implemented as a user level backup file system modification.

[0016]

BEST MODE FOR CARRYING OUT THE INVENTION

[Changing the interface using the library: Figure 2]
As already mentioned, computer systems are layered. Each layer provides an interface to the next higher layer. The upper layer specifies the operations to be performed by the lower layer as required by the lower layer interface. If the upper layer does not follow the interface required by the lower layer, an adapter layer must be added between the upper and lower layers. The purpose of the adapter layer is to specify the operational specifications made according to the interfaces expected by the upper layers,
It is to convert to the operation specification required by the lower layer interface. Using this technique, for example, M
PC running SDOS operating system
However, it is possible for the user to appear as though it were a computer running the UNIX operating system.

When the adapter layer is required by many application programs, it is often implemented as a set of library routines. As the name implies, library routines are routines provided by the manufacturer of a subsystem of a computer system to users of the computer system for use with application programs. FIG. 2 shows how the library routines are used to form the adapter layer. The user program 201 has an interface 206 to the next layer (in this case, a set of system routines). However, the system routine of the computer system in which the user program 201 is used is the interface 213.
Having. Interface 206 and interface 21
The difference 3 is represented in FIG. 2 by the difference in the shape of the lines representing the interfaces. The adapter layer is composed of library routines 207, and is an interface 206 to the next upper layer required by the user program 201.
And an interface 213 to the next lower layer required by the system routine 205. The interface actually consists of a function call, and the routines in the library routine 207 respond to the function call 203 required by the interface 206 to generate the function call required by the interface 213 and to call the function call 203. It works by performing the operation specified by. When finished, the system routine 215 returns the result of its execution to the library routine 207 as indicated by arrow 211, which in turn returns the result to the user program 201 as indicated by the return 205.

Redefining Interfaces Using Dynamic Link Library Routines The usefulness of library routines for redefining interfaces is that in conventional systems, when the executable code for user program 201 is generated by the user. It was limited by the fact that it had to be linked to program 201. Linking in this case refers to the process by which the invocation of the library routine in the user program 201 is related to the position of the library routine in the copy of the library routine 207. The link had to be done when the executable code was generated, so
A user who has only a copy of the executable code can copy one library routine 207 to another library routine 20.
It was not possible to replace the 7 set.

Computer systems are now evolving and library routines can be dynamically linked into user programs. In such computer systems, the linking occurs when the process executing the user program is loaded into the memory of the computer system prior to execution. Dynamic linking allows you to replace one set of library routines with another without changing the object code of your program, which can change the behavior of the system on which your program runs. Is. For a description of dynamic linking, see Shared Library.
Libraries), Sun Microsystems, Inc., Mountain View, Calif., USA (May 1988).

FIG. 3 illustrates how dynamic links are used to change the behavior of the system. System 1 (3
In 01), the user process 306 is executing the application program 309, to which the operating system library 1 (315) is dynamically bound. The operating system library 1 (315) provides an interface to the application program 309 indicated by call 311 and return 313, call 317 to kernel server 305 and return 319 from kernel server 305.
To perform the operation specified by call 311. In the system 2, the user process 306 executes the same application program 309 and uses the same kernel server 305, but here, the operating system library 2 (321) replaces the operating system library 1 (315). Operating system library 2 (3
21) is the operating system library 1 (3
Do all that 15) does. That is, what is needed to transform the system 301 into a system 303 that behaves like the system 301 but also produces side effects 323 is the operating system library 2.
Operating system library 1 (321)
It is only dynamically linked to the user program 309 instead of (315).

[Creation of User-Level Namespace Using Dynamic Link Library] Further, FIG. 4 shows how to dynamically link operating system library 40.
3 is used to create the user-level namespace 405 and how the user-level namespace 405 is used to control the side effect 323. function,
Entities in a computer system, such as files and devices, are referred to by names in programs, and it is a function of the computer system's namespace to associate a name used in a program with the entity represented by that name. In conventional computer systems, the namespace used by user programs is created and maintained by the operating system. In the system 401, the operating system library 403 is the user process 409.
Create and maintain one or more user-level namespaces 405 for. Allows user level namespace 405 to be used by library routines 403 1
Two methods behave differently from the file system provided by the kernel server 305 to the user program 309,
Creating a user-level logical file system that differs in structure, or both. Then, using this logical file system, side effect 323
Control. For example, if system 401 is a backup file system, side effect 323 is that required to create a backup file system, and user level namespace 405 is the file system provided by kernel server 305. It is possible to specify which files in the backup file system should be backed up.
As is clear from FIG. 4, the user level namespace 405
Is a part of the environment of the user process 409.

[Overview of User Level Backup File System: FIG. 5 to FIG. 6] Of the files modified by the application process executing the application program using the dynamic link library and the user level namespace described above. It is possible to create a user-level backup file system that automatically backs up only what you select. FIG.
Shows such a user level backup file system 501. System 501 is implemented by two computer systems. The main system 511 runs the application process 503, and the backup system 513 maintains a backup copy of the files modified by the application process 503. The primary system 511 and the backup system 513 are connected by a communication medium, which allows messages from processes running on the primary system 511 to be sent to processes running on the backup system 513.

The elements of system 501 on main system 511 are application process 503 and kernel server 305 (a). Kernel server 305
(A) provides a file system to the main system 511. In FIG. 5, the file system is the main system 51.
Although represented by a disk 307 (a) local to 1, it can be a remote file system located on another system at all. In any case, the kernel server 305 (a) executes the file operation for the file system to be provided in response to the call 317 from the application process 503,
The result 319 is returned to the process 503, and the operation required by itself is executed on the disk 307 (a). The application process 503 uses the dynamically linkable library to perform file operations with the kernel server 305 (a). In the main system 511, this library is the lib. It has been replaced by a new library called 3d (507). Library 507
In addition to providing an appropriate call 317 to kernel server 305 in response to a call 311 that specifies a file operation that modifies some files, sends a backup message 512 to backup system 513. The file that will result in the backup message 512 being sent is the front end replication tree (FR).
T) 505. The replication tree 505 is arrow 5
06, the lib. It is maintained and used by the routines in 3d (507). In this way,
Replication tree 505 defines a user-level logical file system of files that will result in modification of the backup files on system 513 as a result of modification.

The elements of system 501 on backup system 513 are backend server 515, user-level processes, and kernel server 305 (b) and disk 307 (b), the standard file system for backup system 513. Server and disk drive. Kernel server 305
(B) provides the backend server 515 with a file system. In FIG. 5, the data for the file system is stored on local disk 307 (b). However, it can also be stored on a remote system. The backend server 515 performs a file operation by a call 317 to the kernel server 305 (b) and receives the result of the call from the server 305 (b). The backend server 515 maintains a backend map 517. Backend map 517
Maps the file specified by the front-end replication tree 505 onto a file in the file system of the backup system 513 used as a backup. File system generated by kernel server 305 (a) and kernel server 305
In the embodiment in which the file systems generated by (b) have the same namespace, the backend map 517 is unnecessary.

It will be clear from FIG. 6 how the system 501 operates. FIG. 6 shows a general outline of the format of routine 601 in library 507 that modifies a file. The routine name 603 and the arguments 605 it takes are the same as the names and arguments of the functions used to perform file operations in the library replaced by the library 507. As a result, the calling of this routine in the application program 509 calls the routine 601. After performing the necessary preparations, routine 601 executes kernel server 3
05 (a) executes the same file operation as the routine replaced by routine 601. If this operation is successful, routine 613 calls function 613 with the name of the modified file to determine if front-end replication tree 505 indicates that the modified file should be backed up. . If the front end replication tree indicates so, function 615 returns message 512 with argument 617.
To the backup system 513. Message 51
2, the backup system 513 is the server 305.
Request the backup file system to perform exactly the same operations that were just performed on the file system provided by (a). After sending this message, routine 601 returns. This is also the case if the file was not in the front end replication tree 505, or if the operation specified by function 607 was unsuccessful. The section of code labeled 611 in FIG. 6 specifies a side effect (message 512 in this case). A feature of routine 601 to be noted here is that message 512 is sent to backup system 513 only if the file operation is successful on primary system 511. This is because unsuccessful operations do not need to be backed up.

There are generally two classes of file operations in system 501. Front-end replication tree 50
5 and the user level namespace 405 implemented by the backend map 517 are modified and not modified. An example of a second class of operation is writing to a file specified in the front end replication tree 505. lib. The write function in 3d (507) is the lib. It has the same interface as the write function in the library replaced by 3d. In the preferred embodiment, this is the kernel server 305 to specify the file as an argument.
An integer file descriptor used by (a) and a pointer to a buffer containing the data to be written,
Takes an integer indicating the size of the data to be written. lib.
The write function in 3d is the kernel server 3
05 (a) requests that the system write function be performed on the file specified by the file descriptor, and if the operation is successful, this function returns the file specified by that file descriptor as a front-end replica. Tree 5
Check if it exists in 05. If there,
The function sends a write message 512 to the backend server 515 in the backup system 513 and returns. This message is sent to the kernel server 305
Required to specify a file just written by (a) and perform a write operation just performed by a system write operation in the file system provided by kernel server 305 (a) on the backup file system just as well. Including information. Upon receiving this message, the backend server 515 uses the backend map 517 to determine the file descriptor that the kernel server 305 (b) uses for the backup file, and then the kernel server 305 (b) determines The system write function provided is used to perform a write operation on the backup file with the data and location information provided by this message.

A simple case of the operation of changing the user level namespace 405 is file deletion. lib. The delete function provided by 3d first requests the kernel server 305 (a) to delete the file. When this delete is complete, the delete function checks if information about the deleted file needs to be deleted from the front end replication tree 505. If it does, the function deletes that information. The function then sends the message needed for deletion to the backend server 515 and returns. Upon receiving this message, the backend server 515 finds the file in the backend map 517, requests the kernel server 305 (b) to delete the file, and backends the operation requested by this deletion. Execute on map 517.

A more complex example is renaming. Renaming a file in the file system provided by the kernel server 305 (a) causes the user level namespace 40
There are three possible outcomes to cause in 5.

1. If the old name of the file is part of the user-level namespace 405 and the new name is also part of the user-level namespace 405, then the file remains in the user-level namespace 405. 2. The old name of the file is user-level namespace 4
If the new name is not part of 05 but is part of the user level namespace 405, then the file is added to the user level namespace 405. 3. The old name of the file is user-level namespace 4
05 but the new name is not part of the user level namespace 405, the file is deleted from the user level namespace 405.

In the first case, lib. The rename function of 3d requests the kernel server 305 (a) to perform the rename in its file system.
The function then checks if the renamed file is in the user-level namespace 405, and if it is in the user-level namespace 405, the rename function fronts it to reflect the change. It modifies the end replication tree 505, sends a message to the backend server 515 requesting a name change at the backend server 515, and returns. This message will of course include the old and new pathnames. When the backend server 515 receives this message,
Request a name change from the kernel server 305 (b).

In the second case, the rename function is
Request the server 305 (a) to rename and, as before,
Renamed file is in user-level namespace 405
, But this time the function deletes the renamed file from the frontend replication tree 505, sends a message to the backend server 515, and returns. Back-end server 515
Is a delete message for the file. In response to this message, the backend server 515 causes the kernel server 305 (b) to delete the backup file.

In the third case, as before, the rename function requests a rename, but now it has to send two messages. The first message backs up the file with the name of the file that was moved to the user-level namespace 405, the backup system 513.
Request to create in. Back-end server 51
5 responds to this message by the kernel server 305
(B) requests that the file be created and creates an entry for that file in backend map 517. The rename function then sends a write message with the current contents of the file moved to the user level namespace 405. Back-end server 5
15 responds to this write message by the kernel server 305 (b), and the backup system 51
Write the contents to the backup file in 3.

As can be seen from the above, a single operation performed by the kernel server 305 (a) in the main system 511 causes the backend server 505 to cause the kernel server 305 (b) to perform a series of operations. Request that. As will be further understood, lib. 3d (5
At the end of the operations performed by the functions in 07), the backend map 517 and the frontend replication tree 505 are always in the same state.

[Implementation of the Preferred Embodiment: FIGS.
1] FIG. 7 shows a detailed block diagram of a preferred embodiment 701 of the user level backup file system. In this preferred embodiment, one processor is U
It was implemented in a system running the SunOS 4.1 version of the UNIX operating system and the other processor running the MIPS 4.5 version of the UNIX operating system. System 701
There are two groups of elements. Elements of one group perform backup file operations and elements of the other group make system 701 fault tolerant. The following description first describes the elements that perform backup file operations, and then the elements that provide fault tolerance.

Starting from the main system 511, the application process 503 consists of an application program 509, a dynamically linkable library lib. 3d (5
07), and the front end replication tree 505. The functions of library 507 perform backup file operations as a side effect of file operations. In the system 501, file operations are executed by the kernel server 305 (a). The message generated by the function in the library 507 is sent by the pipe 710 to the backup system 51.
Carried to 3. The pipe 710 is a pipe process 711.
Provided to the application process 503 by
The pipe process 711 itself and the pipe 709 communicate with the application process 503. As will be described in more detail below, pipe process 711 provides a single pipe 710 used by all application processes 503 to create backups on backup system 513.

Next, in the backup system 513 in the preferred embodiment, the backend server 5
15 is divided into two processes: a backend log process (BLP) 716 and a system call engine (SYSCALL ENG) 715. Both use the kernel server 305 (b) to perform file operations. In addition to the backup files, the file system maintained by the kernel server 305 (b) includes log files 703 (b).

The operation is as follows. When initialized, the application process 503 acquires a file identifier designating the pipe 710 from the pipe process 711. When a file operation is executed as a result of the execution of the application program 509, lib. 3d (50
The function for that operation in 7) causes the kernel server 305 (a) to execute the function for the file system provided by the kernel server 305 (a), and further sends a message to the backup system 513 via the pipe 710. send. Upon arrival at the backup system 513, this message is received by the backup log process 716. The backup log process 716 logs the message in the log file 703 (b) in the file system provided by the kernel server 305 (b). Whenever log file 703 (b) has a message, that message is read by system call engine process 715 in order of arrival. In the preferred embodiment, the backend map 5
17 belongs to the system call engine process 715. When the system call engine process 715 reads the message, it causes the kernel server 305 (b) to perform the file operation requested by the message, and the system call engine process 715 self-backends as requested by the message. Maintain map 517.

[Fault-Tolerant Operation of System 701] Fault-tolerant operation of the system requires that a failure be detected and that the system be able to continue operation in response to the detected failure. . In the preferred embodiment, fault detection and response to that fault is handled by WatchD, a user-level system for making distributed systems fault tolerant. For more information on WatchD,
Wai. Huang, Yi. C. Kintal
a), "Software Implemented Fault Tolerance: Technol
ogies and Experiences), 23rd International Conference on Fault Tolerant Computing (23rd Intern
ational Conference on Fault Tolerant Computing),
Toulouse, France, June 22-24, 1993,
And in US patent application Ser. No. 07 / 954,54.
No. 9 (Inventor: Y. Huang, filing date: 19
It is also the subject of September 30, 1992). For purposes of describing the present invention, it should be understood that the WatchD system includes a library called libft and one monitor process on each node of the distributed system. l
ibft is an operation to register a process in WatchD,
It includes routines that perform operations such as specifying an area of memory for automatic backup, and performing checkpoint operations on that memory area. The monitor process monitors user processes registered in WatchD and monitors each other. When the monitor determines that the registered process has failed, the monitor restarts the process. The process can determine what happened when it was restarted by the libft function. While monitoring a user process on one node of a distributed system, the monitor is able to move a copy of important data (also defined using the libft function) to another node of the distributed system. is there. If that monitor's node fails, the monitor on the other node will detect the failure and restart the user process on that other node using the current copy of the critical data. When the failed node comes back up, the monitor for that node restarts the user process using the critical information from the other node and sends a message indicating that the user process has been restarted. Upon receiving the message, the monitor of the other node terminates the user process executing in the other node. In general, WatchD monitors are arranged in a ring configuration, with each monitor monitoring the next monitor in the ring. The number of nodes in the ring and the number of monitors that receive a copy of the critical data of the user process must dictate how many nodes in the distributed system must fail before the user process registered with WatchD cannot be restarted. Decide

In the preferred embodiment, the main system 51
1 and backup system 513 are Wat
It has a chD monitor. These monitors and systems 70
The relationships between the elements of one are shown by the dashed arrows 721. The monitor of the main system 511 is the monitor 717.
As indicated by the dashed arrow 721, the monitor 717 has a pipe process 711 and a front end log process 70.
5 and monitor 719 in system 513. Monitor 719 monitors monitor 717, system call engine process 715, and backend log process 716.

As shown in FIG. 7, a system 701.
Is a failure in the front-end log process 705, the pipe process 711, the system call engine 715, the back-end log process 716, and the system 5
Thirteen faults can be handled. This design has two parts of the system 701 that provide fault tolerance and has two main purposes.

In terms of performance, ensure that recovery overhead is low. · Failure and recovery are transparent to the application,
Guarantees that running applications are never stopped.

The recovery procedure is based on the assumption that WatchD is the most reliable element in the system. The reason is that WatchD performs a very simple task and is capable of self-healing after a failure.

In the following, the recovery from the failure of the backup system 513 will be described in detail, and the recovery from the failure of other processes will also be reviewed. Starting with the failure of the backup system 513, in such a case, the system 701 operates as follows. Monitor 717
When a failure of the system 513 is detected, the pipe process 7
Notify 11 The pipe process 711 creates the front-end log process 705 and stores the file descriptor of the pipe 710 in the front-end log process 705.
Replace with the file descriptor of the pipe 707 to. When the message function used by application process 503 detects a failure of pipe 710, it requests pipe process 711 for a new file descriptor for the pipe. Pipe process 71
1 gives its message function the file descriptor of the pipe 707 connected to the front-end log process 705, and the message sent by the message function goes to the front-end log process 705 instead of the back-end log process 716. When the front-end log process 705 receives the message, it sends the message to the main system 51.
It is put in the log file 703 (a) in 1.

In the preferred embodiment, the message function detects a failure of pipe 710 as follows. Process 503 sends a message through pipe 710 using the TCP / IP protocol. The protocol allows the next message to be sent only if the previous message was received. Thus, the message function used by the function in library routine 507 sends a message through pipe 710 by sending two messages, an actual message and a dummy message. If the message function can send a dummy message, then the actual message has arrived. If system 513 fails, the message sent through pipe 710 will not arrive and a dummy message cannot be sent.

When the backup file system 513 is restored, the monitor 719 displays the system call engine 7
15 and the backend log process 716 is restarted and the monitor 717 is notified. The monitor 717 notifies the pipe process 711, and the pipe process 711 acquires the file descriptor of the pipe 710 and terminates the front end log process 705. When the backend log process 716 is restarted in the system 513, it gets a copy of the log file 703 (a) from the kernel server 305 (a) and attaches it to the log file 703 (b). Subsequently, the system call engine 715 resumes execution of the message in the log file 703 (b).

Lib. The message function used by 3d gets the file descriptor of pipe 710 the same way it got the file descriptor of pipe 707. The message function then attempts to send the message using the file descriptor of pipe 707, and if this attempt fails, the message function again requests the pipe process 711 for the pipe file descriptor. The message function receives the file descriptor of pipe 710 and connects again to the backend.

The remaining failure scenarios are treated as follows.

When the pipe process 711 fails. Monitor 717 detects the failure and restarts the server.
The newly restarted process gets the connection to pipe 710 from the process state saved by WatchD. Other processes know nothing about this failure and recovery.

When the system call engine 715 fails. Monitor 719 detects the failure and restarts system call engine 715. Checkpoint and recovery functions provided by libft have newly restarted system call engine 715
Can recover from an external file to a previously checkpointed status. Other processes know nothing about this failure and recovery.

If the backend log process 716 fails. Monitor 719 detects the failure and restarts backend log process 716. Again, the process 716
Restore status from checkpoint file.
Further, the monitor 719 notifies the monitor 717 that the backend log process 716 has been restarted, and subsequently the monitor 717 notifies the pipe process 711. Process 711 then connects pipe 710 to a new backend log process 716. The next write of each application fails and the lib. 3d gets a new connection from the pipe process 711.

When the front-end log process 705 fails. The front end log process 705 is the system 513.
Present only during the period of failure. When the monitor 717 detects a failure in the front end log process 705, the monitor 717 notifies the pipe process 711. Subsequently, the pipe process 711 restarts the front end log process 705 and reconnects the pipe 708 to it. The next write of the application program 509 fails, and
b. The message transmission function in 3d acquires the file descriptor of the new pipe 708 from the pipe process 711.

[Implementation of User-Level Namespace 405: FIGS. 8-11] The user-level namespace 405 specifies an arbitrary set of files from the file system provided to the application process 503 by the kernel server 305 (a). Can be used to
FIG. 8 shows the relationship between the file system namespace 801 provided by the kernel server 305 (a) and the user level namespace 405 in the user level backup file system 701.

In the name space 801, file names are arranged in a tree. The files (B, D, E, G, I, M, N) forming the leaves of the tree in FIG. 8 contain data or programs. The remaining files are a list of other files. Such files are called directories. An arbitrary file in the namespace 801 can be designated by a path name to the kernel server 305 (a). The path name begins with the root "/" and contains the names of all files from the root to the name of the file specified by the path name. Therefore, the path name of the file D is / A / C / D, and the path name of the file L is / J / K / L.

The user level backup file system 701 specifies a file to be backed up by specifying the subtree of the namespace 801 that contains the file. Next, an operation on the file that modifies the file in the subtree is performed on the backup file in the backup system 513. In FIG. 8, three subtrees, 803 (a), 8
03 (b) and 803 (c) have been selected to be backed up. As a result, the namespace 801
As a result of a change to a data file D, E, G, I, M, or N in a backup file for that data file is made,
Changes to F, H, and L also cause changes to their backup files. Since all files in a subtree are backed up, the files to be backed up can be specified in the user-level namespace 405 by the pathname of the directory that is the root of that subtree. Thus, the subtree 803 (a) is specified in the user level namespace 405 by pathname / A / C (805 (a)).

Of course, the user level namespace 405
Must also be mapped to the file system provided to the system call engine 715 by the kernel server 305 (b). This is done by the backend map 517. As shown in FIG. 9, backend map 517 includes an entry 901 for each open file in user level namespace 405. This entry has two parts. The user level name space information 903 specifies a file in the user level name space 405, and the backup system information 9
Reference numeral 05 designates a file corresponding to the file designated by the user level namespace information in the file system provided by the kernel server 305 (b).

According to the backend map 517, the kernel server 305 (b) is set to the backend log process 71.
6 and the system call engine 715, it is possible to map the subtree of the namespace 801 to the subtree of the namespace 907 of the file system.
This mapping is done by mapping the pathname of the root of the subtree of namespace 801 to the pathname of the root of the corresponding subtree of namespace 907. The root pathname is called the file pathname prefix in the subtree. Thus, subtree 803
The path name in (a) has the prefix / A / C,
The path name of the file E in the subtree 803 (a) is E. In namespace 907, subtree 909 corresponds to subtree 803 (a) by mapping prefix / A / C of namespace 801 to prefix / Z of namespace 907. After mapping, as a result of the change of the file designated by the path name / A / C / E in the namespace 801, the namespace 907
In, the file specified by the path name / Z / E will be changed.

[Details of Front-End Replication Tree 505: FIG. 10] In the preferred embodiment, the user-level namespace 405 is implemented as a front-end replication tree 505. FIG. 10 shows the front end replication tree 50.
5 shows the details. Frontend replication tree 505-2
The two main components are the RTREE 1015 and the file descriptor (FD) cache 1027. R
TREE 1015 is a linked list of pathnames of roots of the subtree 803 composed of files to be backed up. The file descriptor cache 1027 is
It is an array that associates a file descriptor with a device and an inode identifier. The format of this implementation is UNI
The file system provided by the X operating system processes files in three ways: by path name, by an integer file descriptor, and by the identifier of the device where the file resides and the U.
It is the result of specification by the entry (i-node) for that file in the NIX file system table. The file descriptor of a file is valid only for the process that opened the file, and only while that process has the file open. The UNIX file system table allows translations between pathnames and devices and inodes, and between devices and inodes and current file descriptors, but between pathnames and current file descriptors. No direct conversion is possible.

More specifically, MAXTRY1
003 and INIT 1005 are used in initializing the front end replication tree 505. MAXTR
Y1003 indicates the number of times the initialization function attempts to set the pipe 710 in the backup system 513 and gives up. INIT 1005 indicates whether the pipe has been set. RPLOP array 1009
Is the name of the operation 1 that can be performed on the replication tree 505
011 is the sequence.

RTREE PTR1013 is an RTRE
This is a pointer to the first element of the E list 1015. RT
The REE list 1015 is a linked list including one element 1017 for each replication tree 803. Each element 1017
Contains the pathname 1021 of the root of the replication tree 803, the length 1019 of the pathname 1021, and a pointer 1023 to the next element in this linked list. The connection server 1025 uses the pipe 7 to the backup system 513.
10 is a path name in the namespace 801.

The FD cache 1027 is an array of file descriptor cache entries 1029.
The array has as many entries 1029 as there are file descriptors available to the application process 503. The index of the entry for a given file descriptor in the FD cache 1027 is that file descriptor. Entry 1
029 contains a status flag that indicates whether the entry is currently valid and also indicates whether the child process was created while the file was open. The entry 1029 is an identifier 1101 of the device in which the file exists in the main system 511.
And an identifier 1103 of the inode of the file in the main system 511. In the subtree 803 designated by the entry in the RTREE 1015, there is a valid entry 1029 for each file that is currently open.

[Details of Backend Map 517] The backend map 517 has two parts, that is, the path name map 1113 and the open duplicate file list 11.
Seventeen. The path name map 1113 simply maps the path name in the namespace 801 of the primary system 511 to the path name in the namespace 907 of the backup system 513. Each entry 1115 in the map establishes a relationship between the front end path name 1118 and the back end path name 1119. In the path name map 1113, the root of the subtree 803 in the front end namespace 907,
Contains an entry that maps to the root of a subtree in namespace 907. The backend path name 1119 is a part of the backend system information 905. In the preferred embodiment, these mappings are specified in the system configuration file.

The open duplicate file list 1117 is
Application process 503 contains an entry 1120 for each file currently open in replication tree 803. The user level namespace information 903 in the entry 1120 includes the front end file identifier (FFID) 1105 and the front end path name (F
P) 1106 is included. Front-end file identifier 1
Reference numeral 105 includes a device identifier and an inode identifier for a file in the main system 511. The front end path name 1106 is divided into a front end prefix (FPR) 1107 and a subtree path name 1108. Front end prefix (FPR) 1
107 is a prefix for the subtree of the file in the front end namespace 801. The subtree path name 1108 is the path name of a file in the subtree. The backup system information 905 in the entry 1120 is made up of a backend file descriptor 1111. The backend file descriptor 1111 is a file descriptor of the file in the file system provided by the kernel server 305 (b). In the preferred embodiment, the backend map 517 includes a frontend file identifier 1105 and a frontend pathname 11
It is implemented as a hash table that can be accessed by any of 06.

[Operation on Data Structures 505 and 517] Below, how is the data structure 505?
And 517, and how these data structures are affected by various file operations. In the preferred embodiment, application process 503 is timed on a UNIX operating system using a Korn shell. The Korn shell allows a process to set an ENV variable that specifies a file to be executed whenever the process calls the Korn shell. EN in application process 503
The file specified by the V variable is stored in the front end replication table 50 by the application process 503.
It contains the information necessary to configure and initialize 5. Once created, the table 505 becomes part of the application process 503's address space and is available to any child process of the application 503 that is created with the UNIX operating system fork system call and thereby inherits its parent's environment. Becomes On the other hand, the exec system call gives the child process a new environment. In order to make the front end replication tree 505 available to the child process of the application process 503 created by the exec system call, l
ib. 3d has an exec function that copies the front end replication tree 505 to the ENV variable of the new process. This allows the new process to
The front end replication tree 505 can be used without inheriting the parent address space. In other embodiments, named pipes or external files can be used to pass the front end replication tree 505 to the child process created by exec.

Proceeding to the description of the file operation, the first file operation is a mount operation. UNI
In the X operating system, mount adds a tree of names from the file system to the operating system namespace. In a preferred embodiment, lib. The version of mount implemented in 3d has a mode in which a subtree of the frontend namespace 801 is added to the user level namespace 405 as a replication tree 805. When mount is used in this mode, the pathname argument is the user-level namespace 40
5 is the path name of the root of the subtree 803 added to item 5. The function adds the subtree 803 to the user-level namespace 405 by creating a replication tree entry 1017 for that pathname and adding that entry to the replication tree 1015. There is also a umount (unmount) operation that deletes the replication tree entry 1017 having the specified path name from the replication tree 1015.

When the application process 503 executes an open operation on a file in the replication tree 805, lib. The open function in 3d creates a file descriptor cache entry 1029 for the newly opened file and sends an open message to the backend log process 716.
This open message includes the path name, device identifier, and inode identifier in the main system 511 of the opened file. When this message is executed by the system call engine 715, the result is:
An entry 901 is created in the backend map 517. The main system 5 using the path name map 1113
The file in the backend system 513 corresponding to the open file in 11 is found and the file descriptor for the corresponding file is placed in the backend file descriptor 1111.

When a file is opened, file operations in main system 511 use a file descriptor that identifies the file. The message for the corresponding operation on the backup file in the backup system 513 is the device identifier and i
Use the node identifier to identify the file. To execute such a message, the system call engine 715 uses the open duplicate file list 1117 for the device and inode specified in the message.
You only need to access entry 1119 in. This entry is a file descriptor 1111 required to execute an operation in the backup system 513.
including.

When the application process 503 closes a file in the replication tree 505, lib. 3
The close function of d determines from the status field 1033 whether the child process is using the file. If no child process is in use, the close function returns the replication tree 505.
Invalidates the file descriptor cache entry 1029 for that file in and sends a close message to the backup system 513 containing the device identifier and the inode identifier. When the system call engine 715 executes this message, it uses the device identifier and the inode identifier to find the entry 1119 for this file. Subsequently, the backend file descriptor 1111 is used to identify this file, the file in the backup system 513 is closed, and finally, the entry 1119 is deleted from the open duplicate file list 1117.

[Implementing Duplicate Files Using User Level Backup File System: FIG. 13] The backup file system 501 specifies the current file for each file from the primary system 511 specified in the front end replication tree 505 of the application process 503. It is useful in that the copy is guaranteed to reside on the backup system 513. However, the result of the operation in the backup system 513 that modifies the copy of the file from the main system 511 is not reflected in the file in the main system 511. In fact, this means that only the primary system 511 can modify the files backed up on the backup system 513.

In order for both primary system 511 and backup system 513 to be able to modify the copy of the file, as required if the file is a duplicate file, each system must be on its own system. You have to back up the changes made.
That is, the two systems must be peers with respect to operations on copies of duplicate files. With respect to FIG. 5, the two systems 511 and 513 must each have a channel 512 for backup messages to the other system and a backend server 515. Further, the process on the system that wants to change the file is lib. 3d (507)
And the file must have a front end replication tree 505 listed as a replication file. In addition, changes to the duplicate file copies occur in the same order on both systems, and read operations on the local copy of the duplicate file are provided taking into account the writes made on the remote copy of the duplicate file. A synchronization system is required to ensure.

FIG. 13 shows two peer hosts 1302.
(A) and 1302 (B) and duplicate file 13
25 shows an overview of a distributed system 1301 with 25. Each host has a kernel server 305 (not shown) and mass storage. The mass storage device is here disk 307 (a) for host 1302 (A) and disk 307 for host 1302 (B).
(B). Each disc has an identical copy of the duplicate file 1325. The copy on host 1302 (A) is copy 1325 (A) and host 1302 (B)
The top copy is copy 1325 (B). Further, each host 1302 has a backend server 515,
A backup message can be received from the other host 1302. 3 in host 1302 (A)
Two processes 1309 (A, 1..3) are associated with lib. It has a front end replication tree 505 that contains the 3d code 507 and designates the file 1325 as a replication file. One process 1309 on host 1302 (B)
(B, 1) has code 506 and such a front end replication tree. Since each host 1302 functions as a backup for each other, process 1309
(A, 1..3) is a write operation (ie, host 13
Copy 1325 of duplicate file 1325 on 02 (A)
(Operation to change (A)), a backup message 512 (A) is generated as a result of the write operation, and the backend server 5 on the host 1302 (B)
15 (B) responds to this by copying 1 of the duplicate file.
Perform the same write operation on 325 (B). When process 1309 (B, 1) performs a write operation on copy 1325 (B), this write operation results in a backup message 512 (B), which backend server 515 (A) responds to with Copy 1
Perform the same write operation on 325 (A). The backup message 512 is sent over a channel that ensures that the messages arrive in the order in which they were sent, resulting in copies 1325 (A) and 1325.
The write operations for (B) are done in the same order. One way to implement such a channel is TCP / I.
Sending a backup message 512 through P.

Of course, the backend server 513 is
At the same time as the process 1309 performs a write operation on the duplicate file 1325 (A), the duplicate file 13
No write operation is performed on 25 (B). As a result, a read operation on file 1325 (B) may have different results than a simultaneous read operation on file 1325 (A). In some cases this is not the case, but in others it may be different. As a result, the system 1301 has a duplicate file 1
Two types of read operations for 325 are possible. The first read operation is an "atomic read" operation. Atomic read operations are duplicate files 1325
Used when the copy of does not need to be consistent.
This operation simply waits until the current write operation to the local copy of the replicated file 1325 is complete before reading that local copy. The second read operation is a "sequential read" operation. This operation is used when the copy of the duplicate file 1325 must be consistent, so that the copy of the duplicate file being read is consistent with all other copies of that duplicate file. , Synchronize with the write operation to the duplicate file.

The synchronization of the write operation and the sequential read operation is realized by two tokens for the duplicate file 1325, that is, a write token 1327 and a read token 1328. Write token 1
Host 1302, which has 327, can perform read or write operations on the local copy of the duplicate file. Read token 1328
The host 1302 having the host can perform a read operation for the local copy, but cannot perform a write operation. Host 1302, which does not have any token, can only perform atomic read operations. If the host 1302 does not have the required token, it requests the token from another host 1302. If the write operation is incomplete on any of the other hosts 1302, the backup message 51 is sent after sending the last write backup message 512.
Send the token to the channel used for 2. By sending the token in this way, the host 1302
It ensures that write operations occur in the same order in all local copies of duplicate file 1325, and that sequential reads are performed on the same local copy of duplicate file 1325.

Only one host 13 at a given moment
02 only has a write token 1327, and at that moment all other hosts 1302 have no token. If neither host has a write token 1327, then all hosts have a read token 1328.
Having. A host 1302 that has neither a read token nor a write token can request either. A host with a write token can give either a read token or a write token. A host with a read token can request or grant a write token.

The system 1301 has three or more hosts 13.
When it has a 02, the write operation is broadcast to all hosts 1302 that do not have a write token 1327. Token requests and token grants are also broadcast to all hosts 1302. Broadcasting is done using a reliable broadcast package that provides a reliable broadcast order of requests and awards. An example of such a package is the ISI provided by Cornell University.
S. ISIS refers to "Reliable Communication in the Presence of F
ailures) '', ACM Transactions on Computer Systems,
5, 1, February 1987, pp. 47-76. For read token 1328, the only host 1302 that can present the read token.
Is a host that has a write token. As a result, there will never be more than one message granting a read token. For write token 1327, a single host 1302 having write token 1327
, Or all hosts have a read token 1328. In the former case, there is only one message granting the write token 1327. In the latter case, the requesting host 1302 may have read token 13 before it actually has write token 1327.
The award message must be received from all hosts 1302 with 28.

Process 1309 is own host 1302
In order to write to the copy of duplicate file 1325 above, two conditions must be met. The host 1302 on which this process is running must have a write token 1327 for the duplicate file. There must be no uncompleted write operations from another host 1302 to the copy of the duplicate file 1325 in the host 1302.

The host 1302 uses the write token 13
Write token 1 to another host if you do not have 27
Must request 327. Another host 1302
Sends a write token 1327 in the backup message 512 so that the second condition is met, ie, the receiving host 1302 will have a copy of the duplicate file 1325 until the modification specified in the last backup message 512 is complete. Guaranteed not to change.

In the preferred embodiment, synchronization using the write token 1327 is performed by the token server 1311 on each host 1302 and the token file 1307.
And the channels that send out the backup messages 512 in the order in which they are received. Token file 1307 has an area for each duplicate file 1325 that has a copy on host 1302. A standard operating system locking subsystem can lock an area of a file. Two
There are different types of locks. Exclusive locks allow only one process to access the file. Shared locks can be accessed by any number of processes. In general, a process must have an exclusive lock on an area to write to it and a shared lock to read from it. In the preferred embodiment, an operating system lock on the area of the duplicate file 1325 in the token file 1307 is used to copy the duplicate file 132.
Implements the token used to synchronize write and sequential read operations for 5.

For example, in the preferred embodiment, the write token is implemented as a write token lock formed from an operating system lock on an area of the duplicate file 1325. Token server 13
Upon receiving the message indicating that it has a write token, 11 acquires the write token lock.
Process 130 running on host 1302 as long as token server 1311 has write token lock
9 can acquire locks for atomic read operations, sequential read operations, or write operations on the local copy of the duplicate file. These locks are also implemented using operating system locks in token file 1307.

When another host 1302 requests a write token, the token server 1311 releases the write token lock and acquires a tokenless lock (implemented like any other lock). Token server 131
As long as 1 has a tokenless lock, backend server 515 is the only process in host system 1302 that can write to duplicate file 1325. Of course, the backend server 515 responds to the backup message 512 from the host 1302 that currently has the write token 1327.

The operation of system 1301 is as follows. User level backup file system 501
As already described in the explanation of Lib. 3d (507)
Is statically or dynamically bound to the code of application process 509 that performs operations on the duplicate files. The file is then designated as a replicated file in the front end replication tree 505. Used in the system 1301. The 3d (507) version replaces the standard I / O library write routine with the write operation shown in FIG. Third
The get_write_token () function 1201 of the line writes the write token 132 to the token server 1311 of the related host.
Request 7 If the host's token server 1311 has a write token 1327, this function returns immediately. If the token server 1311 does not have the write token 1327, it requests another host and returns when the write token arrives. When the token server 1311 has the write token 1327, the process 1309 executes the write system call syscall (SYS_write, fildes, buf, nbyte) in the fourth line. The function then determines from the front end replication tree 505 whether the file has been replicated, as described in the description of system 501. If duplicated, the write message 512 is sent to another host 1302.
And the write token is released (120
3). The write token 1327 is the same as
Any host 130 that modifies the duplicate file 1325
Must also be acquired for write operations on 2. As a result, all changes are duplicated in file 1325.
All copies are made and all changes are made in the same order.

[Detailed Implementation of Synchronization: FIG. 14] In the preferred embodiment, the lock file 1307 belonging to the copy of the duplicate file 1325 on a given host 1302.
Region has two locks associated with the write operation. The first lock indicates whether the write token 1327 is on the host 1302 and the second lock is
A copy of the duplicate file 1325 is the copy of the host 1302.
Indicates if it is available for writing by the above process 1309. FIG. 14 shows how these two locks are used in the preferred embodiment. The pseudo code 1401 in the figure is again the lib. 3d (5
07) write operation. The region containing the lock is represented by the variable TOKEN_REGION (1403), which has two fields. STATE of lock
Indicates whether the write token 1327 is on host 1302, and the lock TOKEN is process 130
9 indicates whether writing can be performed.
The lock represented by STATE is held exclusively locked by the local token server 1311 when the token 1327 is on another host 1302.

The operation described by the pseudo code 1401 is as follows. As shown in the third line, the area of the token file 1307 containing the lock 1403 is searched by the function fd2token. This function takes the file descriptor of the local copy of the duplicate file 1325 and returns the area 1403. The next step is to determine if the write token 1327 for the replicated file 1325 is in the local host 1302.
This is done by requesting a non-blocking shared lock in the STATE field of field 1403 in line 4. If this lock is retrievable, the write token 1327 is on the local host 1302. If this lock is not obtainable, the pseudo code 1401
The token server 1311 sends a message requesting the token 1327 to the corresponding token server on the other host 1302 and calls a function (not shown) that waits for the message providing the token to return. In the fourth line, whether the write token 1327 is on the local host 1302
09 attempts to acquire a shared lock so that 09 can determine.

Once the write token 1327 is locally available, proceed to the next step. Another lock request is made to the STATE field of area 1403, as shown in line 6. This time it is blocking and process 1 executing code 1401
309 waits until a shared lock on STATE can be acquired (ie, until write token 1327 is locally available) (line 6), then blocks until it acquires an exclusive lock on TOKEN. When the process 1309 receives the exclusive lock, the write system call is actually made, and the message 51
2 is sent to the other host 1302, with the contents of writing to the local copy of the duplicate file 1325. When this is done, area 1403 is unlocked and the write operation ends.

Of course, any other process 1309 attempting to write to the duplicate file 1325 on the local host 1302 will have a shared lock on STATE and will be in the queue of processes waiting for an exclusive lock on TOKEN. be able to. When the process 1309 that completed writing unlocks TOKEN_REGION, the next such process in the queue
Can get an exclusive lock on and perform write operations. Furthermore, the local token server 1311
From the other token server 1311
When the message 512 requesting the write token 1327 for 25 is received, the local token server 1
311 requests an exclusive lock for STATE. The local token server 1311 receives its exclusive lock only after all processes 1309 that have a shared lock on STATE have completed write operations. When the local token server receives the exclusive lock on STATE, it sends a message informing the other host 130 through the same channel that the message generated by the write operation was sent to the other host 1302.
Send to 2. The messages put on the channel arrived in the order in which they were sent, so that the token server 1311 on the other host 1302 had the token 1325 on the backend server 515 on the other host 1302. Only after all write messages 512 from host 1302 have been processed will the exclusive lock on STATE be released.

Synchronization of sequential read and write operations performed on the other host 1302 is accomplished by the read token 1328, much as described with respect to the write token 1327. The process 1309 of performing a sequential read is a read token 1328.
Or first try to get a shared lock on a portion of the token file 1307 that indicates whether the write token 1327 is on the host 1302, li
b. Execute the code in 3d (507). If this attempt is unsuccessful, process 1309 proceeds to token server 1
311 reads the read token 1328 to another host 130.
Request to get from 2. The token is obtained in the same manner as described for the write token 1327. Next, process 1309 attempts to acquire an exclusive lock on the region representing the token and blocks until read token 1328 arrives at host 1302. When the token arrives, process 1
309 requests a shared lock on the local copy of the duplicate file 1325. Process 1309 is not an exclusive lock on its local copy whose local copy is being executed by another process 1309, and is also an exclusive lock on a remote write operation being executed by backup server 515 in response to backup message 512. The shared lock can only be taken if it is not.

As already indicated, the write token 13
Token server 1311 with 27 can present read token 1328. When the token server 1311 with the write token 1327 receives the request, waits for the write operation on the duplicate file to finish, changes the lock on the local copy of the duplicate file from the exclusive lock to the shared lock, and for the write backup message 512. Send a read token depending on the channel used for. All of this causes the read token 1328 to arrive at the requesting host 1302 after the last write backup message 512.

[State Machine for Duplicate Files: FIG. 1
5] Application process 13 in host 1302
09, token server 1311, and backend server 515 cooperation, as well as token server 1311 cooperation between hosts 1302, is better understood by considering process 1309, token server 1311, and backend server 515 as state machines. It For a given duplicate file 1325, the state of each of these machines depends on the write token 1327 and read token 1328 for that file and changes as the token moves between hosts 1302.

Application process 1309 has four states with respect to duplicate file 1325.

1. No operation. In this state, the application process 1309 has the duplicate file 1325.
It does not have any kind of lock on its local copy, and therefore cannot perform read or write operations on the local copy.

2. Atomic read state. In this state, process 1309 has only a shared lock on the local copy, and therefore can only perform atomic read operations.

3. Sequential read state. In this state, the read token 1328 for the duplicate file 1325
Alternatively, the write token 1327 is on the host 1302 and the process 1309 has a shared lock on the local copy, so it can perform sequential read operations as well as atomic reads.

[0092] 4. Write state. In this state, the write token 1327 for the replicated file 1325 is on the host 1302, and the process 1309 has an exclusive lock on the local copy, so it can perform write operations as well as sequential read and atomic read operations. The write operation is backed up with another copy of the duplicate file 1325.

As is clear from the description of states, the transition from one state to another involves the acquisition and loss of tokens and locks. For example, the transition from the no-operation state to the sequential read state requires acquisition of the read token 1328 at the host 1302 and acquisition of the shared lock by the process 1309.

The state of the process implementing the backend server 515 is closely related to the states 1 and 4 of the application process 1309.

1. No operation. In this state, the backend server 515 does not have any kind of lock on the local copy of the replicated file 1325 and therefore cannot perform read or write operations on the local copy.

2. Write state. In this state, the backend server 515 has an exclusive lock on the local copy of the replicated file 1325 and can therefore write to the local copy.

As can be seen from the above, the state change is caused by the movement of the token between the hosts 1302.

In the most complicated case, the token server 131
1 is a process for realizing (A) and (B). The token server 1311 is the host 1302 (A)
And (B) must cooperate with each other to pass the token. FIG. 15 shows two token servers 131.
3 is a state diagram for 1. FIG. In FIG. 15, each state has the numbers 1501, 1502, ..., 1506. State transitions are indicated by arrows. The last two digits of the arrow reference number indicate the start state and end state in which the transition indicated by the arrow is made. Therefore, the arrow 1531 indicates the transition from the state 1503 to the state 1501. Each state transition is the result of a message received at the token server 1311 so that the message is sent to another token server 1311.
May be sent to 11.

In FIG. 15, the message received for a given transition is shown in italics next to the transition arrow, and the message sent is shown in block. For example, the transition indicated by arrow 1531 is the result of a TSgetRtokenTS (italicized) message from another token server 1311. This transition also generates a TSgrantRtokenRP_TS (block body) message to another token server. The name of a message indicates the origin, destination, content of the message and the route taken by the message. For example, TSgrantRtokenRP_TS (block body) must be sent over the channel used for backup message 512 (RP_), from one token server (first TS) to another token server (second TS). Read token grant to (grantRtoke
n) A message. Similarly, TSgetRtokenTS (italics)
Is a read token request message from one token server to another token server, but this message need not be sent through the channel for backup messages.

Starting with the overview of FIG. 15, there are three main states.

Write token state 1503. In this state, the local host 1302 has the write token 1
327 and the process 1309 has a duplicate file 132
It is possible to perform all read and write operations for the local copy of 5, and the token server 1311 can grant both read and write tokens.

Read token state 1501. In this state, the local host 1302 has only read tokens, and the process 1309 can perform all read operations on the local copy of the replicated file 1325 but cannot write to the local copy and the token server 1311 writes. Token 132
7 requests or awards can be made.

No token state 1505. In this state, the local host 1302 has no token, the process 1309 can only perform atomic read operations on the local copy of the replicated file 1325, and only the backend server 515 has the local copy of the replicated file 1325. It is writable, and the token server 1311 can only request read tokens or write tokens.

The initial state in FIG. 15 is the read token state 1501. Application process 130
When 9 attempts a write operation during read token state 1501, state transition 1512 occurs. The token server 1311 receives the APgetWtokenTS (italicized) request from the process 1309, and responds to this request to the token server 1311 of the other host 1302 by TSgetWtokenTS.
(Block type) Send a message. Here, the local token server 1311, in the read token / write token wait (RwaitW) state 1502, gives a write token from another token server TSgrantW.
Wait until you receive the tokenTS message.
As the state name implies, the local host holds a read token while waiting for a write token. Upon receiving the TSgrantWtokenTS (italicized) message, transition 1523 occurs and the token server 1311
Enters the write token state 1503 and the write token 1327 comes to the local host 1302. Here, the write operation that made the request for the write token is performed, resulting in a write to the local copy of the duplicate file 1325 and a write backup message 512 sent to the other host 1302.

Of course, any token server 131
Neither 1 has write token 1327, and each may request write token 1327 from the other. In that case, the tiebreaking algorithm (represented by state transition 1522) is executed in state 1502 to determine which host 1302 receives the write token 1327. This algorithm requires that one host be designated as the primary host and the other host as the secondary host.
In response to the request from the other host 1302, the primary host ignores the request and continues in state 1503. Secondary host does not have token / write token wait (Nwai
tW) Transition 1524 to state 1504.

The token server 1311, state 1503
To the requesting token server 1311 with the write token 1327.
Alternatively, either read token 1328 can be provided. In response to the request for read token 1328, the transition indicated by arrow 1531 occurs. TS
In response to the getRtokenTS (italicized) message, the token server 1311 abandons the write token 1327,
To the requesting token server 1311, TSgrantRtokenRP_TS
Send a (block body) message, but retain the read token 1328. As a result, any token server 1311 has a read token and is in state 1501.

When the request is for a write token 1327, the transition indicated by arrow 1535 occurs. In response to the TSgetWtokenTS request, the token server 1311 abandons its token and TSgrantWto through the channel used for message 512.
The kenRP_TS (block body) message is sent to the backend server, and the token server 1311 enters the state 1505. State 1505 may be reached from state 1501 by transition 1515. This transition is in state 1501
Token server 1311 at receives the TSgetWtokenTS (italicized) message and responds to it by transitioning now 15
Occurs when operating as described for 35. However, since the host of the token server 1311 is not writing, it is not necessary to send the message to the backend server through the channel of the message 512.

Token server 1311 remains in state 1505 until application process 1309 on local host 1302 attempts to perform a read or write operation on the local copy of replicated file 1325. For read operations, this attempt results in a transition 1556, from process 1309.
APgetRtokenTS (Italic) message is token server 1
311 and the token server 1311 responds with TSgetR to the other token server 1311.
Send a tokenTS (block body) message. Next, the token server 1311 enters the no token / wait for read token (NwaitR) state 1506 to wait for the read token 1328. Waiting, write token 13
Local requests for 27 are queued. When a TSgrantRtokenRP_TS (Italic) message granting this token arrives through the channel of message 512,
As a result, transition 1 to read token state 1501
561 happens.

When application process 1309 attempts a write operation on a duplicate file, transition 1554 results. In this transition, the token server 1311 responds to the APgetWtokenTS (italicized) message by sending a TSgetWtokenTS (blocked) message to the other token server 1311 resulting in state 1504. The token server then states 150
4, the TSgr from the other token server 1311
Wait for antWtokenRP_TS (italicized) message. The channel to which this message is sent is the backend server 5
It is for message 512 to 15. TSgran
When a tWtokenRP_TS (italicized) message arrives, status 1
Transition 1543 to 503 occurs.

[Implementation of Sync: FIGS. 1, 16 and 17] In the preferred embodiment, the sync is a duplicate file 1325.
Is implemented by a set of 7 locks for each local copy of In FIG. 16, the type of lock, the elements of system 1301 that must have the lock, and
Here is a list of the meanings of locks. Lock 1601, 160
3 and 1605 are locks that the process 1309 must have to perform atomic read operations, sequential read operations, and write operations on the local copy of the replicated file 1325, respectively. Lock 1607 is the local backend server 5
15 is a lock that must be held in order to write to the local copy of duplicate file 1325. Locks 1609, 1611, and 1613 are requested by the local token server 1311 as requested by the token for the duplicate file 1325 currently residing on the local host 1302. For example, if the local host 1302 has a write token and receives a request for that write token, the local token server 1311 will write the write token lock 1
Release 609 and acquire tokenless lock 1613.

FIG. 17 shows a lock semantic rule. FIG.
There are rows and columns for each of the six locks. If there is an x at the intersection of a row and a column, different requesters cannot simultaneously hold the lock for that row and the lock for that column. For example, if the token server 1311 has a tokenless lock 16
If so, process 1309 will not have sequential read lock 1603 or local write lock 1605, as required for the situation where the local host 1302 has no token.

In the preferred embodiment, the locks of FIG. 16 are implemented by shared and exclusive locks provided by an operating system such as the SunOS operating system of the UNIX operating system or System V Release 4 (SunOS is a Sun Mic
is a trademark of rosystems, Inc.). The lock provided by the operating system allows the process to
You can get a shared or exclusive lock on a byte of a file. In addition, this byte can also be used to represent locks on other files. Thus, in the preferred embodiment, FIG.
In the lock of the token file 1327, the byte string in the token file 1327 is copied to each duplicate file 1
It is implemented by allocating 325 and expressing the lock of the duplicate file using the bytes in the byte sequence of the duplicate file. The byte string for the duplicate file 1325 is called a slot. In the preferred embodiment, each slot has 3 bytes. Each lock in FIG. 16 is represented by a combination of locks on the bytes of the slot.

FIG. 1 shows the combinations used in the preferred embodiment. The first column is a list of locks used on the local copy of duplicate file 1325 in system 1301. The second column shows what kind of operating system lock is used. R
Indicates a shared lock, W indicates an exclusive lock, and NL indicates no lock. The remaining columns show the bytes of the slot. A dash in a column for each byte indicates that the OS lock specified in the second column has been acquired for that byte.
Thus, process 1309 uses atomic read lock 16 on the local copy of replicated file 1325.
When acquiring 01, the operating system has a shared lock on byte 0 of the slot for duplicate file 1325. Similarly, when process 1309 has acquired the write lock 1605 for the local copy, the operating system has a shared lock on bytes 1 and 2 of the slot and an exclusive lock on byte 0.

This mapping of OS locks implements the conflict table of FIG. 17 for system 1301 locks. The conflicting system 1301 locks are mapped to conflicting OS locks and overlapping offsets, while the non-conflicting system 1301 locks are mapped to non-conflicting OS locks or non-overlapping offsets. . In the preferred embodiment, this mapping is automatically generated by a synchronization tool called SLEVE. (A. Skarra,
"SLEVE: Semantic lock for event synchronization (S
LEVE: Semantic Locking for EVEnt synchronizatio
n) '', Proceedings of Ninth International Conferenc
See e on Data Engineering (1993). )

Performing High Level Operations on Duplicate Files An important consequence of each local copy of a duplicate file being equivalent to all other copies of the same is the high level operation just like the write operation above. Can handle the operation of. For a write operation, host 1302 with write token 1327 will copy file 1
Performs a write to the local copy of 325 and then sends a message specifying the write and the data to be written to another host 1302, where the backend server 515 performs the write operation specified in the message. This is executed for the local copy of the duplicate file 1325 in the host 1302. Exactly the same can be done for high-level operations, such as sorting and merging two duplicate files. Duplicate files for all hosts 130
Write token 1327 because they are equivalent in 2
A host 1302 with a can proceed as follows. Write backup message 51 for all writes associated with sort / merge operations
Instead of sending 2, the host 1302 performs a sort-merge on the local copy and then
A message 512 specifying the merge operation is sent to another host 1
It is possible to send to 302. Duplicate file 1325
Are equivalent on all other hosts 1302, so the result of this specified sort-merge operation is the same on all hosts 1302. The advantage of this approach is that
Sending a specification of a sort-merge operation requires much less time and network resources than sending all write operations resulting from a sort-merge on host 1302 that has a write token to other hosts 1302. is there.

More specifically, high level operations can be used in the following situations. Each host 13
02 has two duplicate files, a sorted master list file and an update file. The update to the master list file is performed on the update file, and the sort / merge operation is periodically performed on the update file and the master list file to generate a new master list file. The update file is then deleted, recreated and the cycle restarted. Updates to the update file are made using write operations as described above. Therefore, at the time of the sort-merge operation, all hosts 1302 have the same update file and master list file. The host 1302 having the write token performs the sort-merge operation by itself, and then sends the message 512 having the designation of the sort-merge operation to each of the other hosts 13 in exactly the same manner as when the write backup message 512 is sent.
Send to 02. This specification can be code for a sort-merge operation, but this operation is performed iteratively, so this specification is generally a sort-merge operation performed on update files and masterlist files. Is a command line that specifies that Next, sort merge is executed for each of the other hosts 1302.
Host 13 with write token proceeding in a similar manner
02 deletes and recreates the update file and sends a message designating the same operation to another host 1302. Of course, the operation can be specified at any level. For example, in the above example, it is possible to specify the sorting / merging and the deletion / re-creation of the update files in the shell script, and the operation specification may be that shell script, or all other hosts 1
If 302 has a copy of the shell script, it could be the command line used to invoke the shell script.

[Transactional Synchronization of Access to Duplicate Files] As described, the duplicate file system guarantees consistency between duplicate files for a single access to a file. The replicated file system can also be used to ensure consistency for transactional access to files. A transaction is a single logical action consisting of a single access sequence. An example of a transaction is as follows. Each record r in the file f has an attribute readnu that stores the number of times the program P has read r.
Suppose that m is included. P is rea in the pseudo code below
Increment dnum. However, the parameter list of the function is omitted.

For every r in f read (f); increment_readnum (); write (f);

When several instances of P are executed in parallel, access level synchronization alone is not sufficient to guarantee consistency in the records of f.
Immediately after a P reads r, no other program should read or write r until P increments readnum and writes r to f. The read-increment-write sequence is a single logical action (i.e., transaction) on r to be synchronized as an atomic unit.

If there is no separate transaction mechanism,
Programmers can use the locking primitives of the operating system to implement transaction synchronization isolation for such sequences. The programmer simply takes the sequence exclusive_lock (f) ... unlock
All you have to do is add a primitive like (f). In the general case, calculations involve multiple files and the program
Multiple locks must be required when guaranteeing isolation. For example, the calculation has two files f1 and f
2, while the result is another file f3
Is stored. read (f1); read (f2); computation (); write (f3); Then, the program encloses this sequence with a lock request as follows. share_lock (f1); share_lock (f2); exclusive_lock (f
3) ... unlock (f3); unlock (f2); unlock (f1); But if a process requests multiple locks at one time,
Deadlocks can occur. The synchronization protocol is provided by the transaction mechanism, or
Deadlocks must be prevented or detected and resolved depending on whether they are defined by the application using the operating system lock.

Existing operating systems support primitives to lock (remote) files in local area networks. However, in a distributed environment, locks are effective due to high communication costs, and deadlock detection is more difficult, especially since it requires a global algorithm. Currently available operating system lock primitives do not fully support transactional synchronization in a distributed environment.

The duplicate file system described here is
Provides a service called TX that supports deadlock-free transactional synchronization of program sequences that access replicated files across local or wide area networks. The replicated file system implements a protocol that minimizes the number of messages needed to synchronize parallel processes.

In the preferred embodiment, the duplicate file system generates a current sequence number (curSeqN _f ) for each file f it manages. The duplicate file system is
CurSeq when the process creates f in the replica tree 505 (or moves f to this replica tree 505)
Initialize N _f to 0 and increment curSeqN _f when the process modifies _f in a write () operation. Process P
Receives curSeqN _f as part of getting a read or write token for _f . This value is global. Ie curSeq
N _f reflects all changes to f regardless of whether the process involved is local or remote to P.

In order to use TX, process P is flagged
Open the duplicate file f with O_TX. When P first reads or writes f, TX stores curSeqN _f in the local variable locSeqN _f . The next time P reads or writes f, TX will compare locSeqN _f with curSeqN _f . If they are not equal, then another process has changed f in the meantime and TX returns an error. Otherwise, TX is the operating system read or
Call the write function and return the result.

The duplicate file system defines the order of the entire files to be duplicated to avoid deadlock. When a TX function requests multiple tokens, it makes the requests in a predefined order.

[Interface] The control structure of TX is UN
IX interface Encapsulated in a function interface that redefines and extends the open / close / read / write names and return values. TX defines ETX as a new value for errno. This means an error due to transactional synchronization. TX also defines two new functions, readtx () and writetx (). They are,
It accesses a set of files and has parameters of type fd_set * (as in the system call select) and tx_t *. However, struct txt (char * buf; int nbyt
e;}. The program calls the function with the bit set in fd_set for the file to be read or written, and in the case of ETX error return, the function is called fd_set.
On set, clear all bits except those for which curSeqN has changed. The interface and pseudo code are as follows:

Open (char * path; int flags; mode_t m
ode) If flags include O_TX, initialize the local variable locSeqN _path to 0 and remove O_TX from flags. System call ope
Invokes n and returns the result.

Read (int fd; char * buf; int nbyte) Acquires the read token for the file f opened with fd. If O_TX is not set for fd, call the system call read, release the read token, and return. Otherwise, if the locSeqN _f = 0, substituting the curSeqN _f to locSeqN _f. locSeqN _f ≠ cu
If rSeqN _f , set errno = ETX, otherwise call system call read. Release the read token and return.

Write (int fd; char * buf; int nbyte) Acquires the write token for the file f opened with fd. If O_TX is not set for fd, the system call write for the local copy of f
And, if successful, sends the update to the remote copy, frees the write token, and returns.
Otherwise, if locSeqN _f = 0, then curSeqN _f is lo
Substitute in cSeqN _f . errno if locSeqN _f ≠ curSeqN _f
= ETX, otherwise update the copy of f as above, and for O_TX locSeqN _f (and curSeq
N _f ) is incremented. Release the write token and return.

Readtx (fd_set * readfds; struct tx_t
* txp) For each bit readfds [fd] set, get the read tokens for file f opened with fd in the order defined by the duplicate file system,
If O_TX is set for fd, do the following: If locSeqN _f = 0, mark f as _null,
Otherwise, if locSeqN _f ≠ curSeqN _f , mark f as modified. If any f has changed,
Set errno = ETX and in * readfds clear all bits except those for modified files. If there are no changes in any of the f, for each file f, called the system call read, if f is _null substitutes CurSeqN _f to LocSeqN _f. Release all tokens and return. This function does not read any files if they have changed, otherwise it reads all files.

Writetx (fd_set * depends_on, * writefd
s; struct tx_t * txp) Each bit set depends_on [fd] or writefds [f
For each d], get the read token or write token for the file f opened by fd in the order defined by the duplicate file system, respectively, and if O_TX is set for fd, then To execute. If it is locSeqNf ≠ 0 and ≠ locSeqN _f ≠ curSeqN _f, to mark that there is a change the f. If any f has changed, set errno = ETX and * depends_on
Or in * writefds, clear all bits except those for modified files. If there is no change in any f, call the system call write for each fd in * writefds. If this succeeds,
Send the update to the remote copy of f, and set locSeqN _f to curSeqN _f +1 (and increment curSeqN _f ) _if O_TX is set. Release all tokens and return. This function does not write if any files have changed, otherwise it does all writes.

Resettx (fd_set * fds) For each set bit fds [fd], re-initialize locSeqN _f to 0 for file f opened at fd.

[Usage] TX supports an application defined retry protocol. For illustration purposes, recalling the above example of an access that initiates a new transaction with each new record, the pseudo code using TX would be as follows for each record r in file f: if (fd = open (f, O_RDWR O_TX)) <0 exit; FD_ZERO (&fdset); FD_SET (fd, &fdset); for every r in f resettx (&fdset); for (try = TRY_NUM; try>0; try- -) (if read (fd, buf, nbyte) <0 exit; increment_readnum (); if write (fd, buf, nbyte)> = 0 break; if errno! = ETX exit; / * otherwise reset and re-run To try. * / / * Some other process changed f after read (). * / resettx (&fdset);}

The program resets each time it starts a new read-increment-write sequence.
Reinitialize locSeqN _f with x (). This is because this sequence consists of a single logical action that does not depend on the (previous) state of f.

In contrast, the pseudo-code using TX for the second example for files f1, f2 and f3 shows that locSeqN _f1 or locSeqN _f1 Reset locSeqN _f2 or both.

/ * Open f1 to f3 with fd1 to fd3 together with O_TX flag. * / / * Initialize the tx_t array with the buffer. * / FD_ZERO (&readset); FD_ZERO (&writeset); for (try = TRY_NUM; try> 0; try--) {FD_SET (fd1, &readset); FD_SET (fd2, &readset); if readtx (& readset, txp) <0 {if errno! = ETX exit; resettx (&readset);continue;} Computation (); FD_SET (fd3, &writeset); if writetx (& readset, & writeset, txp)> = 0 break; if errno! = ETX exit; / * Otherwise reset and try again. * / / * Some other process changed f1 or f2 after read (). * / resettx (&fdset);}

The program later resets the file.
If you access f1, f2, or f3 without calling x (), the access will fail if another process has modified the file during this time.

[Applications] TX supports a type of transactional synchronization similar to the optimistic, timestamp-based algorithms used in database systems. In an optimistic manner, transactions read data, record timestamps, perform calculations, and create a list of database updates. At the commit point,
The transaction sends the update to the database system with a time stamp. If the combination of update and timestamp satisfies the transaction synchronization isolation, the database commits the transaction. If not, the transaction is suspended (and restarted).

In contrast, in the pessimistic, lock-based approach, transaction T gets permission (ie, lock) on the object before each read or write. Locks preclude changes in that object by other transactions. If the lock request conflicts with another transaction's lock, T does no further computation and simply waits until the other transaction releases the lock. The pessimistic approach is good for applications where transactions perform intensive long-running calculations on database objects, as work is less likely to be lost by suspending and restarting.

The optimistic approach has important effects in the area of operating systems. It should be noted that the operating system typically serves several applications simultaneously, thus providing each application with fair access to the resources controlled by the operating system. The optimistic method does not wait for locks, so applications are less likely to be denied access to resources.

[Conclusion] Above, how to improve the user-level backup file system,
It has been described how to create a distributed system with duplicate files. Sequential read and write operations performed on local copies of duplicate files have the same semantics as read and write operations on files where there is only one copy. This semantic is implemented by a distributed synchronization system using read and write tokens managed by a token server on elements of the distributed system. In the preferred embodiment, the distributed synchronization system is implemented with seven locks for each local copy of the replicated file. In addition, these locks are implemented using standard operating system locks on 3-byte vectors.

Although the above is the optimum form for realizing the distributed system having the user-level duplicate file system that the present inventor is currently aware of, many variations are possible. In particular, the principles of the present invention can be used in systems that are unrelated to the user level backup file system described above. For example, the synchronization technique disclosed here is almost independent of the technique used to specify duplicate files or perform file backup operations, and in fact, synchronizes the operations on duplicate files. It can also be used for other purposes. Furthermore, synchronization can be achieved by methods other than the lock protocol used in the preferred embodiment.

[0143]

As described above, according to the present invention, the operation required for maintaining a duplicate file can be implemented at the user level of the distributed system. As a result, the present invention does not require special hardware or special operating systems. The preferred embodiment is implemented as a user level backup file system modification.

[Brief description of drawings]

FIG. 1 is a diagram of an implementation of locks used to achieve synchronization in the preferred embodiment.

FIG. 2 is an overview of how the library redefines the interface to a user program.

FIG. 3 illustrates that dynamically linked libraries can be used to redefine operating system interfaces.

FIG. 4 illustrates that dynamically linked libraries can be used to provide a user level namespace.

FIG. 5 is a schematic diagram of a user level backup file system using a dynamic link library.

FIG. 6 is a schematic diagram of a routine in a dynamic link library.

FIG. 7 is a schematic diagram of a preferred embodiment of a user level backup file system.

FIG. 8 is a diagram showing a relationship between a namespace provided by a kernel server 305 (a) and a user level namespace.

FIG. 9: User-level namespace and kernel server 305
FIG. 6 is a diagram showing relationships between namespaces provided by (b).

FIG. 10 is a detailed view of a front end replication tree 505.

FIG. 11 is a detailed diagram of a back end map 517.

FIG. 12 is a pseudo code diagram of a portion of a synchronization system for replicated files.

FIG. 13 is a block diagram of a token mechanism of a synchronization system.

FIG. 14 shows pseudo code for a call that replaces the write system call in the preferred embodiment.

FIG. 15 is a state diagram showing synchronization of operations on a duplicate file in a distributed system having a two-element system.

FIG. 16 is a diagram of a table of locks used in the preferred environment.

FIG. 17 is a diagram of a table of lock semantic rules of FIG. 16;

[Explanation of symbols]

201 user program 203 function call 205 return 206 interface 207 library routine 209 function call 211 return 213 interface 215 system routine 301 system 1 305 kernel server 306 user process 307 disk 309 application program (user program) 311 call 313 return 315 operating system library 1 317 Call 319 Return 321 Operating System Library 2 323 Side Effects 401 System 403 Operating System Library 405 User Level Namespace 409 User Process 501 User Level Backup File System 503 Application Process 505 File Front end replication tree (FRT) 507 lib. 3d library 509 Application program 511 Main system 512 Backup message 513 Backup system 515 Back-end server 517 Back-end map 601 Routine 603 Routine name 605 Argument 701 User-level backup file system 703 Log file 709 Pipe 710 Pipe 711 Pipe process 715 System call engine (SYSCALL) E
NG) 716 Backend Log Process (BLP) 717 Monitor 719 Monitor 801 Namespace 803 Subtree 901 Entry 903 User Level Namespace Information 905 Backup System Information 1003 MAXTRY 1005 INIT 1009 RPLOP Array 1013 RTREE PTR 1015 RTREE 10 File Reserver 1025 Connection Server 1105 Front End File Identifier (FFID) 1106 Front End Path Name (FP) 1107 Front End Prefix (FPR) 1108 Subtree Path Name 1111 Back End File Descriptor 1113 Path Name Map 1117 Open Replication File List 1118 Front End Path Name 1119 Back End Path name 1301 distributed system 1302 peer host 1307 token file 1309 process 1311 token server 1325 duplicate file 1327 write token 1328 read token

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Andrea H. Scarla United States, 07928 New Jersey, Chatham, Orchard Road 26

Claims (5)

[Claims] 1. A method for synchronizing operations performed on a resource by a process executing on an element of a distributed data processing system, wherein a token required to execute the operation in the element is within the element. Determining whether the token is in the element, in the process, if the token is in the element, wait for the token to become available and perform the operation when the token becomes available; If not, the token server in the element requests the token from another element in the distributed data processing system; and in the process, when the token arrives from the other element, the token is An operation executing step for executing the operation when available, in the token server, Responding to a request for the token to the other element only after completion of the operation, synchronizing the operations performed on the resource by the process executing on the element of the distributed data processing system. How to make. 2. The operation executing step comprises: stopping the process until the process acquires a lock on a resource requested by the operation, and then executing the operation; and after executing the operation. Releasing the lock. 3. The method of claim 2, wherein the operation is a write operation to a file in the element that has been duplicated in the other element. 4. An apparatus for generating a duplicate file in a distributed system, comprising: first local operation executing means for performing an operation on the first copy of the duplicate file; and for the first copy of the duplicate file. Second local operation executing means for executing the above-mentioned operation, and operation notifying means for notifying the other local operation executing means of the operation for the copy of the duplicate file created by its own file system means In each of the local operation executing means, in response to the operation notifying means in the other local operation executing means, a notification operation executing means for executing the notified operation for the copy of the duplicate file in the local operation executing means. In each local operation execution means, the operation for the copy of the duplicate file is executed for which copy. Characterized in that comprising a means for synchronizing the operation for copying the replicated file to have the same results regardless of Luke, apparatus for generating a replicated files in a distributed system. 5. In a distributed computing system having a plurality of element computing systems, a duplicate file located in the element computing system and having a copy on different element computing systems is executed for the copy. A distributed computing system comprising distributed means for maintaining the same result regardless of which copy the operation is performed on.