US20030051117A1 - Method and apparatus for extending a file size beyond a file size limitation - Google Patents
Method and apparatus for extending a file size beyond a file size limitation Download PDFInfo
- Publication number
- US20030051117A1 US20030051117A1 US09/951,113 US95111301A US2003051117A1 US 20030051117 A1 US20030051117 A1 US 20030051117A1 US 95111301 A US95111301 A US 95111301A US 2003051117 A1 US2003051117 A1 US 2003051117A1
- Authority
- US
- United States
- Prior art keywords
- storage area
- allocated
- accordance
- file
- instructions
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
- G06F3/0665—Virtualisation aspects at area level, e.g. provisioning of virtual or logical volumes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F2003/0697—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers device management, e.g. handlers, drivers, I/O schedulers
Definitions
- the present invention relates generally to file systems for computers, and more particularly, to a method and apparatus for extending the size of a file beyond a file size limitation imposed by a computer file system.
- An operating system is a program that is used to manage other programs (i.e., application programs) in a computer system.
- the operating system is initially loaded into the computer by a boot program. Once loaded, the operating system can perform a number of services for the application programs. Such services include determining the order in which certain applications can run, managing the sharing of resources (e.g., memory) between the applications, and managing input and output to and from hardware devices, such as disk drives.
- the application programs make use of the operating system by requesting services, for example, through the use of an application programming interface (API).
- API application programming interface
- a user of the computer system can also interact directly with the operating system, for example, through the use of a graphical user interface (GUI).
- GUI graphical user interface
- each type of operating system is typically closely related to and may be designed to work with a specific file system that manages the data on the disk drives.
- Some examples of operating systems include Unix, Linux (a variant of Unix) and Windows.
- a file system typically specifies a convention for naming files, including, for example, the maximum number of characters in a file name, the type of characters that can be used, the format of file extensions that are permitted, etc.
- the file system also specifies the algorithmic or logical locations where a file can be placed.
- Windows and Unix-based operating systems typically employ file systems that use a hierarchical or tree-like structure wherein a file is placed in a directory or subdirectory located at a particular position in the hierarchical structure.
- a file system can possess two different but related constraints: a limitation on the maximum size of an individual file, and a limitation on the maximum size of the file system itself.
- the size of an individual file may be physically limited by the number of bits used in describing an address space of the file.
- some versions of Linux which were designed for use with a hardware architecture of 32 bits, use a four byte integer to address the contents of a file.
- the maximum size of a file is limited to 2 31 bytes minus some number of bytes, i.e., about 2 gigabytes.
- the size of the file system itself is also limited. While the address space of a file system is typically represented by all or part of an eight byte integer, the maximum size of the file system is usually set to a predetermined limit. Limiting the maximum size of the file system provides a number of advantages. For example, the computer system may require less memory and may be able to locate files faster than if the file system were larger. On older Linux systems, for example, the maximum size of the file system was set at one terabyte. That is, regardless of the size of a physical disk, the disk must be divided into a plurality of partitions, each of which is less than or equal to the maximum allowable size of the file system.
- a maximum file size of, for example, 2 gigabytes can be very restrictive. Seismic work typically involves the processing of large volumes of seismic data. These volumes of data often span hundreds of magnetic tapes, and can be several hundred gigabytes in size. Typically, as seismic data tapes are entered into a system, the contents of several tapes are copied to multiple files in one directory or data storage area. Then, when that data storage area becomes full, subsequent tapes are copied to many other data storage areas, which can be scattered around the computer system. Because the maximum size of the file system may be limited, for example, to 1 terabyte, one data storage area may not have sufficient free space to hold all of the incoming data.
- the user is required to manage the data by recording the locations (in the various data storage areas) of each part of the data. Accordingly, a need exists for a method for extending the size of a file beyond a file size limitation imposed by the file system used by a computer system.
- One aspect of the present invention is directed to a method and apparatus for creating a virtual storage volume having a size that is independent of a file size limitation of a computer system.
- a storage area is randomly selected from among a plurality of available storage areas.
- a determination is made as to whether the selected storage area contains at least a predetermined amount of free space. If so, free space corresponding to the predetermined amount is allocated on the selected storage area.
- a symbolic link to the allocated storage area is written in a directory associated with the virtual storage volume. Data destined for the virtual storage volume can then be written in the allocated storage area. Once the allocated storage area has been exhausted, the above steps can be repeated to create another allocated storage area, with a symbolic link thereto being written in the directory associated with the virtual storage volume.
- the size of the virtual storage volume can exceed the file size limitation of the computer system.
- FIG. 1 is a schematic illustration of a computer network in accordance with the present invention
- FIG. 2 is a flow diagram illustrating a method for creating a virtual storage volume in accordance with the present invention
- FIG. 3 is a block diagram illustrating a user's directory and showing symbolic links from the user's directory to a plurality of file systems in accordance with the present invention.
- FIG. 4 is a flow diagram illustrating a method for performing a file operation with respect to a virtual storage volume in accordance with the present invention.
- FIG. 1 schematically illustrates a hardware environment of an embodiment of the present invention.
- a first communications network 110 provides an electronic communications medium that connects a user at a client computer 120 to first and second server computers 130 , 140 .
- the first and second server computers 130 , 140 respectively run first and second operating systems have respective file systems associated therewith for interfacing to first and second storage arrays 135 , 145 .
- the first and second operating systems can be different operating systems or can be the same operating system.
- the client computer 120 is also connected to a third server computer 150 via a second communications network 160 .
- the third server computer 150 runs a third operating system having an associated file system for interfacing to a third storage array 155 .
- Each of the first, second, and third storage arrays 135 , 145 , 155 can include a plurality of storage devices or can be a single storage device.
- Storage device as used herein can encompass hard disk drives, tape drives, solid state memory devices, or other types of storage devices.
- the third operating system can be different from the first and second operating systems or can be the same operating system.
- the client computer 120 can be a personal computer or a workstation.
- the server computers 130 , 140 , 150 can be personal computers, workstations, minicomputers, or mainframes.
- the client computer 120 and the server computers 130 , 140 , 150 can be bi-directionally coupled to the first and second communications networks 110 , 160 over communications lines, via wireless systems, or any combination thereof.
- client computer 120 and the server computers 130 , 140 , 150 can be coupled to one another by various private networks, public networks or any combination thereof, including local-area networks (LANs), wide-area networks (WANs), or the Internet.
- LANs local-area networks
- WANs wide-area networks
- a virtual storage volume can be created in accordance with the present invention, for example, by a user at the client computer 120 creating a file in the user's home directory, or loading data from a tape or other storage device to a file in the user's home directory.
- files located in the user's home directory are conventionally subject to a number of constraints: the maximum file size, the maximum file-system size, and the size of the user's local hard disk drive.
- the present invention advantageously allows the user to disregard these conventional constraints and to treat the user's home directory as a substantially unlimited data storage area.
- the user can process and load data by manipulating what appears to the user to be a single file within the user's home directory structure.
- FIG. 2 is a flow diagram illustrating the operation of a computer, in creating a virtual storage volume in accordance with the present invention.
- the virtual storage volume can be created in a directory associated with a particular user, and can be created by the computer as a result of the user creating a file or writing to a file in the user's home directory.
- the method of creating the virtual storage volume can be implemented in a computer program, or alternatively, can be implemented as a subroutine library, which is called by a computer program and integrated into the computer program.
- the computer determines whether the selected storage area contains at least a minimum amount of free (e.g., unused or unallocated) space.
- the amount of free space required can be determined by a system administrator based on the total amount of storage space available, or can be based on other considerations. For example, the use of about a 250 megabyte block of space can result in more efficient utilization of available storage space, while the use of about a 2 gigabyte block can result in more efficient data transfer.
- the system administrator can set the required amount of free space via a parameter (e.g., a block size parameter) associated with the virtual storage volume.
- the computer can then look for an amount of free space in the selected storage area based on information representing the block size, which is stored in the parameter.
- the system administrator can also adjust the size of the block (e.g., based on currently available resources, such as the amount of available hard drive space) by editing the data stored in the block size parameter.
- step 220 If the selected storage area has at least the required amount of free space available (Yes in step 220 ), the process proceeds to step 225 . If the selected storage area does not have at least the predetermined amount available (No in step 220 ), the process returns to step 215 , wherein the computer randomly selects another storage area.
- the process of the present invention advantageously distributes input/output activity among the various server computers 130 , 140 , 150 and associated storage arrays 135 , 145 , 155 .
- the random selection of a storage area provides the advantage that, even when multiple users are creating their respective virtual storage volumes, the allocated storage areas are likely to be in different file systems. Thus, multiple users are not likely to be accessing the same file systems at the same time, which can cause an input/output bottleneck to occur.
- step 225 the computer allocates an amount of free space in the selected storage area that is equal to the required amount.
- the amount of free space allocated can be determined, for example, by the block size parameter.
- step 230 the computer writes a symbolic link to the allocated storage area in a directory associated with the virtual storage volume. The symbolic link points to and enables access to the allocated storage area.
- step 235 the computer writes the data to be stored in the virtual storage volume in the allocated storage area. As data is written into the allocated storage area, the amount of space remaining will diminish. As long as there is space available in the allocated storage area, additional data can be placed there. However, once there is no more space remaining in the allocated storage area (No in step 240 ), the process returns to step 215 wherein another storage area is randomly selected. Alternatively, the process can return to step 210 to receive a list of available storage areas, which list can be changed, for example, due to network conditions, since the last time that a storage area was selected.
- the process thus allows a large number of allocated storage areas to be linked to a single virtual storage volume, such that the total amount of space available for a single file (i.e., the aggregate of the allocated storage areas) is greater than the maximum allowable file size as determined by the operating system. Further, as the allocated storage areas can be on different file systems, the total amount of space available can be greater than the maximum size of any particular file system.
- FIG. 3 is a block diagram illustrating the relationship between a virtual storage volume, which is located in a user's directory 300 , and the actual data, which are distributed between various ones of the first, second, and third storage devices 135 , 145 , 155 .
- the user's directory 300 contains a virtual storage volume named “UserFile”, which consists of a plurality of files, respectively named “UserFile.DAT”, “UserFile.001”, “UserFile.002”, . . . “UserFile.xxx”.
- Each of the files “UserFile.DAT”, “UserFile.001”, “UserFile.002”, . . . “UserFile.xxx” contains a symbolic link that points to a storage area on one of the first, second, or third storage arrays 135 , 145 , 155 .
- the file “UserFile.DAT” contains a symbolic link 325 that points to a disk DISK 6 in the first storage array 135 .
- the files “UserFile.001”, “UserFile.002”, and “UserFile.xxx” respectively contain symbolic links 330 , 335 , 340 that respectively point to DISK 1 in the third storage array 155 , DISK 2 in the first storage array 135 , and DISK 3 in the second storage array 145 .
- the files associated with the virtual storage volume are randomly interspersed among the first, second, and third storage devices 135 , 145 , 155 .
- the allocated storage areas and the directory containing the virtual storage volume are located in different file systems. In other embodiments, depending on the size of the user directory, at least a portion of the allocated storage areas can be located in the same file system as the directory containing the virtual storage volume.
- FIG. 3 allows the user to see each of the plurality of files that make up the virtual storage volume
- the constituent files of the virtual storage volume can be hidden from the user. The user can thus only see one file in the user directory for each virtual storage volume.
- FIG. 4 is a flow diagram illustrating the operation of a computer, in performing a file operation with respect to a virtual storage volume in accordance with the present invention.
- the virtual storage volume can be located in a directory associated with a particular user, and can appear to the user to be a file in his home directory.
- the method of performing a file operation with the virtual storage volume can be implemented in a computer program, or alternatively, can be implemented as a subroutine library, which is called by a computer program and integrated into the computer program.
- step 410 the computer receives an instruction to perform a file operation with respect to the virtual storage volume.
- the instruction can be to read or load into memory a virtual storage volume named “UserFile”.
- step 420 the computer locates a symbolic link associated with the virtual storage volume.
- virtual volume “UserFile” includes a file named “UserFile.DAT” containing a symbolic link 325 .
- the computer reads the symbolic link (e.g., symbolic link 325 of the file “UserFile.DAT”), which points to an allocated storage area (e.g., DISK 6 of the first data storage area 135 ).
- the file containing the symbolic link is preferably located in a directory associated with the virtual storage volume, such as the user's directory.
- step 440 the computer locates the allocated storage area pointed to by the symbolic link.
- the computer then performs the requested file operation (step 450 ) with respect to the allocated storage area pointed to by the symbolic link (e.g., DISK 6 of the first data storage area 135 ).
- the process returns to step 430 to read a second file associated with the virtual storage volume (e.g., “UserFile.001”), locate the next allocated storage area, and perform the file operation with respect to the allocated storage area.
- the process repeats steps 430 through 440 until the file operation has been performed with respect to all the files of the virtual storage volume (No in step 460 ).
- the process then ends at step 470 .
Abstract
Description
- The present invention relates generally to file systems for computers, and more particularly, to a method and apparatus for extending the size of a file beyond a file size limitation imposed by a computer file system.
- An operating system is a program that is used to manage other programs (i.e., application programs) in a computer system. In a typical computer system, the operating system is initially loaded into the computer by a boot program. Once loaded, the operating system can perform a number of services for the application programs. Such services include determining the order in which certain applications can run, managing the sharing of resources (e.g., memory) between the applications, and managing input and output to and from hardware devices, such as disk drives. The application programs make use of the operating system by requesting services, for example, through the use of an application programming interface (API). A user of the computer system can also interact directly with the operating system, for example, through the use of a graphical user interface (GUI).
- With respect to managing input and output between the application programs and one or more hardware devices, each type of operating system is typically closely related to and may be designed to work with a specific file system that manages the data on the disk drives. Some examples of operating systems include Unix, Linux (a variant of Unix) and Windows.
- A file system typically specifies a convention for naming files, including, for example, the maximum number of characters in a file name, the type of characters that can be used, the format of file extensions that are permitted, etc. The file system also specifies the algorithmic or logical locations where a file can be placed. Windows and Unix-based operating systems typically employ file systems that use a hierarchical or tree-like structure wherein a file is placed in a directory or subdirectory located at a particular position in the hierarchical structure.
- Depending on the addressing structure used, a file system can possess two different but related constraints: a limitation on the maximum size of an individual file, and a limitation on the maximum size of the file system itself.
- The size of an individual file may be physically limited by the number of bits used in describing an address space of the file. For example, some versions of Linux, which were designed for use with a hardware architecture of 32 bits, use a four byte integer to address the contents of a file. Thus, the maximum size of a file is limited to 231 bytes minus some number of bytes, i.e., about 2 gigabytes.
- The size of the file system itself is also limited. While the address space of a file system is typically represented by all or part of an eight byte integer, the maximum size of the file system is usually set to a predetermined limit. Limiting the maximum size of the file system provides a number of advantages. For example, the computer system may require less memory and may be able to locate files faster than if the file system were larger. On older Linux systems, for example, the maximum size of the file system was set at one terabyte. That is, regardless of the size of a physical disk, the disk must be divided into a plurality of partitions, each of which is less than or equal to the maximum allowable size of the file system.
- In many applications, particularly seismic applications, a maximum file size of, for example, 2 gigabytes can be very restrictive. Seismic work typically involves the processing of large volumes of seismic data. These volumes of data often span hundreds of magnetic tapes, and can be several hundred gigabytes in size. Typically, as seismic data tapes are entered into a system, the contents of several tapes are copied to multiple files in one directory or data storage area. Then, when that data storage area becomes full, subsequent tapes are copied to many other data storage areas, which can be scattered around the computer system. Because the maximum size of the file system may be limited, for example, to 1 terabyte, one data storage area may not have sufficient free space to hold all of the incoming data. Thus, the user is required to manage the data by recording the locations (in the various data storage areas) of each part of the data. Accordingly, a need exists for a method for extending the size of a file beyond a file size limitation imposed by the file system used by a computer system.
- One aspect of the present invention is directed to a method and apparatus for creating a virtual storage volume having a size that is independent of a file size limitation of a computer system. A storage area is randomly selected from among a plurality of available storage areas. A determination is made as to whether the selected storage area contains at least a predetermined amount of free space. If so, free space corresponding to the predetermined amount is allocated on the selected storage area. A symbolic link to the allocated storage area is written in a directory associated with the virtual storage volume. Data destined for the virtual storage volume can then be written in the allocated storage area. Once the allocated storage area has been exhausted, the above steps can be repeated to create another allocated storage area, with a symbolic link thereto being written in the directory associated with the virtual storage volume. Thus, the size of the virtual storage volume can exceed the file size limitation of the computer system.
- Another aspect of the present invention is directed to a method and apparatus for performing a file operation with respect to a virtual storage volume having a file size that is independent of a file size limitation of a computer system. A symbolic link, which is located in a directory associated with the virtual storage volume, is read. The symbolic link points to an allocated storage area, which had been randomly selected from among a plurality of available storage areas. The file operation is then performed with respect to the allocated storage area.
- A more complete understanding of the present invention and its advantages will be readily apparent from the following Detailed Description of the Preferred Embodiments taken in conjunction with the accompanying drawings. Throughout the accompanying drawings, like parts are designated by like reference numbers and in which:
- FIG. 1 is a schematic illustration of a computer network in accordance with the present invention;
- FIG. 2 is a flow diagram illustrating a method for creating a virtual storage volume in accordance with the present invention;
- FIG. 3 is a block diagram illustrating a user's directory and showing symbolic links from the user's directory to a plurality of file systems in accordance with the present invention; and
- FIG. 4 is a flow diagram illustrating a method for performing a file operation with respect to a virtual storage volume in accordance with the present invention.
- FIG. 1 schematically illustrates a hardware environment of an embodiment of the present invention. A
first communications network 110 provides an electronic communications medium that connects a user at aclient computer 120 to first andsecond server computers second server computers second storage arrays - The
client computer 120 is also connected to athird server computer 150 via asecond communications network 160. Thethird server computer 150 runs a third operating system having an associated file system for interfacing to athird storage array 155. Each of the first, second, andthird storage arrays - The
client computer 120 can be a personal computer or a workstation. Theserver computers client computer 120 and theserver computers second communications networks client computer 120 and theserver computers - A virtual storage volume can be created in accordance with the present invention, for example, by a user at the
client computer 120 creating a file in the user's home directory, or loading data from a tape or other storage device to a file in the user's home directory. Conventionally, files located in the user's home directory are conventionally subject to a number of constraints: the maximum file size, the maximum file-system size, and the size of the user's local hard disk drive. The present invention advantageously allows the user to disregard these conventional constraints and to treat the user's home directory as a substantially unlimited data storage area. Thus, the user can process and load data by manipulating what appears to the user to be a single file within the user's home directory structure. - FIG. 2 is a flow diagram illustrating the operation of a computer, in creating a virtual storage volume in accordance with the present invention. The virtual storage volume can be created in a directory associated with a particular user, and can be created by the computer as a result of the user creating a file or writing to a file in the user's home directory. The method of creating the virtual storage volume can be implemented in a computer program, or alternatively, can be implemented as a subroutine library, which is called by a computer program and integrated into the computer program.
- The process starts at
step 200. Instep 210, the computer receives a list of available storage areas which can be located in various file systems. The list can indicate only those storage areas that a particular user is permitted to access, or can indicate at least a portion of the storage areas that are connected to and accessible by the computer via one or more networks, such as the first andsecond communications networks step 215, the computer randomly selects a storage area from among the available storage areas listed. The computer can also randomly select a storage area without employing the list of available storage areas. In this case, the computer can select a storage area from among the storage areas to which it is connected or has access. - Then, in
step 220, the computer determines whether the selected storage area contains at least a minimum amount of free (e.g., unused or unallocated) space. The amount of free space required can be determined by a system administrator based on the total amount of storage space available, or can be based on other considerations. For example, the use of about a 250 megabyte block of space can result in more efficient utilization of available storage space, while the use of about a 2 gigabyte block can result in more efficient data transfer. The system administrator can set the required amount of free space via a parameter (e.g., a block size parameter) associated with the virtual storage volume. The computer can then look for an amount of free space in the selected storage area based on information representing the block size, which is stored in the parameter. The system administrator can also adjust the size of the block (e.g., based on currently available resources, such as the amount of available hard drive space) by editing the data stored in the block size parameter. - If the selected storage area has at least the required amount of free space available (Yes in step220), the process proceeds to step 225. If the selected storage area does not have at least the predetermined amount available (No in step 220), the process returns to step 215, wherein the computer randomly selects another storage area.
- By randomly selecting a storage area, rather than selecting the storage area having the most amount of free space, the process of the present invention advantageously distributes input/output activity among the
various server computers storage arrays - Then, in
step 225, the computer allocates an amount of free space in the selected storage area that is equal to the required amount. The amount of free space allocated can be determined, for example, by the block size parameter. Instep 230, the computer writes a symbolic link to the allocated storage area in a directory associated with the virtual storage volume. The symbolic link points to and enables access to the allocated storage area. - In
step 235, the computer writes the data to be stored in the virtual storage volume in the allocated storage area. As data is written into the allocated storage area, the amount of space remaining will diminish. As long as there is space available in the allocated storage area, additional data can be placed there. However, once there is no more space remaining in the allocated storage area (No in step 240), the process returns to step 215 wherein another storage area is randomly selected. Alternatively, the process can return to step 210 to receive a list of available storage areas, which list can be changed, for example, due to network conditions, since the last time that a storage area was selected. - The process thus allows a large number of allocated storage areas to be linked to a single virtual storage volume, such that the total amount of space available for a single file (i.e., the aggregate of the allocated storage areas) is greater than the maximum allowable file size as determined by the operating system. Further, as the allocated storage areas can be on different file systems, the total amount of space available can be greater than the maximum size of any particular file system.
- FIG. 3 is a block diagram illustrating the relationship between a virtual storage volume, which is located in a user's
directory 300, and the actual data, which are distributed between various ones of the first, second, andthird storage devices directory 300 contains a virtual storage volume named “UserFile”, which consists of a plurality of files, respectively named “UserFile.DAT”, “UserFile.001”, “UserFile.002”, . . . “UserFile.xxx”. Each of the files “UserFile.DAT”, “UserFile.001”, “UserFile.002”, . . . “UserFile.xxx” contains a symbolic link that points to a storage area on one of the first, second, orthird storage arrays - More specifically, in this embodiment, the file “UserFile.DAT” contains a
symbolic link 325 that points to a disk DISK6 in thefirst storage array 135. Likewise, the files “UserFile.001”, “UserFile.002”, and “UserFile.xxx” respectively containsymbolic links third storage array 155, DISK2 in thefirst storage array 135, and DISK3 in thesecond storage array 145. As the allocated storage areas are randomly selected from among the available storage areas, the files associated with the virtual storage volume are randomly interspersed among the first, second, andthird storage devices - In the embodiment of FIG. 3, the allocated storage areas and the directory containing the virtual storage volume are located in different file systems. In other embodiments, depending on the size of the user directory, at least a portion of the allocated storage areas can be located in the same file system as the directory containing the virtual storage volume.
- While the embodiment of FIG. 3 allows the user to see each of the plurality of files that make up the virtual storage volume, in other embodiments, the constituent files of the virtual storage volume can be hidden from the user. The user can thus only see one file in the user directory for each virtual storage volume.
- FIG. 4 is a flow diagram illustrating the operation of a computer, in performing a file operation with respect to a virtual storage volume in accordance with the present invention. The virtual storage volume can be located in a directory associated with a particular user, and can appear to the user to be a file in his home directory. The method of performing a file operation with the virtual storage volume can be implemented in a computer program, or alternatively, can be implemented as a subroutine library, which is called by a computer program and integrated into the computer program.
- The process starts at
step 400. Instep 410, the computer receives an instruction to perform a file operation with respect to the virtual storage volume. For example, the instruction can be to read or load into memory a virtual storage volume named “UserFile”. - In
step 420, the computer locates a symbolic link associated with the virtual storage volume. In the embodiment of FIG. 3, virtual volume “UserFile” includes a file named “UserFile.DAT” containing asymbolic link 325. Then, instep 430, the computer reads the symbolic link (e.g.,symbolic link 325 of the file “UserFile.DAT”), which points to an allocated storage area (e.g., DISK6 of the first data storage area 135). The file containing the symbolic link is preferably located in a directory associated with the virtual storage volume, such as the user's directory. - Then, in
step 440, the computer locates the allocated storage area pointed to by the symbolic link. The computer then performs the requested file operation (step 450) with respect to the allocated storage area pointed to by the symbolic link (e.g., DISK 6 of the first data storage area 135). Once the file operation has been performed with respect to a first allocated storage area of the virtual storage volume, the process returns to step 430 to read a second file associated with the virtual storage volume (e.g., “UserFile.001”), locate the next allocated storage area, and perform the file operation with respect to the allocated storage area. Thus, the process repeatssteps 430 through 440 until the file operation has been performed with respect to all the files of the virtual storage volume (No in step 460). The process then ends atstep 470. - Although the present invention has been fully described by way of examples and with reference to the accompanying drawings, it is to be understood that various changes and modifications will be apparent to those skilled in the art without departing from the spirit and scope of the invention. Therefore, unless such changes and modifications depart from the scope of the present invention, they should be construed as being included therein.
Claims (36)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/951,113 US7136981B2 (en) | 2001-09-13 | 2001-09-13 | Method and apparatus for creating a virtual storage volume with a file size independent of a file size limitation |
PCT/US2002/027219 WO2003023624A1 (en) | 2001-09-13 | 2002-08-26 | Extending a file size beyond a file size limitation |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/951,113 US7136981B2 (en) | 2001-09-13 | 2001-09-13 | Method and apparatus for creating a virtual storage volume with a file size independent of a file size limitation |
Publications (2)
Publication Number | Publication Date |
---|---|
US20030051117A1 true US20030051117A1 (en) | 2003-03-13 |
US7136981B2 US7136981B2 (en) | 2006-11-14 |
Family
ID=25491280
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/951,113 Active 2024-07-26 US7136981B2 (en) | 2001-09-13 | 2001-09-13 | Method and apparatus for creating a virtual storage volume with a file size independent of a file size limitation |
Country Status (2)
Country | Link |
---|---|
US (1) | US7136981B2 (en) |
WO (1) | WO2003023624A1 (en) |
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040002990A1 (en) * | 2002-06-28 | 2004-01-01 | Sander Michael R. | Shared drive that provides shared access to editable files in a database |
US20040093356A1 (en) * | 2002-11-12 | 2004-05-13 | Sharpe Edward J. | Process file systems having multiple personalities and methods therefor |
US20050021822A1 (en) * | 2003-06-23 | 2005-01-27 | Ludmila Cherkasova | System and method for modeling the memory state of a streaming media server |
US20060041619A1 (en) * | 2004-08-19 | 2006-02-23 | International Business Machines Corporation | System and method for an on-demand peer-to-peer storage virtualization infrastructure |
US20060129537A1 (en) * | 2004-11-12 | 2006-06-15 | Nec Corporation | Storage management system and method and program |
US20060179106A1 (en) * | 2005-02-10 | 2006-08-10 | Turner Bryan C | Distributed computing based on multiple nodes with determined capacity selectively joining resource groups having resource requirements |
US20080127181A1 (en) * | 2006-06-30 | 2008-05-29 | Microsoft Corporation | Use of fixed-function device as general-purpose platform through virtualization |
US7546354B1 (en) * | 2001-07-06 | 2009-06-09 | Emc Corporation | Dynamic network based storage with high availability |
US20090193110A1 (en) * | 2005-05-05 | 2009-07-30 | International Business Machines Corporation | Autonomic Storage Provisioning to Enhance Storage Virtualization Infrastructure Availability |
US20090287825A1 (en) * | 2005-02-10 | 2009-11-19 | Cisco Technology, Inc. | Distributed client services based on execution of service attributes and data attributes by multiple nodes in resource groups |
US20100017456A1 (en) * | 2004-08-19 | 2010-01-21 | Carl Phillip Gusler | System and Method for an On-Demand Peer-to-Peer Storage Virtualization Infrastructure |
US20100262803A1 (en) * | 2009-04-14 | 2010-10-14 | Fujitsu Limited | Storage system and storage control method |
US20120221611A1 (en) * | 2009-12-17 | 2012-08-30 | Hitachi, Ltd. | Management server, management method, and management program for virtual hard disk |
US20190065527A1 (en) * | 2017-08-30 | 2019-02-28 | Fujitsu Limited | Information processing device and information processing system |
Families Citing this family (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8996586B2 (en) | 2006-02-16 | 2015-03-31 | Callplex, Inc. | Virtual storage of portable media files |
US10303783B2 (en) * | 2006-02-16 | 2019-05-28 | Callplex, Inc. | Distributed virtual storage of portable media files |
US7890541B2 (en) * | 2006-02-17 | 2011-02-15 | International Business Machines Corporation | Partition by growth table space |
US8122452B2 (en) * | 2007-06-29 | 2012-02-21 | Oracle America, Inc. | Swap cap resource control for use in virtualization |
US20090210647A1 (en) | 2008-02-15 | 2009-08-20 | Madhusudanan Kandasamy | Method for dynamically resizing file systems |
US8140807B2 (en) * | 2008-02-15 | 2012-03-20 | International Business Machines Corporation | System and computer program product for dynamically resizing file systems |
US8209343B2 (en) * | 2008-10-06 | 2012-06-26 | Vmware, Inc. | Namespace mapping to central storage |
US8171278B2 (en) * | 2008-08-11 | 2012-05-01 | Vmware, Inc. | Booting a computer system from central storage |
US8392361B2 (en) * | 2008-08-11 | 2013-03-05 | Vmware, Inc. | Centralized management of virtual machines |
US8176105B2 (en) | 2008-12-08 | 2012-05-08 | International Business Machines Corporation | Automated file relocation |
KR101754998B1 (en) * | 2011-01-27 | 2017-07-06 | 삼성전자주식회사 | Multi Core System and Method for Processing Data |
CN104572242A (en) * | 2013-10-24 | 2015-04-29 | 华为技术有限公司 | Method and device for expanding disk space of virtual machine and virtual machine system |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6374336B1 (en) * | 1997-12-24 | 2002-04-16 | Avid Technology, Inc. | Computer system and process for transferring multiple high bandwidth streams of data between multiple storage units and multiple applications in a scalable and reliable manner |
US6745286B2 (en) * | 2001-01-29 | 2004-06-01 | Snap Appliance, Inc. | Interface architecture |
-
2001
- 2001-09-13 US US09/951,113 patent/US7136981B2/en active Active
-
2002
- 2002-08-26 WO PCT/US2002/027219 patent/WO2003023624A1/en not_active Application Discontinuation
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6374336B1 (en) * | 1997-12-24 | 2002-04-16 | Avid Technology, Inc. | Computer system and process for transferring multiple high bandwidth streams of data between multiple storage units and multiple applications in a scalable and reliable manner |
US6745286B2 (en) * | 2001-01-29 | 2004-06-01 | Snap Appliance, Inc. | Interface architecture |
US6754773B2 (en) * | 2001-01-29 | 2004-06-22 | Snap Appliance, Inc. | Data engine with metadata processor |
Cited By (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7546354B1 (en) * | 2001-07-06 | 2009-06-09 | Emc Corporation | Dynamic network based storage with high availability |
US20040002990A1 (en) * | 2002-06-28 | 2004-01-01 | Sander Michael R. | Shared drive that provides shared access to editable files in a database |
US7505971B2 (en) * | 2002-06-28 | 2009-03-17 | Hewlett-Packard Development Company, L.P. | Shared drive that provides shared access to editable files in a database |
US20040093356A1 (en) * | 2002-11-12 | 2004-05-13 | Sharpe Edward J. | Process file systems having multiple personalities and methods therefor |
US7107289B2 (en) * | 2002-11-12 | 2006-09-12 | Hewlett-Packard Development Company | Process file systems having multiple personalities and methods therefor |
US7310681B2 (en) * | 2003-06-23 | 2007-12-18 | Hewlett-Packard Development Company, L.P. | System and method for modeling the memory state of a streaming media server |
US20050021822A1 (en) * | 2003-06-23 | 2005-01-27 | Ludmila Cherkasova | System and method for modeling the memory state of a streaming media server |
US20100017456A1 (en) * | 2004-08-19 | 2010-01-21 | Carl Phillip Gusler | System and Method for an On-Demand Peer-to-Peer Storage Virtualization Infrastructure |
US20060041619A1 (en) * | 2004-08-19 | 2006-02-23 | International Business Machines Corporation | System and method for an on-demand peer-to-peer storage virtualization infrastructure |
US7499980B2 (en) * | 2004-08-19 | 2009-03-03 | International Business Machines Corporation | System and method for an on-demand peer-to-peer storage virtualization infrastructure |
US8307026B2 (en) * | 2004-08-19 | 2012-11-06 | International Business Machines Corporation | On-demand peer-to-peer storage virtualization infrastructure |
US20060129537A1 (en) * | 2004-11-12 | 2006-06-15 | Nec Corporation | Storage management system and method and program |
US7818287B2 (en) * | 2004-11-12 | 2010-10-19 | Nec Corporation | Storage management system and method and program |
US8051170B2 (en) * | 2005-02-10 | 2011-11-01 | Cisco Technology, Inc. | Distributed computing based on multiple nodes with determined capacity selectively joining resource groups having resource requirements |
US8239540B2 (en) | 2005-02-10 | 2012-08-07 | Cisco Technology, Inc. | Distributed computing based on multiple nodes with determined capacity selectively joining resource groups having resource requirements |
US8639816B2 (en) * | 2005-02-10 | 2014-01-28 | Cisco Technology, Inc. | Distributed computing based on multiple nodes with determined capacity selectively joining resource groups having resource requirements |
US20060179106A1 (en) * | 2005-02-10 | 2006-08-10 | Turner Bryan C | Distributed computing based on multiple nodes with determined capacity selectively joining resource groups having resource requirements |
US7991835B2 (en) | 2005-02-10 | 2011-08-02 | Cisco Technology, Inc. | Distributed client services based on execution of service attributes and data attributes by multiple nodes in resource groups |
US20090287825A1 (en) * | 2005-02-10 | 2009-11-19 | Cisco Technology, Inc. | Distributed client services based on execution of service attributes and data attributes by multiple nodes in resource groups |
US8195742B2 (en) | 2005-02-10 | 2012-06-05 | Cisco Technology, Inc. | Distributed client services based on execution of service attributes and data attributes by multiple nodes in resource groups |
US20090193110A1 (en) * | 2005-05-05 | 2009-07-30 | International Business Machines Corporation | Autonomic Storage Provisioning to Enhance Storage Virtualization Infrastructure Availability |
US20080127181A1 (en) * | 2006-06-30 | 2008-05-29 | Microsoft Corporation | Use of fixed-function device as general-purpose platform through virtualization |
US20100262803A1 (en) * | 2009-04-14 | 2010-10-14 | Fujitsu Limited | Storage system and storage control method |
US8856485B2 (en) * | 2009-04-14 | 2014-10-07 | Fujitsu Limited | Storage system and storage control method |
US20120221611A1 (en) * | 2009-12-17 | 2012-08-30 | Hitachi, Ltd. | Management server, management method, and management program for virtual hard disk |
US8825725B2 (en) * | 2009-12-17 | 2014-09-02 | Hitachi, Ltd. | Management server, management method, and management program for execution on a management server |
US20190065527A1 (en) * | 2017-08-30 | 2019-02-28 | Fujitsu Limited | Information processing device and information processing system |
US11055263B2 (en) * | 2017-08-30 | 2021-07-06 | Fujitsu Limited | Information processing device and information processing system for synchronizing data between storage devices |
Also Published As
Publication number | Publication date |
---|---|
WO2003023624A1 (en) | 2003-03-20 |
US7136981B2 (en) | 2006-11-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7136981B2 (en) | Method and apparatus for creating a virtual storage volume with a file size independent of a file size limitation | |
US7082497B2 (en) | System and method for managing a moveable media library with library partitions | |
US6609187B1 (en) | Method and apparatus for supporting resizing of file system partitions | |
US5790886A (en) | Method and system for automated data storage system space allocation utilizing prioritized data set parameters | |
US6665772B1 (en) | Data storage method and device for storing streaming and non-streaming data in common memory space | |
US7392365B2 (en) | Dynamically changeable virtual mapping scheme | |
KR100951107B1 (en) | Method for managing files for optimal performance | |
US6834324B1 (en) | System and method for virtual tape volumes | |
US5802599A (en) | System and method for allocating storage in a fragmented storage space | |
US6094605A (en) | Virtual automated cartridge system | |
US5956745A (en) | System and method for automatically resizing a disk drive volume | |
USRE43437E1 (en) | Storage volume handling system which utilizes disk images | |
US8621174B2 (en) | Relocating data within a file system | |
US6421767B1 (en) | Method and apparatus for managing a storage system using snapshot copy operations with snap groups | |
US6816957B1 (en) | Management of virtual tape volumes using data page atomic units | |
US20070078914A1 (en) | Method, apparatus and program storage device for providing a centralized policy based preallocation in a distributed file system | |
JPH06318168A (en) | Hierarchical data storage management device, method and its network | |
JP2005501317A (en) | External data storage management system and method | |
US6591356B2 (en) | Cluster buster | |
US6804761B1 (en) | Memory allocation system and method | |
JPH09114710A (en) | Method and apparatus for segmentation at inside of partitionin disk file storage device | |
US20200310962A1 (en) | Ordering data updates for improving garbage collection being performed while performing the set of data updates | |
US7765378B1 (en) | Utilization of memory storage | |
US7424574B1 (en) | Method and apparatus for dynamic striping | |
US7089392B2 (en) | Method, system, and program for allocating storage units to a data set |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: CONOCO INC., TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BURCH, CHARLES CARROLL JR.;MENGER, WILLIAM MEREDITH;HEFLIN, ROGER A.;AND OTHERS;REEL/FRAME:012474/0736;SIGNING DATES FROM 20010914 TO 20011004 |
|
AS | Assignment |
Owner name: CONOCOPHILLIPS COMPANY, TEXAS Free format text: MERGER;ASSIGNOR:CONOCO INC.;REEL/FRAME:015696/0976 Effective date: 20021212 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553) Year of fee payment: 12 |