US20210117332A1 - Minimizing data written to disk and enabling directory change notifications in multi-volume filter environments - Google Patents
Minimizing data written to disk and enabling directory change notifications in multi-volume filter environments Download PDFInfo
- Publication number
- US20210117332A1 US20210117332A1 US16/656,900 US201916656900A US2021117332A1 US 20210117332 A1 US20210117332 A1 US 20210117332A1 US 201916656900 A US201916656900 A US 201916656900A US 2021117332 A1 US2021117332 A1 US 2021117332A1
- Authority
- US
- United States
- Prior art keywords
- overlay
- request
- dcn
- file
- minifilter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0875—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches with dedicated cache, e.g. instruction or stack
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0617—Improving the reliability of storage systems in relation to availability
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
- G06F3/0649—Lifecycle management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1016—Performance improvement
Definitions
- Operating systems such as Windows
- Windows 10 provide functionality for preventing the content of a storage medium from being changed.
- Windows 10 provides the Unified Write Filter (UWF) which can be used to redirect all writes that target the operating system image to a RAM or disk cache called an overlay.
- UWF Unified Write Filter
- This overlay stores changes made to the operating system at runtime but is removed when the device is restarted thereby restoring the device to its original state.
- a volume should be construed as a logical unit that represents an area of persistent storage to the file system.
- a volume can correspond to a single physical storage device, such as a hard disk, but can also correspond to a single partition on a physical storage device with multiple partitions.
- a volume can also span across multiple physical storage devices.
- a “protected volume” is a volume that a write filter protects. In other words, when a write filter is employed, the write filter will protect the content (e.g., files, directories, registry entries, etc.) of the volume from being changed.
- FIG. 1 illustrates a simplified Windows-based I/O system architecture 100 when a write filter 140 is employed.
- Write filter 140 may typically be the Windows UWF but may equally be another third-party write filter.
- Architecture 100 includes a disk 105 which may oftentimes be a solid-state drive (SSD) but may represent a physical hard disk, an internal USB device, an external SATA device or any other physical storage medium (or media) that may be compatible with a write filter.
- SSD solid-state drive
- disk 105 includes an area of persistent storage that is represented by a volume.
- Disk stack 110 represents the various drivers that enable disk 105 to be accessed and are not critical to an understanding of the present invention.
- a “protected I/O stack” that includes volume stack 120 a and file system stack 130 a
- an “unprotected I/O stack that includes volume stack 120 b and file system stack 130 b
- the protected I/O stack can be viewed as the primary I/O stack and handles I/O requests that do not modify content on the protected volume (e.g., read requests).
- the unprotected I/O stack can be viewed as a pseudo I/O stack and handles I/O requests that modify content on the protected volume (e.g., write requests).
- a write request should be construed as encompassing I/O requests that modify existing content and I/O requests that create new content. The limited scenarios where such write requests would occur to the protected volume are described below.
- Write filter 140 is loaded above file system stack 130 a on the protected I/O stack so that it can intercept all requests to access the content of the protected volume.
- write filter 140 receives a request to read content from the protected volume, and assuming the content was not previously modified and stored in overlay 150 , it can send the request down the protected I/O stack so that the content will be read from disk 105 in a typical manner.
- write filter 140 receives a request to write content on the protected volume, it redirects the request so that the write is made in overlay 150 (i.e., the new/modified content is stored in overlay 150 while the content stored on disk 105 remains unchanged).
- write filter 140 will ensure that any subsequent request to access the modified content will be completed using the modified content stored in overlay 150 .
- write filter 140 will allow content on the protected volume to be modified.
- an administrator can register an exclusion with write filter 140 for particular content on the protected volume (e.g., particular files or folders).
- write filter 140 will not redirect any request that would modify the excluded content to overlay 150 , but instead will cause the request to be passed down the unprotected I/O stack thereby causing the excluded content to be modified on the protected volume.
- write filter 140 enables an administrator to cause modified content stored in overlay 150 to be committed to disk 105 . In such cases, when write filter 140 detects the write request pertaining to the commit, it will cause it to be passed down the unprotected I/O stack. Accordingly, even when write filter 140 is employed, it is still possible to modify some content on the protected volume.
- U.S. Pat. Publ. No. 2018/0217940 (the '940 Publication) describes how an overlay-managing write filter (or an “overlay optimizer”) can be employed to optimize the performance of write filter 140 .
- an overlay optimizer which can be loaded above write filter 140 , can accomplish this by moving content that is stored in overlay 150 to a separate overlay cache.
- This separate overlay cache may be implemented as a directory on the protected volume that is registered as an exclusion or as a separate partition of disk 105 (i.e., an unprotected volume). In either case, write filter 140 will not block writes to the overlay cache.
- the overlay optimizer can redirect subsequent requests to access the modified content.
- FIGS. 2A and 2B generally illustrate why this is.
- FIGS. 2A and 2B show portions of architecture 100 with the addition of overlay optimizer 200 which can be in the form of a file system minifilter driver that is loaded above write filter 140 .
- overlay optimizer 200 can be in the form of a file system minifilter driver that is loaded above write filter 140 .
- a file, File A is stored on the protected volume on disk 105 and has a size of 100 MB. It is also assumed that, during use of the computing device, a request to modify file A was made and that write filter 140 redirected this request to overlay 150 so that File A on the protected volume remains unmodified.
- write filter 140 when redirecting a request to modify a file on the protected volume, write filter 140 does not store the entire file in overlay 150 but only stores the modified portions of the file (e.g., each page that is modified). In the depicted example, it is assumed that the request to modify File A only modified a single 4 KB page of the file. Accordingly, FIG. 2A shows that overlay 150 stores only the single 4 KB modified page. Additionally, even though write filter 140 only stores the modified portions (e.g., pages) in overlay 150 , it does not provide any way to detect which modified portions are stored in overlay 150 . To the contrary, write filter 140 only provides functionality for determining which files are cached in overlay 150 (or more correctly, which files have modified portions stored in overlay 150 ).
- the UWF_Overlay.GetOverlayFiles method can be called to obtain a list of files that are cached in overlay 150 , but the UWF does not provide a way to determine which pages of these files are actually stored in overlay 150 .
- overlay optimizer 200 can detect that overlay 150 is becoming full as is represented by step 1. Upon determining that overlay 150 is becoming full, overlay optimizer 200 can commence moving files from overlay 150 to the overlay cache (which is assumed to be on disk 105 ). For the reasons given above, overlay optimizer 200 can only detect which files have modified portions stored in overlay 150 but cannot identify only the modified portions. Accordingly, to move File A to the overlay cache, overlay optimizer 200 will issue a request to read File A as represented in step 2.
- step 3 shown in FIG. 2B write filter 140 will complete the read request by providing to overlay optimizer 200 the entire modified File A (i.e., the 100 MB version on the protected volume with the 4 KB modified page from overlay 150 ).
- overlay optimizer 200 will issue a request to write the 100 MB modified File A to the overlay cache. Assuming the overlay cache is implemented as a folder on the protected volume that is registered as an exclusion, write filter 140 will redirect this write down the unprotected I/O stack rather than redirecting it to overlay 150 .
- overlay optimizer 200 will request deletion of File A which will cause write filter 140 to discard the modified page of File A from overlay 150 (but will not alter the unmodified File A stored on the protected volume).
- overlay optimizer 200 has no way of identifying that only a single page of File A exists in overlay 150 , it must copy the entire File A to the overlay cache as part of freeing up overlay 150 .
- overlay optimizer 200 may typically move many files in this manner to free up overlay 150 , a large amount of data will be written to disk 105 which can create a number of problems.
- SSDs which are typically comprised of NAND flash, wear out over time. Therefore, SSD manufacturers typically provide a warranty up to a specified total amount of data written to the SSD which is usually defined as terabytes written or TBW. With the use of overlay optimizer 200 , it is possible that the amount of data written to disk 105 over the warranty period may exceed the TBW.
- overlay optimizer 200 may be configured to forego moving files from overlay 150 after a certain amount of data has been written each day. Although this may preserve the life of the SSD, it increases the likelihood of a forced reboot as overlay 150 becomes full. Even if disk 105 is not an SSD, the large amount of data that overlay optimizer 200 may write to disk 105 when freeing up overlay 150 can still degrade overall system performance. In short, it would be more efficient if overlay optimizer 200 could move only the modified portions of files from overlay 150 rather than copying the entire file.
- Certain applications such as Windows Explorer, custom shells, file system watchers, antivirus software, source code editors, etc., are configured to monitor the contents of a directory. Oftentimes, these applications will register for directory change notifications (DCNs) rather than enumerating the directory on demand.
- DCNs directory change notifications
- an application can employ the FindFirstChangeNotification function to register to be notified when a specified change occurs to/within a directory. These changes may include any change to the name or size of a file in the directory, a change to the name or attributes of the directory, a change in the security of the directory, etc. Once registered, the application will be notified whenever any of the specified changes occur and can take whatever action is appropriate.
- Waiting for a DCN is similar to having a read operation pending against a directory and, if necessary, its subdirectories. When something changes within the directory being watched, the read operation is completed. Accordingly, the DCN functionality is primarily implemented within the file system stack for the volume containing the directory (or more specifically, by the file system driver in the file system stack).
- DCNs will not work properly.
- the DCN request when an application sends a DCN request for a particular directory on the protected volume, the DCN request will be provided to file system stack 130 a only (e.g., similar to pending a read request with the file system driver in file system stack 130 a ). However, the DCN request will not be provided to file system stack 130 b . As a result, the application will only receive a DCN when an I/O request that passes through file system stack 130 a causes the specified change(s) to the directory.
- Windows Explorer is configured to use DCNs to be immediately notified when a folder or file is created in a particular directory. This ensures that the newly created folder or file will be immediately displayed within the Explorer window.
- a user may open Windows Explorer to view the contents of a particular directory (or folder) and right click to select the option to create a new folder. If the directory whose contents is being displayed is part of the protected volume, the request to create the new folder will be handled through file system stack 130 b which is unaware of Windows Explorer's DCN. Windows Explorer will therefore not receive a DCN even though a new folder was created in the directory for which it registered for DCNs.
- deduplication, mirroring and caching solutions typically employ a filter that is loaded on the primary I/O stack, similar to write filter 140 and the protected I/O stack in FIG. 1 , and one or more shadow volumes where each shadow volume would have a corresponding I/O stack similar to the unprotected I/O stack shown in FIG. 1 . Therefore, as a practical matter, applications/services that rely on DCNs only work properly when a directory is not spread across more than one volume.
- the present invention extends to methods, systems and computer program products for minimizing the amount of data that is written to disk when an overlay optimizer is used in conjunction with a write filter to prevent the overlay from becoming full.
- a first instance of an overlay optimizer minifilter can be loaded above the write filter on the protected I/O stack and a second instance of the overlay optimizer filter can be loaded on the unprotected I/O stack.
- the second instance of the overlay optimizer minifilter can be used to intercept writes that were initiated by the overlay optimizer's request to commit files cached in the write filter's overlay to thereby extract only the modified portions of the files that are actually stored in the overlay.
- the second instance of the overlay optimizer minifilter can then write these modified portions of the files, as opposed to the entire files, in the overlay cache. In this way, a minimal amount of data is written to disk when freeing up the overlay.
- the present invention also extends to methods, systems and computer program products for enabling directory change notifications when a write filter is employed as well as in other multi-volume filter environments.
- a directory change notification When a directory change notification is received by the first instance of the overlay optimizer minifilter, it can create and issue duplicate DCN requests to each I/O stack while pending the original DCN request.
- a DCN map can be used to associate the duplicate DCN requests with the pended DCN request so that contents of a completed DCN request can be copied to the DCN request and then the DCN request can be completed. In this way, any changes made to the directory, regardless of which volume the change may have been made on, will trigger a DCN.
- the present invention is implemented—by an overlay optimizer that includes an overlay optimizer service, a first overlay optimizer minifilter that is loaded on a protected I/O stack and a second overlay optimizer minifilter that is loaded on an unprotected I/O stack—as a method for minimizing an amount of data that is written to disk when the overlay optimizer service reduces consumption of the write filter's overlay.
- the overlay optimizer service identifies a first file that is cached in the overlay and sends a first notification to the second overlay optimizer minifilter.
- the first notification identifies the first file.
- the overlay optimizer service then submits a request to commit the first file.
- the second overlay optimizer minifilter intercepts a write request that is being passed down the unprotected I/O stack. Based on the first notification, the second overlay optimizer minifilter determines that the write request pertains to the request to commit the first file. In response, the second overlay optimizer minifilter reparses the write request to an overlay cache.
- the present invention is implemented by a minifilter in a multi-volume filter environment as a method for enabling directory change notifications when a directory spans the multiple volumes.
- the minifilter receives a directory change notification (DCN) request that targets a first directory.
- DCN directory change notification
- the minifilter creates a plurality of duplicate DCN requests corresponding to a plurality of I/O stacks and sends each of the plurality of duplicate DCN requests down the corresponding I/O stack.
- the minifilter also pends the DCN request.
- the minifilter copies content of the completed duplicate DCN request to the DCN request and then completes the DCN request.
- FIG. 1 illustrates a Windows-based I/O system in which a write filter is employed to redirect writes targeting a protected volume to an overlay;
- FIGS. 2A and 2B illustrate why a large amount of data may be written to disk when an overlay optimizer is employed to prevent the write filter's overlay from becoming full;
- FIG. 3 illustrates an example I/O system architecture that may exist on a client device when the overlay optimizer of the present invention is implemented
- FIGS. 4A-4F illustrate functionality that the overlay optimizer can perform to identify the changed portions of files that are actually stored in the write filter's overlay to avoid having to copy the entire file to an overlay cache;
- FIGS. 5A-5C provide flow diagrams representing functionality that the overlay optimizer can perform to identify the changed portions of files that are actually stored in the write filter's overlay to avoid having to copy the entire file to an overlay cache;
- FIG. 6 illustrates an example multi-volume filter architecture in which embodiments of the present invention may be implemented
- FIGS. 7A-7F illustrate functionality that a minifilter can perform to enable directory change notifications when a directory is spread across multiple volumes
- FIGS. 8A and 8B provide flow diagrams representing functionality that a minifilter can perform to enable directory change notifications when a directory is spread across multiple volumes.
- client device should be construed as any user computing device that is capable of executing a write filter.
- a client device would therefore include desktops, laptops, tablets, thin clients, smart phones, etc.
- write filter should be construed as a software component that is employed to protect the content of a storage device from being modified by redirecting I/O requests that target the content so that the modifications are stored elsewhere. Embodiments of the present invention will be described in the context of a write filter provided with the Windows operating system (e.g., the UWF).
- Windows operating system e.g., the UWF
- embodiments of the present invention can equally be implemented on a client device that runs a non-Windows operating system as well as on a client device that runs Windows but employs another third party (i.e., non-Windows) write filter.
- minifilter is used in a Windows context, its use is exemplary and should not be construed as limiting the invention to Windows-based implementations.
- overlay optimizer minifilter should therefore be construed generally as a software component that can perform at least some of the functionality described herein.
- FIG. 3 illustrates an example I/O system architecture 300 that may exist on a client device when the present invention is implemented.
- architecture 300 includes an overlay optimizer 310 which comprises: a first instance of an overlay optimizer minifilter 311 which is loaded on the protected I/O stack above write filter 140 ; a second instance of an overlay optimizer minifilter 312 which is loaded on the unprotected I/O stack; an overlay cache 313 implemented on disk 105 ; and an overlay optimizer service 314 which may be a user-mode component that executes with administrator privileges.
- disk 105 need not represent a single physical storage device and overlay cache 313 may or may not be part of the protected volume.
- Architecture 300 can be employed to enable overlay optimizer 310 to identify and move only the changed portions of files that are stored in overlay 150 to overlay cache 313 as opposed to copying the entire files to overlay cache 313 .
- Overlay optimizer 310 can be configured to monitor the consumption of overlay 150 (e.g., by using overlay optimizer service 314 to periodically read the OverlayConsumption or AvailableSpace members of the UWF_Overlay class or to watch for the UWF_OVERLAY_REACHED_WARNING_LEVEL and/or UWF_OVERLAY_REACHED_CRITICAL_LEVEL events) to thereby detect when overlay 150 is becoming full.
- overlay optimizer 310 could be configured to detect when consumption of overlay 150 has exceeded 60%.
- overlay optimizer 310 When overlay optimizer 310 detects that the consumption of overlay 150 has exceeded some threshold (i.e., that overlay 150 is becoming full), it can commence moving files from overlay 150 to overlay cache 313 to thereby prevent a reboot of the computing device (or severely degraded performance) which would otherwise occur once overlay 150 is full. In contrast to what is described in the background, overlay optimizer 310 can perform functionality for identifying the changed portions of files that are stored in overlay 150 so that only these changed portions will be written to overlay cache 313 .
- FIGS. 4A-4F illustrate the functionality that overlay optimizer 310 performs to identify the changed portions of files that are actually stored in overlay 150 to avoid having to copy the entire file to overlay cache 313 .
- overlay optimizer service 314 can request the files that are cached in overlay 150 .
- overlay optimizer service 314 would submit this request upon detecting that overlay 150 is becoming full.
- overlay optimizer 314 could equally submit such requests on a periodic basis with or without regard to the consumption of overlay 150 .
- the manner in which overlay optimizer service 314 identifies which files are cached in overlay 150 is not essential to the present invention.
- overlay optimizer service 314 could employ the Windows Management Instrumentation (WMI) GetOverlayFiles( ) function to retrieve a listing of files that are cached in overlay 150 .
- WMI Windows Management Instrumentation
- overlay optimizer service 314 receives a list of the files cached in overlay 150 in step 1b.
- FIG. 4A assumes that overlay 150 stores changed pages from File A, File B and File C among other files, and therefore, the list of files received in step 1b identifies these three files. Notably, this list does not provide any information about which portions of the three files are actually stored in overlay 150 but merely provides a filename (or path) to each file.
- overlay optimizer service 314 can identify a particular file in the list in step 2a and send the filename of the file to overlay optimizer service 314 in step 2b. These steps can be repeated for each file in the list (or at least until the overlay's consumption has been reduced to a satisfactory level). As shown, this filename can be in the form of a complete file path to the file (which for File A is assumed to be C: ⁇ Dir1 ⁇ File A.txt). In this way, overlay optimizer minifilter 312 becomes aware of a file that should be moved from overlay 150 to overlay cache 313 .
- overlay optimizer minifilter 312 can request a file ID for the file as represented as step 3a in FIG. 4C .
- overlay optimizer minifilter 312 can employ the filename of File A to obtain a pointer to the file object for File A and can then call the FltQueryInformationFile function and provide as inputs the pointer to the file object for File A and FileInternalInformation as the FILE INFORMATION CLASS value.
- overlay optimizer minifilter 312 will receive back the file ID (or file reference number) as the value of the IndexNumber member of the FILE INTERNAL INFORMATION structure.
- overlay optimizer minifilter 312 stores the file ID. Steps 3a and 3b can be performed for each filename that overlay optimizer minifilter 312 receives from overlay optimizer service 314 in step 2b. As a result, overlay optimizer minifilter 312 will have created a listing of file IDs for files that are to be moved from overlay 150 to overlay cache 313 . As shown, in conjunction with obtaining the file ID of a file, overlay optimizer minifilter 312 can also create a shadow file in overlay cache 313 for the file and maintain a mapping between the file ID and the filename of the corresponding shadow file.
- overlay optimizer service 314 can request that the files be committed as represented as step 4a in FIG. 4D . It is noted that, in normal operation, a request to commit a file from overlay 150 to the protected volume would cause the changed portions of the file to be written to the version of the file on the protected volume thereby permanently modifying the file. However, as will become apparent below, overlay optimizer minifilter 312 prevents this normal handling of a commit from being carried out for any file that overlay optimizer 310 is moving to overlay cache 313 .
- write filter 140 In response to overlay optimizer service 314 's request to commit a file, write filter 140 will read the changed portions of the file in step 4b and issue a request to write the change portions in step 4c. As described in the background, this write request will be directed to the unprotected I/O stack. Given that overlay optimizer minifilter 312 is loaded on the unprotected I/O stack, it will intercept this write request in step 4d. Steps 4a-4d can be performed for each file to be moved.
- overlay optimizer minifilter 312 when overlay optimizer minifilter 312 intercepts a write request, it does not yet know whether the write request is a result of overlay optimizer service 314 's request to commit a file or a result of some other function (e.g., a write to an excluded file/directory on the protected volume, a request to commit a file made by another service, etc.). Accordingly, whenever overlay optimizer minifilter 312 receives a write request, it can perform step 5a to retrieve the context of the file to which the write request pertains (e.g., by calling the FltGetFileContext function).
- some other function e.g., a write to an excluded file/directory on the protected volume, a request to commit a file made by another service, etc.
- overlay optimizer minifilter 312 will receive a file context that includes a file ID of 0xABCD . . . in response to its call to FltGetFileContext. Then, in step 5b, overlay optimizer minifilter 312 can compare the file ID retrieved in step 5a to its list of file IDs that it created in step 3b to thereby determine whether the current write request pertains to a file that overlay optimizer service 314 indicated that it would be committing. In the depicted example, overlay optimizer minifilter 312 will determine that the file ID obtained in step 5a (0xABCD . . . ) matches the file ID of File A (0xABCD . . . ) that it had previously stored in its list.
- overlay optimizer minifilter 312 may retrieve the context of the file and verify that the context includes the file ID of the file. If not, overlay optimizer minifilter 312 could perform steps similar to steps 3a and 3 b to retrieve the file ID and store it in the file's context. In this way, even if the creator of the write request did not create a file context that included the file ID, the file ID will be stored in the file context before step 5b is performed.
- overlay optimizer minifilter 312 Using the file ID to identify write requests that are a result of overlay optimizer service 314 's request to commit a file ensures that overlay optimizer minifilter 312 will detect such write requests regardless of how they are performed. For example, this technique will detect such writes even when the file is committed to disk via multiple handles or threads or when the file is opened via a short name or relative path.
- Step 6 in FIG. 4F represents how overlay optimizer minifilter 312 may reparse (or redirect) a write request when it determines that the write request is a result of overlay optimizer service 314 's request to commit a file (i.e., when overlay optimizer minifilter 312 determines in step 5b that the file ID of the file to which the write request pertains matches a file ID contained in its list). Because overlay optimizer minifilter 312 has mapped the file ID to the filename of the corresponding shadow file in overlay cache 313 , it can replace the filename that is currently associated with the write request with the filename of the corresponding shadow file.
- overlay optimizer minifilter 312 could call the IoReplaceFileObjectName function with inputs of the pointer to the file object for file A (which would be provided in the write request) and the filename of the corresponding shadow file. As represented in FIG. 4F , this will cause the write request to target the corresponding shadow file in overlay cache 313 rather than the actual file stored on the protected volume. In this way, the changed portions of the file will be stored in overlay cache 313 but will not be committed to the actual unmodified file that exists on the protected volume.
- overlay optimizer minifilter 312 determines in step 5b that the file ID of the file to which the write request pertains does not match a file ID contained in its list, it can simply allow the write request to be passed down the stack for normal handling. In short, overlay optimizer minifilter 312 will pass each write request down towards volume stack 120 b but selectively modifies any write request that are determined to be a result of overlay optimizer service 314 's request to commit a file cached in overlay 150 .
- write filter 140 When any write request that resulted from a request to commit a file is completed successfully (whether or not the commit request was issued by overlay optimizer service 314 ), write filter 140 will complete the commit by discarding the changed portions of the file from overlay 150 . Accordingly, for a commit initiated by overlay optimizer service 314 , the end result is that only the changed portions of the file are copied to overlay cache 313 (but are not actually committed to the corresponding file on the protected volume) and then the changed portions of the file are discarded from overlay 150 thereby reducing its consumption.
- overlay optimizer 310 can update a hash table or “map” that reflects that the changed portions to the file exist in overlay cache 313 .
- the '940 Publication describes how an overlay optimizer can employ a map for this purpose and therefore the creation, update and use of a map will not be described in detail.
- overlay optimizer minifilter 311 can employ the map when handling an I/O request to determine whether content from overlay cache 313 needs to be accessed to complete the I/O request properly (e.g., by merging a file's changed pages stored in and retrieved from overlay cache 313 with the file's unmodified pages retrieved from the protected volume.
- FIGS. 5A-5C provide flow diagrams corresponding to the above-described functionality.
- FIG. 5A represents functionality that overlay optimizer service 314 performs to determine when to initiate the process of freeing up overlay 150 , to notify overlay optimizer minifilter 312 of the files that will be moved from overlay 150 and to initiate the process of moving the files.
- FIG. 5B represents functionality that overlay optimizer minifilter 312 performs when it receives a notification from overlay optimizer service 314 that a file will be moved from overlay 150 .
- FIG. 5A represents functionality that overlay optimizer service 314 performs to determine when to initiate the process of freeing up overlay 150 , to notify overlay optimizer minifilter 312 of the files that will be moved from overlay 150 and to initiate the process of moving the files.
- FIG. 5B represents functionality that overlay optimizer minifilter 312 performs when it receives a notification from overlay optimizer service 314 that a file will be moved from overlay 150 .
- FIG. 5A represents functionality that overlay optimizer service 314 performs to determine when to initiate the
- 5C represents functionality that overlay optimizer 313 performs when it receives an I/O request to determine whether the I/O request pertains to a commit initiated by overlay optimizer service 314 , and if so, to rep arse the I/O request to a corresponding shadow file in overlay cache 313 .
- overlay optimizer 310 may also be configured to enable directory change notifications (DCNs) when the content of a directory is spread across more than one volume.
- DCNs directory change notifications
- the protected volume and overlay cache 313 may each store files (or portions of files) that exist (or are represented as existing) in the same directory.
- the request will only be provided to file system stack 130 a , and therefore, any changes to the directory that are made through file system stack 130 b will not trigger a DCN.
- FIG. 6 illustrates a multi-volume filter architecture 600 that is substantially the same as architecture 300 but more closely represents the architecture that would exist when a deduplication, mirroring or caching solution is implemented.
- a filter is loaded on the primary I/O stack in architecture 600 .
- This filter is labeled and will be referred to as reparse filter 140 a to indicate that it could represent write filter 140 or a file system filter of a deduplication, mirroring, caching or similar solution.
- a supporting I/O stack (which includes file system stack 130 b - 1 and volume stack 120 b - 1 ) will also exist in architecture 600 and corresponds with shadow volume 1 .
- Architecture 600 may also include zero or more additional supporting I/O stacks which would include a file system stack 130 b - n and a volume stack 120 b - n and would correspond with shadow volume n (where n is intended to represent any reasonable integer including 0).
- a minifilter 311 a can be loaded on the primary I/O stack above reparse filter 140 a and a minifilter ( 312 a - 1 through 312 a - n ) may be loaded on each supporting I/O stack above the file system driver.
- These minifilters may or may not be the same as the overlay optimizer minifilters described above.
- FIGS. 7A-7F illustrate how a minifilter loaded on the primary I/O stack in a multi-volume filter environment can enable DCNs when a directory is spread across multiple volumes.
- FIGS. 7A-7F will be based on architecture 300 and will show overlay optimizer minifilter 311 performing the functionality, but the depicted functionality can equally be performed in architecture 600 using minifilter 311 a.
- the protected volume includes a directory, C: ⁇ Dir1, and that overlay optimizer 310 maintains a corresponding directory, D: ⁇ Dir1, in overlay cache 313 where modified files in Dir1 have been moved.
- a file named File 2.txt is shown as being stored in overlay cache 313 .
- File 2.txt could have originally been stored in overlay 150 (e.g., when File 2.txt was created or modified) but moved to overlay cache 313 .
- overlay optimizer 310 can maintain a map that defines that file 2.txt is actually part of C: ⁇ Dir1 (at least from the user's perspective).
- a DCN should be triggered whenever any changes are made to this directory on the protected volume as well as whenever any changes are made to the corresponding directory in overlay cache 313 .
- step 1a an application, which is assumed to be Windows Explorer in this example, submits a request to register for a DCN for C: ⁇ Dir1.
- this may be accomplished by calling the FindFirstChangeNotificationA function and specifying the full directory path as input.
- the I/O manager of the operating system can generate an I/O request (or DCN request) 700 and send it to the file system driver in file system stack 130 a .
- DCN request 700 may be in the form of an IRP_MN_NOTIFY_CHANGE_DIRECTORY request. Because it is loaded above file system stack 130 a , in step 1c, overlay optimizer minifilter 311 will receive DCN request 700 before it reaches the file system driver.
- FIG. 7B illustrates the functionality that overlay optimizer minifilter 311 can perform when it receives DCN request 700 (e.g., as part of its pre-operation callback routine for IRP_MN_NOTIFY_CHANGE_DIRECTORY requests).
- overlay optimizer minifilter 311 can obtain a context for the directory identified in DCN request 700 .
- overlay optimizer minifilter 311 could call the FltGetStreamHandleContext (or similar) function and specify a pointer to the file object for C: ⁇ Dir1 as input (which pointer could be obtained from DCN request 700 ) to retrieve the context for C: ⁇ Dir1.
- overlay optimizer minifilter 311 can create one (e.g., via a call to the FltAllocateContext function). For example, if DCN request 700 is a result of a call to the FindFirstChangeNotificationA function, it typically would not yet have a context. In contrast, if I/O request 700 were a result of a call to the FindNextChangeNotification function, it typically would have a context since calls to this function reuse the handle obtained by calling FindFirstChangeNotificationA. In any case, after performing step 2, overlay optimizer minifilter 311 will have obtained a context 701 for the directory that is the target of DCN request 700 .
- FIG. 7C illustrates how overlay optimizer minifilter 311 can create a duplicate DCN request 700 a - 1 through 700 a - n (or collectively 700 a ) for each I/O stack and employ context 701 to maintain a DCN map 750 that links DCN request 700 with each duplicate DCN request 700 a .
- overlay optimizer minifilter 311 can create a duplicate DCN request 700 a - 1 to be sent to the file system driver in file system stack 130 a and a duplicate DCN request 700 a - 2 to be sent to the file system driver in file system stack 130 b .
- Each duplicate DCN request 700 a can specify the same changes as DCN request 700 .
- Overlay optimizer minifilter 311 can also configure each duplicate DCN request 700 a so that it targets the appropriate directory.
- DCN request 700 a - 1 can target the same directory, C: ⁇ Dir1, as DCN request 700 , but DCN request 700 a - 2 will need to target the corresponding directory D: ⁇ Dir1 on overlay cache 313 .
- duplicate DCN request 700 a - 1 will cause the file system driver in file system stack 130 a to generate a DCN when a specified change occurs to C: ⁇ Dir1 while duplicate DCN request 700 a - 2 will cause the file system driver in file system stack 130 b to generate a DCN when a specified change occurs to D: ⁇ Dir1.
- overlay optimizer minifilter 311 can create or update map 750 to link DCN request 700 to each duplicate DCN request 700 a . Any suitable technique can be employed to create such links including by mapping an identifier of DCN request 700 to an identifier of each duplicate DCN request 700 a .
- overlay optimizer minifilter 311 can store DCN map 750 in the context of the file object for C: ⁇ Dir1. For example, overlay optimizer minifilter 311 may call FltSetStreamHandleContext and provide as inputs a pointer to the file object for C: ⁇ Dir1 and a pointer to context 701 which contains map 750 .
- overlay optimizer minifilter 311 passes the duplicate DCN requests 700 a down to the file system driver in the corresponding file system stacks. This can be accomplished using asynchronous I/O such as by calling the FltPerformAsynchronousIo function. As part of passing duplicate DCN requests 700 a to the respective file system driver, overlay optimizer minifilter 311 can set context 701 containing DCN map 750 as the callback context for the asynchronous I/O request (e.g., by passing a pointer to context 701 to the FltPerformAsynchronousIo function).
- Including DCN map 750 in the callback context ensures that overlay optimizer minifilter 311 will have access to DCN map 750 when a duplicate DCN request 700 a is completed.
- duplicate DCN request 700 a - 1 will be passed to the file system driver in file system stack 130 a and duplicate DCN request 700 a - 2 will be passed to the file system driver in file system stack 130 b .
- the file system drivers will handle the duplicate DCN requests 700 a in a manner similar to how they handle any DCN request (i.e., they will complete the DCN request when any of the specified changes occur to the specified directory).
- overlay optimizer minifilter 311 pends DCN request 700 (e.g., by returning FLT_PREOP_PENDING from its pre-operation callback routine). While DCN request 700 is pended, overlay optimizer minifilter 311 can wait for the completion of any of the duplicate DCN requests 700 a.
- FIG. 7E illustrates the functionality that is performed when a file system driver completes one of the duplicate DCN requests 700 a .
- step 5a it is assumed that the file system driver in file system stack 130 b detects a specified change to D: ⁇ Dir1 and therefore completes duplicate DCM request 700 a - 2 .
- overlay optimizer minifilter 311 will receive the completed duplicate DCM request 700 a - 2 . Because duplicate DCM request 700 a - 2 was passed down using asynchronous I/O, duplicate DCM request 700 a - 2 will be passed up to overlay optimizer minifilter 311 by calling overlay optimizer minifilter 311 's callback routine.
- This call to overlay optimizer minifilter 311 's callback routine will specify the callback context that overlay optimizer minifilter 311 set when it called the FltPerformAsynchronousIo function.
- the call to the callback routine will provide a pointer to context 701 which contains map 750 .
- overlay optimizer minifilter 311 can retrieve map 750 from the callback context and use it to identify DCN request 700 as the DCN request to which duplicate DCN request 700 a - 2 is mapped. It is noted that, at any given time, overlay optimizer minifilter 311 may be managing many DCN requests pertaining to many different directories. The use of DCN maps therefore enables overlay optimizer minifilter 311 to identify to which pended DCN request any given completed duplicate DCN request pertains.
- overlay optimizer minifilter 311 copies the notification results from duplicate DCN request 700 a - 2 to DCN request 700 .
- the notification results are populated by the file system driver into duplicate DCN request 700 a - 2 's completion buffer and identify which change occurred to the directory.
- overlay optimizer minifilter 311 completes DCN request 700 which will cause the DCN to be provided to the requesting application, Windows Explorer.
- overlay optimizer minifilter 311 updates map 750 to remove DCN request 700 and duplicate DCN requests 700 a - 2 and sets context 701 as the context for C: ⁇ Dir1.
- overlay optimizer minifilter 311 This ensures that if another duplicate DCN request 700 a is subsequently completed, overlay optimizer minifilter 311 will be able to determine from map 750 that DCN request 700 has already been completed. Additionally, by maintaining map 750 in context 701 , it can be reused when/if another DCN request targeting C: ⁇ Dir1 is received (e.g., if Windows Explorer immediately calls FindNextChangeNotification which would typically be the case).
- overlay optimizer minifilter 311 can leave the remaining duplicate DCN request(s) 700 a pending so that they need not be recreated if/when another DCN request is received for the same directory. For example, in the example shown in FIG. 7A-7F , overlay optimizer 311 can leave duplicate DCN request 700 a - 1 pending after it completes DCN request 700 .
- overlay optimizer minifilter 311 will only need to create a duplicate DCN request to send to file system stack 130 b and can update map 750 to link both duplicate DCN request 700 a - 1 and the newly created duplicate DCN request to the newly received DCN request.
- FIGS. 8A and 8B provide flowcharts which correspond with the above-described DCN functionality.
- FIG. 8A represents functionality that overlay optimizer minifilter 311 (or minifilter 311 a ) can perform as part of its pre-operation callback routine for IRP_MN_NOTIFY_CHANGE_DIRECTORY requests.
- FIG. 8B represents functionality that overlay optimizer minifilter 311 (or minifilter 311 a ) can perform as part of its asynchronous I/O callback routine when a duplicate DCN request is completed.
- the above-described DCN functionality can be implemented to ensure that applications that rely on DCNs will function properly even when overlay optimizer 310 is used or deduplication, mirroring, caching etc. are implemented.
- Embodiments of the present invention may comprise or utilize special purpose or general-purpose computers including computer hardware, such as, for example, one or more processors and system memory.
- Embodiments within the scope of the present invention also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures.
- Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system.
- Computer-readable media is categorized into two disjoint categories: computer storage media and transmission media.
- Computer storage media devices include RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other similarly storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.
- Transmission media include signals and carrier waves.
- Computer-executable instructions comprise, for example, instructions and data which, when executed by a processor, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions.
- the computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language or P-Code, or even source code.
- the invention may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks.
- program modules may be located in both local and remote memory storage devices.
- An example of a distributed system environment is a cloud of networked servers or server resources. Accordingly, the present invention can be hosted in a cloud environment.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
- N/A
- Operating systems, such as Windows, provide functionality for preventing the content of a storage medium from being changed. In a typical example, it may be desirable to prevent the operating system image, which may be stored on a particular disk partition or on flash media, from being changed at runtime. As one example only, Windows 10 provides the Unified Write Filter (UWF) which can be used to redirect all writes that target the operating system image to a RAM or disk cache called an overlay. This overlay stores changes made to the operating system at runtime but is removed when the device is restarted thereby restoring the device to its original state.
- In this specification, a volume should be construed as a logical unit that represents an area of persistent storage to the file system. A volume can correspond to a single physical storage device, such as a hard disk, but can also correspond to a single partition on a physical storage device with multiple partitions. A volume can also span across multiple physical storage devices. A “protected volume” is a volume that a write filter protects. In other words, when a write filter is employed, the write filter will protect the content (e.g., files, directories, registry entries, etc.) of the volume from being changed.
-
FIG. 1 illustrates a simplified Windows-based I/O system architecture 100 when awrite filter 140 is employed.Write filter 140 may typically be the Windows UWF but may equally be another third-party write filter.Architecture 100 includes adisk 105 which may oftentimes be a solid-state drive (SSD) but may represent a physical hard disk, an internal USB device, an external SATA device or any other physical storage medium (or media) that may be compatible with a write filter. In accordance with the definition of a volume,disk 105 includes an area of persistent storage that is represented by a volume.Disk stack 110 represents the various drivers that enabledisk 105 to be accessed and are not critical to an understanding of the present invention. - In the Windows operating system, when write
filter 140 is employed to protect the content of a volume from modification, two different I/O stacks that represent the protected volume are loaded above disk stack 110: a “protected I/O stack” that includesvolume stack 120 a andfile system stack 130 a; and an “unprotected I/O stack that includesvolume stack 120 b andfile system stack 130 b. The protected I/O stack can be viewed as the primary I/O stack and handles I/O requests that do not modify content on the protected volume (e.g., read requests). The unprotected I/O stack can be viewed as a pseudo I/O stack and handles I/O requests that modify content on the protected volume (e.g., write requests). For purposes of this description and the claims, a write request should be construed as encompassing I/O requests that modify existing content and I/O requests that create new content. The limited scenarios where such write requests would occur to the protected volume are described below. -
Write filter 140 is loaded abovefile system stack 130 a on the protected I/O stack so that it can intercept all requests to access the content of the protected volume. When writefilter 140 receives a request to read content from the protected volume, and assuming the content was not previously modified and stored inoverlay 150, it can send the request down the protected I/O stack so that the content will be read fromdisk 105 in a typical manner. In contrast, when writefilter 140 receives a request to write content on the protected volume, it redirects the request so that the write is made in overlay 150 (i.e., the new/modified content is stored inoverlay 150 while the content stored ondisk 105 remains unchanged). Once modified content is stored inoverlay 150, writefilter 140 will ensure that any subsequent request to access the modified content will be completed using the modified content stored inoverlay 150. - There are some scenarios where write
filter 140 will allow content on the protected volume to be modified. For example, an administrator can register an exclusion with writefilter 140 for particular content on the protected volume (e.g., particular files or folders). In such cases, writefilter 140 will not redirect any request that would modify the excluded content tooverlay 150, but instead will cause the request to be passed down the unprotected I/O stack thereby causing the excluded content to be modified on the protected volume. As another example, writefilter 140 enables an administrator to cause modified content stored inoverlay 150 to be committed todisk 105. In such cases, when writefilter 140 detects the write request pertaining to the commit, it will cause it to be passed down the unprotected I/O stack. Accordingly, even when writefilter 140 is employed, it is still possible to modify some content on the protected volume. - U.S. Pat. Publ. No. 2018/0217940 (the '940 Publication) describes how an overlay-managing write filter (or an “overlay optimizer”) can be employed to optimize the performance of write
filter 140. As explained in the '940 Publication, a primary role that an overlay optimizer plays is to prevent a system reboot due tooverlay 150 becoming full. In general terms, an overlay optimizer, which can be loaded above writefilter 140, can accomplish this by moving content that is stored inoverlay 150 to a separate overlay cache. This separate overlay cache may be implemented as a directory on the protected volume that is registered as an exclusion or as a separate partition of disk 105 (i.e., an unprotected volume). In either case, writefilter 140 will not block writes to the overlay cache. Once modified content is moved fromoverlay 150 to the overlay cache, the overlay optimizer can redirect subsequent requests to access the modified content. - Although this technique is effective to prevent the overlay from becoming full, it oftentimes results in a large amount of data being written to disk.
FIGS. 2A and 2B generally illustrate why this is.FIGS. 2A and 2B show portions ofarchitecture 100 with the addition ofoverlay optimizer 200 which can be in the form of a file system minifilter driver that is loaded above writefilter 140. In these figures, it is assumed that a file, File A, is stored on the protected volume ondisk 105 and has a size of 100 MB. It is also assumed that, during use of the computing device, a request to modify file A was made and that writefilter 140 redirected this request to overlay 150 so that File A on the protected volume remains unmodified. - Notably, when redirecting a request to modify a file on the protected volume, write
filter 140 does not store the entire file inoverlay 150 but only stores the modified portions of the file (e.g., each page that is modified). In the depicted example, it is assumed that the request to modify File A only modified a single 4 KB page of the file. Accordingly,FIG. 2A shows thatoverlay 150 stores only the single 4 KB modified page. Additionally, even though writefilter 140 only stores the modified portions (e.g., pages) inoverlay 150, it does not provide any way to detect which modified portions are stored inoverlay 150. To the contrary, writefilter 140 only provides functionality for determining which files are cached in overlay 150 (or more correctly, which files have modified portions stored in overlay 150). For example, in Windows, the UWF_Overlay.GetOverlayFiles method can be called to obtain a list of files that are cached inoverlay 150, but the UWF does not provide a way to determine which pages of these files are actually stored inoverlay 150. - Although not depicted in
FIG. 2A , it is assumed that the operation ofwrite filter 140 has caused many other modified pages of files to be stored inoverlay 150 such thatoverlay 150 has grown beyond some threshold percentage of its capacity (e.g., 60% of its maximum size). In such scenarios,overlay optimizer 200 can detect thatoverlay 150 is becoming full as is represented bystep 1. Upon determining thatoverlay 150 is becoming full,overlay optimizer 200 can commence moving files fromoverlay 150 to the overlay cache (which is assumed to be on disk 105). For the reasons given above,overlay optimizer 200 can only detect which files have modified portions stored inoverlay 150 but cannot identify only the modified portions. Accordingly, to move File A to the overlay cache,overlay optimizer 200 will issue a request to read File A as represented instep 2. - This request will be intercepted by write
filter 140. Then, instep 3 shown inFIG. 2B , writefilter 140 will complete the read request by providing tooverlay optimizer 200 the entire modified File A (i.e., the 100 MB version on the protected volume with the 4 KB modified page from overlay 150). Instep 4,overlay optimizer 200 will issue a request to write the 100 MB modified File A to the overlay cache. Assuming the overlay cache is implemented as a folder on the protected volume that is registered as an exclusion, writefilter 140 will redirect this write down the unprotected I/O stack rather than redirecting it to overlay 150. Finally, instep 5,overlay optimizer 200 will request deletion of File A which will causewrite filter 140 to discard the modified page of File A from overlay 150 (but will not alter the unmodified File A stored on the protected volume). - As can be seen, because
overlay optimizer 200 has no way of identifying that only a single page of File A exists inoverlay 150, it must copy the entire File A to the overlay cache as part of freeing upoverlay 150. Considering thatoverlay optimizer 200 may typically move many files in this manner to free upoverlay 150, a large amount of data will be written todisk 105 which can create a number of problems. For example, SSDs, which are typically comprised of NAND flash, wear out over time. Therefore, SSD manufacturers typically provide a warranty up to a specified total amount of data written to the SSD which is usually defined as terabytes written or TBW. With the use ofoverlay optimizer 200, it is possible that the amount of data written todisk 105 over the warranty period may exceed the TBW. To avoid this,overlay optimizer 200 may be configured to forego moving files fromoverlay 150 after a certain amount of data has been written each day. Although this may preserve the life of the SSD, it increases the likelihood of a forced reboot asoverlay 150 becomes full. Even ifdisk 105 is not an SSD, the large amount of data thatoverlay optimizer 200 may write todisk 105 when freeing upoverlay 150 can still degrade overall system performance. In short, it would be more efficient ifoverlay optimizer 200 could move only the modified portions of files fromoverlay 150 rather than copying the entire file. - Certain applications, such as Windows Explorer, custom shells, file system watchers, antivirus software, source code editors, etc., are configured to monitor the contents of a directory. Oftentimes, these applications will register for directory change notifications (DCNs) rather than enumerating the directory on demand. For example, in Windows, an application can employ the FindFirstChangeNotification function to register to be notified when a specified change occurs to/within a directory. These changes may include any change to the name or size of a file in the directory, a change to the name or attributes of the directory, a change in the security of the directory, etc. Once registered, the application will be notified whenever any of the specified changes occur and can take whatever action is appropriate.
- Waiting for a DCN is similar to having a read operation pending against a directory and, if necessary, its subdirectories. When something changes within the directory being watched, the read operation is completed. Accordingly, the DCN functionality is primarily implemented within the file system stack for the volume containing the directory (or more specifically, by the file system driver in the file system stack).
- In multi-volume architectures, such as Windows-based I/
O system architecture 100, DCNs will not work properly. With reference toFIG. 1 , when an application sends a DCN request for a particular directory on the protected volume, the DCN request will be provided to filesystem stack 130 a only (e.g., similar to pending a read request with the file system driver infile system stack 130 a). However, the DCN request will not be provided to filesystem stack 130 b. As a result, the application will only receive a DCN when an I/O request that passes throughfile system stack 130 a causes the specified change(s) to the directory. In contrast, if an I/O request that passes throughfile system stack 130 b causes the specified change(s) to the directory, no DCN will be generated becausefile system stack 130 b will be unaware of the DCN request. In short, whenwrite filter 140 is used, an application that relies on DCNs will not be notified of all directory changes for which it has registered. - One specific example highlights the difficulties that this problem can create. Windows Explorer is configured to use DCNs to be immediately notified when a folder or file is created in a particular directory. This ensures that the newly created folder or file will be immediately displayed within the Explorer window. In write filter environments, however, a user may open Windows Explorer to view the contents of a particular directory (or folder) and right click to select the option to create a new folder. If the directory whose contents is being displayed is part of the protected volume, the request to create the new folder will be handled through
file system stack 130 b which is unaware of Windows Explorer's DCN. Windows Explorer will therefore not receive a DCN even though a new folder was created in the directory for which it registered for DCNs. As a result, a new folder will be created but Windows Explorer will not update its user interface to include the new folder. Most users in this situation would be unfamiliar with write filters and may assume that a new folder was not created. Even if a user suspected that the write filter may be the cause of the problem, he or she would likely be unaware of how to force Windows Explorer to enumerate the contents of the directory and update its user interface to display the existence of the new folder. - This issue is not limited to the multi-volume write filter architecture depicted in
FIG. 1 but exists in any multi-volume filter environment. For example, deduplication, mirroring and caching solutions typically employ a filter that is loaded on the primary I/O stack, similar to writefilter 140 and the protected I/O stack inFIG. 1 , and one or more shadow volumes where each shadow volume would have a corresponding I/O stack similar to the unprotected I/O stack shown inFIG. 1 . Therefore, as a practical matter, applications/services that rely on DCNs only work properly when a directory is not spread across more than one volume. - The present invention extends to methods, systems and computer program products for minimizing the amount of data that is written to disk when an overlay optimizer is used in conjunction with a write filter to prevent the overlay from becoming full. To minimize the amount of data written to disk, a first instance of an overlay optimizer minifilter can be loaded above the write filter on the protected I/O stack and a second instance of the overlay optimizer filter can be loaded on the unprotected I/O stack. The second instance of the overlay optimizer minifilter can be used to intercept writes that were initiated by the overlay optimizer's request to commit files cached in the write filter's overlay to thereby extract only the modified portions of the files that are actually stored in the overlay. The second instance of the overlay optimizer minifilter can then write these modified portions of the files, as opposed to the entire files, in the overlay cache. In this way, a minimal amount of data is written to disk when freeing up the overlay.
- The present invention also extends to methods, systems and computer program products for enabling directory change notifications when a write filter is employed as well as in other multi-volume filter environments. When a directory change notification is received by the first instance of the overlay optimizer minifilter, it can create and issue duplicate DCN requests to each I/O stack while pending the original DCN request. A DCN map can be used to associate the duplicate DCN requests with the pended DCN request so that contents of a completed DCN request can be copied to the DCN request and then the DCN request can be completed. In this way, any changes made to the directory, regardless of which volume the change may have been made on, will trigger a DCN.
- In one embodiment, the present invention is implemented—by an overlay optimizer that includes an overlay optimizer service, a first overlay optimizer minifilter that is loaded on a protected I/O stack and a second overlay optimizer minifilter that is loaded on an unprotected I/O stack—as a method for minimizing an amount of data that is written to disk when the overlay optimizer service reduces consumption of the write filter's overlay. The overlay optimizer service identifies a first file that is cached in the overlay and sends a first notification to the second overlay optimizer minifilter. The first notification identifies the first file. The overlay optimizer service then submits a request to commit the first file. The second overlay optimizer minifilter intercepts a write request that is being passed down the unprotected I/O stack. Based on the first notification, the second overlay optimizer minifilter determines that the write request pertains to the request to commit the first file. In response, the second overlay optimizer minifilter reparses the write request to an overlay cache.
- In another embodiment, the present invention is implemented by a minifilter in a multi-volume filter environment as a method for enabling directory change notifications when a directory spans the multiple volumes. The minifilter receives a directory change notification (DCN) request that targets a first directory. In response, the minifilter creates a plurality of duplicate DCN requests corresponding to a plurality of I/O stacks and sends each of the plurality of duplicate DCN requests down the corresponding I/O stack. The minifilter also pends the DCN request. In response to any of the plurality of duplicate DCN requests being completed, the minifilter copies content of the completed duplicate DCN request to the DCN request and then completes the DCN request.
- This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter.
- Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
-
FIG. 1 illustrates a Windows-based I/O system in which a write filter is employed to redirect writes targeting a protected volume to an overlay; -
FIGS. 2A and 2B illustrate why a large amount of data may be written to disk when an overlay optimizer is employed to prevent the write filter's overlay from becoming full; -
FIG. 3 illustrates an example I/O system architecture that may exist on a client device when the overlay optimizer of the present invention is implemented; -
FIGS. 4A-4F illustrate functionality that the overlay optimizer can perform to identify the changed portions of files that are actually stored in the write filter's overlay to avoid having to copy the entire file to an overlay cache; -
FIGS. 5A-5C provide flow diagrams representing functionality that the overlay optimizer can perform to identify the changed portions of files that are actually stored in the write filter's overlay to avoid having to copy the entire file to an overlay cache; -
FIG. 6 illustrates an example multi-volume filter architecture in which embodiments of the present invention may be implemented; -
FIGS. 7A-7F illustrate functionality that a minifilter can perform to enable directory change notifications when a directory is spread across multiple volumes; and -
FIGS. 8A and 8B provide flow diagrams representing functionality that a minifilter can perform to enable directory change notifications when a directory is spread across multiple volumes. - In this specification and the claims, the term “client device” should be construed as any user computing device that is capable of executing a write filter. A client device would therefore include desktops, laptops, tablets, thin clients, smart phones, etc. The term “write filter” should be construed as a software component that is employed to protect the content of a storage device from being modified by redirecting I/O requests that target the content so that the modifications are stored elsewhere. Embodiments of the present invention will be described in the context of a write filter provided with the Windows operating system (e.g., the UWF). However, embodiments of the present invention can equally be implemented on a client device that runs a non-Windows operating system as well as on a client device that runs Windows but employs another third party (i.e., non-Windows) write filter. Although the term “minifilter” is used in a Windows context, its use is exemplary and should not be construed as limiting the invention to Windows-based implementations. The term “overlay optimizer minifilter” should therefore be construed generally as a software component that can perform at least some of the functionality described herein.
-
FIG. 3 illustrates an example I/O system architecture 300 that may exist on a client device when the present invention is implemented. In comparison toarchitecture 100,architecture 300 includes anoverlay optimizer 310 which comprises: a first instance of anoverlay optimizer minifilter 311 which is loaded on the protected I/O stack abovewrite filter 140; a second instance of anoverlay optimizer minifilter 312 which is loaded on the unprotected I/O stack; anoverlay cache 313 implemented ondisk 105; and anoverlay optimizer service 314 which may be a user-mode component that executes with administrator privileges. As noted in the background,disk 105 need not represent a single physical storage device andoverlay cache 313 may or may not be part of the protected volume.Architecture 300 can be employed to enableoverlay optimizer 310 to identify and move only the changed portions of files that are stored inoverlay 150 tooverlay cache 313 as opposed to copying the entire files tooverlay cache 313. - Prior to describing how
overlay optimizer 310 moves only changed portions of files, the following overview is given to describe howoverlay optimizer 310 can detect when it should start moving files fromoverlay 150.Overlay optimizer 310 can be configured to monitor the consumption of overlay 150 (e.g., by usingoverlay optimizer service 314 to periodically read the OverlayConsumption or AvailableSpace members of the UWF_Overlay class or to watch for the UWF_OVERLAY_REACHED_WARNING_LEVEL and/or UWF_OVERLAY_REACHED_CRITICAL_LEVEL events) to thereby detect whenoverlay 150 is becoming full. As one example only,overlay optimizer 310 could be configured to detect when consumption ofoverlay 150 has exceeded 60%. Whenoverlay optimizer 310 detects that the consumption ofoverlay 150 has exceeded some threshold (i.e., thatoverlay 150 is becoming full), it can commence moving files fromoverlay 150 tooverlay cache 313 to thereby prevent a reboot of the computing device (or severely degraded performance) which would otherwise occur onceoverlay 150 is full. In contrast to what is described in the background,overlay optimizer 310 can perform functionality for identifying the changed portions of files that are stored inoverlay 150 so that only these changed portions will be written tooverlay cache 313. -
FIGS. 4A-4F illustrate the functionality thatoverlay optimizer 310 performs to identify the changed portions of files that are actually stored inoverlay 150 to avoid having to copy the entire file tooverlay cache 313. Instep 1a shown inFIG. 4A ,overlay optimizer service 314 can request the files that are cached inoverlay 150. Typically,overlay optimizer service 314 would submit this request upon detecting thatoverlay 150 is becoming full. However,overlay optimizer 314 could equally submit such requests on a periodic basis with or without regard to the consumption ofoverlay 150. Accordingly, the manner in whichoverlay optimizer service 314 identifies which files are cached inoverlay 150 is not essential to the present invention. In Windows-based implementations,overlay optimizer service 314 could employ the Windows Management Instrumentation (WMI) GetOverlayFiles( ) function to retrieve a listing of files that are cached inoverlay 150. - In response to its request,
overlay optimizer service 314 receives a list of the files cached inoverlay 150 instep 1b. For the sake of illustration,FIG. 4A assumes thatoverlay 150 stores changed pages from File A, File B and File C among other files, and therefore, the list of files received instep 1b identifies these three files. Notably, this list does not provide any information about which portions of the three files are actually stored inoverlay 150 but merely provides a filename (or path) to each file. - Turning to
FIG. 4B , onceoverlay optimizer service 314 obtains the list of files, it can identify a particular file in the list instep 2a and send the filename of the file tooverlay optimizer service 314 instep 2b. These steps can be repeated for each file in the list (or at least until the overlay's consumption has been reduced to a satisfactory level). As shown, this filename can be in the form of a complete file path to the file (which for File A is assumed to be C:\Dir1\File A.txt). In this way,overlay optimizer minifilter 312 becomes aware of a file that should be moved fromoverlay 150 tooverlay cache 313. - Once
overlay optimizer minifilter 312 receives the filename of a file fromoverlay optimizer service 314, it can request a file ID for the file as represented asstep 3a inFIG. 4C . For example, in Windows-based implementations,overlay optimizer minifilter 312 can employ the filename of File A to obtain a pointer to the file object for File A and can then call the FltQueryInformationFile function and provide as inputs the pointer to the file object for File A and FileInternalInformation as the FILE INFORMATION CLASS value. As a result,overlay optimizer minifilter 312 will receive back the file ID (or file reference number) as the value of the IndexNumber member of the FILE INTERNAL INFORMATION structure. This file ID is a unique 8 byte value assigned by the file system to represent the file. Instep 3b,overlay optimizer minifilter 312 stores the file ID.Steps overlay optimizer minifilter 312 receives fromoverlay optimizer service 314 instep 2b. As a result,overlay optimizer minifilter 312 will have created a listing of file IDs for files that are to be moved fromoverlay 150 tooverlay cache 313. As shown, in conjunction with obtaining the file ID of a file,overlay optimizer minifilter 312 can also create a shadow file inoverlay cache 313 for the file and maintain a mapping between the file ID and the filename of the corresponding shadow file. - Once
overlay optimizer service 314 notifiesoverlay optimizer minifilter 312 of each file that is to be moved fromoverlay 150,overlay optimizer service 314 can request that the files be committed as represented asstep 4a inFIG. 4D . It is noted that, in normal operation, a request to commit a file fromoverlay 150 to the protected volume would cause the changed portions of the file to be written to the version of the file on the protected volume thereby permanently modifying the file. However, as will become apparent below,overlay optimizer minifilter 312 prevents this normal handling of a commit from being carried out for any file thatoverlay optimizer 310 is moving tooverlay cache 313. - In response to
overlay optimizer service 314's request to commit a file, writefilter 140 will read the changed portions of the file instep 4b and issue a request to write the change portions instep 4c. As described in the background, this write request will be directed to the unprotected I/O stack. Given thatoverlay optimizer minifilter 312 is loaded on the unprotected I/O stack, it will intercept this write request instep 4d.Steps 4a-4d can be performed for each file to be moved. - Turning to
FIG. 4E , whenoverlay optimizer minifilter 312 intercepts a write request, it does not yet know whether the write request is a result ofoverlay optimizer service 314's request to commit a file or a result of some other function (e.g., a write to an excluded file/directory on the protected volume, a request to commit a file made by another service, etc.). Accordingly, wheneveroverlay optimizer minifilter 312 receives a write request, it can performstep 5a to retrieve the context of the file to which the write request pertains (e.g., by calling the FltGetFileContext function). In the depicted example, it is assumed that the received write request pertains to File A and therefore,overlay optimizer minifilter 312 will receive a file context that includes a file ID of 0xABCD . . . in response to its call to FltGetFileContext. Then, instep 5b,overlay optimizer minifilter 312 can compare the file ID retrieved instep 5a to its list of file IDs that it created instep 3b to thereby determine whether the current write request pertains to a file thatoverlay optimizer service 314 indicated that it would be committing. In the depicted example,overlay optimizer minifilter 312 will determine that the file ID obtained instep 5a (0xABCD . . . ) matches the file ID of File A (0xABCD . . . ) that it had previously stored in its list. - Although not shown, to ensure that the comparison in
step 5b will be possible, in some embodiments,overlay optimizer minifilter 312 may retrieve the context of the file and verify that the context includes the file ID of the file. If not,overlay optimizer minifilter 312 could perform steps similar tosteps step 5b is performed. - Using the file ID to identify write requests that are a result of
overlay optimizer service 314's request to commit a file ensures thatoverlay optimizer minifilter 312 will detect such write requests regardless of how they are performed. For example, this technique will detect such writes even when the file is committed to disk via multiple handles or threads or when the file is opened via a short name or relative path. -
Step 6 inFIG. 4F represents howoverlay optimizer minifilter 312 may reparse (or redirect) a write request when it determines that the write request is a result ofoverlay optimizer service 314's request to commit a file (i.e., whenoverlay optimizer minifilter 312 determines instep 5b that the file ID of the file to which the write request pertains matches a file ID contained in its list). Becauseoverlay optimizer minifilter 312 has mapped the file ID to the filename of the corresponding shadow file inoverlay cache 313, it can replace the filename that is currently associated with the write request with the filename of the corresponding shadow file. - For example,
overlay optimizer minifilter 312 could call the IoReplaceFileObjectName function with inputs of the pointer to the file object for file A (which would be provided in the write request) and the filename of the corresponding shadow file. As represented inFIG. 4F , this will cause the write request to target the corresponding shadow file inoverlay cache 313 rather than the actual file stored on the protected volume. In this way, the changed portions of the file will be stored inoverlay cache 313 but will not be committed to the actual unmodified file that exists on the protected volume. - Although not shown, if
overlay optimizer minifilter 312 determines instep 5b that the file ID of the file to which the write request pertains does not match a file ID contained in its list, it can simply allow the write request to be passed down the stack for normal handling. In short,overlay optimizer minifilter 312 will pass each write request down towardsvolume stack 120 b but selectively modifies any write request that are determined to be a result ofoverlay optimizer service 314's request to commit a file cached inoverlay 150. - When any write request that resulted from a request to commit a file is completed successfully (whether or not the commit request was issued by overlay optimizer service 314), write
filter 140 will complete the commit by discarding the changed portions of the file fromoverlay 150. Accordingly, for a commit initiated byoverlay optimizer service 314, the end result is that only the changed portions of the file are copied to overlay cache 313 (but are not actually committed to the corresponding file on the protected volume) and then the changed portions of the file are discarded fromoverlay 150 thereby reducing its consumption. - To ensure that the user will still see the changed file (i.e., to ensure that the file with the changed portions will still be presented to the user),
overlay optimizer 310 can update a hash table or “map” that reflects that the changed portions to the file exist inoverlay cache 313. The '940 Publication describes how an overlay optimizer can employ a map for this purpose and therefore the creation, update and use of a map will not be described in detail. Suffice it to say thatoverlay optimizer minifilter 311 can employ the map when handling an I/O request to determine whether content fromoverlay cache 313 needs to be accessed to complete the I/O request properly (e.g., by merging a file's changed pages stored in and retrieved fromoverlay cache 313 with the file's unmodified pages retrieved from the protected volume. -
FIGS. 5A-5C provide flow diagrams corresponding to the above-described functionality.FIG. 5A represents functionality thatoverlay optimizer service 314 performs to determine when to initiate the process of freeing upoverlay 150, to notifyoverlay optimizer minifilter 312 of the files that will be moved fromoverlay 150 and to initiate the process of moving the files.FIG. 5B represents functionality thatoverlay optimizer minifilter 312 performs when it receives a notification fromoverlay optimizer service 314 that a file will be moved fromoverlay 150.FIG. 5C represents functionality thatoverlay optimizer 313 performs when it receives an I/O request to determine whether the I/O request pertains to a commit initiated byoverlay optimizer service 314, and if so, to rep arse the I/O request to a corresponding shadow file inoverlay cache 313. - In some embodiments,
overlay optimizer 310 may also be configured to enable directory change notifications (DCNs) when the content of a directory is spread across more than one volume. For example, inarchitecture 300 shown inFIG. 3 , the protected volume andoverlay cache 313 may each store files (or portions of files) that exist (or are represented as existing) in the same directory. In this scenario, and with prior art techniques as described in detail in the background, when an application requests a DCN for changes to this directory, the request will only be provided to filesystem stack 130 a, and therefore, any changes to the directory that are made throughfile system stack 130 b will not trigger a DCN. - This same problem exists in other multi-volume filter architectures. For example,
FIG. 6 illustrates amulti-volume filter architecture 600 that is substantially the same asarchitecture 300 but more closely represents the architecture that would exist when a deduplication, mirroring or caching solution is implemented. As inarchitecture 300, a filter is loaded on the primary I/O stack inarchitecture 600. This filter is labeled and will be referred to as reparse filter 140 a to indicate that it could represent writefilter 140 or a file system filter of a deduplication, mirroring, caching or similar solution. As inarchitecture 300, a supporting I/O stack (which includesfile system stack 130 b-1 andvolume stack 120 b-1) will also exist inarchitecture 600 and corresponds withshadow volume 1.Architecture 600 may also include zero or more additional supporting I/O stacks which would include afile system stack 130 b-n and avolume stack 120 b-n and would correspond with shadow volume n (where n is intended to represent any reasonable integer including 0). In accordance with embodiments of the present invention, a minifilter 311 a can be loaded on the primary I/O stack abovereparse filter 140 a and a minifilter (312 a-1 through 312 a-n) may be loaded on each supporting I/O stack above the file system driver. These minifilters may or may not be the same as the overlay optimizer minifilters described above. -
FIGS. 7A-7F illustrate how a minifilter loaded on the primary I/O stack in a multi-volume filter environment can enable DCNs when a directory is spread across multiple volumes. For illustrative purposes,FIGS. 7A-7F will be based onarchitecture 300 and will show overlay optimizerminifilter 311 performing the functionality, but the depicted functionality can equally be performed inarchitecture 600 using minifilter 311 a. - In
FIGS. 7A-7F , it is assumed that the protected volume includes a directory, C:\Dir1, and thatoverlay optimizer 310 maintains a corresponding directory, D:\Dir1, inoverlay cache 313 where modified files in Dir1 have been moved. For example, a file named File 2.txt is shown as being stored inoverlay cache 313. As described above, File 2.txt could have originally been stored in overlay 150 (e.g., when File 2.txt was created or modified) but moved tooverlay cache 313. As also described above,overlay optimizer 310 can maintain a map that defines that file 2.txt is actually part of C:\Dir1 (at least from the user's perspective). Of importance to the present discussion, if an application desires to be notified of changes to C:\Dir1, a DCN should be triggered whenever any changes are made to this directory on the protected volume as well as whenever any changes are made to the corresponding directory inoverlay cache 313. - Turning to
FIG. 7A , instep 1a, an application, which is assumed to be Windows Explorer in this example, submits a request to register for a DCN for C:\Dir1. In Windows-based implementations, this may be accomplished by calling the FindFirstChangeNotificationA function and specifying the full directory path as input. In response, the I/O manager of the operating system can generate an I/O request (or DCN request) 700 and send it to the file system driver infile system stack 130 a. In Windows-based implementations,DCN request 700 may be in the form of an IRP_MN_NOTIFY_CHANGE_DIRECTORY request. Because it is loaded abovefile system stack 130 a, instep 1c,overlay optimizer minifilter 311 will receiveDCN request 700 before it reaches the file system driver. -
FIG. 7B illustrates the functionality thatoverlay optimizer minifilter 311 can perform when it receives DCN request 700 (e.g., as part of its pre-operation callback routine for IRP_MN_NOTIFY_CHANGE_DIRECTORY requests). Instep 2a,overlay optimizer minifilter 311 can obtain a context for the directory identified inDCN request 700. For example,overlay optimizer minifilter 311 could call the FltGetStreamHandleContext (or similar) function and specify a pointer to the file object for C:\Dir1 as input (which pointer could be obtained from DCN request 700) to retrieve the context for C:\Dir1. Although not shown, a context may not have been created yet for C:\Dir1 in which caseoverlay optimizer minifilter 311 can create one (e.g., via a call to the FltAllocateContext function). For example, ifDCN request 700 is a result of a call to the FindFirstChangeNotificationA function, it typically would not yet have a context. In contrast, if I/O request 700 were a result of a call to the FindNextChangeNotification function, it typically would have a context since calls to this function reuse the handle obtained by calling FindFirstChangeNotificationA. In any case, after performingstep 2,overlay optimizer minifilter 311 will have obtained acontext 701 for the directory that is the target ofDCN request 700. -
FIG. 7C illustrates howoverlay optimizer minifilter 311 can create aduplicate DCN request 700 a-1 through 700 a-n (or collectively 700 a) for each I/O stack and employcontext 701 to maintain aDCN map 750 that linksDCN request 700 with eachduplicate DCN request 700 a. As represented bystep 3a,overlay optimizer minifilter 311 can create aduplicate DCN request 700 a-1 to be sent to the file system driver infile system stack 130 a and aduplicate DCN request 700 a-2 to be sent to the file system driver infile system stack 130 b. Eachduplicate DCN request 700 a can specify the same changes asDCN request 700.Overlay optimizer minifilter 311 can also configure eachduplicate DCN request 700 a so that it targets the appropriate directory. For example,DCN request 700 a-1 can target the same directory, C:\Dir1, asDCN request 700, butDCN request 700 a-2 will need to target the corresponding directory D:\Dir1 onoverlay cache 313. In this way,duplicate DCN request 700 a-1 will cause the file system driver infile system stack 130 a to generate a DCN when a specified change occurs to C:\Dir1 whileduplicate DCN request 700 a-2 will cause the file system driver infile system stack 130 b to generate a DCN when a specified change occurs to D:\Dir1. - In
step 3b,overlay optimizer minifilter 311 can create or updatemap 750 to linkDCN request 700 to eachduplicate DCN request 700 a. Any suitable technique can be employed to create such links including by mapping an identifier ofDCN request 700 to an identifier of eachduplicate DCN request 700 a. Instep 3c,overlay optimizer minifilter 311 can storeDCN map 750 in the context of the file object for C:\Dir1. For example,overlay optimizer minifilter 311 may call FltSetStreamHandleContext and provide as inputs a pointer to the file object for C:\Dir1 and a pointer tocontext 701 which containsmap 750. - Turning to
FIG. 7D , instep 4a,overlay optimizer minifilter 311 passes the duplicate DCN requests 700 a down to the file system driver in the corresponding file system stacks. This can be accomplished using asynchronous I/O such as by calling the FltPerformAsynchronousIo function. As part of passing duplicate DCN requests 700 a to the respective file system driver,overlay optimizer minifilter 311 can setcontext 701 containingDCN map 750 as the callback context for the asynchronous I/O request (e.g., by passing a pointer tocontext 701 to the FltPerformAsynchronousIo function). IncludingDCN map 750 in the callback context ensures thatoverlay optimizer minifilter 311 will have access toDCN map 750 when aduplicate DCN request 700 a is completed. In this example,duplicate DCN request 700 a-1 will be passed to the file system driver infile system stack 130 a andduplicate DCN request 700 a-2 will be passed to the file system driver infile system stack 130 b. The file system drivers will handle the duplicate DCN requests 700 a in a manner similar to how they handle any DCN request (i.e., they will complete the DCN request when any of the specified changes occur to the specified directory). Instep 4b,overlay optimizer minifilter 311 pends DCN request 700 (e.g., by returning FLT_PREOP_PENDING from its pre-operation callback routine). WhileDCN request 700 is pended,overlay optimizer minifilter 311 can wait for the completion of any of the duplicate DCN requests 700 a. -
FIG. 7E illustrates the functionality that is performed when a file system driver completes one of the duplicate DCN requests 700 a. Instep 5a, it is assumed that the file system driver infile system stack 130 b detects a specified change to D:\Dir1 and therefore completesduplicate DCM request 700 a-2. In response,overlay optimizer minifilter 311 will receive the completedduplicate DCM request 700 a-2. Becauseduplicate DCM request 700 a-2 was passed down using asynchronous I/O,duplicate DCM request 700 a-2 will be passed up tooverlay optimizer minifilter 311 by callingoverlay optimizer minifilter 311's callback routine. This call tooverlay optimizer minifilter 311's callback routine will specify the callback context thatoverlay optimizer minifilter 311 set when it called the FltPerformAsynchronousIo function. In other words, the call to the callback routine will provide a pointer tocontext 701 which containsmap 750. - In step 5c,
overlay optimizer minifilter 311 can retrieve map 750 from the callback context and use it to identifyDCN request 700 as the DCN request to whichduplicate DCN request 700 a-2 is mapped. It is noted that, at any given time,overlay optimizer minifilter 311 may be managing many DCN requests pertaining to many different directories. The use of DCN maps therefore enablesoverlay optimizer minifilter 311 to identify to which pended DCN request any given completed duplicate DCN request pertains. - In
step 6a shown inFIG. 7E ,overlay optimizer minifilter 311 copies the notification results fromduplicate DCN request 700 a-2 toDCN request 700. The notification results are populated by the file system driver intoduplicate DCN request 700 a-2's completion buffer and identify which change occurred to the directory. Once the notification results are copied toDCN request 700,overlay optimizer minifilter 311 completesDCN request 700 which will cause the DCN to be provided to the requesting application, Windows Explorer. Instep 6c,overlay optimizer minifilter 311 updates map 750 to removeDCN request 700 andduplicate DCN requests 700 a-2 and setscontext 701 as the context for C:\Dir1. This ensures that if anotherduplicate DCN request 700 a is subsequently completed,overlay optimizer minifilter 311 will be able to determine frommap 750 thatDCN request 700 has already been completed. Additionally, by maintainingmap 750 incontext 701, it can be reused when/if another DCN request targeting C:\Dir1 is received (e.g., if Windows Explorer immediately calls FindNextChangeNotification which would typically be the case). - In some embodiments, when one
duplicate DCN request 700 a is completed,overlay optimizer minifilter 311 can leave the remaining duplicate DCN request(s) 700 a pending so that they need not be recreated if/when another DCN request is received for the same directory. For example, in the example shown inFIG. 7A-7F ,overlay optimizer 311 can leaveduplicate DCN request 700 a-1 pending after it completesDCN request 700. Then, if another DCN request is received that targets C:\Dir1, in this subsequent iteration ofstep overlay optimizer minifilter 311 will only need to create a duplicate DCN request to send to filesystem stack 130 b and can update map 750 to link bothduplicate DCN request 700 a-1 and the newly created duplicate DCN request to the newly received DCN request. -
FIGS. 8A and 8B provide flowcharts which correspond with the above-described DCN functionality.FIG. 8A represents functionality that overlay optimizer minifilter 311 (or minifilter 311 a) can perform as part of its pre-operation callback routine for IRP_MN_NOTIFY_CHANGE_DIRECTORY requests.FIG. 8B represents functionality that overlay optimizer minifilter 311 (or minifilter 311 a) can perform as part of its asynchronous I/O callback routine when a duplicate DCN request is completed. To summarize, the above-described DCN functionality can be implemented to ensure that applications that rely on DCNs will function properly even whenoverlay optimizer 310 is used or deduplication, mirroring, caching etc. are implemented. - Embodiments of the present invention may comprise or utilize special purpose or general-purpose computers including computer hardware, such as, for example, one or more processors and system memory. Embodiments within the scope of the present invention also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system.
- Computer-readable media is categorized into two disjoint categories: computer storage media and transmission media. Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other similarly storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Transmission media include signals and carrier waves.
- Computer-executable instructions comprise, for example, instructions and data which, when executed by a processor, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language or P-Code, or even source code.
- Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like.
- The invention may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices. An example of a distributed system environment is a cloud of networked servers or server resources. Accordingly, the present invention can be hosted in a cloud environment.
- The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/656,900 US11204877B2 (en) | 2019-10-18 | 2019-10-18 | Minimizing data written to disk and enabling directory change notifications in multi-volume filter environments |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/656,900 US11204877B2 (en) | 2019-10-18 | 2019-10-18 | Minimizing data written to disk and enabling directory change notifications in multi-volume filter environments |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210117332A1 true US20210117332A1 (en) | 2021-04-22 |
US11204877B2 US11204877B2 (en) | 2021-12-21 |
Family
ID=75491152
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/656,900 Active 2040-09-14 US11204877B2 (en) | 2019-10-18 | 2019-10-18 | Minimizing data written to disk and enabling directory change notifications in multi-volume filter environments |
Country Status (1)
Country | Link |
---|---|
US (1) | US11204877B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11113393B2 (en) * | 2019-11-04 | 2021-09-07 | Dell Products L.P. | Providing security features in write filter environments |
US20210294910A1 (en) * | 2020-03-18 | 2021-09-23 | Veritas Technologies Llc | Systems and methods for protecting a folder from unauthorized file modification |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11843682B1 (en) * | 2022-08-31 | 2023-12-12 | Adobe Inc. | Prepopulating an edge server cache |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6170050B1 (en) * | 1998-04-22 | 2001-01-02 | Sun Microsystems, Inc. | Length decoder for variable length data |
US8904115B2 (en) * | 2010-09-28 | 2014-12-02 | Texas Instruments Incorporated | Cache with multiple access pipelines |
-
2019
- 2019-10-18 US US16/656,900 patent/US11204877B2/en active Active
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11113393B2 (en) * | 2019-11-04 | 2021-09-07 | Dell Products L.P. | Providing security features in write filter environments |
US20210294910A1 (en) * | 2020-03-18 | 2021-09-23 | Veritas Technologies Llc | Systems and methods for protecting a folder from unauthorized file modification |
US12056251B2 (en) * | 2020-03-18 | 2024-08-06 | Veritas Technologies Llc | Systems and methods for protecting a folder from unauthorized file modification |
Also Published As
Publication number | Publication date |
---|---|
US11204877B2 (en) | 2021-12-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10353636B2 (en) | Write filter with dynamically expandable overlay | |
US11204877B2 (en) | Minimizing data written to disk and enabling directory change notifications in multi-volume filter environments | |
US10621101B2 (en) | Mechanism to free up the overlay of a file-based write filter | |
US20190334984A1 (en) | Data storage system with lun archiving to cloud using volume-to-object translation | |
US11327924B2 (en) | Archiving data sets in a volume in a primary storage in a volume image copy of the volume in a secondary storage | |
US9547655B1 (en) | Filesystem independent snapshot driver | |
US20160132529A1 (en) | Systems and methods for cloud safe storage and data retrieval | |
WO2009123342A1 (en) | Database system, database update method, database, and database update program | |
US11113393B2 (en) | Providing security features in write filter environments | |
US10303562B2 (en) | Using metadata extracted from proxy files to access data stored in secondary storage | |
US9152545B1 (en) | Read-write access in a read-only environment | |
US7516133B2 (en) | Method and apparatus for file replication with a common format | |
US11194765B2 (en) | Accelerating and atomically moving files in an overlay optimizer | |
US8819657B1 (en) | Method and apparatus for maintaining data consistency in a virtualized application during software update installation | |
US10824598B2 (en) | Handling file commit and commit-delete operations in an overlay optimizer | |
US20220011938A1 (en) | System and method for selectively restoring data | |
US10235187B2 (en) | Merging application configurations to enhance multi-layer performance | |
US8255675B1 (en) | System and method for storage management of file system configuration data | |
US11307800B2 (en) | Supporting file exclusions and commits in disk-based write filters | |
US10146467B1 (en) | Method and system for archival load balancing | |
US10657054B2 (en) | Handling renames in an overlay optimizer | |
US10789014B2 (en) | Preventing cross-volume file moves in an overlay optimizer | |
US20240202332A1 (en) | System and Method for Ransomware Scan Using Incremental Data Blocks | |
US11226933B2 (en) | Enhancing write filter features using an auxiliary overlay | |
US11675668B2 (en) | Leveraging a cloud-based object storage to efficiently manage data from a failed backup operation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: DELL PRODUCTS L.P., TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:VAJRAVEL, GOKUL THIRUCHENGODE;BANDAKKA, JYOTHI;KUMAR, ANKIT;REEL/FRAME:050758/0615 Effective date: 20191007 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT, TEXAS Free format text: PATENT SECURITY AGREEMENT (NOTES);ASSIGNORS:DELL PRODUCTS L.P.;EMC IP HOLDING COMPANY LLC;WYSE TECHNOLOGY L.L.C.;AND OTHERS;REEL/FRAME:051302/0528 Effective date: 20191212 |
|
AS | Assignment |
Owner name: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, NORTH CAROLINA Free format text: SECURITY AGREEMENT;ASSIGNORS:DELL PRODUCTS L.P.;EMC IP HOLDING COMPANY LLC;WYSE TECHNOLOGY L.L.C.;AND OTHERS;REEL/FRAME:051449/0728 Effective date: 20191230 |
|
AS | Assignment |
Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., TEXAS Free format text: SECURITY AGREEMENT;ASSIGNORS:CREDANT TECHNOLOGIES INC.;DELL INTERNATIONAL L.L.C.;DELL MARKETING L.P.;AND OTHERS;REEL/FRAME:053546/0001 Effective date: 20200409 |
|
AS | Assignment |
Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT, TEXAS Free format text: SECURITY INTEREST;ASSIGNORS:DELL PRODUCTS L.P.;EMC CORPORATION;EMC IP HOLDING COMPANY LLC;REEL/FRAME:053311/0169 Effective date: 20200603 |
|
AS | Assignment |
Owner name: EMC CORPORATION, MASSACHUSETTS Free format text: RELEASE OF SECURITY INTEREST AT REEL 051449 FRAME 0728;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058002/0010 Effective date: 20211101 Owner name: SECUREWORKS CORP., DELAWARE Free format text: RELEASE OF SECURITY INTEREST AT REEL 051449 FRAME 0728;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058002/0010 Effective date: 20211101 Owner name: WYSE TECHNOLOGY L.L.C., CALIFORNIA Free format text: RELEASE OF SECURITY INTEREST AT REEL 051449 FRAME 0728;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058002/0010 Effective date: 20211101 Owner name: EMC IP HOLDING COMPANY LLC, TEXAS Free format text: RELEASE OF SECURITY INTEREST AT REEL 051449 FRAME 0728;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058002/0010 Effective date: 20211101 Owner name: DELL PRODUCTS L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST AT REEL 051449 FRAME 0728;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058002/0010 Effective date: 20211101 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: EMC IP HOLDING COMPANY LLC, TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (053311/0169);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060438/0742 Effective date: 20220329 Owner name: EMC CORPORATION, MASSACHUSETTS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (053311/0169);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060438/0742 Effective date: 20220329 Owner name: DELL PRODUCTS L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (053311/0169);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060438/0742 Effective date: 20220329 Owner name: SECUREWORKS CORP., DELAWARE Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (051302/0528);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060438/0593 Effective date: 20220329 Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO WYSE TECHNOLOGY L.L.C.), TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (051302/0528);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060438/0593 Effective date: 20220329 Owner name: EMC IP HOLDING COMPANY LLC, TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (051302/0528);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060438/0593 Effective date: 20220329 Owner name: DELL PRODUCTS L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (051302/0528);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060438/0593 Effective date: 20220329 |