WO2023087231A1

WO2023087231A1 - Directory reading system

Info

Publication number: WO2023087231A1
Application number: PCT/CN2021/131620
Authority: WO
Inventors: 霍尔德阿比吉特; 李楚
Original assignee: 华为技术有限公司
Priority date: 2021-11-19
Filing date: 2021-11-19
Publication date: 2023-05-25
Also published as: CN117280333A

Abstract

The present application provides a query system and apparatus. The query system comprises a client and a storage device. The client is used for sending, to the storage device, a read request for requesting to read directory entries in a target path. The storage device is used for obtaining the directory entries in the target path in response to the read request; and compressing some or all of the directory entries in the target path, and sending the compressed directory entries to the client by means of a read response. According to the mode, the storage device compresses the directory entries, and then sends the compressed directory entries, so that more entries can be borne in one read response, the number of roundtrips for the interaction of the directory entries between the storage device and the client can be reduced, the delay of obtaining the complete directory entries by the client is reduced, and the directory reading performance is improved.

Description

A directory reading system

technical field

The present application relates to the field of computer technology, in particular to a directory reading system.

Background technique

A distributed or shared storage system (e.g., a network-attached storage (NAS) system or cluster) typically consists of multiple NAS devices that provide file-based data storage services to other devices (clients, such as an application server).

Clients can use data sharing or file sharing protocols such as Network File System (Network File System, NFS) or Common Internet File System (Common Internet File System, CIFS) to access the file system on the NAS device via the network. If the NAS client requests to access the directory entry under the specified path in the NAS device, the NAS device will return the directory entry of the specified path to the NAS client. However, this may require multiple round trips between the NAS client and the NAS device, because when the directory entries are large, the NAS device can only return some entries to the client each time.

To sum up, as the number of entries in the current directory increases, the cost of reading will become higher and higher, and the overall delay of reading the directory is relatively high.

Contents of the invention

The present application provides a directory reading system, which is used to reduce the time delay for a NAS client to obtain directory entries and improve the performance of directory reading.

Description of drawings

FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of a file system on a NAS device provided by an embodiment of the present application;

FIG. 3 is a schematic flowchart of a method executed by a directory reading system provided in an embodiment of the present application;

FIG. 4 is a schematic flow diagram of obtaining a file handle provided by an embodiment of the present application;

FIG. 5 is a schematic flowchart of an interactive directory entry provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of the format of a read response provided by the embodiment of the present application;

Fig. 7 is a schematic diagram of comparison of different directory reading methods provided in the embodiment of the present application.

Detailed ways

FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application. The system includes a NAS client 10 and a NAS cluster. Wherein, a NAS cluster generally includes a plurality of NAS devices 20 (only one NAS device 20 is shown in FIG. 1 , but the present application is not limited to one NAS device 20 ).

Wherein, the NAS client 10 is a computing device on the user side, which may be a physical machine or a virtual machine. Physical machines include, but are not limited to, desktop computers, servers, laptops, and mobile devices. The NAS client 10 can communicate with any NAS device 20 in the NAS cluster through the network 150 . Wherein, the network 150 generally represents any telecommunications or computer network, including, for example, an enterprise intranet, a wide area network (wide area network, WAN), a local area network (local area network, LAN), a personal area network (personal area network, PAN) or the Internet.

The NAS device 20 may refer to a device connected to the network with a data storage function, on which a file system (see FIG. 2 ) is deployed to provide file-level sharing services. In other words, the NAS device 20 uses file-level Data access and sharing provides storage resources to different clients 10 . FIG. 2 is a schematic diagram of a file system deployed in the NAS device 20 . First, an introduction to the file system:

A file system is a structured form of data file storage and organization. We know that all the data in the computer are 0 and 1, and a series of 01 combinations stored on the hardware media are completely indistinguishable and manageable for us. Therefore, we use the concept of "file" to organize these data, and the data used for the same purpose can be composed of different types of files according to the structure required by different applications. Usually different suffixes are used to refer to different types, and then we give each file a name that is easy to understand and remember. And when there are many files, we group these files according to a certain division method, and each group of files is placed in the same directory (or folder). In addition, there may be a subdirectory (subdirectory or subfolder) under the directory except files, and all files and directories form a tree structure. This tree structure has a dedicated name: File System (File System). There are many types of file systems, the common ones are FAT/FAT32/NTFS of Windows, EXT2/EXT3/EXT4/XFS/BtrFS of Linux, etc. In order to facilitate the search, start from the root node and go down to the file itself, and use special characters for the names of these directories, subdirectories, and files (such as "\" for Windows/DOS, and "/" for Unix-like systems. ) together, such a string of characters is called a file path, such as "/etc/systemd/system.conf" in Linux or "C:\Windows\System32\taskmgr.exe" in Windows. A path is a unique identifier for accessing a specific file. For example, D:\data\file.exe under Windows is the path of a file, which represents the file.exe file under the data directory under the D partition.

The file system is built on the block device. The file system not only records the file path, but also records which blocks form a file, and which blocks record directory/subdirectory information. Different file systems have different organizational structures. In order to facilitate management, a block device such as a hard disk can usually be divided into multiple logical block devices, that is, a hard disk partition (Partition). Conversely, the capacity and performance of a single medium are limited, and multiple physical block devices can be combined into a logical block device through certain technical means, such as various levels of RAID, JBOD, etc. File systems can also be built on top of these logical block devices. In any case, the application server application does not need to care about the specific location of the underlying block device where the file to be accessed is located. It only needs to send the file name/ID of the file to the file system, and the file system will query the file according to the file name/ID. Just the path.

The NAS client 10 can access the file system of the NAS device 20 via the network 150 by using various file access protocols (such as NFS, CIFS or SMB, etc., which are not limited in this embodiment). For example, the client 10 initiates a remote access request to the file system in the NAS device 20 , such as a directory read request, and the client 10 sends the directory read request to the NAS device 20 . In response to the directory read request, the NAS device 20 acquires the directory entries under the specified path requested by the client 10 and returns these directory entries to the client 10 .

In an implementation manner, the NAS client 10 may iteratively call the read directory request, and the number of calls depends on the number of directory entries under the specified path and the configuration of the file access protocol. For example, the directory entries under the specified path include 1000 entries, and the file access protocol configuration returns 100 entries at a time, then for the first read directory request of the NAS client 10, the NAS device 20 first returns the 1000 directory entries to the NAS client 10 The 1-100th directory entries in the entry, then, the NAS client 10 will send the second read directory request, and the NAS device 20 will return the 101-200th directory entries to the NAS client 10, and so on, until all Return done.

When the above method accesses the remote file system, frequent calls and round trips may occur, and the network delay is high.

Therefore, the embodiment of the present application provides a method for reading a directory. By implementing the method, the amount of data transmission in the network can be reduced, the time delay for reading the directory can be reduced, and the access performance to the remote file system can be improved.

In the following, with reference to FIG. 3 , it will be described by taking the directory reading method provided by the embodiment of the present application applied to the system architecture shown in FIG. 1 as an example. The NAS client mentioned in this method may be the NAS client 10 (or a component of the NAS client 10 such as a processor) in FIG. ).

FIG. 3 is a schematic flow chart corresponding to a directory reading method provided by an embodiment of the present application. As shown in Figure 3, the method includes the following steps:

In step 301, the NAS client sends a read request to the NAS device. Correspondingly, the NAS device receives the read request sent by the NAS client.

The read request is used to request to read directory entries of a specified directory in the NAS device.

In step 302, the NAS device obtains the directory entry of the specified directory.

In step 303, the NAS device compresses part or all of the directory entries of the specified directory.

In step 304, the NAS device sends a read response to the NAS client, and the read response includes data after the NAS device compresses some or all of the directory entries of the specified directory. Correspondingly, the NAS client receives the read response.

In the above method, after the NAS device obtains the directory entries, it compresses the entries, so that more entries can be carried in a read response, thereby reducing the number of round trips between the NAS device and the NAS client for interactive directory entries, reducing data The transmission volume reduces the delay for the NAS client to obtain complete directory entries from the NAS device, and improves the performance of directory entry reading.

The complete flow of the directory reading method in the embodiment of the present application is introduced as follows in conjunction with FIG. 4 and FIG. 5:

Those skilled in the art know that in the NFS protocol, a computer device addresses a file or directory via an opaque file handle. In the file system, each file or each directory has its own file handle, and the file handle is the only identification basis for the file or directory. Exemplarily, the NAS client may use the file handle to indicate the specified directory to be read.

Please refer to FIG. 4, which is a schematic flowchart of obtaining a file handle.

Exemplarily, the process may include:

Step 1. An application program on the NAS client calls the opendir() function to pass the path of the specified directory, so as to request to open the specified directory. The libc library intercepts the opendir() call, and queries the corresponding file handle based on the path of the specified directory: in one possible implementation, the NAS client may cache the file handle corresponding to the specified directory locally, then the libc library The file handle can be obtained locally directly. In another possible implementation, referring to step 2 in Figure 4, the NAS client can send the path of the specified directory to the NAS device when the file handle is not found locally. Of course, the NAS client can also Instead of querying locally, the path of the specified directory is directly sent to the NAS device, and the NAS device determines the file handle of the specified directory, and returns the queried file handle to the NAS device 10, and then the file handle will be returned to the NAS client. libc library.

Step 3, the libc library allocates a file descriptor for the file handle, and maintains the mapping relationship between the file handle and the file descriptor, wherein the file descriptor is used for the application, and the Libc library uses the file descriptor and the file descriptor as the The information of the buffer allocated by the application program (hereinafter referred to as the application buffer for short) is returned to the application program. The information of the application buffer may include a pointer for indicating the address of the application buffer and/or information for indicating the length of the application buffer. In this application, the application buffer is used to carry the directory entries of the specified directory that the application program requests to read. The application buffer length may be configured by the user or allocated by the NAS client, which is not limited in this embodiment of the present application.

It is worth noting that the libc library is a new function library provided by the embodiment of this application, which can be used to intercept the aforementioned opendir() function and the subsequent readdir() function, so that the original function library can be Completing the query of the directory entries in the embodiment of the present application makes the technical solution provided by the present application have better portability and compatibility.

Referring to FIG. 5, FIG. 5 is a schematic flowchart of querying directory entries. FIG. 5 is a subsequent query process executed on the basis of FIG. 4. Exemplarily, the process may include:

Step 4: On the NAS client, after the application obtains the file descriptor corresponding to the specified directory and the application buffer information, it calls the readdir() function to request to read the directory entries of the specified directory, and calls the readdir() function to pass the file descriptor and application buffer information. Correspondingly, the libc library gets the readdir() call.

In step 5, the NAS client sends a read request to the NAS device based on the readdir() call, and correspondingly, the NAS device receives the read request.

Exemplarily, the process of sending a read request includes: the libc library determines the file handle corresponding to the file descriptor passed by the readdir() call according to the previously maintained mapping relationship between the file descriptor and the file handle, and sends the file handle and readdir () calls are passed to the VFS together, and the VFS sends the readdir call to the corresponding file system client, such as the FS client. After that, in one embodiment, the FS client sends the read request to the NAS device 20 through the RPC client. Exemplarily, the read request may be a message conforming to the frame format of the RPC protocol encapsulated according to the information transmitted by the readdir() call. Exemplarily, the read request includes, but is not limited to, one or more of the following parameters: allocated buffer length, cookie.

1) The length of the first storage area: used to indicate the size of the storage area (that is, the application buffer) allocated by the NAS client for this read call. The NAS device returns the corresponding amount of data according to the size of the storage area, which can also be understood as the upper limit of the storage space occupied by the data part in a read response. For example, if the size of the storage area allocated by the NAS client is 64KB, the NAS device returns The amount of data no larger than 64kB is sent to the NAS client. In this application, the information in the field used to indicate the length of the first storage area in the read request may be referred to as first indication information. For ease of description, the first storage area is referred to as an allocation buffer as follows.

2) cookie: it is both an input parameter and an output parameter of the NAS device. Cookie is transparent data for NAS client, it does not need to be parsed by NAS client, and NAS client does not care about its content. For directory entries, you can first mark the location of the returned directory entry. Usually, the cookie contains the starting address of the next entry of the returned directory entry (such as the offset of the entry). The NAS client receives the cookie Afterwards, it will be returned intact to the NAS device for parsing by the NAS device, which is used to instruct the NAS device where to start reading next time. It should be understood that the first read request for a specified directory may not include a cookie or the cookie may be empty.

In addition, the read request may also carry the file handle of the specified directory, or the NAS client may separately send the file handle of the specified directory to the NAS device, that is, the read request does not include the file handle. It should be noted that the foregoing manner is only an optional implementation manner based on the NFS protocol, and this embodiment of the present application is not limited to the manner in which the NAS client indicates a specified directory through a file handle. For example, the NAS device 10 can send a read request carrying the path of the specified directory to the NAS device, or the NAS client can send the read request and the path of the specified directory to the NAS device, etc. Any method that can indicate the specified directory is applicable to this document. Application example.

Step 6: In response to the read request, the NAS device queries the specified directory according to the file handle sent by the NAS client, and obtains the directory entry of the specified directory.

In step 7, the NAS device generates a read response based on.

The NAS device generates a read response according to the obtained directory entry, and returns part or all of the directory entries to the NAS client through the read response.

Referring to FIG. 6, for example, the read response includes a header and a data part, and each part is introduced as follows:

One, the head.

The header includes, but is not limited to, one or more of the following:

1) cookie: As mentioned above, it is both an input parameter and an output parameter of the NAS device. When used as an input parameter, the NAS device determines the starting address of the directory entry to be read according to the cookie. As an output parameter, the NAS device uses the cooker to record the starting address of the directory entry to be read next time. For details, refer to the introduction above, and will not repeat the description here.

2) end-of-file flag (end-of-file): an output parameter of the NAS device, used to indicate whether the directory entries of the specified directory are completely returned. When the NAS device returns all the entries of the specified directory to the NAS client, this field will be set. The field of the end-of-file flag can include 1 bit, and the different values of the 1 bit indicate whether the returned directory entry has reached the directory entry. End, for example, when the value is 0, it means that it has not been reached, and when the value is 1, it means that the end of the directory entry has been reached. It can be understood that usually the NAS device will set this field in the read response of the last entry of the returned directory entry. After the NAS client receives the read response indicating that the end of the directory entry has been reached, it will not initiate a read request for the same directory to the NAS device 10 .

3) Valid data size (valid data size): It is an output parameter of the NAS device, and refers to the length of the valid data carried in the data part of the read response.

4) The length of the second storage area: it is an output parameter of the NAS device. For the convenience of explanation, the second storage area is called shadow buffer size (shadow buffer size) as follows. The shadow buffer is the storage area on the NAS client used to host the decompressed directory entries. The size of the shadow buffer is determined based on the compression ratio of the directory entries compressed by the NAS device. For example, if the valid data in the read response are all compressed directory entries, the size is 64KB, and the compression ratio is 2, then the size of the shadow buffer is It is 128KB. That is to say, the size of the shadow buffer is equal to the size of the valid data in the read response before compression. In this application, the information on the field indicating the length of the shadow buffer in the read response is referred to as second indication information.

2. Data part:

The data part, the effective data part output by the NAS device, is used to carry the directory entries of the specified directory requested by the NAS client to read, and the maximum storage space occupied by the data part is the size of the allocated buffer.

It should be noted that the read requests and read responses listed above are only illustrative, and the embodiment of the present application does not limit the parameters or formats included in the read requests and read responses. For example, in practical applications, the read response may also include the number of entries (the number of entries carried by the data part), or when the number of directory entries in the specified directory is small and does not need to be compressed, the second indication information may not be carried in the read response. etc. In addition, the names of the above parameters are only examples for convenience of description, and may have different names in different scenarios, which is not limited in this embodiment of the present application.

In this application, the NAS device will adopt different compression strategies according to the size of the allocation buffer and the size of the directory entries of the specified directory. The compression strategy refers to the way of filling the directory entries into the read response. Compression strategies include full compression, partial compression, and no compression.

Exemplarily, the NAS device determines the size of the allocation buffer according to the first indication information of the read request, and obtains the size of the full directory entry of the specified directory, and then compares the two, if the size of the full directory entry is not larger than the size of the allocation buffer size, the read response can be generated in an uncompressed manner. If the size of the full directory entry is larger than the size of the allocated buffer, the read response can be generated fully or partially compressed.

A few specific scenarios are listed below to introduce the above compression strategy in detail. For the convenience of explanation, the allocated allocation buffer size is 64KB as an example to illustrate:

Scenario 1. Partial compression/full compression

If the size of the full directory entry of the specified directory is greater than the size of the allocation buffer, the NAS device can fill the data portion of the read response through one or more compressions. For ease of understanding, the data portion is referred to as the allocation buffer as follows:

(1) The first compression:

The NAS device starts from the head of the complete directory entry and fills up the allocation buffer of the read response with partial directory entries. Assuming that the size of the allocation buffer is 64KB, then the size of this part of the directory entry filled for the first time is also 64KB, (such as the complete The entry corresponding to the 0-64KB of the directory entry). After the filling is completed, the NAS device compresses the 64KB directory entries. Assuming that the compression rate is 2, the storage space occupied by these directory entries after compression is 32KB.

Wherein, compression ratio CR=data size So before compression/data size Sc after compression; as mentioned above, the size of the entry before compression is 64KB, the compression ratio is 2, and the size of the entry after compression is 32KB. It should be understood that the compression rate is related to the compression algorithm and the data to be compressed. The embodiment of this application does not limit the compression algorithm, such as Shannon-Fano algorithm, Huffman coding, arithmetic coding, LZ77/LZ78 coding, etc., any existing algorithm that can compress data and possible applications in the future All compression algorithms are applicable to this embodiment of the application.

(2) After the first compression, a compressed data block (chunk) (hereinafter referred to as the compressed block) is obtained. The first compressed block can be called the compressed block 1. As mentioned above, the compressed block 1 The size is 32KB. Since the size of the allocation buffer is 64KB, the remaining available space in the allocation buffer is close to 32KB. The closeness here refers to slightly less than 32KB, because the NAS device can also generate header information for the compressed block 1 , this part of the header information will occupy part of the space in the allocation buffer, therefore, the remaining storage space in the allocation buffer is actually less than 32KB. It should be understood that the NAS device can generate a header information for each compressed block, The header information is not generated only for compressed block 1. For ease of illustration, the header of each compressed block is ignored here, assuming that the remaining storage space is 32KB.

Exemplarily, if the remaining unfilled directory entries, that is, the directory entries corresponding to the 64 KB of the complete directory entry to the end of the directory can be completely placed in the remaining storage space of the allocation buffer, then this part of the remaining directory entries will be Does not need to be compressed and can be placed directly in the allocated buffer of the read response.

In another example, if the remaining unfilled directory entries are larger than the remaining storage space of the allocation buffer, then the NAS device continues to sequentially select a directory entry that is as large as the remaining storage space from the position of the directory entry selected last time, and fills in the directory entry again. Full allocation buffer, if the remaining storage space is 32KB, then sequentially take 32KB (such as the entry corresponding to the 64KB-96KB of the complete directory entry) to fill the remaining storage space, and then compress the newly filled 32KB directory entry , again to get a new compressed block, called compressed block 2. Assuming that the size of the compressed block 2 is 20KB, the remaining storage space of the allocation buffer after this round of compression is 12KB (64KB-32KB-20KB), and the NAS device continues to judge whether the remaining directory entries are greater than 12KB. Similarly, if not greater than , the remaining directory entries are filled directly into the allocation buffer, ie no compression is required. If it is larger, continue to sequentially fetch 12KB entries for filling, then compress the 12KB, and so on, until the end filling condition is satisfied.

Wherein, the end filling condition includes but not limited to: (1) There is no remaining storage space in the allocation buffer, that is, it has been filled to the end of the allocation buffer. (2) The remaining storage space in the allocation buffer satisfies the set threshold (such as the first threshold). If the first threshold is 2KB, that is, when the allocation buffer is 64KB and the effective data length reaches 62KB, then no Then further filling compression, the end of the filling. It is worth noting that the effective data length of 62KB here means that the compressed data reaches 62KB, and the allocation buffer can be filled before compression, see the example above. (3) There are no remaining directory entries, that is, the complete directory entries of the specified directory have all been filled.

So far, the data part of the read response (allocation buffer) has been filled, and the NAS device also needs to generate relevant parameters in the read response header according to the effective data length, such as the effective data length and the shadow buffer length; where the effective data length is For the storage space occupied by the data in the data part of the read response, it can be understood that when the allocation buffer is full, the effective data length is the length of the allocation buffer, such as 64KB.

As mentioned above, the shadow buffer is used to carry the decompressed directory entries in the NAS client, so the length of the shadow buffer is actually the actual length of all directory entries filled into the allocation buffer by the NAS device before compression. Therefore, the length of the shadow buffer is related to the compression rate and the effective data length. It is worth noting that there may be uncompressed data in the allocated buffer, and the length of the shadow buffer is satisfied by the length of the compressed data * the compression rate + the length of the uncompressed data ; Length of compressed data + length of uncompressed data = effective data length. For example, the effective data length is 64KB, wherein the compressed data length and the uncompressed data length are both 32KB, and the compression rate is 2, then the shadow buffer length=32KB*2+32KB=96KB. For another example, if the effective data length is 64KB, and the effective data are all compressed data, then the shadow buffer length is 64KB*2=128KB.

In this way, the read response generation is completed.

Scenario two, no compression.

It can be understood that if the size of the full directory entry of the specified directory is less than or equal to the size of the allocation buffer, for example, when the number of full directory entries is small, the NAS device can directly place the full directory entry in the allocation buffer of the read response, that is, The directory entry requested by the user can be completely returned through a read response, so the NAS device does not need to compress the directory entry.

In summary, a partial compression strategy is implemented, and there are both compressed and uncompressed data in the allocation buffer. The full compression strategy is implemented, and the directory entries in the allocation buffer are all compressed data. If the uncompressed policy is enforced, the allocation buffer is filled with actual uncompressed directory entries.

Execute full compression or partial compression strategy, the data part of the read response may have multiple data blocks, including compressed blocks and uncompressed blocks, the embodiment of this application can generate a compression header for each data block, the compression header includes The indication information used to indicate whether the data in the data block is compressed (for example, referred to as third indication information), the third indication may be used by the NAS client to determine whether the data in the data block needs to be decompressed. Optionally, the compression header may also include other information, such as compression algorithm, verification information (such as checksum), block length information, and the like.

In step 8, the NAS device sends a read response to the NAS client, and correspondingly, the NAS client receives the read response sent by the NAS device.

Step 9. The data abstraction layer of the NAS client detects whether there is a compressed block in the read response, and if it exists, allocates a shadow buffer for carrying decompressed data. This process may include: the data abstraction layer The field determines the length of the shadow buffer, and allocates a shadow buffer of equal length for the read response, and then writes the pointer used to indicate the address of the shadow buffer into the corresponding field of the read response header. This field may be reserved by the NAS device, or generated by the NAS client itself.

Then the data abstraction layer will carry the read response and send it to the decompression module to decompress the data in the read response. Alternatively, decompression can be performed directly by the data abstraction layer. The data abstraction layer and the decompression module here can be a software module or a hardware module, or a combination of software modules and hardware modules, which is not limited in this application. If the data abstraction layer directly decompresses, the field for indicating the shadow buffer pointer may not be added in the read response, and the data abstraction layer maintains the corresponding relationship between the pointer of the shadow buffer and the read response.

The NAS client can determine whether the data block needs to be decompressed according to the information in the compression header of each data block. For example, when the third indication information indicates that the data in the data block is compressed data, the NAS client determines that the data The data in the block needs to be decompressed. Optionally, when the compression header includes compression information such as compression algorithm and checksum, the NAS client can use the corresponding decompression algorithm to decompress the data in the data block, or use the checksum to decompress the data in the data block. The data is verified, and the data block is decompressed under the condition of ensuring the integrity and accuracy of the data.

Decompression strategies include serial decompression and parallel decompression. The serial decompression process includes: according to the arrangement position of the compressed blocks, take a compressed block in order each time, decompress the compressed data in it, obtain multiple directory entries, and fill them in the shadow buffer sequentially. After a compressed block is decompressed, the next compressed block is sequentially decompressed from the read response, and the decompressed directory entries are filled into the shadow buffer, and so on, until all the directory entries of all data blocks are Fill up to the shadow buffer. It should be noted that some data blocks are uncompressed and do not need to be decompressed.

Parallel decompression, obtain multiple compressed blocks for decompression at the same time, and fill the decompressed directory entries into the shadow buffer in order according to the order of the data blocks, ensuring that the directory entries in the shadow buffer are arranged in the order of the actual entries .

If the third indication information indicates that the data in the data block is uncompressed, then the NAS client can directly put the data in the data block into the shadow buffer.

Continuing to refer to FIG. 6, there are sequentially decompressed directory entries stored in the shadow buffer. Exemplarily, each directory entry includes but is not limited to one or more of the following information: entry type (D_type), entry The inode (D_INO), entry length (D_RECLEN), entry offset (D_OFFSET), entry name (D_NAME), etc. It should be noted that the information included in the above directory entries and the ordering of the information are only examples, which are not limited in this embodiment of the present application.

Step 10, return the directory entry to the application.

Write the directory entries of the shadow buffer sequentially to the application buffer.

Referring to FIG. 7 , FIG. 7 is a schematic diagram showing the interactive process of reading directories without compression and with compression. The uncompressed method may require multiple round trips between the NAS client and the NAS server to exchange complete directory entries, which takes a long time and consumes a lot of network bandwidth. The compression method is the method of the embodiment of the present application, that is, the NAS device compresses the directory entries and sends them to the NAS device 10. Those skilled in the art have determined through experiments that this can be achieved under a considerable workload (about 1 million files per directory). directory entries) about a 2x performance gain. Some standard compression techniques such as LZ4 can provide a compression ratio of about 2, and the time required to compress and decompress a 64KB buffer is less than 100μs, which is less than the 10% overhead of the total response time of a readdir() request. The directory reading method in the embodiment of the present application can significantly reduce the number of round trips, shorten the delay, and improve the directory reading performance.

Claims

A query system, characterized in that it includes a client and a storage device:

The client is configured to send a read request to the storage device, and the read request is used to request to read the directory entry of the target path;

The storage device is configured to respond to the read request and obtain a directory entry of the target path;

The storage device is further configured to send a read response to the client, the read response includes a data part, and the data part includes data after the storage device compresses some or all of the directory entries;

The client is configured to receive the read response sent by the storage device.
The system according to claim 1, wherein the client is further configured to decompress the compressed data in the read response to obtain part or all of the directory entries.
The system according to claim 1 or 2, wherein the read request includes first indication information, or the read request includes the first indication information and a cookie;

The first indication information is used to indicate the maximum value of the storage area occupied by the data part in the read response; the cookie is used to indicate the starting address of the entry to be read in the directory entries of the specified directory.
The system according to claim 3, wherein the data portion in the read response includes at least a first data block and a second data block;

The data in the first data block is obtained by compressing the first data by the storage device, and the first data includes a first partial entry of the directory entry; the size of the first partial entry is not larger than the data The maximum value of the storage area occupied by the part;

The data in the second data block is obtained by compressing the second data by the storage device, the second data includes a second part of the directory entry, and the second part of the entry is in the first part after the entry; or the second data is a remaining entry in the directory entry except the first data.
The system of claim 4, wherein the first data block includes a block header; the header includes compression information for data in the first data block, and the compression information includes among the following parameters One or more of:

The third indication information, compression algorithm, and checksum used to indicate whether the data in the first data block is compressed.
The system according to any one of claims 1-5, wherein the read response further includes a header, and the header includes one or more of the following:

Second indication information, end-of-file mark, cookie, effective data length;

Wherein, the second indication information is used to indicate the size of the storage area, and the storage area is determined by the storage device according to the total length of the data part of the read response before compression; the storage area is used to carry the storing some or all of the directory entries after the client decompresses the data in the data part of the read response;

The end-of-file flag is used to indicate whether the directory entry of the specified directory has been completely returned;

The valid data length is used to indicate the length of the data part in the read response.
The system according to claim 6, wherein the client is further configured to determine the storage area according to the second indication information, and obtain part or all of the directory entries decompressed based on the read response stored in the storage area.