WO2017107948A1 - 文件的写聚合、读聚合方法及系统和客户端 - Google Patents

文件的写聚合、读聚合方法及系统和客户端 Download PDF

Info

Publication number
WO2017107948A1
WO2017107948A1 PCT/CN2016/111473 CN2016111473W WO2017107948A1 WO 2017107948 A1 WO2017107948 A1 WO 2017107948A1 CN 2016111473 W CN2016111473 W CN 2016111473W WO 2017107948 A1 WO2017107948 A1 WO 2017107948A1
Authority
WO
WIPO (PCT)
Prior art keywords
small file
client
file
small
aggregation
Prior art date
Application number
PCT/CN2016/111473
Other languages
English (en)
French (fr)
Inventor
杨洪章
王志坤
罗圣美
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2017107948A1 publication Critical patent/WO2017107948A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G06F16/285—Clustering or classification

Definitions

  • the present disclosure relates to the field of distributed storage technologies, for example, to a file aggregation, read aggregation method, and system.
  • the present disclosure provides a file write aggregation, read aggregation method and system, which reduce the time of reading and writing during write aggregation and read aggregation of small files.
  • a file write aggregation method provided by the present disclosure includes:
  • the client initiates a write operation of a small file
  • the client calculates an aggregation location of the small file in the object storage device OSD according to a global unique number assigned by the metadata server for the small file and a storage area unique number;
  • the client sends a small file aggregation request carrying the aggregation location to the object storage device OSD; wherein the small file refers to a file whose size is less than a threshold.
  • the global unique number is an inode number ino
  • the storage area unique number is an object number ono
  • the client initiates a write operation of the small file, including:
  • the client writes the content of the small file to a local cache and sends a request to the metadata server to create metadata for the small file.
  • the global unique number assigned by the metadata server to the small file is allocated according to a timing increasing allocation principle
  • the small files consecutive to the N globally unique numbers aggregated in the aggregation location are prejudged as the same object, and the N small files of the same object are respectively assigned different storage area unique numbers; wherein N is less than or equal to A positive integer that presets the number of aggregates.
  • the client sends a small file aggregation request carrying the aggregation location to the object storage device OSD
  • the small file is written into a number.
  • where. - is a negative sign.
  • the present disclosure also provides a file write aggregation system, including:
  • the client is configured to initiate a write operation of a small file
  • the metadata server is configured to allocate a global unique number and a storage area unique number to the small file
  • the client is further configured to calculate, according to the global unique number and the storage area unique number, an aggregation location of the small file in the object storage device OSD;
  • the object storage device OSD is arranged to aggregate small files according to the aggregated location.
  • the global unique number is an inode number ino
  • the storage area unique number is an object number ono
  • the metadata server is further configured to:
  • the small files consecutive to the N globally unique numbers aggregated in the aggregation location are prejudged as the same object, and the N small files of the same object are respectively assigned different storage area unique numbers; wherein N is less than or equal to A positive integer that presets the number of aggregates.
  • the object storage device OSD is further configured to:
  • the small file write aggregation request detecting a storage area unique number of the N globally consecutive consecutive small files according to the small file write aggregation request, and when the storage area of the small file is uniquely numbered, the small file is Writes the unique numbered area of the first-storage area in the object numbered as the aggregation location; where - is the negative number; the small file write aggregation request contains the small file, the aggregate location, the global unique number, and the storage area unique number; N is A positive integer less than or equal to the preset number of aggregates.
  • the present disclosure provides a non-transitory computer readable storage medium storing computer executable instructions that are set to any of the above file write aggregation methods.
  • the present disclosure also provides a client, including:
  • At least one processor At least one processor
  • the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to:
  • the client initiates a write request operation for a small file
  • the client calculates an aggregation location of the small file in the object storage device OSD according to a global unique number assigned by the metadata server for the small file and a storage area unique number;
  • the client sends a small file aggregation request carrying the aggregation location to the object storage device OSD; wherein the small file refers to a file whose size is less than a threshold.
  • the disclosure initiates a write operation of a small file by the client, and the client calculates an aggregate location of the small file in the object storage device OSD according to the global unique number assigned to the small file by the metadata server and the storage area unique number, and the client is oriented toward the object.
  • the storage device OSD sends a small file aggregation request carrying the aggregation location, where the small file refers to a file whose size is less than a threshold. Since the small file storage location is determined by calculating the global unique number and the storage area unique number, and writing the small file according to the small file storage location, there is no need to additionally create a mapping table of the small file, thereby Reduced write time for small files.
  • a file read aggregation method provided by the present disclosure includes:
  • the client initiates a read request for a small file
  • the small file is obtained from the object storage device OSD according to the aggregation location.
  • the global unique number is an inode number ino
  • the storage area unique number is an object number ono
  • the method further includes:
  • the global unique number and the storage area unique number of the small file are obtained from the metadata server.
  • the obtaining the small file from the object storage device OSD according to the aggregation location includes:
  • the present disclosure also provides a client, including:
  • Read request module set to initiate a read request for a small file
  • a number obtaining module configured to obtain a global unique number and a storage area unique number of the small file from a metadata server
  • a calculating module configured to calculate an aggregated location of the small file in the object storage device OSD according to the global unique number and the storage area unique number;
  • a small file acquisition module configured to acquire the small file from the object storage device OSD according to the aggregation location.
  • the global unique number is an inode number ino
  • the storage area unique number is an object number ono
  • the client further includes:
  • a detecting module configured to detect whether the small file exists in a cache of the client
  • the small file obtaining module is further configured to acquire the small file if the small file exists in the cache of the client;
  • the number obtaining module is further configured to: if the small file does not exist in the cache of the client, obtain a global unique number and a storage area unique number of the small file from the metadata server.
  • the small file obtaining module includes:
  • a sending unit configured to send a read aggregation request to the object storage device OSD;
  • a receiving unit configured to receive all small files in the aggregated location sent by the object storage device OSD and save;
  • the obtaining unit is configured to acquire a small file in the corresponding area of the absolute value of the unique number of the storage area.
  • the present disclosure also provides a non-transitory computer readable storage medium storing computer executable instructions arranged to perform a file read aggregation method.
  • the present disclosure also provides a client, including:
  • At least one processor At least one processor
  • the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to:
  • the small file is obtained from the object storage device OSD according to the aggregation location.
  • the disclosure initiates a read request of a small file by the client, obtains a global unique number of the small file and a unique number of the storage area from the metadata server according to the read request of the small file, and calculates a small file according to the global unique number and the unique number of the storage area.
  • the aggregate location in the object storage device OSD causes the client to retrieve small files from the OSD based on the aggregated location. Since the small file storage location is determined by calculating the global unique number and the storage area unique number, and reading the small file according to the small file storage location, it is not necessary to read the small file by checking the mapping table. , which reduces the reading time of small files.
  • FIG. 1 is a schematic flow chart of a first embodiment of a file write aggregation method
  • FIG. 2 is a schematic diagram of a refinement process in which a metadata server allocates a global unique number and a unique number of a storage area for a small file in the second embodiment of the file aggregation method;
  • FIG. 3 is a schematic diagram of a refinement process of an object storage device OSD aggregating small files according to an aggregation location in a third embodiment of the file aggregation method;
  • FIG. 4 is a schematic diagram of functional modules of a first embodiment of a file write aggregation system
  • FIG. 5 is a schematic flowchart diagram of a first embodiment of a file read aggregation method
  • FIG. 6 is a schematic flow chart of a second embodiment of a file read aggregation method
  • FIG. 7 is a schematic diagram of a refinement process of acquiring a small file according to an aggregation location in a third embodiment of a file read aggregation method
  • FIG. 8 is a schematic diagram of functional modules of a first embodiment of a client
  • FIG. 9 is a schematic diagram of functional modules of a second embodiment of a client.
  • FIG. 10 is a schematic diagram of a refinement function module of a file acquisition module in a third embodiment of a client.
  • Figure 11 is a schematic diagram of the hardware structure of the client.
  • FIG. 1 is a schematic flowchart of a first embodiment of a file write aggregation method.
  • the file write aggregation method of the present disclosure can be applied to small files, which are files whose size is less than a threshold (for example, 1 MB).
  • step 110 the client initiates a write operation of the small file.
  • the file size of a small file may be less than 64 KB or 1 MB, and the small file may also be a file smaller than the file system stripe size.
  • the client initiates a small file write operation The client writes the contents of the small file to the client's local cache. After the client writes the contents of the small file to the client's local cache, the client sends a metadata request to the metadata server to create the small files.
  • the client initiates a write operation of the small file, including writing the content of the small file into the cache, creating a metadata request for the small file, and writing the content in the cache to the storage device according to the metadata.
  • the metadata server assigns a globally unique number and a storage area unique number to the small file.
  • the global unique number may be an inode number (ino)
  • the storage area unique number may be an object number (ono).
  • the metadata server will assign the global unique number and the storage area unique number of the small file in the object one by one to the small file, and then return the global unique number and the unique number of the storage area to the small file.
  • the client receives the global unique number and the unique number of the storage area.
  • the object is a basic unit for constructing a data storage in a storage system based on an Object-based Storage Device (OSD).
  • An object is a combination of file data and a set of attribute information (Meta Data), and the object storage structure includes an object. , object storage device, metadata server, client of the object storage system.
  • the client calculates an aggregate location of the small file in the object storage device OSD based on the global unique number and the storage area unique number.
  • the aggregation location may be determined by an Object Identifier (OID), and the object identifier may be a 64-bit data.
  • OID Object Identifier
  • 1, calculate the aggregation location of the small file in the object storage device OSD, that is, the number of the small file object ((the file number of the small file + the area number of the small file in the object +1) ⁇ 32)
  • ⁇ 32 indicates that 32 bits are shifted to the left, and
  • the aggregation location is a unique number of the object in the global object after the small file is aggregated into an object.
  • HASH generalizes the computer Common data distribution algorithms in the field, including but not limited to: squared intermediate value, modular operation remainder, distributed hash table (DHT) algorithm, CRUSH (Controlled Replication Under Scalable Hashing) algorithm.
  • DHT distributed hash table
  • CRUSH Controlled Replication Under Scalable Hashing
  • step 140 the object storage device OSD aggregates the small files according to the aggregated location.
  • the object storage device After receiving the aggregation location of the small file, the object storage device aggregates the small files having the same aggregation location into the same object and writes them into different regions. At the same time, after the object storage device OSD aggregates the small file, the object storage device sends a small file aggregation write completion instruction to the client, and after receiving the small file aggregation write completion instruction, the client sends the small message to the client.
  • the data server sends a metadata update request update instruction, notifies the metadata server to update the metadata information of the small file, updates the file number of the small file, the area number, the creation time of the small file, the modification time of the small file, the access right, File size.
  • the client initiates a write operation of the small file, and the global unique number and the storage area unique number allocated by the metadata server for the small file are calculated by the client according to the global unique number and the unique number of the storage area.
  • the aggregation location in the storage device OSD is such that the object storage device OSD aggregates the small files according to the aggregation location. Since the small file storage location is determined by calculating the global unique number and the storage area unique number, and writing the small file according to the small file storage location, there is no need to additionally create a mapping table of the small file, thereby reducing the small The time the file was written.
  • the foregoing step 120 includes steps 1210-1220.
  • step 1210 the metadata server assigns a globally unique number to the small file in a timed incremental allocation principle.
  • the client sends a request for creating the small file metadata to the metadata server, and the metadata server generates a request according to the small file metadata.
  • Assigning a global unique number to the small files one by one the order of assigning the global unique numbers of the small files may be allocated in an order of increasing order, and the global unique number preset in the metadata server is assigned to the earliest small file.
  • the created small file, the globally unique number of the subsequently created small file is incremented by 1 based on the preset unique file number.
  • the metadata server allocates the global unique number of the small file according to the creation time of the small file, and the later the creation time of the small file, the larger the file number to be assigned.
  • small files include B, C, D, E, F, G, H, I, J, K, L, M, Their creation time increases in turn, and the file number 2015 preset in the metadata server is assigned to the earliest created small file B.
  • the file number of the small file C is 2016, and the file number of the small file D is 2017, small.
  • the file numbers of the files E, F, G, H, I, J, K, L, and M can be analogized by analogy.
  • step 1220 the N globally consecutive consecutive small files aggregated in the aggregation location are pre-determined as the same object, and the N small files of the same object are respectively assigned different storage area unique numbers.
  • N is a positive integer less than or equal to the preset number of aggregations.
  • An object has a memory capacity of 4 MB. Up to 4 small files are aggregated in one object, and at least one small file is aggregated. Among them, four storage areas are allocated in one object. After the metadata server allocates the global unique number of the small file one by one for the small file, the metadata server predicts four small files with consecutive global unique numbers as the same object, for example, (B, C, D, E), (F, G, H, I), (J, K, L, M) 12 small file decisions are combined in three objects.
  • the unaggregated small files may be aggregated into the same object, and the storage area is allocated to the objects.
  • Unique number The allocation principle may be that the small file storage area in the Kth area is uniquely numbered -k. For example, if small files B, C, D, E, F, G, H, I, J, and K are aggregated, then (B, C, D, E), (F, G, H, I), ( J, K) 10 small files are aggregated into three objects, and the size of the object containing 2 small files (J, K) is 4MB.
  • the metadata server allocates the area number in the object one by one for the small file.
  • the area number of the small file B in the object is -1
  • the area number of the small file C in the object is -2
  • the area number of the small file D in the object is -3
  • the area number of the small file E in the object is -4
  • the area of the small file F in the object is numbered -1
  • the area of the small file G in the object is -2
  • the area of the small file H in the object is -3
  • the small file I is in the object.
  • the area number is -4, the area number of the small file J in the object is -1, the area number of the small file K in the object is -2, the area number of the small file L in the object is -3, and the small file M is in the object.
  • the area number in is -4.
  • the metadata server can create metadata for the above small files one by one, such as the creation time, access rights, last modification time, and file size of the small files.
  • the global unique number is assigned to the small file by the metadata server according to the principle of increasing the timing, and the storage area unique number is allocated for the N globally consecutive consecutive small files that are prejudged to be the same object. Since the global unique number of the small file and the unique number of the storage area are allocated according to the principle of increasing the time series, and are sequentially aggregated in the object according to the area number, it is not necessary to additionally create these small files. A mapping table of files to improve the efficiency of small files.
  • a third embodiment of the write aggregation method of the file is proposed, with reference to FIG. 3.
  • step 1410 the object storage device OSD receives the small file write aggregation request sent by the client.
  • the client after the client determines the OSD to be written by the client, the client sends the small file write aggregation request to the OSD, and the OSD receives the small file write aggregation request sent by the client.
  • the object storage device OSD detects the storage area unique number of the N globally unique number consecutive small files one by one according to the small file write aggregation request, and writes the small file number to when the unique number of the small file storage area is negative.
  • the first storage area in the object of the aggregation location is uniquely numbered, where - is a negative number and N is a positive integer less than or equal to the preset aggregation number.
  • the OSD receives a small file write aggregation request sent by the client, and the OSD detects whether the storage area unique number of the received small file is negative one by one, and determines the OSD after the unique number of the storage area of the received small file is negative.
  • the OSD small file aggregation module is notified to perform a small file aggregation write operation on the small file.
  • the small file aggregation module aggregates consecutive globally unique numbered small files in the same object; or, according to the absolute value of the unique number of the storage area, determines that the small file is in the storage area of the object, and writes the small file into the small file.
  • the absolute value of the storage area of the file is the absolute value corresponding to the storage area.
  • the object storage device OSD receives the small file write aggregation request sent by the client, and detects the storage area unique number of the N global unique number consecutive small files one by one according to the small file write aggregation request, and uniquely numbers in the storage area of the small file. When the number is negative, the small file is written into the unique numbered area of the first storage area in the object numbered as the aggregation position, so that the small file storage location is oriented and unique, thereby reducing the reading and writing time of the small file.
  • the present disclosure optionally provides a write aggregation system for files.
  • FIG. 4 is a schematic diagram of functional modules of a first embodiment of a file write aggregation system.
  • the system may include: a client 10, a metadata server 20, and an object storage device OSD 30.
  • the client 10 is arranged to initiate a write operation of a small file.
  • the file size of a small file may be less than 1 MB
  • the client initiates a small file write operation, and the client writes the content of the small file into the local cache of the client.
  • the client After the client writes the contents of the small file to the client's local cache, the client sends the metadata to the metadata server. Create metadata requests for these small files.
  • the metadata server 20 is arranged to assign a globally unique number and a storage area unique number to the small file.
  • the metadata server After receiving the metadata request, the metadata server will assign the global unique number and the storage area unique number of the small file in the object one by one to the small file, and transmit the global unique number and the unique number of the storage area to the customer. End, the client receives the global unique number and the unique number of the storage area.
  • the client 10 is further arranged to calculate an aggregate location of the small file in the object storage device OSD based on the global unique number and the storage area unique number.
  • the global unique number may be an inode number ino
  • the storage area unique number may be an object number ono
  • the client passes the aggregation.
  • Position calculation formula aggregation position ((ino+ono+1) ⁇ 32)
  • ⁇ 32 indicates that 32 bits are shifted to the left
  • Small files in any storage area in the same object are calculated by the same aggregation location.
  • HASH generalizes the common data distribution algorithms common in the computer field, including but not limited to: squared intermediate value, modular operation remainder, DHT algorithm, CRUSH algorithm.
  • the object storage device OSD 30 is arranged to aggregate small files according to the aggregated location.
  • the object storage device After receiving the aggregation location of the small file, the object storage device aggregates the small files having the same aggregation location into the same object and writes them into different regions. At the same time, after the object storage device OSD aggregates the small file, the object storage device sends a small file aggregation write completion instruction to the client, and after receiving the small file aggregation write completion instruction, the client sends the small message to the client.
  • the data server sends a metadata update request update instruction, notifies the metadata server to update the metadata information of the small file, updates the file number of the small file, the area number, the creation time of the small file, the modification time of the small file, the access right, File size.
  • the metadata server 20 is also arranged to assign a globally unique number to the small file in a time-increasing allocation principle.
  • the client sends a request for creating the small file metadata to the metadata server, and the metadata server generates a request according to the small file metadata.
  • Assigning a global unique number to the small files one by one the order of assigning the global unique numbers of the small files may be allocated in an order of increasing order, and the global unique number preset in the metadata server is assigned to the earliest small file.
  • the created small file, the globally unique number of the subsequently created small file is incremented by 1 based on the preset unique file number.
  • the metadata server allocates the global unique number of the small file according to the creation time of the small file, and the later the creation time of the small file, the larger the file number to be assigned.
  • small files include B, C, D, E, F, G, H, I, J, K, L, M, and their creation time increases in turn, and the file number 2015 preset in the metadata server is allocated.
  • the file number of the small file C is 2016, the file number of the small file D is 2017, and the file number of the small file E, F, G, H, I, J, K, L, M Can be analogized by analogy.
  • the metadata server 20 is further configured to predict that the N globally consecutive consecutive small files aggregated in the aggregation location are predicted to be the same object, and the N small files of the same object are respectively assigned different storage area unique numbers.
  • N is a positive integer less than or equal to the preset number of aggregations.
  • An object has a memory capacity of 4 MB. Up to 4 small files are aggregated in one object, and at least one small file is aggregated. Among them, four storage areas are allocated in one object. After the metadata server allocates the global unique number of the small file one by one for the small file, the metadata server predicts four small files with consecutive global unique numbers as the same object, for example, (B, C , D, E), (F, G, H, I), (J, K, L, M) 12 small file decisions are combined in three objects.
  • the unaggregated small files may be aggregated into the same object, and the storage area is uniquely allocated to the objects. Numbering.
  • the allocation principle may be that the small file storage area in the Kth area is uniquely numbered -k. For example, if small files B, C, D, E, F, G, H, I, J, and K are aggregated, then (B, C, D, E), (F, G, H, I), ( J, K) 10 small files are aggregated into three objects, and the size of the object containing 2 small files (J, K) is 4MB.
  • the metadata server allocates the area number in the object one by one for the small file.
  • the area number of the small file B in the object is -1
  • the area number of the small file C in the object is -2
  • the area number of the small file D in the object is -3
  • the area of the small file E in the object is The domain number is -4
  • the area number of the small file F in the object is -1
  • the area number of the small file G in the object is -2
  • the area number of the small file H in the object is -3
  • the small file I is in the object.
  • the area number in the middle is -4, the area number of the small file J in the object is -1, the area number of the small file K in the object is -2, the area number of the small file L in the object is -3, and the small file M
  • the area number in the object is -4.
  • the metadata server can create metadata for the above small files one by one, such as the creation time, access rights, last modification time, and file size of the small files.
  • the object storage device OSD 30 is further configured to receive a small file write aggregation request sent by the client.
  • the client after the client determines the OSD to be written by the client, the client sends the small file write aggregation request to the OSD, and the OSD receives the small file write aggregation request sent by the client, where
  • the small file write aggregation request includes a small file, an aggregate location, a global unique number, and a storage area unique number.
  • the object storage device OSD 30 is further configured to detect the storage area unique number of the N globally unique number consecutive small files one by one according to the small file write aggregation request, and write the small file number when the unique number of the small file storage area is negative.
  • the first storage area in the object for the aggregation location is uniquely numbered, where - is a negative sign.
  • the OSD receives a small file write aggregation request sent by the client, and the OSD detects whether the storage area unique number of the received small file is negative one by one, and determines the OSD after the unique number of the storage area of the received small file is negative.
  • the OSD small file aggregation module is notified to perform a small file aggregation write operation on the small file.
  • the small file aggregation module aggregates the consecutive globally uniquely numbered small files in the same object; or, according to the absolute value of the unique number of the storage area, determines that the small file is in the storage area of the object, and writes the small file into the small file.
  • the absolute value of the storage area of the file is the absolute value corresponding to the storage area.
  • the client 10 initiates a write operation of a small file
  • the metadata server 20 assigns a global unique number and a storage area unique number to the small file
  • the client 10 calculates a small file according to the global unique number and the unique number of the storage area.
  • the aggregate location in the object storage device OSD 30 causes the object storage device OSD 30 to aggregate small files according to the aggregated location. Since the small file storage location is determined by calculating the global unique number and the storage area unique number, and writing the small file according to the small file storage location, there is no need to additionally create a mapping table of the small file, thereby Reduced write time for small files.
  • FIG. 4 is a schematic flowchart of a first embodiment of a file read aggregation method.
  • step 510 the client initiates a read request for a small file.
  • the file size of a small file may be less than 1 MB, and the client initiates a small file read request to the small file, that is, the small file is sequentially read according to the client.
  • step 520 the global unique number of the small file and the storage area unique number are obtained from the metadata server.
  • the client obtains metadata of the currently read small file from the metadata server.
  • the metadata server detects the access right of the client to the small file according to the read request of the small file initiated by the client, and if the metadata server determines that the client is illegally accessing the small file, then the client Feedback to the client that the file is illegally accessed, the client checks whether the reading of the small file is completed, if not, continues to read the next small file in the small file, and if completed, ends the small file Reading. If the metadata server determines that the client is a legitimate access to the small file, the metadata server sends the global unique number of the read small file and the storage area unique number to the client, and the client receives the metadata. The globally unique number sent by the server and the unique number of the storage area.
  • step 530 the aggregated location of the small file in the object storage device OSD is calculated based on the global unique number and the storage area unique number.
  • the global unique number may be an inode number ino
  • the storage area unique number may be an object number ono
  • 1, calculate the aggregation position of the small file in the object storage device OSD, that is, the number of the object to which the small file belongs ((small file File number + area number of the small file in the object +1) ⁇ 32)
  • ⁇ 32 indicates that 32 bits are shifted to the left, and
  • the aggregation location is a unique number of the object in the global object after the small file is aggregated into an object. Small files in any storage area in the same object are calculated by the same aggregation location.
  • HASH generalizes the common data distribution algorithms common in the computer field, including but not limited to: squared intermediate value, modular operation remainder, DHT algorithm, CRUSH algorithm.
  • step 540 a small file is obtained from the object storage device OSD according to the aggregation location.
  • the small file aggregation module in the OSD sends all the small files aggregated in the aggregation location to the client. end.
  • the client feeds back the data in the small file to the application of the client according to the unique number of the storage area of the small file, and other small files included in the aggregation location are stored in the local cache of the client.
  • the client initiates a read request of a small file, obtains a global unique number of the small file and a unique number of the storage area from the metadata server according to the read request of the small file, and calculates a small file according to the global unique number and the unique number of the storage area.
  • the aggregation location in the object storage device OSD causes the client to retrieve small files from the OSD based on the aggregation location. Since the small file storage location is determined by calculating the global unique number and the storage area unique number, and reading the small file according to the small file storage location, it is not necessary to read the small file by checking the mapping table. , which reduces the time for small file reading.
  • the first embodiment of the file-based read aggregation method proposes a second embodiment of the file read aggregation method.
  • the read aggregation method further includes: steps 550-560.
  • step 550 it is detected whether there is a small file in the cache of the client.
  • the client detects whether the currently read small file exists in the local cache of the client, and it can be understood that the small file is also aggregated into the object that is aggregated into the object. All files are read out and saved, so you can reduce the read time of the small file by looking up the existence of the small file in the client's cache.
  • step 560 is executed to obtain a small file.
  • the client acquires the small file and returns the data in the small file to the application of the client.
  • step 520 is executed, that is, the global unique number of the small file and the unique number of the storage area are obtained from the metadata server.
  • the client sends an ordered read request of the small file to the metadata server, so that the client obtains the file number allocated by the metadata server for the small file and The area number of the small file in the object. For example, when the client initiates an ordered read request for small files B, C, D, E, F, G, H, I, J, K, L, M, the client is detected in the cache. If the small file does not exist, the client sends an ordered read request of the small file to the metadata server, obtains a global unique number assigned by the metadata server for the small file, and a unique number of the small file in the object storage area.
  • the small file is detected in the cache of the client, and if the small file exists in the cache of the client, the small file is obtained; if the small file does not exist in the cache of the client, the secondary data is obtained.
  • the server obtains the global unique number of the small file and the unique number of the storage area, so that the client can quickly read the small file after initiating an ordered read request to the small file, thereby improving the client's reading of the small file. Write efficiency.
  • a third embodiment of the file read aggregation method is proposed.
  • the foregoing step 540 includes: steps 5410-5430.
  • step 5410 a read aggregation request is sent to the object storage device OSD.
  • the client determines an OSD stored after the small file is aggregated into an object, and the client sends a read aggregation request of the small file to the OSD, and the OSD determines whether the storage area unique number of the small file is A negative number, after determining that the storage area unique number of the small file is a negative number, the OSD notifies the small file aggregation module in the OSD to perform a small file aggregation read operation on the small file.
  • step 5420 all of the small files in the aggregated location sent by the object storage device OSD are received and saved.
  • the small file aggregation module sends all the small files in the object described by the small file to the client from the OSD, and the client saves all the small files in the object.
  • the object storage device OSD aggregates and reads the small file
  • the object storage device sends a small file aggregation read completion instruction to the client, and after receiving the small file aggregation read completion instruction, the client receives the small file aggregation read completion instruction.
  • the client sends a metadata update request update instruction to the metadata server to notify the metadata server to update the metadata information of the small file, such as the last access time of the small file and the last access time of the object.
  • step 5430 the small file in the absolute value corresponding area of the storage area unique number is obtained.
  • the client reads the absolute value of the unique number of the storage area of the small file, that is, the small file of the unique numbered area of the first storage area in the object, and reads the data in the small file to the Client application.
  • the client sends a read aggregation request to the OSD, and the OSD determines whether the unique number of the storage area of the small file is a negative number. After determining that the unique number of the storage area of the small file is a negative number, The client receives all the small files in the aggregation location sent by the OSD and saves, so that the client obtains the small file in the corresponding area of the absolute value of the unique number of the storage area, so that the small file is not read by checking the mapping table, thereby saving The time to read small files.
  • the present disclosure also provides a client.
  • FIG. 8 is a schematic diagram of functional modules of a first embodiment of a client according to the present disclosure.
  • the client includes a read request module 40, a number acquisition module 50, a calculation module 60, and a small file acquisition module 70.
  • the read request module 40 is arranged to initiate a read request for a small file.
  • the file size of a small file is less than 1 MB
  • the client initiates a small file read request to the small file, that is, the small file is sequentially read according to the client.
  • the number obtaining module 50 is configured to acquire a global unique number of the small file and a storage area unique number from the metadata server.
  • the client obtains metadata of the currently read small file from the metadata server.
  • the metadata server detects the access right of the client to the small file according to the read request of the small file initiated by the client, and if the metadata server determines that the client is illegally accessing the small file, then the client Feedback to the client that the file is illegally accessed, the client checks whether the reading of the small file is completed, if not, continues to read the next small file in the small file, and if completed, ends the small file Reading. If the metadata server determines that the client is a legitimate access to the small file, the metadata server sends the global unique number of the read small file and the storage area unique number to the client, and the client receives the metadata. The globally unique number sent by the server and the unique number of the storage area.
  • the calculation module 60 is configured to calculate an aggregate location of the small file in the object storage device OSD according to the global unique number and the storage area unique number.
  • the global unique number may be an inode number ino
  • the storage area unique number may be an object number ono
  • 1, calculate the aggregation position of the small file in the object storage device OSD, that is, the number of the object to which the small file belongs ((small file File number + area number of the small file in the object +1) ⁇ 32)
  • ⁇ 32 Indicates that 32 bits are shifted to the left, and
  • the aggregation location is a unique number of the object in the global object after the small file is aggregated into an object. Small files in any storage area in the same object are calculated by the same aggregation location.
  • HASH generalizes the common data distribution algorithms common in the computer field, including but not limited to: squared intermediate value, modular operation remainder, DHT algorithm, CRUSH algorithm.
  • the small file acquisition module 70 is configured to acquire a small file from the object storage device OSD according to the aggregation location.
  • the small file aggregation module in the OSD sends all the small files aggregated in the aggregation location to the client. end.
  • the client feeds back the data in the small file to the application of the client according to the unique number of the storage area of the small file, and other small files included in the aggregation location are stored in the local cache of the client.
  • the read request module 40 initiates a read request of the small file
  • the number obtaining module 50 obtains the global unique number and the storage area unique number of the small file from the metadata server according to the read request of the small file
  • the calculation module 60 is based on the global unique number.
  • the storage area unique number calculates an aggregation location of the small file in the object storage device OSD, so that the small file acquisition module 70 acquires a small file from the OSD according to the aggregated location. Since the small file storage location is determined by calculating the global unique number and the storage area unique number, and reading the small file according to the small file storage location, it is not necessary to read the small file by checking the mapping table. , which reduces the time for small file reading.
  • the client further includes: a detecting module 50.
  • the detecting module 80 is configured to detect whether a small file exists in the cache of the client.
  • the client detects whether the currently read small file exists in the local cache of the client, and it can be understood that the small file is also aggregated into the object that is aggregated into the object. All files are read out and saved, so you can reduce the read time of the small file by looking up the existence of the small file in the client's cache.
  • the small file obtaining module 70 is further configured to acquire a small file when a small file exists in the cache of the client;
  • the client acquires the small file and returns the data in the small file to the application of the client.
  • the number obtaining module 50 is further configured to acquire a global unique number of the small file and a storage area unique number from the metadata server when there is no small file in the cache of the client.
  • the client When detecting that the small file does not exist in the cache of the client, the client sends an ordered read request of the small file to the metadata server, so that the client obtains the file number allocated by the metadata server for the small file. And the area number of the small file in the object. For example, when the client initiates an ordered read request of the small files B, C, D, E, F, G, H, I, J, K, L, and M, the small file is not found in the cache of the client. Then, the client sends an ordered read request of the small file to the metadata server, obtains a global unique number assigned by the metadata server for the small file, and a unique number of the small file in the object storage area.
  • the small file is detected in the cache of the client, and if the small file exists in the cache of the client, the small file is obtained; if the small file does not exist in the cache of the client, the secondary data is obtained.
  • the server obtains the global unique number of the small file and the unique number of the storage area, so that the client can quickly read the small file after initiating an ordered read request to the small file, thereby improving the client's reading of the small file. Write efficiency.
  • the small file obtaining module 70 includes a sending unit 71, a receiving unit 72, and an obtaining unit. 73.
  • the transmitting unit 71 is arranged to send a read aggregation request to the object storage device OSD.
  • the client determines an OSD stored after the small file is aggregated into an object, and the client sends a read aggregation request of the small file to the OSD, and the OSD determines whether the storage area unique number of the small file is A negative number, after determining that the storage area unique number of the small file is a negative number, the OSD notifies the small file aggregation module in the OSD to perform a small file aggregation read operation on the small file.
  • the receiving unit 72 is configured to receive all the small files in the aggregated location sent by the object storage device OSD and save them.
  • the small file aggregation module sends all the small files in the object described by the small file to the client from the OSD, and the client saves all the small files in the object.
  • the object storage device OSD aggregates and reads the small file
  • the object storage device sends a small file aggregation read to the client.
  • the client receives the small file aggregation read completion instruction, and the client sends a metadata update request update instruction to the metadata server to notify the metadata server to update the metadata information of the small file, such as a small file.
  • the last access time the last access time of the object.
  • the obtaining unit 73 is configured to acquire a small file in the absolute value corresponding area of the storage area unique number.
  • the client reads the absolute value of the unique number of the storage area of the small file, that is, the small file of the unique numbered area of the first storage area in the object, and reads the data in the small file to the Client application.
  • the client sends a read aggregation request to the OSD, and the OSD determines whether the unique number of the storage area of the small file is a negative number. After determining that the unique number of the storage area of the small file is a negative number, the client receives the aggregation sent by the OSD. All the small files in the location are saved, so that the client obtains the small file in the corresponding area of the absolute value of the unique number of the storage area, so that the small file is not read by checking the mapping table, thereby saving the reading of the small file. time.
  • the present disclosure also provides a non-transitory computer readable storage medium storing computer executable instructions arranged to perform the file read aggregation method of any of the above embodiments.
  • the present disclosure also provides a schematic diagram of a hardware structure of a client.
  • the client includes:
  • At least one processor 101 which is exemplified by a processor 101 in FIG. 11; and a memory 102, may further include a communication interface 104 and a bus 103.
  • the processor 101, the communication interface 104, and the memory 103 can complete communication with each other through the bus 103.
  • Communication interface 104 can be used for information transfer.
  • the processor 101 can call logic instructions in the memory 102 to perform a file read aggregation method.
  • logic instructions in the memory 102 described above may be implemented in the form of software functional units and sold or used as separate products, and may be stored in a computer readable storage medium.
  • the memory 102 is a computer readable storage medium and can be used to store software programs, computer executable programs, program instructions or modules corresponding to the methods in the embodiments of the present disclosure.
  • the processor 101 performs a function application and data processing by executing a software program, an instruction, or a module stored in the memory 102, that is, implementing a file read aggregation method.
  • the memory 102 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function; and the storage data area may be stored according to the terminal device Use the data created, etc. Further, the memory 102 may include a high speed random access memory, and may also include a nonvolatile memory.
  • the technical solution of the present disclosure may be embodied in the form of a software product stored in a storage medium, including one or more instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) Performing all or part of the steps of the method of the embodiments of the present disclosure.
  • the foregoing storage medium may be a non-transitory storage medium, including: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, and the like, and may be a temporary storage medium.
  • the present disclosure also provides a non-transitory computer readable storage medium storing computer executable instructions arranged to perform the file write aggregation method of any of the above embodiments.
  • the present disclosure also provides a client having the same hardware structure as the client in FIG. 11, which is different from the client in FIG. 10 in that the processor of the client can call the logic instruction in the memory to Execute the file write aggregation method.
  • the write aggregation, read aggregation method, system and client of the file provided by the present disclosure reduce the time of reading and writing in the write aggregation and read aggregation process of small files.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)

Abstract

本文公开了一种文件的写聚合方法及系统，该写聚合方法包括：客户端发起小文件的写操作；客户端根据元数据服务器为所述小文件分配的全局唯一编号和存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置；以及客户端向对象存储设备OSD发送携带所述聚合位置的小文件聚合请求；其中，所述小文件是指大小小于一阈值的文件。

Description

文件的写聚合、读聚合方法及系统和客户端 技术领域
本公开涉及分布式存储技术领域，例如涉及一种文件的写聚合、读聚合方法及系统。
背景技术
随着云计算、移动互联网等技术的发展，需要存储及读取的数据量日益增大，而不同的数据的存储与读取方式是不同的，如只有数千字节的小文件，广泛采用随机聚合、查表索引的方式来进行存储与读取。由于聚合策略中的小文件在聚合时，需要额外创建这些小文件的映射表，并通过查询映射表才能读取这些小文件，因此耗费较多小文件的读取和写入的时间。
发明内容
本公开提供一种文件写聚合、读聚合方法及系统，减少了小文件的写聚合和读聚合过程中读取和写入的时间。
本公开提供的一种文件写聚合方法，包括：
客户端发起小文件的写操作；
所述客户端根据元数据服务器为所述小文件分配的全局唯一编号和存储区域唯一编号计算所述小文件在对象存储设备OSD中的聚合位置；
所述客户端向所述对象存储设备OSD发送携带所述聚合位置的小文件聚合请求；其中，所述小文件是指大小小于一阈值的文件。
可选地，所述全局唯一编号为索引节点编号ino，存储区域唯一编号为对象编号ono，所述聚合位置的计算公式为：聚合位置＝((ino+ono+1)＜＜32)|1，其中，＜＜32表示左移32个比特位，|表示按位或运算。
可选地，所述客户端发起小文件的写操作包括：
所述客户端将所述小文件的内容写入本地缓存，以及向所述元数据服务器发送创建所述小文件的元数据的请求。
可选地，所述元数据服务器为所述小文件分配的全局唯一编号是按照时序递增的分配原则分配的；以及
在所述聚合位置中聚合的N个全局唯一编号连续的所述小文件被预判为同一对象，同一对象的N个小文件分别被分配不同的存储区域唯一编号；其中，N为小于或等于预设聚合数量的正整数。
可选地，所述客户端向所述对象存储设备OSD发送携带所述聚合位置的小文件聚合请求后，在所述小文件的存储区域唯一编号为负数时，所述小文件被写入编号为聚合位置的对象中的第一聚合位置个区域中，其中。-为负号。
本公开还提供一种文件写聚合系统，包括：
客户端、元数据服务器及对象存储设备OSD，其中，
所述客户端，设置为发起小文件的写操作；
所述元数据服务器，设置为为所述小文件分配全局唯一编号和存储区域唯一编号；
所述客户端，还设置为根据所述全局唯一编号和所述存储区域唯一编号计算所述小文件在所述对象存储设备OSD中的聚合位置；以及
所述对象存储设备OSD，设置为根据聚合位置将小文件聚合。
可选地，所述全局唯一编号为索引节点编号ino，存储区域唯一编号为对象编号ono，所述聚合位置的计算公式为：聚合位置＝((ino+ono+1)＜＜32)|1，其中，＜＜32表示左移32个比特位，|表示按位或运算。
可选地，所述元数据服务器还设置为：
按时序递增的分配原则为所述小文件分配全局唯一编号；以及
在所述聚合位置中聚合的N个全局唯一编号连续的所述小文件被预判为同一对象，同一对象的N个小文件分别被分配不同的存储区域唯一编号；其中，N为小于或等于预设聚合数量的正整数。
可选地，所述对象存储设备OSD还设置为：
所述对象存储设备OSD接收所述客户端发送的小文件写聚合请求；以及
根据所述小文件写聚合请求逐个检测N个全局唯一编号连续的小文件的存储区域唯一编号，在所述小文件的存储区域唯一编号为负数时，将所述小文件 写入编号为聚合位置的对象中的第-存储区域唯一编号个区域中；其中，-为负号；小文件写聚合请求包含小文件、聚合位置、全局唯一编号和存储区域唯一编号；N为小于或等于预设聚合数量的正整数。
本公开提供一种非暂态计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令设置为上述任一文件写聚合方法。
本公开还提供了一种客户端，包括：
至少一个处理器；以及
与所述至少一个处理器通信连接的存储器；其中，
所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器执行：
客户端发起小文件的写请求操作；
所述客户端根据元数据服务器为所述小文件分配的全局唯一编号和存储区域唯一编号计算所述小文件在对象存储设备OSD中的聚合位置；以及
客户端向所述对象存储设备OSD发送携带所述聚合位置的小文件聚合请求；其中，所述小文件是指大小小于一阈值的文件。
本公开通过客户端发起小文件的写操作，该客户端根据元数据服务器为小文件分配的全局唯一编号和存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置，客户端向向对象存储设备OSD发送携带所述聚合位置的小文件聚合请求，其中，所述小文件是指大小小于一阈值的文件。由于该小文件存储位置的是通过计算获取到的全局唯一编号和存储区域唯一编号而确定，并根据该小文件存储位置写入该小文件，因此不需要额外创建该小文件的映射表，从而减少了小文件的写入时间。
本公开提供的一种文件读聚合方法，包括：
客户端发起小文件的读请求；
从元数据服务器获取所述小文件的全局唯一编号和存储区域唯一编号；
根据所述全局唯一编号和所述存储区域唯一编号计算所述小文件在对象存储设备OSD中的聚合位置；以及
根据所述聚合位置从所述对象存储设备OSD中获取所述小文件。
可选地，所述全局唯一编号为索引节点编号ino，存储区域唯一编号为对象编号ono，所述聚合位置的计算公式为：聚合位置＝((ino+ono+1)＜＜32)|1，其中，＜＜32表示左移32个比特位，|表示按位或运算。
可选地，在所述客户端发起小文件的读请求之后，所述方法还包括：
检测所述客户端的缓存中是否存在所述小文件；
若所述客户端的缓存中存在所述小文件，则获取所述小文件；以及
若所述客户端的缓存中未存在所述小文件，则从元数据服务器获取所述小文件的全局唯一编号和存储区域唯一编号。
可选地，所述根据所述聚合位置从所述对象存储设备OSD中获取所述小文件包括：
向所述对象存储设备OSD发送读聚合请求；
接收所述对象存储设备OSD发送的所述聚合位置中所有的小文件并保存；以及
获取所述存储区域唯一编号的绝对值对应区域中的小文件。
本公开还提供一种客户端，包括：
读请求模块，设置为发起小文件的读请求；
编号获取模块，设置为从元数据服务器获取所述小文件的全局唯一编号和存储区域唯一编号；
计算模块，设置为根据所述全局唯一编号和所述存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置；以及
小文件获取模块，设置为根据所述聚合位置从所述对象存储设备OSD中获取所述小文件。
可选地，所述全局唯一编号为索引节点编号ino，存储区域唯一编号为对象编号ono，所述聚合位置的计算公式为：聚合位置＝((ino+ono+1)＜＜32)|1，其中，＜＜32表示左移32个比特位，|表示按位或运算。
可选地，所述客户端还包括：
检测模块，设置为检测所述客户端的缓存中是否存在所述小文件；
所述小文件获取模块，还设置为若所述客户端的缓存中存在所述小文件，则获取所述小文件；以及
所述编号获取模块，还设置为若所述客户端的缓存中未存在所述小文件，则从元数据服务器获取所述小文件的全局唯一编号和存储区域唯一编号。
可选地，所述小文件获取模块包括：
发送单元，设置为向所述对象存储设备OSD发送读聚合请求；
接收单元，设置为接收所述对象存储设备OSD发送的所述聚合位置中所有的小文件并保存；以及
获取单元，设置为获取所述存储区域唯一编号的绝对值对应区域中的小文件。本公开还提供了一种非暂态计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令设置为执行文件读聚合方法。
本公开还提供了一种客户端，包括：
至少一个处理器；以及
与所述至少一个处理器通信连接的存储器；其中，
所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器执行：
发起小文件的读请求；
从元数据服务器获取所述小文件的全局唯一编号和存储区域唯一编号；
根据所述全局唯一编号和所述存储区域唯一编号计算所述小文件在对象存储设备OSD中的聚合位置；以及
根据所述聚合位置从所述对象存储设备OSD中获取所述小文件。
本公开通过客户端发起小文件的读请求，根据该小文件的读请求从元数据服务器获取小文件的全局唯一编号和存储区域唯一编号，并根据全局唯一编号和存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置，使得该客户端根据聚合位置从OSD中获取小文件。由于该小文件存储位置的是通过计算获取到的全局唯一编号和存储区域唯一编号而确定，并根据该小文件存储位置读取该小文件，因此不需要通过查映射表来读取该小文件，从而减少了小文件的读取时间。
附图说明
图1为文件的写聚合方法的第一实施例的流程示意图；
图2为文件的写聚合方法的第二实施例中元数据服务器为小文件分配全局唯一编号和存储区域唯一编号的细化流程示意图；
图3为本文件写聚合方法的第三实施例中对象存储设备OSD根据聚合位置将小文件聚合的细化流程示意图；
图4为文件写聚合系统的第一实施例的功能模块示意图；
图5为文件读聚合方法的第一实施例的流程示意图；
图6为文件读聚合方法的第二实施例的流程示意图；
图7为文件读聚合方法的第三实施例中根据聚合位置获取小文件的细化流程示意图；
图8为客户端的第一实施例的功能模块示意图；
图9为客户端的第二实施例的功能模块示意图；
图10为客户端的第三实施例中文件获取模块的细化功能模块示意图；以及
图11为客户端的硬件结构示意图。
具体实施方式
应当理解，此处所描述的实施例仅仅用以解释本公开，并不用于限定本公开。
参照图1，图1为文件的写聚合方法的第一实施例的流程示意图。本公开的文件写聚合方法可以应用于小文件，小文件是指大小小于一阈值(例如1MB)的文件。
在步骤110中，客户端发起小文件的写操作。
在本实施例中，一个小文件的文件大小可以是小于64KB或1MB，小文件还可以是小于文件系统条带粒度(Stripe Size)的文件。客户端发起小文件写操 作，客户端将该小文件的内容写入客户端的本地缓存中。在该客户端将该小文件的内容写入该客户端的本地缓存中后，该客户端向元数据服务器发送创建这些小文件的元数据请求。
可选的，所述客户端发起小文件的写操作包括将小文件的内容写入缓存、创建小文件的元数据请求以及将缓存中的内容根据元数据写入存储设备。
在步骤120中，元数据服务器为小文件分配全局唯一编号和存储区域唯一编号。其中，全局唯一编号可以为索引节点编号(inode number，ino)，存储区域唯一编号可以为对象编号(object number，ono)。该元数据服务器接收到元数据请求后，将会为该小文件逐一分配全局唯一编号和小文件在对象中的存储区域唯一编号，再将该全局唯一编号及该存储区域唯一编号回传给该客户端，该客户端接收该全局唯一编号和该存储区域唯一编号。其中，对象是基于对象存储设备(Object-based Storage Device，OSD)构建存储系统中数据存储的基本单位，一个对象是文件的数据和一组属性信息(Meta Data)的组合，对象存储结构包括对象、对象存储设备、元数据服务器、对象存储系统的客户端。
在步骤130中，客户端根据全局唯一编号和存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置。其中，聚合位置可以是由对象标识符(Object Identifier，OID)决定，对象标识符可以是一个64位比特的数据，当全局唯一编号为ino，存储区域唯一编号为ono时，OID可以通过公式OID＝((ino+ono+1)＜＜32)|1计算得到，OID的数据决定了小文件在对象存储设备中的聚合位置。
在该客户端接收到该元数据服务器回传该全局唯一编号和该存储区域唯一编号后，该客户端可以通过该聚合位置的计算公式聚合位置＝((ino+ono+1)＜＜32)|1，计算该小文件在对象存储设备OSD中的聚合位置，即小文件对象的编号＝((小文件的文件编号+小文件在对象中的区域编号+1)＜＜32)|1，计算出该小文件聚合成对象后，该对象的编号。其中，＜＜32表示左移32个比特位，|表示按位或运算。该聚合位置为该小文件在聚合成对象后，该对象在全局对象中的唯一编号。在同一对象中任一存储区域的小文件通过计算后得出的聚合位置均相同。该对象存储设备OSD的确定可以通过公式OSD_num＝HASH(聚合位置)，计算得到的，即对象存储设备的编号＝HASH(文件的对象编号)，计算出该对象所处的对象存储设备的编号，其中，HASH泛化了计算机 领域常见的、公开的数据分布算法，包括但不限于：平方取中间值、模运算取余、分布式哈希表(Distributed Hash Table，DHT)算法、CRUSH(Controlled Replication Under Scalable Hashing)算法。计算得到聚合位置后，客户端将聚合位置发送给对象存储设备OSD。
在步骤140中，对象存储设备OSD根据聚合位置将小文件聚合。
在接收到该小文件的聚合位置后，该对象存储设备将具有相同的聚合位置的小文件聚合在同一对象中，分别写入不同区域中。同时，在对象存储设备OSD将该小文件聚合完成后，该对象存储设备向该客户端发送小文件聚合写完成指令，该客户端接收到该小文件聚合写完成指令后，该客户端向元数据服务器发送元数据更新请求更新指令，通知该元数据服务器更新该小文件的元数据信息，更新该小文件的文件编号、区域编号、小文件的创建时间、小文件的修改时间、访问权限、文件大小。
本实施例通过客户端发起小文件的写操作，以供元数据服务器为小文件分配的全局唯一编号和存储区域唯一编号该客户端根据该全局唯一编号和该存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置，使得该对象存储设备OSD根据该聚合位置将小文件聚合。由于该小文件存储位置的是通过计算全局唯一编号和存储区域唯一编号而确定，并根据该小文件存储位置写入该小文件，因此不需要额外创建该小文件的映射表，从而减少了小文件的写入时间。
基于文件的写聚合方法的第一实施例，提出文件的写聚合方法的第二实施例，在本实施例中，参照图2，上述步骤120包括：步骤1210-1220。
在步骤1210中，元数据服务器按时序递增的分配原则为小文件分配全局唯一编号。
在本实施例中，在该客户端发起该小文件的写操作后，该客户端向该元数据服务器发送该小文件元数据的创建请求，该元数据服务器根据该小文件元数据的创建请求为该小文件逐一分配全局唯一编号，该小文件的全局唯一编号的分配顺序可以按照时序递增的顺序来分配的，将该元数据服务器中预设的全局唯一编号，分配给该小文件中最早创建的小文件，后续创建的小文件的全局唯一编号在预设的唯一的文件编号基础上依次增加1。该元数据服务器按照小文件的创建时间来分配小文件的全局唯一编号，小文件的创建时间越晚，被分配到的文件编号越大。例如，小文件包括B、C、D、E、F、G、H、I、J、K、L、M， 它们的创建时间依次增大，将该元数据服务器中预设的文件编号2015分配给最早创建的小文件B，则小文件C的文件编号为2016，则小文件D的文件编号为2017，小文件E、F、G、H、I、J、K、L、M的文件编号可以依次类推。
在步骤1220中，将在聚合位置中聚合的N个全局唯一编号连续的小文件预判为同一对象，为同一对象的N个小文件分别分配不同的存储区域唯一编号。其中，N为小于或等于预设聚合数量的正整数。
一个对象的内存容量为4MB，一个对象中最多聚合4个小文件，至少要聚合一个小文件，其中，一个对象中分配4个存储区域。在该元数据服务器为该小文件逐一分配该小文件的全局唯一编号后，该元数据服务器将具有连续全局唯一编号的4个小文件预判为同一个对象，例如，将(B、C、D、E)、(F、G、H、I)、(J、K、L、M)12个小文件判定分别合并在三个对象中。
当未聚合完毕的小文件的数量小于预设聚合数量(例如上述的预设聚合数量为4)时，可以将所述未聚合完毕的小文件聚合于同一对象中，并为其对象分配存储区域唯一编号。分配原则可以为：第K个区域中的小文件存储区域唯一编号为-k。例如，对小文件B、C、D、E、F、G、H、I、J、K进行聚合，则将(B、C、D、E)、(F、G、H、I)、(J、K)10个小文件分别聚合成三个对象中，其中包含(J、K)2个小文件的对象占用内存大小是4MB。在该小文件聚合成对象后，该元数据服务器为该小文件逐一分配在该对象中的区域编号。例如，小文件B在对象中的区域编号为-1、小文件C在对象中的区域编号为-2、小文件D在对象中的区域编号为-3、小文件E在对象中的区域编号为-4、小文件F在对象中的区域编号为-1、小文件G在对象中的区域编号为-2、小文件H在对象中的区域编号为-3、小文件I在对象中的区域编号为-4、小文件J在对象中的区域编号为-1、小文件K在对象中的区域编号为-2、小文件L在对象中的区域编号为-3、小文件M在对象中的区域编号为-4。
该元数据服务器可以逐一为上述小文件创建元数据，如小文件的创建时间、访问权限、最后修改时间、文件大小。
本实施例通过元数据服务器按时序递增的分配原则为小文件分配全局唯一编号，并为预判为同一对象的N个全局唯一编号连续的小文件分配存储区域唯一编号。由于该小文件的全局唯一编号及该存储区域唯一编号是按时序递增的原则分配的，并按照区域编号有序聚合在对象中，因此不需要额外创建这些小 文件的映射表，从而提高对小文件的效率。
可选的，基于第一实施例，提出文件的写聚合方法的第三实施例，参照图3。
在步骤1410中，对象存储设备OSD接收客户端发送的小文件写聚合请求。
在本实施例中，在该客户端确定该小文件将要写入的OSD后，该客户端向该OSD发送该小文件写聚合请求，该OSD接收客户端发送的小文件写聚合请求。
在步骤1420中，对象存储设备OSD根据小文件写聚合请求逐个检测N个全局唯一编号连续小文件的存储区域唯一编号，在小文件的存储区域唯一编号为负数时，将小文件写入编号为聚合位置的对象中的第-存储区域唯一编号个区域中，其中，-为负号，N为小于或等于预设聚合数量的正整数。
在该OSD接收客户端发送的小文件写聚合请求，该OSD逐个检测接收到的小文件的存储区域唯一编号是否为负数，在确定接收到的小文件的存储区域唯一编号为负数后，该OSD通知该OSD中小文件聚合模块对该小文件进行小文件聚合写操作。该小文件聚合模块将连续个全局唯一编号小文件聚合在同一个对象中；或者，根据该存储区域唯一编号的绝对值确定该小文件在该对象的存储区域，将该小文件写入该小文件的存储区域唯一编号的绝对值对应的存储区域中。本实施例通过对象存储设备OSD接收客户端发送的小文件写聚合请求，并根据小文件写聚合请求逐个检测N个全局唯一编号连续小文件的存储区域唯一编号，在小文件的存储区域唯一编号为负数时，将小文件写入编号为聚合位置的对象中的第-存储区域唯一编号个区域中，使得小文件存储位置是定向的、唯一的，从而减少了小文件的读写时间。
本公开可选提供一种文件的写聚合系统。
参照图4，图4为文件的写聚合系统的第一实施例的功能模块示意图。
在本实施例中，该系统可以包括：客户端10、元数据服务器20、对象存储设备OSD 30。
所述客户端10设置为发起小文件的写操作。
在本实施例中，一个小文件的文件大小可以小于1MB，客户端发起小文件写操作，客户端将该小文件的内容写入客户端的本地缓存中。在该客户端将该小文件的内容写入该客户端的本地缓存中后，该客户端向元数据服务器发送创 建这些小文件的元数据请求。
所述元数据服务器20设置为为小文件分配全局唯一编号和存储区域唯一编号。
该元数据服务器接收到元数据请求后，将会为该小文件逐一分配全局唯一编号和小文件在对象中的存储区域唯一编号，将该全局唯一编号及该存储区域唯一编号回传给该客户端，该客户端接收该全局唯一编号和该存储区域唯一编号。
所述客户端10还设置为根据全局唯一编号和存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置。
在该客户端接收到该元数据服务器回传该全局唯一编号和该存储区域唯一编号后，全局唯一编号可以为索引节点编号ino，存储区域唯一编号可以为对象编号ono，该客户端通过该聚合位置的计算公式聚合位置＝((ino+ono+1)＜＜32)|1，计算该小文件在对象存储设备OSD中的聚合位置，即小文件对象的编号＝((小文件的文件编号+小文件在对象中的区域编号+1)＜＜32)|1，计算出该小文件聚合成对象后，该对象的编号。其中，＜＜32表示左移32个比特位，|表示按位或运算，该聚合位置为该小文件在聚合成对象后，该对象在全局对象中的唯一编号。在同一对象中任一存储区域的小文件通过计算后得出的聚合位置均相同。该对象存储设备OSD的确定可以通过公式OSD_num＝HASH(聚合位置)，计算得到的，即对象存储设备的编号＝HASH(文件的对象编号)，计算出该对象所处的对象存储设备的编号，其中，HASH泛化了计算机领域常见的、公开的数据分布算法，包括但不限于：平方取中间值、模运算取余、DHT算法、CRUSH算法。
所述对象存储设备OSD 30设置为根据聚合位置将小文件聚合。
在接收到该小文件的聚合位置后，该对象存储设备将具有相同的聚合位置的小文件聚合在同一对象中，分别写入不同区域中。同时，在对象存储设备OSD将该小文件聚合完成后，该对象存储设备向该客户端发送小文件聚合写完成指令，该客户端接收到该小文件聚合写完成指令后，该客户端向元数据服务器发送元数据更新请求更新指令，通知该元数据服务器更新该小文件的元数据信息，更新该小文件的文件编号、区域编号、小文件的创建时间、小文件的修改时间、访问权限、文件大小。
所述元数据服务器20还设置为按时序递增的分配原则为小文件分配全局唯一编号。
在本实施例中，在该客户端发起该小文件的写操作后，该客户端向该元数据服务器发送该小文件元数据的创建请求，该元数据服务器根据该小文件元数据的创建请求为该小文件逐一分配全局唯一编号，该小文件的全局唯一编号的分配顺序可以按照时序递增的顺序来分配的，将该元数据服务器中预设的全局唯一编号，分配给该小文件中最早创建的小文件，后续创建的小文件的全局唯一编号在预设的唯一的文件编号基础上依次增加1。该元数据服务器按照小文件的创建时间来分配小文件的全局唯一编号，小文件的创建时间越晚，被分配到的文件编号越大。例如，小文件包括B、C、D、E、F、G、H、I、J、K、L、M，它们的创建时间依次增大，将该元数据服务器中预设的文件编号2015分配给最早创建的小文件B，则小文件C的文件编号为2016，则小文件D的文件编号为2017，小文件E、F、G、H、I、J、K、L、M的文件编号可以依次类推。
所述元数据服务器20还设置为在聚合位置中聚合的N个全局唯一编号连续的小文件被预判为同一对象，同一对象的N个小文件分别被分配不同的存储区域唯一编号。其中，N为小于或等于预设聚合数量的正整数。
一个对象的内存容量为4MB，一个对象中最多聚合4个小文件，至少要聚合一个小文件，其中，一个对象中分配4个存储区域。在该元数据服务器为该小文件逐一分配该小文件的全局唯一编号后，该元数据服务器将具有连续的全局唯一编号的4个小文件预判为同一个对象，例如，将(B、C、D、E)、(F、G、H、I)、(J、K、L、M)12个小文件判定分别合并在三个对象中。
当未聚合完毕的小文件的数量小于预设聚合数量(例如上述的预设聚合数量为4)时，可以将所述未聚合完毕的小文件聚合于同一对象中，并为对象分配存储区域唯一编号。分配原则可以为：第K个区域中的小文件存储区域唯一编号为-k。例如，对小文件B、C、D、E、F、G、H、I、J、K进行聚合，则将(B、C、D、E)、(F、G、H、I)、(J、K)10个小文件分别聚合成三个对象中，其中包含(J、K)2个小文件的对象占用内存大小是4MB。
在该小文件聚合成对象后，该元数据服务器为该小文件逐一分配在该对象中的区域编号。例如，小文件B在对象中的区域编号为-1、小文件C在对象中的区域编号为-2、小文件D在对象中的区域编号为-3、小文件E在对象中的区 域编号为-4、小文件F在对象中的区域编号为-1、小文件G在对象中的区域编号为-2、小文件H在对象中的区域编号为-3、小文件I在对象中的区域编号为-4、小文件J在对象中的区域编号为-1、小文件K在对象中的区域编号为-2、小文件L在对象中的区域编号为-3、小文件M在对象中的区域编号为-4。
该元数据服务器可以逐一为上述小文件创建元数据，如小文件的创建时间、访问权限、最后修改时间、文件大小。
所述对象存储设备OSD 30还设置为接收客户端发送的小文件写聚合请求。
在本实施例中，在该客户端确定该小文件将要写入的OSD后，该客户端向该OSD发送该小文件写聚合请求，该OSD接收客户端发送的小文件写聚合请求，其中，该小文件写聚合请求包括小文件、聚合位置、全局唯一编号和存储区域唯一编号。
所述对象存储设备OSD 30还设置为根据小文件写聚合请求逐个检测N个全局唯一编号连续小文件的存储区域唯一编号，在小文件的存储区域唯一编号为负数时，将小文件写入编号为聚合位置的对象中的第-存储区域唯一编号个区域中，其中-为负号。
在该OSD接收客户端发送的小文件写聚合请求，该OSD逐个检测接收到的小文件的存储区域唯一编号是否为负数，在确定接收到的小文件的存储区域唯一编号为负数后，该OSD通知该OSD中小文件聚合模块对该小文件进行小文件聚合写操作。该小文件聚合模块将连续全局唯一编号的小文件聚合在同一个对象中；或者，根据该存储区域唯一编号的绝对值确定该小文件在该对象的存储区域，将该小文件写入该小文件的存储区域唯一编号的绝对值对应的存储区域中。
本实施例通过客户端10发起小文件的写操作，元数据服务器20为小文件分配全局唯一编号和存储区域唯一编号，该客户端10根据该全局唯一编号和该存储区域唯一编号计算小文件在对象存储设备OSD 30中的聚合位置，使得该对象存储设备OSD 30根据该聚合位置将小文件聚合。由于该小文件存储位置的是通过计算获取到的全局唯一编号和存储区域唯一编号而确定，并根据该小文件存储位置写入该小文件，因此不需要额外创建该小文件的映射表，从而减少了小文件的写入时间。
参照图4，图4为文件的读聚合方法的第一实施例的流程示意图。
在步骤510中，客户端发起小文件的读请求。
在本实施例中，一个小文件的文件大小可以小于1MB，客户端向小文件发起小文件读请求，即根据该客户端对该小文件按顺序进行读取。
在步骤520中，从元数据服务器获取小文件的全局唯一编号和存储区域唯一编号。
该客户端向元数据服务器获取当前读取的小文件的元数据。该元数据服务器根据该客户端发起的小文件的读请求，检测该客户端对小文件的访问权限，如果该元数据服务器判定该客户端对该小文件的是非法访问，则向该客户端反馈该客户端对该文件是非法访问的信息，该客户端检查对该小文件的读取是否完成，如果没有完成则继续读取该小文件中的下一个小文件，如果完成则结束小文件的读取。如果该元数据服务器判定该客户端对该小文件的是合法访问，则元数据服务器将该读取小文件的全局唯一编号和存储区域唯一编号发送给该客户端，该客户端接收该元数据服务器发送的该全局唯一编号和该存储区域唯一编号。
在步骤530中，根据全局唯一编号和存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置。
在该客户端接收到该元数据服务器回传该全局唯一编号和该存储区域唯一编号后，全局唯一编号可以为索引节点编号ino，存储区域唯一编号可以为对象编号ono，该客户端可以通过该聚合位置的计算公式聚合位置＝((ino+ono+1)＜＜32)|1，计算该小文件在对象存储设备OSD中的聚合位置，即小文件所属对象的编号＝((小文件的文件编号+小文件在对象中的区域编号+1)＜＜32)|1，计算出该小文件聚合成的对象后，该对象的对象编号。其中，＜＜32表示左移32个比特位，|表示按位或运算。该聚合位置为该小文件在聚合成对象后，该对象在全局对象中的唯一编号。在同一对象中任一存储区域的小文件通过计算后得出的聚合位置均相同。该对象存储设备OSD的确定可以通过公式OSD_num＝HASH(聚合位置)，计算得到的，即对象存储设备的编号＝HASH(文件的对象编号)，计算出该对象所处的对象存储设备的编号，其中，HASH泛化了计算机领域常见的、公开的数据分布算法，包括但不限于：平方取中间值、模运算取余、DHT算法、CRUSH算法。
在步骤540中，根据聚合位置从对象存储设备OSD中获取小文件。
在该客户端计算出该小文件的聚合位置后，即计算出该小文件在OSD中的聚合位置后，该OSD中的小文件聚合模块将该聚合位置中聚合的所有小文件发送给该客户端。客户端根据该小文件的存储区域唯一编号，将该小文件中的数据反馈给客户端的应用，该聚合位置中包含的其他小文件则存储在该客户端的本地缓存中。
本实施例通过客户端发起小文件的读请求，根据该小文件的读请求从元数据服务器获取小文件的全局唯一编号和存储区域唯一编号，并根据全局唯一编号和存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置，使得该客户端根据聚合位置从OSD中获取小文件。由于该小文件存储位置的是通过计算获取到的全局唯一编号和存储区域唯一编号而确定，并根据该小文件存储位置读取该小文件，因此不需要通过查映射表来读取该小文件，从而减少了小文件读取的时间。
可选的，基于文件的读聚合方法的第一实施例，提出文件的读聚合方法的第二实施例。在本实施例中，参照图6，上述步骤510之后，读聚合方法还包括：步骤550-560。
在步骤550中，检测客户端的缓存中是否存在小文件。
在本实施例中，该客户端检测当前读取的该小文件是否存在该客户端的本地缓存中，可以理解的是，对该小文件读取的同时也将该小文件聚合成的对象中的所有文件读取出来并保存，因此可以通过查找该客户端的缓存中是否存在该小文件，可以减少对该小文件的读取时间。
若客户端的缓存中存在小文件，则执行步骤560，获取小文件。
在该检测到该客户端的缓存中存在该小文件时，该客户端获取该小文件，并将该小文件中的数据返回给该客户端的应用。
若客户端的缓存中未存在小文件，则执行步骤520，即从元数据服务器获取小文件的全局唯一编号和存储区域唯一编号。
若检测到客户端的缓存中未存在该小文件，则该客户端向该元数据服务器发送该小文件的有序读请求，使得该客户端获取该元数据服务器为该小文件分配的文件编号及该小文件在对象中的区域编号。例如，客户端发起小文件B、C、D、E、F、G、H、I、J、K、L、M的有序读请求时，检测到该客户端的缓存中 未存在该小文件，则该客户端向该元数据服务器发送该小文件的有序读请求，获取元数据服务器为小文件分配的全局唯一编号及小文件在对象存储区域唯一编号。
本实施例通过检测该客户端的缓存中是否存在该小文件，如果该客户端的缓存中存在该小文件时，则获取该小文件；如果客户端的缓存中未存在所述小文件，则从元数据服务器获取小文件的全局唯一编号和存储区域唯一编号，使得该客户端在对该小文件发起有序读请求后能够快速的对该小文件进行读取，从而提高了客户端对小文件的读写效率。
可选的，基于文件的读聚合方法的第一实施例，提出文件的读聚合方法的第三实施例，在本实施例中，参照图7，上述步骤540包括：步骤5410-5430。
在步骤5410中，向对象存储设备OSD发送读聚合请求。
在本实施例中，该客户端确定该小文件聚合成对象后所存储的OSD，该客户端向该OSD发送该小文件的读聚合请求，该OSD判断该小文件的存储区域唯一编号是否为负数，在确定该小文件的存储区域唯一编号为负数后，该OSD通知该OSD中的小文件聚合模块对该小文件进行小文件聚合读操作。
在步骤5420中，接收对象存储设备OSD发送的聚合位置中所有的小文件并保存。
该小文件聚合模块从该OSD中将该小文件所述的对象中的所有小文件发送给该客户端，该客户端并保存该对象中所有小文件。同时，在对象存储设备OSD将该小文件聚合读取完成后，该对象存储设备向该客户端发送小文件聚合读取完成指令，该客户端接收到该小文件聚合读取完成指令后，该客户端向元数据服务器发送元数据更新请求更新指令，通知该元数据服务器更新该小文件的元数据信息，如小文件的最后访问时间、对象的最后访问时间。
在步骤5430中，获取存储区域唯一编号的绝对值对应区域中的小文件。
该客户端将该小文件的存储区域唯一编号的绝对值，即该对象中第-存储区域唯一编号个区域的小文件从该对象中读取出，并将该小文件中的数据反馈给该客户端的应用。
本实施例通过客户端向OSD发送读聚合请求，该OSD判断该小文件的存储区域唯一编号是否为负数，在确定该小文件的存储区域唯一编号为负数后， 该客户端接收OSD发送的聚合位置中所有的小文件并保存，从而该客户端获取存储区域唯一编号的绝对值对应区域中的小文件，使得不用通过查映射表来读取该小文件，节约了对小文件的读取的时间。
本公开还提供一种客户端。
参照图8，图8为本公开客户端的第一实施例的功能模块示意图。
在本实施例中，该客户端包括：读请求模块40、编号获取模块50、计算模块60以及小文件获取模块70。
所述读请求模块40设置为发起小文件的读请求。
本实施例中，一个小文件的文件大小小于1MB，客户端向小文件发起小文件读请求，即根据该客户端对该小文件按顺序进行读取。
所述编号获取模块50设置为从元数据服务器获取小文件的全局唯一编号和存储区域唯一编号。
该客户端向元数据服务器获取当前读取的小文件的元数据。该元数据服务器根据该客户端发起的小文件的读请求，检测该客户端对小文件的访问权限，如果该元数据服务器判定该客户端对该小文件的是非法访问，则向该客户端反馈该客户端对该文件是非法访问的信息，该客户端检查对该小文件的读取是否完成，如果没有完成则继续读取该小文件中的下一个小文件，如果完成则结束小文件的读取。如果该元数据服务器判定该客户端对该小文件的是合法访问，则元数据服务器将该读取小文件的全局唯一编号和存储区域唯一编号发送给该客户端，该客户端接收该元数据服务器发送的该全局唯一编号和该存储区域唯一编号。
所述计算模块60设置为根据全局唯一编号和存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置。
在该客户端接收到该元数据服务器回传该全局唯一编号和该存储区域唯一编号后，全局唯一编号可以为索引节点编号ino，存储区域唯一编号可以为对象编号ono，该客户端可以通过该聚合位置的计算公式聚合位置＝((ino+ono+1)＜＜32)|1，计算该小文件在对象存储设备OSD中的聚合位置，即小文件所属对象的编号＝((小文件的文件编号+小文件在对象中的区域编号+1)＜＜32)|1，计算出该小文件聚合成的对象后，该对象的对象编号。其中，＜＜32 表示左移32个比特位，|表示按位或运算。该聚合位置为该小文件在聚合成对象后，该对象在全局对象中的唯一编号。在同一对象中任一存储区域的小文件通过计算后得出的聚合位置均相同。该对象存储设备OSD的确定可以通过公式OSD_num＝HASH(聚合位置)，计算得到的，即对象存储设备的编号＝HASH(文件的对象编号)，计算出该对象所处的对象存储设备的编号，其中，HASH泛化了计算机领域常见的、公开的数据分布算法，包括但不限于：平方取中间值、模运算取余、DHT算法、CRUSH算法。
所述小文件获取模块70设置为根据聚合位置从对象存储设备OSD中获取小文件。
在该客户端计算出该小文件的聚合位置后，即计算出该小文件在OSD中的聚合位置后，该OSD中的小文件聚合模块将该聚合位置中聚合的所有小文件发送给该客户端。客户端根据该小文件的存储区域唯一编号，将该小文件中的数据反馈给客户端的应用，该聚合位置中包含的其他小文件则存储在该客户端的本地缓存中。
本实施例通过读请求模块40发起小文件的读请求，编号获取模块50根据该小文件的读请求从元数据服务器获取小文件的全局唯一编号和存储区域唯一编号，计算模块60根据全局唯一编号和存储区域唯一编号计算小文件在对象存储设备OSD中的聚合位置，使得该小文件获取模块70根据聚合位置从OSD中获取小文件。由于该小文件存储位置的是通过计算获取到的全局唯一编号和存储区域唯一编号而确定，并根据该小文件存储位置读取该小文件，因此不需要通过查映射表来读取该小文件，从而减少了小文件读取的时间。
可选的，基于客户端的第一实施例，提出本公开客户端第二实施例。参照图8，在本实施例中，所述客户端还包括：检测模块50。
所述检测模块80设置为检测客户端的缓存中是否存在小文件。
在本实施例中，该客户端检测当前读取的该小文件是否存在该客户端的本地缓存中，可以理解的是，对该小文件读取的同时也将该小文件聚合成的对象中的所有文件读取出来并保存，因此可以通过查找该客户端的缓存中是否存在该小文件，可以减少对该小文件的读取时间。
所述小文件获取模块70，还设置为在客户端的缓存中存在小文件时，则获取小文件；
在该检测到该客户端的缓存中存在该小文件时，该客户端获取该小文件，并将该小文件中的数据返回给该客户端的应用。
所述编号获取模块50还设置为在客户端的缓存中未存在小文件时，则从元数据服务器获取小文件的全局唯一编号和存储区域唯一编号。
在检测到客户端的缓存中未存在该小文件时，则该客户端向该元数据服务器发送该小文件的有序读请求，使得该客户端获取该元数据服务器为该小文件分配的文件编号及该小文件在对象中的区域编号。例如，客户端发起小文件B、C、D、E、F、G、H、I、J、K、L、M的有序读请求时，检测到该客户端的缓存中未存在该小文件，则该客户端向该元数据服务器发送该小文件的有序读请求，获取元数据服务器为小文件分配的全局唯一编号及小文件在对象存储区域唯一编号。
本实施例通过检测该客户端的缓存中是否存在该小文件，如果该客户端的缓存中存在该小文件时，则获取该小文件；如果客户端的缓存中未存在所述小文件，则从元数据服务器获取小文件的全局唯一编号和存储区域唯一编号，使得该客户端在对该小文件发起有序读请求后能够快速的对该小文件进行读取，从而提高了客户端对小文件的读写效率。
可选的，基于客户端的第一实施例，提出本公开客户端第三实施例，参照图10，在本实施例中，所述小文件获取模块70包括发送单元71、接收单元72以及获取单元73。
所述发送单元71设置为向对象存储设备OSD发送读聚合请求。
在本实施例中，该客户端确定该小文件聚合成对象后所存储的OSD，该客户端向该OSD发送该小文件的读聚合请求，该OSD判断该小文件的存储区域唯一编号是否为负数，在确定该小文件的存储区域唯一编号为负数后，该OSD通知该OSD中的小文件聚合模块对该小文件进行小文件聚合读操作。
所述接收单元72设置为接收对象存储设备OSD发送的聚合位置中所有的小文件并保存。
该小文件聚合模块从该OSD中将该小文件所述的对象中的所有小文件发送给该客户端，该客户端并保存该对象中所有小文件。同时，在对象存储设备OSD将该小文件聚合读取完成后，该对象存储设备向该客户端发送小文件聚合读取 完成指令，该客户端接收到该小文件聚合读取完成指令后，该客户端向元数据服务器发送元数据更新请求更新指令，通知该元数据服务器更新该小文件的元数据信息，如小文件的最后访问时间、对象的最后访问时间。
所述获取单元73设置为获取存储区域唯一编号的绝对值对应区域中的小文件。
该客户端将该小文件的存储区域唯一编号的绝对值，即该对象中第-存储区域唯一编号个区域的小文件从该对象中读取出，并将该小文件中的数据反馈给该客户端的应用。
本实施例通过客户端向OSD发送读聚合请求，该OSD判断该小文件的存储区域唯一编号是否为负数，在确定该小文件的存储区域唯一编号为负数后，该客户端接收OSD发送的聚合位置中所有的小文件并保存，从而该客户端获取存储区域唯一编号的绝对值对应区域中的小文件，使得不用通过查映射表来读取该小文件，节约了对小文件的读取的时间。
本公开还提供一种非暂态计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令设置为执行上述任一实施例中的文件读聚合方法。
本公开还提供了一种客户端的硬件结构示意图，参见图11，该客户端包括：
至少一个处理器(Processor)101，图11中以一个处理器101为例；和存储器(Memory)102，还可以包括通信接口(Communications Interface)104和总线103。其中，处理器101、通信接口104、存储器103可以通过总线103完成相互间的通信。通信接口104可以用于信息传输。处理器101可以调用存储器102中的逻辑指令，以执行文件读聚合方法。
此外，上述的存储器102中的逻辑指令可以通过软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。
存储器102作为一种计算机可读存储介质，可用于存储软件程序、计算机可执行程序，如本公开实施例中的方法对应的程序指令或模块。处理器101通过运行存储在存储器102中的软件程序、指令或模块，从而执行功能应用以及数据处理，即实现文件读聚合方法。
存储器102可包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序；存储数据区可存储根据终端设备的使 用所创建的数据等。此外，存储器102可以包括高速随机存取存储器，还可以包括非易失性存储器。
本公开的技术方案可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括一个或多个指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本公开实施例所述方法的全部或部分步骤。而前述的存储介质可以是非暂态存储介质，包括：U盘、移动硬盘、ROM、RAM、磁碟或者光盘等多种可以存储程序代码的介质，也可以是暂态存储介质。
本公开还提供一种非暂态计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令设置为执行上述任一实施例中的文件写聚合方法。
本公开还提供了一种客户端，与图11中的客户端具有相同的硬件结构，与图10中的客户端的不同在于，本实施例中客户端的处理器可以调用存储器中的逻辑指令，以执行文件写聚合方法。
本公开提供的文件的写聚合、读聚合方法及系统和客户端，减少了小文件的写聚合和读聚合过程中读取和写入的时间。

Claims (18)

  1. 一种文件写聚合方法，包括：
    客户端发起小文件的写操作；
    所述客户端根据元数据服务器为所述小文件分配的全局唯一编号和存储区域唯一编号计算所述小文件在对象存储设备OSD中的聚合位置；
    所述客户端向所述对象存储设备OSD发送携带所述聚合位置的小文件聚合请求；其中，所述小文件是指大小小于一阈值的文件。
  2. 如权利要求1所述的方法，其中，所述全局唯一编号为索引节点编号ino，存储区域唯一编号为对象编号ono，所述聚合位置的计算公式为：聚合位置＝((ino+ono+1)＜＜32)|1，其中，＜＜32表示左移32个比特位，|表示按位或运算。
  3. 如权利要求1所述的方法，其中，所述客户端发起小文件的写操作包括：
    所述客户端将所述小文件的内容写入本地缓存，以及向所述元数据服务器发送创建所述小文件的元数据的请求。
  4. 如权利要求1所述的方法，其中，所述元数据服务器为所述小文件分配的全局唯一编号是按照时序递增的分配原则分配的；以及
    在所述聚合位置中聚合的N个全局唯一编号连续的所述小文件被预判为同一对象，同一对象的N个小文件分别被分配不同的存储区域唯一编号；其中，N为小于或等于预设聚合数量的正整数。
  5. 如权利要求4所述的方法，其中，
    所述客户端向所述对象存储设备OSD发送携带所述聚合位置的小文件聚合请求后，在所述小文件的存储区域唯一编号为负数时，所述小文件被写入编号为聚合位置的对象中的第-存储区域唯一编号个区域中，其中，-为负号。
  6. 一种文件写聚合系统，包括：客户端、元数据服务器及对象存储设备OSD，其中，
    所述客户端，设置为发起小文件的写操作；
    所述元数据服务器，设置为为所述小文件分配全局唯一编号和存储区域唯一编号；
    所述客户端，还设置为根据所述全局唯一编号和所述存储区域唯一编号计算所述小文件在所述对象存储设备OSD中的聚合位置；以及
    所述对象存储设备OSD，设置为根据所述聚合位置将所述小文件聚合。
  7. 如权利要求6所述的系统，其中，所述全局唯一编号为索引节点编号ino，存储区域唯一编号为对象编号ono，所述聚合位置的计算公式为：聚合位置＝((ino+ono+1)＜＜32)|1，其中，＜＜32表示左移32个比特位，|表示按位或运算。
  8. 如权利要求6所述的系统，其中，所述元数据服务器还设置为：
    按时序递增的分配原则为所述小文件分配全局唯一编号；以及
    在所述聚合位置中聚合的N个全局唯一编号连续的所述小文件被预判为同一对象，同一对象的N个小文件分别被分配不同的存储区域唯一编号；其中，N为小于或等于预设聚合数量的正整数。
  9. 如权利要求6所述的系统，其中，所述对象存储设备OSD还设置为：
    接收所述客户端发送的小文件写聚合请求；以及
    根据所述小文件写聚合请求逐个检测N个全局唯一编号连续的小文件的存储区域唯一编号，在所述小文件的存储区域唯一编号为负数时，将所述小文件写入编号为聚合位置的对象中的第-存储区域唯一编号个区域中；其中，-为负号；小文件写聚合请求包含小文件、聚合位置、全局唯一编号和存储区域唯一编号；N为小于或等于预设聚合数量的正整数。
  10. 一种文件读聚合方法，包括：
    客户端发起小文件的读请求；
    从元数据服务器获取所述小文件的全局唯一编号和存储区域唯一编号；
    根据所述全局唯一编号和所述存储区域唯一编号计算所述小文件在对象存储设备OSD中的聚合位置；以及
    根据所述聚合位置从所述对象存储设备OSD中获取所述小文件。
  11. 如权利要求10所述的方法，其中，所述全局唯一编号为索引节点编号ino，存储区域唯一编号为对象编号ono，所述聚合位置的计算公式为：聚合位置＝((ino+ono+1)＜＜32)|1，其中，＜＜32表示左移32个比特位，|表示按位或运算。
  12. 如权利要求10所述的方法，在所述客户端发起小文件的读请求之后，所述方法还包括：
    检测所述客户端的缓存中是否存在所述小文件；
    若所述客户端的缓存中存在所述小文件，则获取所述小文件；以及
    若所述客户端的缓存中未存在所述小文件，则从元数据服务器获取所述小文件的全局唯一编号和存储区域唯一编号。
  13. 如权利要求10所述的方法，其中，所述根据所述聚合位置从所述对象存储设备OSD中获取所述小文件包括：
    向所述对象存储设备OSD发送读聚合请求；
    接收所述对象存储设备OSD发送的所述聚合位置中所有的小文件并保存；以及
    获取所述存储区域唯一编号的绝对值对应区域中的小文件。
  14. 一种客户端，包括：
    读请求模块，设置为发起小文件的读请求；
    编号获取模块，设置为从元数据服务器获取所述小文件的全局唯一编号和存储区域唯一编号；
    计算模块，设置为根据所述全局唯一编号和所述存储区域唯一编号计算所述小文件在对象存储设备OSD中的聚合位置；以及
    小文件获取模块，设置为根据所述聚合位置从所述对象存储设备OSD中获取所述小文件。
  15. 如权利要求14所述的客户端，其中，所述全局唯一编号为索引节点编号ino，存储区域唯一编号为对象编号ono，所述聚合位置的计算公式为：聚合位置＝((ino+ono+1)＜＜32)|1，其中，＜＜32表示左移32个比特位，|表示按位或运算。
  16. 如权利要求14所述的客户端，所述客户端还包括：
    检测模块，设置为检测所述客户端的缓存中是否存在所述小文件；
    所述小文件获取模块，还设置为若所述客户端的缓存中存在所述小文件，则获取所述小文件；以及
    所述编号获取模块，还设置为若所述客户端的缓存中未存在所述小文件，则从元数据服务器获取所述小文件的全局唯一编号和存储区域唯一编号。
  17. 如权利要求14所述的客户端，其中，所述小文件获取模块包括：
    发送单元，设置为向所述对象存储设备OSD发送读聚合请求；
    接收单元，设置为接收所述对象存储设备OSD发送的所述聚合位置中所有的小文件并保存；以及
    获取单元，设置为获取所述存储区域唯一编号的绝对值对应区域中的小文件。
  18. 一种非暂态计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令设置为执行权利要求1-5和10-13中任一项的方法。
PCT/CN2016/111473 2015-12-23 2016-12-22 文件的写聚合、读聚合方法及系统和客户端 WO2017107948A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510980277.XA CN106911743B (zh) 2015-12-23 2015-12-23 小文件的写聚合、读聚合方法及系统和客户端
CN201510980277.X 2015-12-23

Publications (1)

Publication Number Publication Date
WO2017107948A1 true WO2017107948A1 (zh) 2017-06-29

Family

ID=59088982

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/111473 WO2017107948A1 (zh) 2015-12-23 2016-12-22 文件的写聚合、读聚合方法及系统和客户端

Country Status (2)

Country Link
CN (1) CN106911743B (zh)
WO (1) WO2017107948A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113590560A (zh) * 2021-06-29 2021-11-02 济南浪潮数据技术有限公司 一种分布式系统的缓存优化方法、系统、设备和存储介质
WO2023179742A1 (zh) * 2022-03-25 2023-09-28 阿里云计算有限公司 数据访问方法及系统、硬件卸载设备、电子设备及介质
CN117648297A (zh) * 2024-01-30 2024-03-05 中国人民解放军国防科技大学 基于对象存储小文件离线合并方法、系统、设备及介质

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107506466B (zh) * 2017-08-30 2020-08-04 郑州云海信息技术有限公司 一种小文件存储方法及系统
CN109241004B (zh) * 2018-09-26 2022-02-18 郑州云海信息技术有限公司 元数据文件大小恢复方法、系统、装置及可读存储介质
CN109597903B (zh) * 2018-11-21 2021-12-28 北京市商汤科技开发有限公司 图像文件处理装置和方法、文件存储系统及存储介质
CN111309687A (zh) * 2020-01-21 2020-06-19 苏州浪潮智能科技有限公司 一种对象存储小文件处理方法、装置、设备及存储介质
CN112463730B (zh) * 2020-12-02 2022-07-19 苏州浪潮智能科技有限公司 一种存储海量小文件分级优化的方法、系统及介质
CN113128175B (zh) * 2021-04-19 2023-01-24 福建福昕软件开发股份有限公司 一种大批量pdf文件合并的方法及系统
CN118132520A (zh) * 2024-05-08 2024-06-04 济南浪潮数据技术有限公司 存储系统文件处理方法、电子设备、存储介质及程序产品

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4660142A (en) * 1981-05-22 1987-04-21 Data General Corporation Digital data processing system employing an object-based addressing system with a single object table
CN104079600A (zh) * 2013-03-27 2014-10-01 中兴通讯股份有限公司 文件存储方法、装置、访问客户端及元数据服务器系统
CN105187502A (zh) * 2015-08-07 2015-12-23 北京思特奇信息技术股份有限公司 一种基于分布式弹性块存储的方法和系统

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103514210B (zh) * 2012-06-28 2017-06-27 华为技术有限公司 小文件处理方法及装置
CN104965845B (zh) * 2014-12-30 2017-04-12 浙江大华技术股份有限公司 一种小文件的定位方法及系统

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4660142A (en) * 1981-05-22 1987-04-21 Data General Corporation Digital data processing system employing an object-based addressing system with a single object table
CN104079600A (zh) * 2013-03-27 2014-10-01 中兴通讯股份有限公司 文件存储方法、装置、访问客户端及元数据服务器系统
CN105187502A (zh) * 2015-08-07 2015-12-23 北京思特奇信息技术股份有限公司 一种基于分布式弹性块存储的方法和系统

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
GUI, LI: "Research on Key Technologies for the Client of Distributed File System", ELECTRONIC TECHNOLOGY & INFORMATION SCIENCE , CHINA MASTER'S THESES FULL-TEXT DATABASE, 15 June 2014 (2014-06-15) *
JONES, M.T.;: "Ceph: A Linux Petabyte-Scale Distributed File System", IBM DEVELOPERWORKS, 4 June 2010 (2010-06-04), pages 1 - 7, XP055397106 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113590560A (zh) * 2021-06-29 2021-11-02 济南浪潮数据技术有限公司 一种分布式系统的缓存优化方法、系统、设备和存储介质
WO2023179742A1 (zh) * 2022-03-25 2023-09-28 阿里云计算有限公司 数据访问方法及系统、硬件卸载设备、电子设备及介质
CN117648297A (zh) * 2024-01-30 2024-03-05 中国人民解放军国防科技大学 基于对象存储小文件离线合并方法、系统、设备及介质
CN117648297B (zh) * 2024-01-30 2024-06-11 中国人民解放军国防科技大学 基于对象存储小文件离线合并方法、系统、设备及介质

Also Published As

Publication number Publication date
CN106911743A (zh) 2017-06-30
CN106911743B (zh) 2019-03-26

Similar Documents

Publication Publication Date Title
WO2017107948A1 (zh) 文件的写聚合、读聚合方法及系统和客户端
US10540323B2 (en) Managing I/O operations in a storage network
US11531482B2 (en) Data deduplication method and apparatus
US9846642B2 (en) Efficient key collision handling
US10831735B2 (en) Processing device configured for efficient generation of a direct mapped hash table persisted to non-volatile block memory
US10374792B1 (en) Layout-independent cryptographic stamp of a distributed dataset
US11249834B2 (en) Storage system with coordinated recovery across multiple input-output journals of different types
WO2018099107A1 (zh) 一种哈希表管理的方法和装置、计算机存储介质
JP6542909B2 (ja) ファイル操作方法及び装置
US9665485B2 (en) Logical and physical block addressing for efficiently storing data to improve access speed in a data deduplication system
WO2016045096A1 (zh) 一种文件迁移方法、装置和存储设备
US10423641B2 (en) Hierarchical coherency log for managing a distributed data storage system
WO2021077745A1 (zh) 一种分布式存储系统的数据读写方法
US11245774B2 (en) Cache storage for streaming data
EP2659377A2 (en) Adaptive index for data deduplication
US10725686B2 (en) Write stream separation into multiple partitions
US9110820B1 (en) Hybrid data storage system in an HPC exascale environment
CN110908589B (zh) 数据文件的处理方法、装置、系统和存储介质
WO2016095761A1 (zh) 缓存的处理方法和装置
CN107153512B (zh) 一种数据迁移方法和装置
CN103970875A (zh) 一种并行重复数据删除方法
US20130232124A1 (en) Deduplicating a file system
US10515055B2 (en) Mapping logical identifiers using multiple identifier spaces
US20150286442A1 (en) Cluster-wide memory management using similarity-preserving signatures
CN110168513B (zh) 在不同存储系统中对大文件的部分存储

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16877758

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16877758

Country of ref document: EP

Kind code of ref document: A1