CN111625515A - Method and device for processing operation request of aggregated small files - Google Patents
Method and device for processing operation request of aggregated small files Download PDFInfo
- Publication number
- CN111625515A CN111625515A CN202010469827.2A CN202010469827A CN111625515A CN 111625515 A CN111625515 A CN 111625515A CN 202010469827 A CN202010469827 A CN 202010469827A CN 111625515 A CN111625515 A CN 111625515A
- Authority
- CN
- China
- Prior art keywords
- metadata
- invalid space
- request
- record
- invalid
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000012545 processing Methods 0.000 title claims abstract description 34
- 230000004931 aggregating effect Effects 0.000 claims abstract description 24
- 230000037430 deletion Effects 0.000 claims abstract description 24
- 238000012217 deletion Methods 0.000 claims abstract description 24
- 230000004048 modification Effects 0.000 claims abstract description 24
- 238000012986 modification Methods 0.000 claims abstract description 24
- 238000003672 processing method Methods 0.000 claims abstract description 12
- 230000002776 aggregation Effects 0.000 claims description 23
- 238000004220 aggregation Methods 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 9
- 230000000694 effects Effects 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 8
- 230000002829 reductive effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a processing method of an operation request of a small aggregated file, which comprises the steps of firstly generating an invalid space record after receiving a deletion request or a modification request aiming at the small aggregated file, then completing a metadata updating operation by using a metadata server, adding the invalid space record into a target queue after the metadata updating operation is completed by the metadata server, and performing batch destaging on the invalid space record in the target queue when a preset time point is reached. Therefore, the method can avoid the phenomenon that the tray-dropping operation of the invalid space is required when the deletion request or the modification request is executed every time, improves the modification and deletion performance of the aggregated small files, reduces the pressure of the metadata server, and can ensure the accuracy of the invalid space to be dropped. In addition, the application also provides a processing device for aggregating operation requests of small files, terminal equipment of a distributed file system and a readable storage medium, and the technical effect of the processing device corresponds to that of the method.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for processing an operation request of an aggregated small file, a terminal device of a distributed file system, and a readable storage medium.
Background
In a small file aggregation scene of a distributed file system based on an object storage technology, if the mapping position of an aggregated small file in an aggregated large file is changed, an invalid space record is generated. In order to ensure that garbage objects can be cleaned subsequently, the generated invalid space records are subjected to a disk dropping process every time. Therefore, when the distributed file system receives a modification request or a deletion request for the aggregated small file, the metadata server not only needs to update the metadata, but also needs to perform a disk-dropping operation of an invalid space, which seriously affects the operation efficiency of the aggregated small file and reduces the performance of the distributed file system.
Therefore, how to avoid that the performance of the distributed file system is affected by executing a disk-dropping operation of an invalid space every time an operation request for aggregating small files is received is a problem to be solved by those skilled in the art.
Disclosure of Invention
The application aims to provide a processing method and device for an operation request of an aggregated small file, a terminal device of a distributed file system and a readable storage medium, which are used for solving the problem that the performance of the distributed file system is seriously influenced because a disk-dropping operation of an invalid space needs to be executed every time the operation request of the aggregated small file is received in the current scheme. The specific scheme is as follows:
in a first aspect, the present application provides a method for processing an operation request of aggregating doclets, which is applied to a client, and includes:
receiving an operation request aiming at the aggregation small file, wherein the operation request is a deletion request or a modification request;
recording the storage space of the aggregated small file as an invalid space, and generating an invalid space record;
executing the operation request, and sending a metadata updating request to a metadata server according to an execution result of the operation request;
after receiving a message of completing metadata updating fed back by the metadata server, adding the invalid space record to a target queue;
and when the preset time point is reached, carrying out a tray falling operation on the invalid space records in the target queue.
Preferably, the recording the storage space of the aggregated small file as an invalid space and generating an invalid space record includes:
recording the storage space of the small aggregated file as an invalid space, and generating an invalid space record, wherein the invalid space record comprises identification information of a large aggregated file corresponding to the small aggregated file, and also comprises offset and length of the small aggregated file in the large aggregated file.
Preferably, the executing the operation request and sending a metadata update request to a metadata server according to an execution result of the operation request includes:
executing the operation request, and updating the number of the metadata of the aggregation small file to obtain a number updating result; sending a metadata update request to a metadata server, wherein the metadata update request comprises the number update result;
correspondingly, after receiving the message that the updating of the metadata fed back by the metadata server is completed, adding the invalid space record to the target queue includes:
and after receiving a message of completing metadata updating fed back by the metadata server, if the message comprises the number updating result, adding an invalid space record corresponding to the number updating result to a target queue.
Preferably, after the executing the operation request, updating the number of the metadata of the aggregated doclet, and obtaining a number update result, the method further includes:
and adding the number updating result in the invalid space record.
Preferably, after receiving a message that metadata update fed back by the metadata server is completed, if the message includes the number update result, adding the invalid space record to a target queue, including:
and after receiving a message of completing metadata updating fed back by the metadata server, determining a target number updating result included in the message, and adding an invalid space record of which the number updating result is less than or equal to the target number updating result to a target queue.
Preferably, when the preset time point is reached, performing a disk dropping operation on the invalid space record in the target queue, including:
and periodically carrying out a disk dropping operation on the invalid space records in the target queue.
Preferably, before performing a disk-dropping operation on the invalid space record in the target queue when the preset time point is reached, the method further includes:
scanning the aggregated large file according to the object granularity according to the invalid space records in the target queue, and judging whether the storage space which is not marked as the invalid space is the invalid space;
and if so, generating a corresponding invalid space record and adding the invalid space record to the target queue.
In a second aspect, the present application provides a processing apparatus for aggregating operation requests of small files, including:
a request receiving module: the method comprises the steps of receiving an operation request aiming at an aggregation small file, wherein the operation request is a deletion request or a modification request;
a record generation module: the storage space used for recording the aggregation small files as an invalid space and generating invalid space records;
a request update module: the metadata server is used for executing the operation request and sending a metadata updating request to the metadata server according to an execution result of the operation request;
a record adding module: the invalid space record is added to a target queue after a message that the updating of the metadata fed back by the metadata server is completed is received;
a recording and landing module: and the method is used for performing a disk dropping operation on the invalid space records in the target queue when a preset time point is reached.
In a third aspect, the present application provides a terminal device of a distributed file system, including:
a memory: for storing a computer program;
a processor: for executing said computer program for implementing the steps of the method for processing an operation request for aggregating doclets as described above.
In a fourth aspect, the present application provides a readable storage medium having stored thereon a computer program for implementing, when being executed by a processor, the steps of the method for processing an operation request for aggregating doclets as described above.
The processing method for aggregating operation requests of small files provided by the application comprises the following steps: receiving an operation request aiming at the aggregated small file, wherein the operation request is a deletion request or a modification request; recording the storage space of the aggregated small files as an invalid space, and generating an invalid space record; executing the operation request, and sending a metadata updating request to the metadata server according to an execution result of the operation request; after receiving a message of completing metadata updating fed back by a metadata server, adding an invalid space record to a target queue; and when the preset time point is reached, carrying out a tray falling operation on the invalid space records in the target queue.
It can be seen that, after receiving a delete request or a modify request for the aggregated small file, the method first generates a corresponding invalid space record, then completes a metadata update operation by using a metadata server, adds the invalid space record into a target queue after the metadata update operation is completed by the metadata server, and performs batch destaging on the invalid space record in the target queue when a preset time point is reached. On one hand, the method avoids the disk dropping operation of an invalid space required for executing the deletion request or the modification request every time, improves the modification performance and deletion performance of the aggregated small files, and reduces the pressure of the metadata server; on the other hand, the method adds the invalid space record into the target queue only after the metadata server completes the metadata updating operation, so that the accuracy of the invalid space to be landed can be ensured.
In addition, the present application also provides a processing apparatus for aggregating operation requests of small files, a terminal device of a distributed file system, and a readable storage medium, where the technical effect corresponds to the technical effect of the above method, and is not described herein again.
Drawings
For a clearer explanation of the embodiments or technical solutions of the prior art of the present application, the drawings needed for the description of the embodiments or prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a flowchart illustrating a first implementation of a method for processing an operation request of an aggregated doclet according to an embodiment of the present disclosure;
fig. 2 is a flowchart illustrating an implementation of a process of dropping an invalid space in a second embodiment of a method for processing operation requests of aggregated small files according to the present application;
fig. 3 is a flowchart illustrating an implementation of a scan merging process according to a second embodiment of a method for processing an operation request of aggregating doclets provided by the present application;
fig. 4 is a functional block diagram of an embodiment of a processing apparatus for aggregating operation requests of small files provided in the present application.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the traditional scheme, after a client receives a deletion request or a modification request for an aggregated small file each time, a metadata server needs to respectively execute metadata updating operation and invalid space disk dropping operation, so that the metadata server has high computing pressure, the aggregated small file deletion and modification efficiency is low, and the performance of a distributed file system is seriously influenced.
In order to solve the problem, the application provides a processing method and device for aggregating operation requests of small files, a terminal device of a distributed file system and a readable storage medium. After a deletion request or a modification request aiming at the small aggregated files is received, generating corresponding invalid space records, after a metadata server finishes metadata updating operation, adding the invalid space records to a target queue to wait for disk dropping, and performing batch disk dropping operation on the invalid space records in the target queue when a preset time point is reached.
Therefore, the application has at least the following advantages: firstly, the method and the device do not need to execute the tray dropping operation of the invalid space after receiving the deletion request or the modification request aiming at the small aggregated files every time, reduce the tray dropping frequency of the invalid space, improve the modification and deletion performance of the small aggregated files and reduce the calculation pressure of a metadata server; secondly, completing the disk dropping operation of the invalid space by using the client, and further reducing the computing pressure of the metadata server; thirdly, if and only after the metadata server completes the metadata updating operation, the invalid space record is added to the target queue, so that the accuracy of the invalid space to be landed can be ensured.
Referring to fig. 1, a first embodiment of a method for processing an operation request of aggregating small files provided in the present application is described below, where the first embodiment includes:
s101, receiving an operation request aiming at the aggregated small file, wherein the operation request is a deletion request or a modification request;
s102, recording the storage space of the aggregated small files as an invalid space, and generating an invalid space record;
s103, executing the operation request, and sending a metadata updating request to a metadata server according to an execution result of the operation request;
s104, after receiving a message of completing metadata updating fed back by the metadata server, adding the invalid space record to a target queue;
and S105, when the preset time point is reached, performing a disk dropping operation on the invalid space records in the target queue.
The embodiment is applied to the client of the distributed file system based on the object storage technology. In a small file application scenario, aggregation storage of small files is an important means for improving the utilization rate of a storage system and increasing the file reading speed. In a small file aggregation scene, a plurality of small files are aggregated by taking an object as a unit, and the object is stored in a disk as data of an aggregated large file.
In this embodiment, the operation request refers to a deletion request or a modification request. When the aggregate small file is deleted, the storage space where the aggregate small file originally exists becomes an invalid space. When the small aggregated file is modified, two processing modes are adopted, wherein one processing mode is writing other positions, and the other processing mode is changing into a non-aggregated file, and the two processing modes can change the storage space where the small aggregated file originally locates into an invalid space.
Therefore, after receiving an operation request for the aggregate small file, the present embodiment generates an invalid space record. The invalid space record is used for recording an original storage space of the aggregate small file, and may specifically include identification information of an aggregate large file in which the aggregate small file is located, an offset and a length of the aggregate small file in the aggregate large file, and the like.
The process of executing the operation request specifically includes: and modifying or deleting the data of the aggregation small file, and updating the metadata of the aggregation small file. Then, a metadata updating request is sent to the metadata server according to the metadata updating result on the client. After the metadata server completes the metadata updating operation, a message is sent to the client. And then, the client knows that the metadata server side completes the metadata updating operation, and further adds the generated invalid space record to the target queue.
It can be understood that the metadata update request sent by the client to the metadata server includes the identification information of the metadata, so the metadata server knows which metadata is updated; correspondingly, the message which is sent by the metadata server to the client and represents that the updating of the metadata is completed also comprises the identification information of the metadata, so that the client knows which metadata completes the updating process at the metadata server side, and the corresponding invalid space record is added to the target queue.
The target queue is configured to store an invalid space record to be landed, and specifically, the target queue may perform a landing operation on the invalid space record in the target queue at regular time, for example, periodically perform a landing operation on the invalid space record in the target queue according to a certain frequency. By the aid of the method, the number of times of falling of invalid space is reduced, the frequency of falling of invalid space is reduced, deletion and modification performance of aggregated small files is improved, the computing pressure of a metadata server is reduced, and the purpose of improving performance of a distributed file system is finally achieved.
After receiving a delete request or a modify request for the aggregated small file, the method in this embodiment first generates a corresponding invalid space record, then completes a metadata update operation by using a metadata server, adds the invalid space record to a target queue after the metadata update operation is completed by the metadata server, and performs batch destaging on the invalid space records in the target queue when a preset time point is reached. On one hand, the method avoids the disk dropping operation of an invalid space required for executing the deletion request or the modification request every time, improves the modification performance and deletion performance of the aggregated small files, and reduces the pressure of the metadata server; on the other hand, the method adds the invalid space record into the target queue only after the metadata server completes the metadata updating operation, so that the accuracy of the invalid space to be landed can be ensured.
The second embodiment of the processing method for aggregating operation requests of small files provided by the present application is described in detail below, and the second embodiment is implemented based on the first embodiment and is expanded to a certain extent on the basis of the first embodiment.
Specifically, the second embodiment defines that the client updates the identification information of the metadata, that is, the number of the metadata when executing the operation request; and defines the invalid space record to include the updated metadata number. Based on the above conditions, considering that the metadata server performs metadata updating operation according to the order of the metadata numbers from small to large, after receiving a message that the metadata updating is completed and fed back by the metadata server, the client first determines the metadata number corresponding to the message, and then directly adds all invalid space records less than or equal to the metadata number to the target queue, thereby avoiding unnecessary waiting process, reducing the number of times of comparing the metadata numbers, and further improving the performance of the distributed file system.
Referring to fig. 2, the second embodiment specifically includes:
s201, receiving an operation request aiming at the aggregated small file, wherein the operation request is a deletion request or a modification request;
s202, recording the storage space of the aggregated small files as an invalid space, and generating an invalid space record;
the invalid space record comprises identification information of an aggregation large file corresponding to the aggregation small file, and also comprises offset and length of the aggregation small file in the aggregation large file. Thus, the invalid space record may specifically be: (aggregate large file ino number, offset, length len). After the invalid space record is generated, it may be placed in a first queue of the local cache.
S203, executing the operation request, and updating the number of the metadata of the aggregation small file to obtain a number updating result;
the number update result may be denoted as tid, specifically, the updated metadata id. And the client increases the metadata id each time the metadata is modified, and the metadata server persists the corresponding metadata according to the tid and then responds to the client.
S204, adding the number updating result in the invalid space record;
specifically, the cache record is extracted from the first queue, the corresponding number updating result is added to the invalid space record, and the added invalid space record is placed into the second queue.
S205, sending a metadata updating request to a metadata server, wherein the metadata updating request comprises the number updating result;
the embodiment does not limit the sequence of S204 and S205.
S206, after receiving the message of completing the metadata updating fed back by the metadata server, determining a target number updating result included in the message, and adding an invalid space record of which the number updating result is less than or equal to the target number updating result to a target queue;
specifically, an invalid space record with the number updating result less than or equal to the target number updating result is selected from the second queue and added to the target queue.
And S207, periodically carrying out a tray falling operation on the invalid space records in the target queue.
In addition, the embodiment also provides a process for supplementing and merging the invalid space through scanning. As shown in fig. 3, the process specifically includes: loading an invalid space corresponding to a previously recorded aggregation large file to a disk, and scanning the aggregation large file at regular time according to the object granularity; for each object, scanning is started from the end of the first invalid record, metadata of the aggregation small file to which the space belongs is obtained according to the characteristics of the object, and whether the current storage space is an invalid space is judged according to the metadata; if the current storage space is an invalid space, generating an invalid space record, and continuously scanning the next data area; if the current storage space is not the invalid space, skipping the current object and continuing to scan the next object. And after the aggregated large file is scanned, combining the newly generated invalid space record with the original invalid space record.
Therefore, according to the method for processing the operation request of the aggregated small file, aiming at the aggregated file, a merging scheme of invalid space recording, batch asynchronous disk dropping and scanning supplement is designed, so that the modification and deletion performance of the aggregated small file is improved, the metadata cluster pressure is reduced, and meanwhile, the invalid garbage data can be completely recorded.
In the following, a processing apparatus for aggregating small file operation requests provided by an embodiment of the present application is described, and a processing apparatus for aggregating small file operation requests described below and a processing method for aggregating small file operation requests described above may be referred to correspondingly.
As shown in fig. 4, the processing apparatus for processing an operation request of aggregating small files according to this embodiment includes:
the request receiving module 401: the method comprises the steps of receiving an operation request aiming at an aggregation small file, wherein the operation request is a deletion request or a modification request;
the record generation module 402: the storage space used for recording the aggregation small files as an invalid space and generating invalid space records;
request update module 403: the metadata server is used for executing the operation request and sending a metadata updating request to the metadata server according to an execution result of the operation request;
the record addition module 404: the invalid space record is added to a target queue after a message that the updating of the metadata fed back by the metadata server is completed is received;
the recording drop module 405: and the method is used for performing a disk dropping operation on the invalid space records in the target queue when a preset time point is reached.
The processing apparatus for processing an operation request of an aggregated small file of this embodiment is used to implement the foregoing processing method for an operation request of an aggregated small file, and therefore specific implementations of the apparatus can be seen in the foregoing embodiment parts of the processing method for an operation request of an aggregated small file, for example, the request receiving module 401, the record generating module 402, the request updating module 403, the record adding module 404, and the record dropping module 405, and steps S101, S102, S103, S104, and S105 of the processing method for an operation request of an aggregated small file are respectively used to implement the foregoing steps of the processing method for an operation request of an aggregated small file, so specific implementations thereof can refer to descriptions of corresponding respective part embodiments, and a description thereof will not be further expanded herein.
In addition, since the processing apparatus for processing an operation request of aggregating small files of this embodiment is used to implement the foregoing processing method for processing an operation request of aggregating small files, the role thereof corresponds to the role of the foregoing method, and details are not described here.
In addition, the present application also provides a terminal device of a distributed file system, including:
a memory: for storing a computer program;
a processor: for executing said computer program for implementing the steps of the method for processing an operation request for aggregating doclets as described above.
Finally, the present application provides a readable storage medium having stored thereon a computer program for implementing the steps of the method for processing an operation request for aggregating doclets as described above when executed by a processor.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The above detailed descriptions of the solutions provided in the present application, and the specific examples applied herein are set forth to explain the principles and implementations of the present application, and the above descriptions of the examples are only used to help understand the method and its core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A processing method for aggregating operation requests of small files is applied to a client and comprises the following steps:
receiving an operation request aiming at the aggregation small file, wherein the operation request is a deletion request or a modification request;
recording the storage space of the aggregated small file as an invalid space, and generating an invalid space record;
executing the operation request, and sending a metadata updating request to a metadata server according to an execution result of the operation request;
after receiving a message of completing metadata updating fed back by the metadata server, adding the invalid space record to a target queue;
and when the preset time point is reached, carrying out a tray falling operation on the invalid space records in the target queue.
2. The method of claim 1, wherein said marking the storage space of the aggregate doclet as invalid space and generating an invalid space record comprises:
recording the storage space of the small aggregated file as an invalid space, and generating an invalid space record, wherein the invalid space record comprises identification information of a large aggregated file corresponding to the small aggregated file, and also comprises offset and length of the small aggregated file in the large aggregated file.
3. The method of claim 2, wherein the executing the operation request, sending a metadata update request to a metadata server based on a result of the executing of the operation request, comprises:
executing the operation request, and updating the number of the metadata of the aggregation small file to obtain a number updating result; sending a metadata update request to a metadata server, wherein the metadata update request comprises the number update result;
correspondingly, after receiving the message that the updating of the metadata fed back by the metadata server is completed, adding the invalid space record to the target queue includes:
and after receiving a message of completing metadata updating fed back by the metadata server, if the message comprises the number updating result, adding an invalid space record corresponding to the number updating result to a target queue.
4. The method of claim 3, wherein after the executing the operation request updates the number of the metadata of the aggregate doclet to obtain a number update result, further comprising:
and adding the number updating result in the invalid space record.
5. The method of claim 4, wherein the adding the invalid space record to a target queue if the message includes the number update result after receiving the message that the metadata update fed back by the metadata server is completed comprises:
and after receiving a message of completing metadata updating fed back by the metadata server, determining a target number updating result included in the message, and adding an invalid space record of which the number updating result is less than or equal to the target number updating result to a target queue.
6. The method of claim 1, wherein performing a destage operation on the invalid space records in the target queue when a preset time point is reached comprises:
and periodically carrying out a disk dropping operation on the invalid space records in the target queue.
7. The method according to any one of claims 1-6, wherein before performing a disk-down operation on the invalid space records in the target queue when the preset time point is reached, the method further comprises:
scanning the aggregated large file according to the object granularity according to the invalid space records in the target queue, and judging whether the storage space which is not marked as the invalid space is the invalid space;
and if so, generating a corresponding invalid space record and adding the invalid space record to the target queue.
8. A device for processing operation requests for aggregating doclets, comprising:
a request receiving module: the method comprises the steps of receiving an operation request aiming at an aggregation small file, wherein the operation request is a deletion request or a modification request;
a record generation module: the storage space used for recording the aggregation small files as an invalid space and generating invalid space records;
a request update module: the metadata server is used for executing the operation request and sending a metadata updating request to the metadata server according to an execution result of the operation request;
a record adding module: the invalid space record is added to a target queue after a message that the updating of the metadata fed back by the metadata server is completed is received;
a recording and landing module: and the method is used for performing a disk dropping operation on the invalid space records in the target queue when a preset time point is reached.
9. A terminal device of a distributed file system, comprising:
a memory: for storing a computer program;
a processor: steps of a method for executing said computer program for carrying out the processing of an operation request for an aggregated doclet according to any one of claims 1 to 7.
10. A readable storage medium, characterized in that it has stored thereon a computer program for implementing, when being executed by a processor, the steps of a method for processing an operation request for aggregating doclets according to any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010469827.2A CN111625515A (en) | 2020-05-28 | 2020-05-28 | Method and device for processing operation request of aggregated small files |
| PCT/CN2021/073259 WO2021238246A1 (en) | 2020-05-28 | 2021-01-22 | Method and apparatus for processing operation request for aggregation small file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010469827.2A CN111625515A (en) | 2020-05-28 | 2020-05-28 | Method and device for processing operation request of aggregated small files |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111625515A true CN111625515A (en) | 2020-09-04 |
Family
ID=72260087
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010469827.2A Withdrawn CN111625515A (en) | 2020-05-28 | 2020-05-28 | Method and device for processing operation request of aggregated small files |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111625515A (en) |
| WO (1) | WO2021238246A1 (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148800A (en) * | 2020-10-20 | 2020-12-29 | 北京天华星航科技有限公司 | Distributed data storage system |
| CN113704027A (en) * | 2021-10-29 | 2021-11-26 | 苏州浪潮智能科技有限公司 | File aggregation compatible method and device, computer equipment and storage medium |
| WO2021238246A1 (en) * | 2020-05-28 | 2021-12-02 | 苏州浪潮智能科技有限公司 | Method and apparatus for processing operation request for aggregation small file |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115576505B (en) * | 2022-12-13 | 2023-03-03 | 浪潮电子信息产业股份有限公司 | Data storage method, device and equipment and readable storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095489A (en) * | 2015-08-18 | 2015-11-25 | 浪潮(北京)电子信息产业有限公司 | Distributed file deletion method, device and system |
| CN107704203B (en) * | 2017-09-27 | 2021-08-31 | 郑州云海信息技术有限公司 | Deletion method, device and equipment for aggregated large file and computer storage medium |
| US20190114082A1 (en) * | 2017-10-17 | 2019-04-18 | HoneycombData Inc. | Coordination Of Compaction In A Distributed Storage System |
| CN111125034A (en) * | 2019-12-27 | 2020-05-08 | 深信服科技股份有限公司 | Aggregation object data processing method, system and related equipment |
| CN111625515A (en) * | 2020-05-28 | 2020-09-04 | 苏州浪潮智能科技有限公司 | Method and device for processing operation request of aggregated small files |
-
2020
- 2020-05-28 CN CN202010469827.2A patent/CN111625515A/en not_active Withdrawn
-
2021
- 2021-01-22 WO PCT/CN2021/073259 patent/WO2021238246A1/en active Application Filing
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021238246A1 (en) * | 2020-05-28 | 2021-12-02 | 苏州浪潮智能科技有限公司 | Method and apparatus for processing operation request for aggregation small file |
| CN112148800A (en) * | 2020-10-20 | 2020-12-29 | 北京天华星航科技有限公司 | Distributed data storage system |
| CN113704027A (en) * | 2021-10-29 | 2021-11-26 | 苏州浪潮智能科技有限公司 | File aggregation compatible method and device, computer equipment and storage medium |
| CN113704027B (en) * | 2021-10-29 | 2022-02-18 | 苏州浪潮智能科技有限公司 | File aggregation compatible method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021238246A1 (en) | 2021-12-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111625515A (en) | Method and device for processing operation request of aggregated small files | |
| US10133679B2 (en) | Read cache management method and apparatus based on solid state drive | |
| CN107943718B (en) | Method and device for cleaning cache file | |
| US20130097402A1 (en) | Data prefetching method for distributed hash table dht storage system, node, and system | |
| CN111176560B (en) | Cache management method and device, computer equipment and storage medium | |
| CN109614377A (en) | File delet method, device, equipment and the storage medium of distributed file system | |
| CN110532201B (en) | Metadata processing method and device | |
| CN113298695B (en) | Image processing assembly and chip, image processing method and storage medium | |
| CN110727404A (en) | Data deduplication method and device based on storage end and storage medium | |
| CN109213450B (en) | Associated metadata deleting method, device and equipment based on flash memory array | |
| CN104092670A (en) | Method for utilizing network cache server to process files and device for processing cache files | |
| US11461239B2 (en) | Method and apparatus for buffering data blocks, computer device, and computer-readable storage medium | |
| CN113110801A (en) | Method, system, equipment and storage medium for accelerating small file reading speed | |
| WO2022048475A1 (en) | Data deduplication method, node, and computer readable storage medium | |
| CN108280123B (en) | HBase column polymerization method | |
| CN114442961B (en) | Data processing method, device, computer equipment and storage medium | |
| CN109582233A (en) | A kind of caching method and device of data | |
| CN112214178B (en) | Storage system, data reading method and data writing method | |
| CN115061630A (en) | Data migration method, device, equipment and medium | |
| CN110362769B (en) | Data processing method and device | |
| CN115809263A (en) | Data processing method, device, equipment and storage medium | |
| CN111625506A (en) | Distributed data deleting method, device and equipment based on deleting queue | |
| CN112463741A (en) | Cleaning method for aggregated large files and related equipment | |
| CN113626382A (en) | Index method, system and related device for aggregation object in distributed file system | |
| CN107506156B (en) | Io optimization method of block device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200904 |