CN107729142B - Thread calling method for self-compaction metadata - Google Patents
Thread calling method for self-compaction metadata Download PDFInfo
- Publication number
- CN107729142B CN107729142B CN201710909927.0A CN201710909927A CN107729142B CN 107729142 B CN107729142 B CN 107729142B CN 201710909927 A CN201710909927 A CN 201710909927A CN 107729142 B CN107729142 B CN 107729142B
- Authority
- CN
- China
- Prior art keywords
- read
- metadata
- thread
- task
- disk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000005056 compaction Methods 0.000 title abstract description 23
- 230000008569 process Effects 0.000 claims abstract description 22
- 230000006870 function Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 101150039208 KCNK3 gene Proteins 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Abstract
The invention discloses a thread calling method for self-compaction metadata, which comprises the following steps: establishing a metadata read-write thread and a read-write disk thread, and adding all tasks of the self-simplified metadata into a linked list in a serial mode; sequentially reading tasks from the linked list through a metadata read-write thread for processing; the metadata read-write thread adopts a B + tree read-write thread; when the nodes of the tree in the metadata read-write thread are not in the memory and the disk needs to be read, the task which is being processed by the metadata read-write thread is put back to the linked list again, the disk reading operation is handed over to the disk read-write thread for processing, and the metadata read-write thread processes the next task. The method improves metadata processing efficiency.
Description
Technical Field
The invention relates to the technical field of metadata, in particular to a thread calling method for self-compaction metadata.
Background
At present, the self-compaction technology is one of the basic technologies applied to a storage system, and is also a basic function of a medium-high end storage system (including part of low end storage), and the key point of the self-compaction technology is that the storage system can allocate the used space as required, that is, the storage system does not need to allocate all the space immediately after creating a volume, but allocates the space as required, so that the utilization rate of a storage system disk can be improved, and the cost can be reduced.
The self-compaction technology can play a great role in space utilization efficiency and improve the disk utilization efficiency, but compared with the common volume, the self-compaction volume does not adopt a pre-allocation mode but uses and allocates space at the same time, so that an additional metadata management mechanism is inevitably required to be added, and the performance loss of the self-compaction volume to the common volume is caused. The main reason for performance loss is that the self-compaction volume needs to read and write metadata first before reading and writing data, and currently, metadata management is usually implemented by using a B + Tree, and metadata is read and written in a multithreading manner. Firstly, the tree needs to be locked to avoid competition when the metadata is read and written in multiple threads, so that the tree cannot be operated in parallel although the tree has multiple threads, the reading and writing is still in a serial mode, the advantage of multiple threads is not exerted, and time is consumed due to frequent locking and unlocking; secondly, because all nodes in the tree only have a part in the memory, when reading and writing the metadata, multiple disk reading operations may occur, so if waiting for the metadata processing to complete each time, much time may be consumed due to waiting for disk reading, the performance is reduced, and the metadata processing efficiency is low. Therefore, how to improve the processing efficiency of metadata is an urgent problem to be solved.
Disclosure of Invention
The invention aims to provide a thread calling method for self-compaction metadata so as to improve the metadata processing efficiency.
In order to solve the technical problem, the invention provides a thread calling method for self-compaction metadata, which comprises the following steps:
establishing a metadata read-write thread and a read-write disk thread, and adding all tasks of the self-simplified metadata into a linked list in a serial mode;
sequentially reading tasks from the linked list through a metadata read-write thread for processing; the metadata read-write thread adopts a B + tree read-write thread;
when the nodes of the tree in the metadata read-write thread are not in the memory and the disk needs to be read, the task which is being processed by the metadata read-write thread is put back to the linked list again, the disk reading operation is handed over to the disk read-write thread for processing, and the metadata read-write thread processes the next task.
Preferably, the number of the metadata read-write threads is one, and the number of the read-write disk threads is multiple.
Preferably, after the task being processed by the metadata read-write thread is placed back to the linked list again, the method further includes:
the task placed back in the linked list is marked as disk-read.
Preferably, the method further comprises:
and when the disk reading operation is finished and the nodes of the tree are stored in the memory, canceling the disk reading state of the task in the chain list, and processing the task through the metadata reading and writing thread.
Preferably, after marking the task put back into the linked list as the disk reading state, the method further comprises:
the task marked as disk-read state is added to the beginning of the linked list.
Preferably, after all tasks of the self-reduced metadata are added to the linked list in a serial manner, the method further includes:
and when a new task is generated, adding the new task to the end of the linked list and waiting for execution.
Preferably, if the task of setting the disk reading mark already exists at the beginning of the linked list, the task of replacing the linked list is inserted from the beginning of the linked list to the position after the task of setting.
The thread calling method of the self-simplified metadata provided by the invention establishes a metadata read-write thread and a read-write disk thread, and adds all tasks of the self-simplified metadata to a linked list in a serial mode; sequentially reading tasks from the linked list through a metadata read-write thread for processing; the metadata read-write thread adopts a B + tree read-write thread; when the nodes of the tree in the metadata read-write thread are not in the memory and the disk needs to be read, the task which is being processed by the metadata read-write thread is put back to the linked list again, the disk reading operation is handed over to the disk read-write thread for processing, and the metadata read-write thread processes the next task. Therefore, a special B + tree read-write thread is arranged, all metadata operations are added to the linked list in a serial mode, the tasks are read from the linked list by the thread and are sequentially executed, and all tree-related operations are completed by the special thread, so that competition among the threads is effectively avoided, and locking and unlocking operations are also saved; and meanwhile, when the nodes of the tree are not in the memory and the disk needs to be read, the task is put back to the linked list again, the disk reading operation is finished by other threads, and the metadata reading and writing thread continues to process the next task. Different thread functions are set, a special metadata read-write thread processes metadata in a non-waiting mode, and related metadata read-write disks are processed by the special read-write disk thread, so that the metadata processing thread is ensured not to need waiting time all the time, the efficiency of metadata processing is improved, and the read-write efficiency of the whole self-compaction volume is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flowchart of a method for thread invocation of self-compaction metadata according to the present invention;
FIG. 2 is a task execution flow diagram;
FIG. 3 is a diagram of a task chain.
Detailed Description
The core of the invention is to provide a thread calling method of self-simplified metadata so as to improve the processing efficiency of the metadata.
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart of a method for thread invocation of self-compaction metadata according to the present invention, the method includes:
s11: establishing a metadata read-write thread and a read-write disk thread, and adding all tasks of the self-simplified metadata into a linked list in a serial mode;
s12: sequentially reading tasks from the linked list through a metadata read-write thread for processing;
the metadata read-write thread adopts a B + tree read-write thread;
s13: when the nodes of the tree in the metadata read-write thread are not in the memory and the disk needs to be read, the task which is being processed by the metadata read-write thread is put back to the linked list again, the disk reading operation is handed over to the disk read-write thread for processing, and the metadata read-write thread processes the next task.
Therefore, a special B + tree read-write thread is arranged, all metadata operations are added to the linked list in a serial mode, the tasks are read from the linked list by the thread and are sequentially executed, and all tree-related operations are completed by the special thread, so that competition among the threads is effectively avoided, and locking and unlocking operations are also saved; and meanwhile, when the nodes of the tree are not in the memory and the disk needs to be read, the task is put back to the linked list again, the disk reading operation is finished by other threads, and the metadata reading and writing thread continues to process the next task. Different thread functions are set, a special metadata read-write thread processes metadata in a non-waiting mode, and related metadata read-write disks are processed by the special read-write disk thread, so that the metadata processing thread is ensured not to need waiting time all the time, the efficiency of metadata processing is improved, and the read-write efficiency of the whole self-compaction volume is improved.
Based on the method, specifically, the number of the metadata read-write threads is one, and the number of the disk read-write threads is multiple.
Different thread functions are set, a special metadata read-write thread processes metadata in a non-waiting mode, and related metadata read-write disks are processed by the special read-write disk thread, so that the thread use efficiency and the metadata processing performance are improved.
Further, after all tasks of the self-reduced metadata are added to the linked list in a serial manner in step S11, the method further includes: and when a new task is generated, adding the new task to the end of the linked list and waiting for execution.
Further, in step S13, after the task being processed by the metadata read-write thread is put back to the linked list, the method further includes: the task placed back in the linked list is marked as disk-read.
Specifically, after the task put back into the linked list again is marked as a disk reading state, the method further includes: the task marked as disk-read state is added to the beginning of the linked list.
If the task of setting the disk reading mark exists at the beginning of the linked list, the task which is put back into the linked list again is inserted into the position behind the set task from the beginning of the linked list.
Further, after step S13, the method further includes: and after the disk reading operation is finished and the nodes of the tree are stored in the memory, the disk reading state of the task in the chain table is canceled and the task is processed through the metadata reading and writing thread.
The method sets a special metadata read-write thread, and processes the self-simplified metadata processing task in a serial mode, thereby avoiding locking and unlocking and waiting for reading the disk. And setting a plurality of disc reading and writing threads, and handing over the type of thread for processing when the operation of reading and writing the disc is needed in the metadata task, so that the metadata reading and writing threads do not need to process and wait for reading and writing the disc. And setting a task linked list for storing the self-simplified metadata task. When a new task is generated, the new task is inserted into the tail of the linked list, and when the new task is inserted back into the linked list, the tasks are placed from the head of the linked list to the rear of the set task, so that all the tasks are guaranteed to have equal priority. The method improves the processing efficiency of the self-compaction metadata and the read-write performance of the self-compaction volume, thereby indirectly improving the performance of the whole storage product and improving the product competitiveness.
At present, since the self-compaction volume is realized through a B + tree, limited by the size of a memory, metadata cannot be completely stored in the memory, but a follow-up and follow-up mode is adopted, namely when a certain node needs to be read, if the node is not in the memory, the node is read from a disk into the memory, and when the use of the memory reaches a threshold value, part of unused nodes are written back into the disk. The method divides the threads into two types, namely a metadata read-write thread and a disk read-write thread, wherein the metadata read-write thread is only one, and the disk read-write thread can be multiple.
The method comprises the steps that a metadata read-write thread is specially used for reading and writing a B + tree, the thread can complete the whole read-write process of metadata only when nodes of the tree are in a memory, if part of nodes in a path from a root node to a leaf node of the tree are found not to be read into the memory in the process of processing, the task is set to be in a disk read state, the operation of the disk read is handed over to a disk read-write thread for processing, then the task is inserted back into a linked list again, and after the nodes are read into the memory, the disk read state of the task is canceled and the task is waited to be processed by the metadata processing thread. In the process, the metadata processing thread processes the next task first, and the metadata processing thread is guaranteed not to need waiting time all the time (except the case of no task processing), so that the efficiency of metadata processing is improved, and the reading and writing efficiency of the whole self-compaction volume is improved. The specific flow chart is shown in fig. 2.
The method sets a special B + tree read-write thread, all metadata operations are added to the linked list in a serial mode, the threads read tasks from the linked list and execute the tasks in sequence, and all tree-related operations are completed by one special thread, so that competition among the threads is effectively avoided, and locking and unlocking operations are also saved; and meanwhile, when the nodes of the tree are not in the memory and the disk needs to be read, the task is put back into the linked list again, the disk reading operation is finished by other threads, and after the disk reading is finished and the needed tree nodes are stored in the memory, the task is processed by the metadata reading and writing thread.
As for the processing of the Task linked list, as shown in fig. 3, the metadata read-write thread sequentially reads the tasks from the linked list for processing, and adds the tasks to the end of the Task linked list for a new generated Task, and waits for execution (e.g., Task 6); for the Task (such as Task1) needing to read the disc, the disc reading mark is set and then added to the beginning of the Task linked list so as to be executed preferentially after the disc reading is finished, if the Task with the disc reading mark already exists at the beginning of the linked list, the Task is inserted from the beginning of the linked list to the position after the set Task, and therefore, each Task is guaranteed to have equal priority; in addition, it should be noted that when the metadata read-write thread reads the linked list, starting from the head of the list, once the task is found to set the disk reading flag, the next task is directly skipped over and read.
Based on the method, different thread functions are set in detail, a special metadata read-write thread processes the metadata in a non-waiting mode, and the related metadata read-write disk is processed by the special read-write disk thread, so that the thread use efficiency and the metadata processing performance are improved. A special metadata reading and writing thread is arranged to process the self-simplified metadata processing task in a serial mode, so that locking and unlocking are avoided, and disk reading is waited. And setting a plurality of disk reading and writing threads, and handing over the type of thread to process when the operation of reading and writing the disk is needed in the metadata task, so that the metadata reading and writing threads do not need to process and wait for reading and writing the disk. When a task reads or writes a disk, the task is set to a flag bit of reading the disk, which indicates that the task is not processed for the moment. For the tasks needing to read the disk, when the tasks are inserted back into the linked list, all the tasks are guaranteed to have equal priority from the head of the linked list to the set tasks. And setting a task linked list for storing the self-simplified metadata task. When a new task is generated, it is inserted at the end of the linked list.
The invention carries out special design optimization aiming at the thread management of the common self-compaction metadata processing, processes the metadata by setting a special thread and hands the read disk to another special thread, thereby reducing the waiting time in the metadata processing process, further improving the reading and writing efficiency of the whole self-compaction volume and improving the system performance.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The thread calling method of the self-compaction metadata provided by the invention is described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (7)
1. A thread calling method for self-compacting metadata is characterized by comprising the following steps:
establishing a metadata read-write thread and a read-write disk thread, and adding all tasks of the self-simplified metadata into a linked list in a serial mode;
sequentially reading tasks from the linked list through a metadata read-write thread for processing; the metadata read-write thread adopts a B + tree read-write thread; the B + tree reading and writing thread is used for reading and writing the B + tree;
when partial nodes of the tree in the metadata read-write thread are not in the memory and the disk needs to be read, the task which is being processed by the metadata read-write thread is put back to the linked list again, the disk reading operation is handed over to the disk reading-writing thread for processing, and the metadata read-write thread processes the next task.
2. The method of claim 1, wherein the number of metadata read-write threads is one, and the number of read-write disk threads is multiple.
3. The method of claim 2, wherein after replacing the tasks being processed by the metadata read-write thread back into the linked list, further comprising:
the task placed back in the linked list is marked as disk-read.
4. The method of claim 3, further comprising:
and when the disk reading operation is finished and the nodes of the tree are stored in the memory, canceling the disk reading state of the task in the chain list, and processing the task through the metadata reading and writing thread.
5. The method of claim 3, wherein after marking the tasks placed back in the linked list as a disk read state, further comprising:
the task marked as disk-read state is added to the beginning of the linked list.
6. The method of claim 1, wherein after all tasks of the self-reducing metadata are added to the linked list in a serial manner, further comprising:
and when a new task is generated, adding the new task to the end of the linked list and waiting for execution.
7. The method of claim 5, wherein the task to be placed back into the linked list is inserted from the beginning of the linked list to a location after the set task if the task to set the disk read flag already exists at the beginning of the linked list.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710909927.0A CN107729142B (en) | 2017-09-29 | 2017-09-29 | Thread calling method for self-compaction metadata |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710909927.0A CN107729142B (en) | 2017-09-29 | 2017-09-29 | Thread calling method for self-compaction metadata |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107729142A CN107729142A (en) | 2018-02-23 |
| CN107729142B true CN107729142B (en) | 2021-06-29 |
Family
ID=61209220
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710909927.0A Active CN107729142B (en) | 2017-09-29 | 2017-09-29 | Thread calling method for self-compaction metadata |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107729142B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110008030B (en) * | 2019-04-16 | 2020-06-19 | 苏州浪潮智能科技有限公司 | Method, system and equipment for accessing metadata |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2751667A4 (en) * | 2011-08-29 | 2015-07-15 | Intel Corp | Parallel operation on b+ trees |
| CN102571925B (en) * | 2011-12-19 | 2014-08-27 | 华中科技大学 | RDMA (Remote Direct Memory Access)-based data access method in fiber channel environment |
| EP2802991B1 (en) * | 2012-01-12 | 2020-05-06 | SanDisk Technologies LLC | Systems and methods for managing cache admission |
| US9218204B2 (en) * | 2012-12-21 | 2015-12-22 | Advanced Micro Devices, Inc. | Processing engine for complex atomic operations |
| CN103294786A (en) * | 2013-05-17 | 2013-09-11 | 华中科技大学 | Metadata organization and management method and system of distributed file system |
| CN104331478B (en) * | 2014-11-05 | 2017-09-22 | 浪潮电子信息产业股份有限公司 | It is a kind of to simplify memory system data consistency management method certainly |
| CN104408128B (en) * | 2014-11-26 | 2017-11-03 | 上海爱数信息技术股份有限公司 | A kind of reading optimization method indexed based on B+ trees asynchronous refresh |
-
2017
- 2017-09-29 CN CN201710909927.0A patent/CN107729142B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN107729142A (en) | 2018-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20150186068A1 (en) | Command queuing using linked list queues | |
| CN106569891B (en) | Method and device for scheduling and executing tasks in storage system | |
| JP2003044296A5 (en) | ||
| CN103164490B (en) | A kind of efficient storage implementation method of not fixed-length data and device | |
| CN105095287A (en) | LSM (Log Structured Merge) data compact method and device | |
| US20150253992A1 (en) | Memory system and control method | |
| WO2021043212A1 (en) | Log data storage method, electronic device and storage medium | |
| CN107122130B (en) | Data deduplication method and device | |
| CN103425435B (en) | Disk storage method and disk storage system | |
| CN103370691A (en) | Managing buffer overflow conditions | |
| CN108733324B (en) | Data reading and writing method, device and equipment of solid state disk and storage medium | |
| CN107391544B (en) | Processing method, device and equipment of column type storage data and computer storage medium | |
| CN103413569B (en) | One reads and one writes static RAM | |
| CN105045632A (en) | Method and device for implementing lock free queue in multi-core environment | |
| CN103150149A (en) | Method and device for processing redo data of database | |
| CN110910921A (en) | Command read-write method and device and computer storage medium | |
| CN101847096B (en) | Optimization method of stack variable-containing function | |
| CN113032162B (en) | Multi-process communication method based on shared memory backup mechanism | |
| US20240086332A1 (en) | Data processing method and system, device, and medium | |
| CN107729142B (en) | Thread calling method for self-compaction metadata | |
| US20160070648A1 (en) | Data storage system and operation method thereof | |
| US7266650B2 (en) | Method, apparatus, and computer program product for implementing enhanced circular queue using loop counts | |
| CN109213450B (en) | Associated metadata deleting method, device and equipment based on flash memory array | |
| CN110288340B (en) | Data set storage method based on Java intelligent contract | |
| CN105653466A (en) | Data storage device and flash memory control method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |