CN106649615B - Log segment management method and device - Google Patents
Log segment management method and device Download PDFInfo
- Publication number
- CN106649615B CN106649615B CN201611085968.4A CN201611085968A CN106649615B CN 106649615 B CN106649615 B CN 106649615B CN 201611085968 A CN201611085968 A CN 201611085968A CN 106649615 B CN106649615 B CN 106649615B
- Authority
- CN
- China
- Prior art keywords
- log
- event
- size
- value
- events
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007726 management method Methods 0.000 title claims abstract description 36
- 238000000034 method Methods 0.000 claims description 9
- 238000010276 construction Methods 0.000 description 4
- 230000007423 decrease Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000013523 data management Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a log segment management method, which comprises the following steps: s1: writing a log event; s2: determining whether the write log event in S1 is a subtree event; if yes, ignoring the size of the log event and not counting the total size of the log event, and updating the event quantity value; if not, acquiring the size information of the log event, counting the total size of the log event, and updating the event quantity value and the event size value; s3: determining whether the event quantity value reaches a log event quantity threshold and determining whether the event size value reaches a log event size threshold; if the event quantity value does not reach the log event quantity threshold value and the event size value does not reach the log event size threshold value, the operation goes to S1; otherwise, go to S4; s4: the log segment is constructed, and the flow proceeds to S1. The invention also discloses a log segment management device.
Description
Technical Field
The invention relates to the technical field of metadata management, in particular to a log segment management method and a log segment management device.
Background
The existing file system is mainly divided into two parts: metadata management and data management. The metadata requests are more than 50% of all requests of the file system, so that the processing performance of the metadata requests is improved, and the performance of the whole file system can be effectively improved.
In order to ensure the recoverability of the stored content of the metadata node, each operation of the metadata node is recorded in the log section in the form of log events. Each log segment can store a certain size (the specific size is related to a disk stripe, which is described as an example in the following 4M) or a certain number (which can be configured, which is described as an example in the following 1024) of log events, and after the log events are recorded in the log segment, the total amount of the log events contained in the log segment is determined. If the total size of the log events in the log segment exceeds 4M or the total size exceeds 1024 log events, the log segment is ended, and the metadata node reconstructs a log segment to prepare for storing the next log event. And the default initial event of each log segment is a sub-tree event, and the contents of a sub-tree root, a sub-tree root boundary and the like of the current metadata node directory structure of the system are recorded in the event.
In a system of multiple metadata nodes, each metadata node has its own directory structure. Under the condition of the dynamic subtree, the newly-built directory can be distributed to relatively-idle metadata nodes according to the cluster metadata heat degree, so that the purposes of balancing metadata load and improving the performance of a file system are achieved.
If the newly created directory and its parent directory are not responsible for the same metadata node, the newly created directory needs to be recorded as the subtree root by the metadata node to which it belongs. Based on the log event storage mechanism just described: the more subtree roots the metadata node is responsible for, the larger the subtree events recorded in the log segment. Under the existing mechanism, when the subtree root reaches a certain order of magnitude, the subtree event size can exceed 4M. Resulting in any metadata operation, such as adding, deleting, renaming, etc., to a file, even if the content that the operation itself needs to store is small, the log segment is still reconstructed due to the oversized sub-tree event, and the start of the new log segment is the oversized sub-tree event. Frequent construction of log segments and excessively large sub-tree events cause a straight-line drop in metadata performance.
Therefore, for those skilled in the art, how to prevent the metadata performance from being degraded due to too frequent construction of log segments in a system supporting a plurality of metadata nodes is an urgent technical problem to be solved.
Disclosure of Invention
Based on the technical problems existing in the background technology, the invention provides a log segment management method, which comprises the following steps:
s1: writing a log event;
s2: determining whether the write log event in S1 is a subtree event; if yes, ignoring the size of the log event and not counting the total size of the log event, and updating the event quantity value; if not, acquiring the size information of the log event, counting the total size of the log event, and updating the event quantity value and the event size value;
s3: determining whether the event quantity value reaches a log event quantity threshold and determining whether the event size value reaches a log event size threshold; if the event quantity value does not reach the log event quantity threshold value and the event size value does not reach the log event size threshold value, the operation goes to S1; otherwise, go to S4;
s4: the log segment is constructed, and the flow proceeds to S1.
Preferably, before step S1, the method further includes the steps of: setting a threshold value of the number of log events and a threshold value of the size of the log events.
Preferably, the log event number threshold is 1024, and/or the log event size threshold is 4M.
The invention also provides a log segment management method, which comprises the following steps:
judging whether the written log event is a subtree event or not;
putting subtree events into a first log event pool, and putting non-subtree events into a second log event pool;
counting the total number of the log events in the first log event pool and the second log event pool as an event number value, and counting the total size of the log events in the second log event pool as an event size value;
and if the event number value is smaller than the threshold value of the event number of the log segment and the event size value is smaller than the threshold value of the event size of the log segment, writing the log event into the current log segment, and otherwise, constructing a new log segment and writing the new log segment into the log event.
Preferably, the method comprises the following steps: setting a threshold value of the number of log events and a threshold value of the size of the log events.
Preferably, the log event number threshold is 1024, and/or the log event size threshold is 4M.
In the two log segment management methods provided by the invention, the key points are that the log events are judged and divided into subtree events and non-subtree events, the subtree events count the total number of the log events and the volume is not counted into the total size of the log events, the non-subtree events count the total number of the log events and the volume is counted into the total size of the log events, so that the frequent construction of the log segments caused by the overlarge subtree events is avoided; the method has at least the following advantages:
1. the log segments are not frequently constructed due to the overlarge subtree events, the linear decline of the metadata performance caused by the overlarge subtree events is avoided, and the metadata clustering performance can be improved;
2. the method is suitable for a file system capable of supporting a plurality of metadata nodes, has good compatibility with the existing metadata cluster dynamic sub-tree directory structure, and can be widely applied to cloud storage systems, distributed file systems and the like.
The invention also provides a log segment management device, comprising:
the event writing module is used for writing the log event into the current log segment;
the first judging module is used for judging whether the log event is a subtree event or not;
the management module is used for counting the total number of the log events in the current log segment as an event number value; the method is also used for counting the sum of the sizes of the non-subtree events as an event size value;
the second judgment module is used for generating a trigger signal when the event number value reaches the log event number threshold; and further configured to generate a trigger signal if the event size value reaches the log event size threshold;
and the log segment generating module is used for constructing a log segment when the second judging module generates the trigger signal.
Preferably, the method further comprises the following steps: and the assignment module is used for setting a threshold value of the number of the log events and a threshold value of the size of the log events.
The beneficial effects of the log segment management device provided by the invention are basically similar to those of the log segment management method, and are not repeated here.
Drawings
FIG. 1 is a schematic flowchart illustrating an embodiment of a log segment management method according to the present invention;
FIG. 2 is a flowchart illustrating another embodiment of a log segment management method according to the present invention;
fig. 3 is a block diagram of an embodiment of a log segment management apparatus according to the present invention.
Detailed Description
As shown in fig. 1-3, fig. 1 is a schematic flowchart of an embodiment of a log segment management method according to the present invention; FIG. 2 is a flowchart illustrating another embodiment of a log segment management method according to the present invention; fig. 3 is a block diagram of an embodiment of a log segment management apparatus according to the present invention.
The invention is described in detail below with reference to the figures and examples.
Example 1
Referring to fig. 1, a log segment management method disclosed by the present invention includes the following steps:
s0: setting a threshold value of the number of log events to be 1024 and a threshold value of the size of the log events to be 4M;
s1: writing a log event;
s2: determining whether the write log event in S1 is a subtree event; if yes, ignoring the size of the log event and not counting the total size of the log event, and updating the event quantity value; if not, acquiring the size information of the log event, counting the total size of the log event, and updating the event quantity value and the event size value;
s3: judging whether the event quantity value reaches 1024 or not and judging whether the event size value reaches 4M or not; if the event quantity value does not reach 1024 and the event size value does not reach 4M, the step is switched to S1; otherwise, go to S4;
s4: the log segment is constructed, and the flow proceeds to S1.
In the above embodiment 1, it can be known that, first, whether a log event is written into a sub-tree event is determined, and if yes, the size of the event is ignored and only the number of events is counted, so that the structure of a log segment caused by an excessively large sub-tree event is effectively prevented, and the linear decline of metadata performance caused by the excessively large sub-tree event is avoided.
Example 2
Referring to fig. 2, the log segment management method disclosed by the present invention includes the following steps:
setting a log event number threshold 1024 and a log event size threshold 4M;
judging whether the written log event is a subtree event or not;
putting subtree events into a first log event pool, and putting non-subtree events into a second log event pool;
counting the total number of the log events in the first log event pool and the second log event pool as an event number value, and counting the total size of the log events in the second log event pool as an event size value;
and if the event number value is smaller than the threshold value of the event number of the log segment and the event size value is smaller than the threshold value of the event size of the log segment, writing the log event into the current log segment, and otherwise, constructing a new log segment and writing the new log segment into the log event.
In the above embodiment 2, it can be known that, in the embodiment 2, after determining whether a log event is a subtree event, a subtree event and a non-subtree event are divided into two parts, when counting the total number of the log events, both the two parts are counted, and when counting the total size of the log events, only the non-subtree event part is counted; the construction of log segments caused by overlarge subtree events can be effectively prevented, and the linear decline of metadata performance caused by the overlarge subtree events is avoided.
Referring to fig. 3, the log segment management apparatus disclosed in the present invention includes:
the event writing module is used for writing the log event into the current log segment;
the first judging module is used for judging whether the log event is a subtree event or not;
the management module is used for counting the total number of the log events in the current log segment as an event number value; the method is also used for counting the sum of the sizes of the non-subtree events as an event size value;
the second judgment module is used for generating a trigger signal when the event number value reaches the log event number threshold; and further configured to generate a trigger signal if the event size value reaches the log event size threshold;
the log segment generation module is used for constructing a log segment when the second judgment module generates a trigger signal;
and the assignment module is used for setting a threshold value of the number of the log events and a threshold value of the size of the log events.
The beneficial effect of the log segment management device is basically similar to that of the log segment management method, and the description is omitted here.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be considered to be within the technical scope of the present invention, and the technical solutions and the inventive concepts thereof according to the present invention should be equivalent or changed within the scope of the present invention.
Claims (8)
1. A log segment management method, comprising the steps of:
s1: writing a log event;
s2: determining whether the write log event in S1 is a subtree event; if yes, ignoring the size of the log event and not counting the total size of the log event, and updating the event quantity value; if not, acquiring the size information of the log event, counting the total size of the log event, and updating the event quantity value and the event size value;
s3: determining whether the event quantity value reaches a log event quantity threshold and determining whether the event size value reaches a log event size threshold; if the event quantity value does not reach the log event quantity threshold value and the event size value does not reach the log event size threshold value, the operation goes to S1; otherwise, go to S4;
s4: the log segment is constructed, and the flow proceeds to S1.
2. The log segment management method according to claim 1, further comprising, before the step S1, the step of: setting a threshold value of the number of log events and a threshold value of the size of the log events.
3. Method for log segment management according to claim 1 or 2, characterised in that the threshold number of log events is 1024 and/or the threshold size of log events is 4M.
4. A log segment management method, comprising the steps of:
judging whether the written log event is a subtree event or not;
putting subtree events into a first log event pool, and putting non-subtree events into a second log event pool;
counting the total number of the log events in the first log event pool and the second log event pool as an event number value, and counting the total size of the log events in the second log event pool as an event size value;
and if the event number value is smaller than the threshold value of the event number of the log segment and the event size value is smaller than the threshold value of the event size of the log segment, writing the log event into the current log segment, and otherwise, constructing a new log segment and writing the new log segment into the log event.
5. The log segment management method of claim 4, comprising the steps of: setting a threshold value of the number of log events and a threshold value of the size of the log events.
6. The log segment management method of claim 5, wherein the log event number threshold is 1024 and/or the log event size threshold is 4M.
7. A log segment management apparatus, comprising:
the event writing module is used for writing the log event into the current log segment;
the first judging module is used for judging whether the log event is a subtree event or not;
the management module is used for counting the total number of the log events in the current log segment as an event number value; the method is also used for counting the sum of the sizes of the non-subtree events as an event size value;
the second judgment module is used for generating a trigger signal when the event number value reaches the log event number threshold; and further configured to generate a trigger signal if the event size value reaches the log event size threshold;
and the log segment generating module is used for constructing a log segment when the second judging module generates the trigger signal.
8. The log segment management device of claim 7, further comprising:
and the assignment module is used for setting a threshold value of the number of the log events and a threshold value of the size of the log events.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611085968.4A CN106649615B (en) | 2016-11-30 | 2016-11-30 | Log segment management method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611085968.4A CN106649615B (en) | 2016-11-30 | 2016-11-30 | Log segment management method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106649615A CN106649615A (en) | 2017-05-10 |
| CN106649615B true CN106649615B (en) | 2020-09-22 |
Family
ID=58813729
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611085968.4A Active CN106649615B (en) | 2016-11-30 | 2016-11-30 | Log segment management method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106649615B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102306168A (en) * | 2011-08-23 | 2012-01-04 | 成都市华为赛门铁克科技有限公司 | Log operation method and device and file system |
| CN102541982A (en) * | 2011-10-25 | 2012-07-04 | 曙光信息产业(北京)有限公司 | Method for organizing and accessing metadata file log |
| US8412689B2 (en) * | 2010-07-07 | 2013-04-02 | Microsoft Corporation | Shared log-structured multi-version transactional datastore with metadata to enable melding trees |
| CN103198088A (en) * | 2011-11-07 | 2013-07-10 | Sap股份公司 | Shadow paging based log segment directory |
| CN104252406A (en) * | 2013-06-28 | 2014-12-31 | 华为技术有限公司 | Method and device for processing data |
| CN105049260A (en) * | 2015-08-24 | 2015-11-11 | 浪潮(北京)电子信息产业有限公司 | Dialog management method and device |
-
2016
- 2016-11-30 CN CN201611085968.4A patent/CN106649615B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8412689B2 (en) * | 2010-07-07 | 2013-04-02 | Microsoft Corporation | Shared log-structured multi-version transactional datastore with metadata to enable melding trees |
| CN102306168A (en) * | 2011-08-23 | 2012-01-04 | 成都市华为赛门铁克科技有限公司 | Log operation method and device and file system |
| CN102541982A (en) * | 2011-10-25 | 2012-07-04 | 曙光信息产业(北京)有限公司 | Method for organizing and accessing metadata file log |
| CN103198088A (en) * | 2011-11-07 | 2013-07-10 | Sap股份公司 | Shadow paging based log segment directory |
| CN104252406A (en) * | 2013-06-28 | 2014-12-31 | 华为技术有限公司 | Method and device for processing data |
| CN105049260A (en) * | 2015-08-24 | 2015-11-11 | 浪潮(北京)电子信息产业有限公司 | Dialog management method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106649615A (en) | 2017-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11226987B2 (en) | System and method for in-place data writes to reduce fragmentation in a multidimensional database environment | |
| CN104615594B (en) | A kind of data-updating method and device | |
| EP2631821A1 (en) | File storage method and device thereof | |
| CN105468642A (en) | Data storage method and apparatus | |
| CN107045531A (en) | A kind of system and method for optimization HDFS small documents access | |
| CN104281717B (en) | A kind of method for setting up magnanimity ID mapping relations | |
| CN101783740B (en) | Method and device for managing message file | |
| CN102158349A (en) | Log management device and method thereof | |
| CN104268088A (en) | Vehicle DVR (Digital Video Recorder) hard disk data storage method | |
| CN109086141B (en) | Memory management method and device and computer readable storage medium | |
| CN107402870B (en) | Method and device for processing log segment in metadata server | |
| CN110399096B (en) | Method, device and equipment for deleting metadata cache of distributed file system again | |
| US11625187B2 (en) | Method and system for intercepting a discarded page for a memory swap | |
| CN104484460A (en) | Metadata heat degree statistical method of distributed file system | |
| US11287996B2 (en) | Method, device and computer program product for storing data | |
| CN104408128B (en) | A kind of reading optimization method indexed based on B+ trees asynchronous refresh | |
| CN106155917A (en) | EMS memory management process and device | |
| CN106649615B (en) | Log segment management method and device | |
| CN104252415B (en) | Method and system for redistributing data | |
| US10002458B2 (en) | Data plot processing | |
| CN106503260B (en) | Method and device for improving effective storage space of database | |
| CN102270179B (en) | For the data Storage and Processing method based on sheet unit and the system thereof of CAD system | |
| CN106873906A (en) | Method and apparatus for managing metamessage | |
| CN106777107A (en) | A kind of method of digital specimen tile data storage | |
| CN107632880B (en) | Method for exporting excel data, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200828 Address after: 215100 No. 1 Guanpu Road, Guoxiang Street, Wuzhong Economic Development Zone, Suzhou City, Jiangsu Province Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd. Address before: 450018 Henan province Zheng Dong New District of Zhengzhou City Xinyi Road No. 278 16 floor room 1601 Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |