CN106649615A - Method and device for managing log segment - Google Patents
Method and device for managing log segment Download PDFInfo
- Publication number
- CN106649615A CN106649615A CN201611085968.4A CN201611085968A CN106649615A CN 106649615 A CN106649615 A CN 106649615A CN 201611085968 A CN201611085968 A CN 201611085968A CN 106649615 A CN106649615 A CN 106649615A
- Authority
- CN
- China
- Prior art keywords
- event
- log
- size
- log event
- daily record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title abstract description 7
- 238000007726 management method Methods 0.000 claims description 33
- 238000010276 construction Methods 0.000 claims description 7
- 238000005516 engineering process Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method for managing a log segment. The method comprises the following steps that S1, a log event is written in; S2, whether the log event written in S1 is a subtree event or not is judged, if yes, the size of the log event is ignored and is not reckoned in the total size of the log event, a data number value is updated, and if not, size information of the log event is acquired and reckoned in the total size of the log event, and the event number value and the event size value are updated; S3, whether the event number value reaches a log event number threshold value or not and whether the event size value reaches a log event size threshold value or not are judged, and if the event number value does not reach the log event number threshold value and the event size value does not reach the log event size threshold value, S1 is executed, otherwise, S4 is executed; S4, the log segment is constructed, and S1 is executed. The invention further discloses a method for managing the log segment.
Description
Technical field
The present invention relates to metadata management technical field, more particularly to a kind of daily record segment management method, further relate to a kind of day
Will segment management device.
Background technology
Existing file system is broadly divided into two large divisions：Metadata management, data management.Wherein, the request of metadata is
More than the 50% of all requests of file system, so lifting the process performance of metadata request, can effectively lift whole file
The performance of system.
In order to ensure the restorability of metadata node storage content, each operation of metadata node all can be with daily record thing
The form of part is recorded in daily record section.Each daily record section can deposit a certain size (specific size is relevant with stripe, hereafter
Illustrate by taking 4M as an example) or certain amount (can configure, hereafter as a example by 1024 illustrate) log event, when log event is remembered
After recording daily record section, the log event total amount included in daily record section can be judged.If log event total size exceeds in daily record section
When 4M or total amount are more than 1024, this daily record section terminates, and metadata node can reconfigure a daily record section and prepare under storage
One log event.And the acquiescence initiation event of each daily record section is exactly subtree event, the current unit's number of system is have recorded in event
According to contents such as the subtree root of nodal directory structure, subtree root borders.
In the system of multiple metadata nodes, each metadata node has oneself responsible bibliographic structure.In dynamic
In the case of subtree, newly-built catalogue can give the metadata node of relative free according to cluster metadata temperature, flat to reach
Weighing apparatus metadata load, lifts the purpose of file system performance.
If newly-built catalogue and its parent directory are not same metadata nodes be responsible for, that newly-built catalogue needs conduct
Subtree root is recorded by affiliated metadata node.Based on log event memory mechanism just described：What metadata node was responsible for
Subtree root is more, and the subtree event recorded in daily record section is bigger.Under existing mechanism, when subtree root reaches certain amount level,
Subtree event size can just exceed 4M.Cause any metadata operation, such as file additions and deletions are changed, renaming, i.e. convenient to operate
Itself needs the content very little of storage, but due to excessive subtree event, still to reconfigure daily record section, and new daily record section
Starting be again excessive subtree event.Frequently constructing the excessive subtree event of daily record section and construction causes metadata performance straight
Line declines.
Therefore, to those skilled in the art, in the system for supporting multiple metadata nodes, how to prevent excessively
Frequently construct daily record section and cause the technical problem that metadata hydraulic performance decline is urgent need to resolve.
The content of the invention
Based on the technical problem that background technology is present, the present invention proposes a kind of daily record segment management method, including following step
Suddenly：
S1：Write log event；
S2：Judge to be write in S1 log event whether as subtree event；If so, then the log event size ignored and not
Count log event total size, and update event quantitative value；If it is not, then obtaining the log event size information and counting daily record thing
Part total size, and update event quantitative value and event size value；
S3：Judge whether event number value reaches log event amount threshold and judge whether event size value reaches daily record
Event size threshold value；If event number value is not up to log event amount threshold and event size value is not up to log event size
Threshold value, then proceed to S1；Otherwise, S4 is proceeded to；
S4：Construction daily record section, proceeds to S1.
Preferably, before step S1, also including step：Setting log event amount threshold and log event size threshold
Value.
Preferably, log event amount threshold is 1024, and/or, log event size threshold value is 4M.
The invention allows for a kind of daily record segment management method, comprises the following steps：
Judge whether write log event is subtree event；
Subtree event is put into the first log event pond, non-subtree event is put into the second log event pond；
Count in the first log event pond and the second log event pond log event total quantity and as event number value, system
Count in the second log event pond log event total size and as event size value；
Event number value is less than daily record section event number threshold value and event size value is less than daily record section event size threshold value, then
Log event is write in current log section, otherwise, new daily record section is constructed and is write log event.
Preferably, comprise the following steps：Setting log event amount threshold and log event size threshold value.
Preferably, log event amount threshold is 1024, and/or, log event size threshold value is 4M.
The two kinds of daily record segment management methods provided in the present invention, key point is log event to be judged and is distinguished
For subtree event and non-subtree event, subtree event count log event total quantity and amount of capacity to be not counted in log event always big
Little, non-subtree event counts log event total quantity and amount of capacity counts log event total size, so as to avoid subtree event
The excessive frequent construction for causing daily record section；Said method at least has advantages below：
1st, will not cause frequently to construct daily record section due to excessive subtree event, it is to avoid the unit that excessive subtree event causes
Data performance straight line declines, and can lift metadata cluster performance；
2nd, suitable for the file system of multiple metadata nodes can be supported, with existing metadata set group motion state subtree catalogue
Structural compatibility is good, can be widely applied to cloud storage system, distributed file system etc..
The invention allows for daily record segment management device, including：
Event writing module, for writing log event in current log section；
Whether the first judge module, judge log event as subtree event；
Management module, for counting current log section in log event total quantity as event number value；It is additionally operable to statistics
The size sum of non-subtree event is used as event size value；
Second judge module, for when event number value reaches log event amount threshold, generating trigger；Also use
It is to generate trigger in log event size threshold value is reached in event size value；
Daily record section generation module, for when the second judge module generates trigger, constructing daily record section.
Preferably, also include：Assignment module, for setting log event amount threshold and log event size threshold value.
The basic class of beneficial effect of daily record segment management device proposed by the present invention, its advantage and daily record segment management method
Seemingly, here is omitted.
Description of the drawings
Fig. 1 is a kind of schematic flow sheet of embodiment in a kind of daily record segment management method proposed by the present invention；
Fig. 2 is the schematic flow sheet of another kind of embodiment in a kind of daily record segment management method proposed by the present invention；
Fig. 3 is a kind of a kind of structured flowchart of embodiment of daily record segment management device proposed by the present invention.
Specific embodiment
As Figure 1-3, Fig. 1 is a kind of flow process signal of embodiment in a kind of daily record segment management method proposed by the present invention
Figure；Fig. 2 is the schematic flow sheet of another kind of embodiment in a kind of daily record segment management method proposed by the present invention；Fig. 3 is the present invention
A kind of a kind of structured flowchart of the embodiment of the daily record segment management device for proposing.
The present invention is described in detail with reference to the accompanying drawings and examples.
Embodiment 1
With reference to Fig. 1, a kind of daily record segment management method disclosed in this invention is comprised the following steps：
S0：Log event amount threshold is set as 1024 and log event size threshold value 4M；
S1：Write log event；
S2：Judge to be write in S1 log event whether as subtree event；If so, then the log event size ignored and not
Count log event total size, and update event quantitative value；If it is not, then obtaining the log event size information and counting daily record thing
Part total size, and update event quantitative value and event size value；
S3：Judge whether event number value reaches 1024 and judge whether event size value reaches 4M；If event number value
Not up to 1024 and event size value is not up to 4M, then proceed to S1；Otherwise, S4 is proceeded to；
S4：Construction daily record section, proceeds to S1.
In above-described embodiment 1, we can know, advanced row write enters whether log event is subtree event, if then
Ignore the event size and only count event number, so as to effectively prevent the excessive construction for causing daily record section of subtree event, it is to avoid
The metadata performance straight line that excessive subtree event causes declines.
Embodiment 2
With reference to Fig. 2, a kind of daily record segment management method disclosed in this invention is comprised the following steps：
Setting log event amount threshold 1024 and log event size threshold value 4M；
Judge whether write log event is subtree event；
Subtree event is put into the first log event pond, non-subtree event is put into the second log event pond；
Count in the first log event pond and the second log event pond log event total quantity and as event number value, system
Count in the second log event pond log event total size and as event size value；
Event number value is less than daily record section event number threshold value and event size value is less than daily record section event size threshold value, then
Log event is write in current log section, otherwise, new daily record section is constructed and is write log event.
In above-described embodiment 2, we can know, whether embodiment 2 uses and judge log event as subtree thing
After part, subtree event is divided into two parts with non-subtree event, in statistical log total number of events amount, two parts are all united
Meter, in statistical log event total size, then only counts non-subtree event section；Can effectively prevent that subtree event is excessive to be caused
The construction of daily record section, it is to avoid the metadata performance straight line that excessive subtree event causes declines.
With reference to Fig. 3, a kind of daily record segment management device disclosed in this invention, including：
Event writing module, for writing log event in current log section；
Whether the first judge module, judge log event as subtree event；
Management module, for counting current log section in log event total quantity as event number value；It is additionally operable to statistics
The size sum of non-subtree event is used as event size value；
Second judge module, for when event number value reaches log event amount threshold, generating trigger；Also use
It is to generate trigger in log event size threshold value is reached in event size value；
Daily record section generation module, for when the second judge module generates trigger, constructing daily record section；
Assignment module, for setting log event amount threshold and log event size threshold value.
Above-mentioned daily record segment management device, its advantage is substantially similar with the beneficial effect of daily record segment management method, herein
Repeat no more.
The above, the only present invention preferably specific embodiment, but protection scope of the present invention is not limited thereto,
Any those familiar with the art the invention discloses technical scope in, technology according to the present invention scheme and its
Inventive concept equivalent or change in addition, all should be included within the scope of the present invention.
Claims (8)
1. a kind of daily record segment management method, it is characterised in that comprise the following steps：
S1：Write log event；
S2：Judge to be write in S1 log event whether as subtree event；If so, then the log event size ignored and is not counted in
Log event total size, and update event quantitative value；If it is not, then obtaining the log event size information and to count log event total
Size, and update event quantitative value and event size value；
S3：Judge whether event number value reaches log event amount threshold and judge whether event size value reaches log event
Size threshold value；If event number value is not up to log event amount threshold and event size value is not up to log event size threshold
Value, then proceed to S1；Otherwise, S4 is proceeded to；
S4：Construction daily record section, proceeds to S1.
2. daily record segment management method according to claim 1, it is characterised in that before step S1, also including step：If
Settled date will event number threshold value and log event size threshold value.
3. daily record segment management method according to claim 1 and 2, it is characterised in that log event amount threshold is 1024,
And/or, log event size threshold value is 4M.
4. a kind of daily record segment management method, it is characterised in that comprise the following steps：
Judge whether write log event is subtree event；
Subtree event is put into the first log event pond, non-subtree event is put into the second log event pond；
Count in the first log event pond and the second log event pond log event total quantity and as event number value, statistics the
Log event total size and as event size value in two log event ponds；
Event number value is less than daily record section event number threshold value and event size value is less than daily record section event size threshold value, then working as
Front daily record section writes log event, otherwise, constructs new daily record section and writes log event.
5. daily record segment management method according to claim 4, it is characterised in that comprise the following steps：Setting log event
Amount threshold and log event size threshold value.
6. daily record segment management method according to claim 5, it is characterised in that log event amount threshold is 1024, and/
Or, log event size threshold value is 4M.
7. a kind of daily record segment management device, it is characterised in that include：
Event writing module, for writing log event in current log section；
Whether the first judge module, judge log event as subtree event；
Management module, for counting current log section in log event total quantity as event number value；It is additionally operable to count non-son
The size sum of tree event is used as event size value；
Second judge module, for when event number value reaches log event amount threshold, generating trigger；It is additionally operable to
Event size value reaches log event size threshold value, generates trigger；
Daily record section generation module, for when the second judge module generates trigger, constructing daily record section.
8. daily record segment management device according to claim 7, it is characterised in that also include：
Assignment module, for setting log event amount threshold and log event size threshold value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611085968.4A CN106649615B (en) | 2016-11-30 | 2016-11-30 | Log segment management method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611085968.4A CN106649615B (en) | 2016-11-30 | 2016-11-30 | Log segment management method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106649615A true CN106649615A (en) | 2017-05-10 |
| CN106649615B CN106649615B (en) | 2020-09-22 |
Family
ID=58813729
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611085968.4A Active CN106649615B (en) | 2016-11-30 | 2016-11-30 | Log segment management method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106649615B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102306168A (en) * | 2011-08-23 | 2012-01-04 | 成都市华为赛门铁克科技有限公司 | Log operation method and device and file system |
| CN102541982A (en) * | 2011-10-25 | 2012-07-04 | 曙光信息产业(北京)有限公司 | Method for organizing and accessing metadata file log |
| US8412689B2 (en) * | 2010-07-07 | 2013-04-02 | Microsoft Corporation | Shared log-structured multi-version transactional datastore with metadata to enable melding trees |
| CN103198088A (en) * | 2011-11-07 | 2013-07-10 | Sap股份公司 | Shadow paging based log segment directory |
| CN104252406A (en) * | 2013-06-28 | 2014-12-31 | 华为技术有限公司 | Method and device for processing data |
| CN105049260A (en) * | 2015-08-24 | 2015-11-11 | 浪潮(北京)电子信息产业有限公司 | Dialog management method and device |
-
2016
- 2016-11-30 CN CN201611085968.4A patent/CN106649615B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8412689B2 (en) * | 2010-07-07 | 2013-04-02 | Microsoft Corporation | Shared log-structured multi-version transactional datastore with metadata to enable melding trees |
| CN102306168A (en) * | 2011-08-23 | 2012-01-04 | 成都市华为赛门铁克科技有限公司 | Log operation method and device and file system |
| CN102541982A (en) * | 2011-10-25 | 2012-07-04 | 曙光信息产业(北京)有限公司 | Method for organizing and accessing metadata file log |
| CN103198088A (en) * | 2011-11-07 | 2013-07-10 | Sap股份公司 | Shadow paging based log segment directory |
| CN104252406A (en) * | 2013-06-28 | 2014-12-31 | 华为技术有限公司 | Method and device for processing data |
| CN105049260A (en) * | 2015-08-24 | 2015-11-11 | 浪潮(北京)电子信息产业有限公司 | Dialog management method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106649615B (en) | 2020-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103412803B (en) | The method and device that data are recovered | |
| CN102467521B (en) | Easily-extensible multi-level classification search method and system | |
| JP2016519810A5 (en) | ||
| CN110781184B (en) | Data table construction method, device, equipment and storage medium | |
| CN103577590A (en) | Data query method and system | |
| CN107590226A (en) | A kind of map vector rendering intent based on tile | |
| TW201232294A (en) | Formatting data by example | |
| CN102646121A (en) | Two-stage storage method combined with RDBMS (relational database management system) and Hadoop cloud storage | |
| CN106599104A (en) | Mass data association method based on redis cluster | |
| CN102982186B (en) | Based on maintaining method and the system of the range partition table of oracle database system | |
| CN108932286B (en) | Data query method and device | |
| CN107391652A (en) | A kind of templating form interface defines system and method | |
| CN102158349A (en) | Log management device and method thereof | |
| CN110750372B (en) | Log system and log management method based on shared memory | |
| CN104268099A (en) | Management data reading and writing method and device | |
| CN102564435A (en) | Navigation electronic map incremental bag and self-checking method | |
| CN107943542A (en) | A kind of configuration information management method, device, computer-readable recording medium and storage control | |
| CN104408128B (en) | A kind of reading optimization method indexed based on B+ trees asynchronous refresh | |
| CN107704200A (en) | A kind of data deposit method | |
| CN110991152B (en) | Method and system for step-by-step data acquisition and automatic summarization | |
| CN110597805B (en) | Memory index structure processing method | |
| CN107480235A (en) | A kind of database framework of data platform | |
| CN106503260B (en) | Method and device for improving effective storage space of database | |
| CN106844312A (en) | A kind of data table transmition method, device, computer-readable recording medium and storage control | |
| CN106649615A (en) | Method and device for managing log segment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200828 Address after: 215100 No. 1 Guanpu Road, Guoxiang Street, Wuzhong Economic Development Zone, Suzhou City, Jiangsu Province Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd. Address before: 450018 Henan province Zheng Dong New District of Zhengzhou City Xinyi Road No. 278 16 floor room 1601 Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |