CN104021088A - Log storage method and device - Google Patents
Log storage method and device Download PDFInfo
- Publication number
- CN104021088A CN104021088A CN201410290387.9A CN201410290387A CN104021088A CN 104021088 A CN104021088 A CN 104021088A CN 201410290387 A CN201410290387 A CN 201410290387A CN 104021088 A CN104021088 A CN 104021088A
- Authority
- CN
- China
- Prior art keywords
- daily record
- log
- disk
- size
- record data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The embodiment of the invention provides a log storage method and device. The log storage method comprises the following steps: receiving a log storage request sent by a log source and determining a sending rate of log data sent by the log source; determining the size of a log containing the log data according to the sending rate of the log data; determining a storage address of the log according to an identifier of the log and an IP (Internet Protocol) address of the log source; distributing a disk space for the log in a disk corresponding to the storage address by adopting a predistribution method according to the size of the log; and storing the log containing the log data into the disk space. According to the method and device which are provided by the embodiment, the log can be stored in the predistributed disk space by determining the log size and the storage address of the log and predistributing the disk space for the log in the disk corresponding to the storage address. As the disk space is distributed according to the log size, disk fragmentations can be reduced and the performances of the disk of a log server can be improved.
Description
Technical field
The embodiment of the present invention relates to computer memory technical, relates in particular to a kind of log storing method and device.
Background technology
Daily record is the logout of Log Source (the such as network equipment, system and service routine etc.) log by name of generation in the time of operation, the set of the data that produce while being Log Source operation, the descriptor of the associative operations such as date, time, user and action is all being recorded in every a line daily record.Log recording the life cycle of Log Source, by consulting and analyzing daily record, can know that Log Source is at residing state sometime.
Log server be a kind of to different Log Sources move the daily record that produces and send record storage server unit; generally there is larger disk storage space; to realize the daily record that different Log Sources the are sent storage of classifying; for example; classification storage by date; in some occasions higher to time statistical demand, conventionally also need by the hour or minute etc. journal file is cut apart to storage.
Existing log store technology, is generally the corresponding disk storage space of the interim distribution of daily record sending according to Log Source, thereby reaches the object to disk by log store.
All there is following defect in above-mentioned log store technology: in centralized log store structure, also in the time that multiple Log Sources send daily record to log server simultaneously, what log server can replace writes daily record data in log store file corresponding to each Log Source, cause aiming at single day storage area on the disk of log server discontinuous, further cause the storage area of the daily record data in each log store file to intersect, thereby produce a large amount of disk fragmentses, reduce the performance of the disk of log server.
Summary of the invention
The embodiment of the present invention provides a kind of log storing method and device, to reduce disk fragments, and improves the performance of the disk of log server.
First aspect, the embodiment of the present invention provides a kind of log storing method, comprising:
Receive the log store request that Log Source sends, described log store request comprises: daily record data, the mark of daily record that includes described daily record data and the IP address of Log Source;
Determine that Log Source sends the transmission rate of daily record data;
According to the transmission rate of daily record data, determine the size of the daily record that includes described daily record data;
According to the IP address of the mark of described daily record and Log Source, determine the memory address of daily record;
According to the size of described daily record, adopt preallocated method, in the disk corresponding with described memory address, be that described daily record distributes disk space;
By the log store that includes described daily record data to described disk space in.
Second aspect, the embodiment of the present invention also provides a kind of log storage, comprising:
Storage resource request receiver module, the log store request sending for receiving Log Source, described log store request comprises: daily record data, the mark of daily record that includes described daily record data and the IP address of Log Source;
Transmission rate determination module, for determining that Log Source sends the transmission rate of daily record data;
Daily record size determination module, for according to the transmission rate of daily record data, determines the size of the daily record that includes described daily record data;
Memory address determination module, for according to the IP address of the mark of described daily record and Log Source, determines the memory address of daily record;
Disk space distribution module, for according to the size of described daily record, adopts preallocated method, is that described daily record distributes disk space in the disk corresponding with described memory address;
Log store module, for by the log store that includes described daily record data to described disk space.
The log storing method that the embodiment of the present invention provides and device, send the transmission rate of daily record data by Log Source, can determine daily record size, by the mark of the daily record in log store request and the IP address of Log Source, can determine the memory address of daily record, by being daily record predistribution disk space in the disk corresponding with this memory address, therefore can be by log store in preallocated disk space.Because preallocated disk space distributes according to daily record size, therefore can reduce disk fragments, and improve the performance of the disk of log server.
Brief description of the drawings
In order to be illustrated more clearly in the present invention, introduce simply the accompanying drawing of required use in the present invention being done to one below, apparently, accompanying drawing in the following describes is some embodiments of the present invention, for those of ordinary skill in the art, do not paying under the prerequisite of creative work, can also obtain according to these accompanying drawings other accompanying drawing.
The process flow diagram of a kind of log storing method that Fig. 1 provides for the embodiment of the present invention one;
The process flow diagram of a kind of log storing method that Fig. 2 provides for the embodiment of the present invention two;
The structural representation of a kind of log storage that Fig. 3 provides for the embodiment of the present invention three.
Embodiment
For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing, the technical scheme in the embodiment of the present invention is described in further detail, obviously, described embodiment is the present invention's part embodiment, instead of whole embodiment.Be understandable that; specific embodiment described herein is only for explaining the present invention; but not limitation of the invention; based on the embodiment in the present invention; those of ordinary skill in the art, not making the every other embodiment obtaining under creative work prerequisite, belong to the scope of protection of the invention.It also should be noted that, for convenience of description, in accompanying drawing, only show part related to the present invention but not full content.
Embodiment mono-
Refer to Fig. 1, the process flow diagram of a kind of log storing method providing for the embodiment of the present invention one.The method of the embodiment of the present invention can be carried out by the log storage realizing with hardware and/or software, and this device is typically and is disposed in the server that log store service can be provided.
As shown in Figure 1, described method comprises:
The log store request that step 110, reception Log Source send, described log store request comprises: daily record data, the mark of daily record that includes described daily record data and the IP address of Log Source;
In this step, Log Source comprises the network equipment, system and service routine etc.Daily record data is that Log Source produces in the time of operation, and daily record data is being recorded the logout of Log Source, for example, and the information such as date, time, user and associative operation.Daily record is the logout of Log Source log by name of generation in the time of operation, is the set of daily record data.Log Source produces daily record data incessantly, and daily record is distributed to a daily record mark, is also daily record ID.The daily record conventionally with identical event body, same time generation has identical daily record mark.
It should be noted that, in this step, can receive the log store request that different Log Sources send simultaneously, and different log store requests is carried out to parallel processing.
Step 120, determine that Log Source sends the transmission rate of daily record data;
In this step, can be by the described log store request of monitoring, determine that Log Source sends the transmission rate of daily record data.In other words, in the time that the request type monitoring is log store request, trigger this log store request of monitoring, thereby determine that Log Source sends the transmission rate of daily record data.
Step 130, according to the transmission rate of daily record data, determine the size of daily record that includes described daily record data;
In this step, can and send duration by the transmission rate of daily record data, determine the size of the daily record that includes described daily record data; Also can be by adding up in advance the corresponding relation of transmission rate and the daily record size of knowing daily record data, according to this corresponding relation, the daily record size corresponding with the transmission rate of current daily record data is defined as to the size of the daily record that includes described daily record data, for example for example, know this corresponding relation by the transmission rate of the daily record data in statistics Preset Time section (, 1 month or 3 weeks etc.) with corresponding daily record size.
Step 140, according to the IP address of the mark of described daily record and Log Source, determine the memory address of daily record;
This step is specifically determined the memory address of daily record.Exemplarily, according to the IP address of the mark of described daily record and Log Source, according to default log store strategy, the memory address of determining daily record is { Disk name/daily record ID/log/IP address/date/time }.
In practical application, the relevant daily record data of a business (, application under a business) a corresponding daily record ID, for example, it is the business diary of QQ browser for business, the corresponding daily record ID of customer registration affair (application under QQ browser) can be set, there is corresponding log data structures, and corresponding another daily record ID of user's browsing event (Another application under QQ browser), corresponding have log data structures corresponding to this daily record ID.Certainly,, in practical application, a business also can only be applied for a daily record ID, daily record ID of certain business application or multiple daily record ID, by business development personnel according to the daily record data amount of business, daily record data kind, and considering in conjunction with the follow-up statistical demand based on daily record data.
Step 150, according to the size of described daily record, adopt preallocated method, in the disk corresponding with described memory address, be that described daily record distributes disk space;
This step in the disk corresponding in the log store address definite with step 140, is specifically to distribute by the definite daily record size of step 130 disk space matching with daily record size.
Step 160, by the log store that includes described daily record data to described disk space in.
The technical scheme of the present embodiment, send the transmission rate of daily record data by Log Source, can determine daily record size, by the mark of the daily record in log store request and the IP address of Log Source, can determine the memory address of daily record, by being daily record predistribution disk space in the disk corresponding with this memory address, therefore can be by log store in preallocated disk space.Because preallocated disk space distributes according to daily record size, therefore can reduce disk fragments, and improve the performance of the disk of log server.
It should be noted that, be generally by the time daily record need to store time in prior art, is this daily record memory allocated space temporarily, cause daily record to be dispersed and store the zones of different of whole disk into, instead of be stored in continuously disk continuous bunch in.In other words, in use for some time, due to the operation that repeatedly writes and delete, the free sector in disk can be distributed in whole disk on discontinuous physical location disk, thereby daily record can not be stored in the continuous sector of disk.Like this, more just need to go to read to different regions while reading and writing daily record, increase moving around of magnetic head, reduce the access speed of disk.
And the technical scheme of the present embodiment, by preallocated mode, the data of individual log file are stored in the continuous space of a slice on disk all the time, because the end to end neat storage of daily record reduces disk fragments, therefore disk read-write speed be can accelerate, and overall performance and the travelling speed of disk improved.
Also it should be noted that, in prior art, need regularly to carry out disk fragments arrangement, capacity and disk fragments degree that the time that disk fragments arranges and disk have used are directly proportional, and during conducting disk defragmentation, the IO interface of disk can be taken by Disk Defragmenter, substantially cannot carry out normal log store service, the life-span that also can reduce disk.
And the technical scheme of the present embodiment, adopt preallocated mode, the data of individual log file are stored in the continuous space of a slice on disk all the time, and minimizing disk fragments, thereby in disk fragments arrangement process, the IO interface of disk can provide disk fragments to arrange service, log store service can be provided again, also ensured the stable of server performance, can be because of not increasing of daily record hydraulic performance decline, and can extend serviceable life of disk.
Embodiment bis-
Refer to Fig. 2, the process flow diagram of a kind of log storing method providing for the embodiment of the present invention two.The present embodiment, on the basis of above-described embodiment, provides the size according to described daily record, adopts preallocated method, is the preferred version that described daily record distributes this step of disk space in the disk corresponding with described memory address.As shown in Figure 2, this method for optimizing, comprising:
Step 210, according to the size of described daily record, determine the grouping of free space in the required disk of this daily record;
Aspect the space management of disk, each free space on disk is divided into several groups according to approximate size, the size of each group can be configured by system manager in the time that operation is initial, also can in file system format, determine according to disk size.
Step 220, in described grouping, between seek area, size is more than or equal to the interval of described daily record size;
Each group comprises interval characterising parameter storage area and at least one daily record data storage area, and this step is specifically searched the interval that daily record data storage area size is more than or equal to described daily record size.
Step 230, using the disk space as this daily record being more than or equal in the interval of described daily record size.
The technical scheme of the present embodiment, by the interval size of determining that the grouping of free space in the disk corresponding with this memory address comprises, using the disk space as this daily record being more than or equal in the interval of described daily record size, it can be daily record predistribution disk space, because preallocated disk space distributes according to daily record size, therefore can reduce disk fragments, and improve the performance of the disk of log server.
Embodiment tri-
Refer to Fig. 3, the structural representation of a kind of log storage providing for the embodiment of the present invention three.As shown in Figure 3, this device comprises: storage resource request receiver module 310, transmission rate determination module 320, daily record size determination module 330, memory address determination module 340, disk space distribution module 350 and log store module 360.
Wherein, the log store request that storage resource request receiver module 310 sends for receiving Log Source, described log store request comprises: daily record data, the mark of daily record that includes described daily record data and the IP address of Log Source; Transmission rate determination module 320 sends the transmission rate of daily record data for determining Log Source; Daily record size determination module 330, for according to the transmission rate of daily record data, is determined the size of the daily record that includes described daily record data; Memory address determination module 340, for according to the IP address of the mark of described daily record and Log Source, is determined the memory address of daily record; Disk space distribution module 350, for according to the size of described daily record, adopts preallocated method, is that described daily record distributes disk space in the disk corresponding with described memory address; Log store module 360 for by the log store that includes described daily record data to described disk space.
The technical scheme of the present embodiment, send the transmission rate of daily record data by Log Source, can determine daily record size, by the mark of the daily record in log store request and the IP address of Log Source, can determine the memory address of daily record, by being daily record predistribution disk space in the disk corresponding with this memory address, therefore can be by log store in preallocated disk space.Because preallocated disk space distributes according to daily record size, therefore can reduce disk fragments, and improve the performance of the disk of log server.
In such scheme, transmission rate determination module 320, specifically for by the described log store request of monitoring, determines that Log Source sends the transmission rate of daily record data.
In such scheme, disk space distribution module 350 comprises: grouping determining unit, range lookup unit and disk space determining unit.
Wherein, grouping determining unit, for according to the size of described daily record, is determined the grouping of free space in the required disk of this daily record; Range lookup unit is in described grouping, between seek area the big or small interval that is more than or equal to described daily record size; Disk space determining unit is for being more than or equal to the disk space as this daily record in interval of described daily record size.
The log storage that the embodiment of the present invention provides can be carried out the log storing method that any embodiment of the present invention provides, and possesses the corresponding functional module of manner of execution and beneficial effect.
Finally it should be noted that: above each embodiment is only for technical scheme of the present invention is described, but not be limited; In embodiment, preferred embodiment, be not limited, to those skilled in the art, the present invention can have various changes and variation.All any amendments of doing, be equal to replacement, improvement etc., within protection scope of the present invention all should be included within spirit of the present invention and principle.
Claims (6)
1. a log storing method, is characterized in that, comprising:
Receive the log store request that Log Source sends, described log store request comprises: daily record data, the mark of daily record that includes described daily record data and the IP address of Log Source;
Determine that Log Source sends the transmission rate of daily record data;
According to the transmission rate of daily record data, determine the size of the daily record that includes described daily record data;
According to the IP address of the mark of described daily record and Log Source, determine the memory address of daily record;
According to the size of described daily record, adopt preallocated method, in the disk corresponding with described memory address, be that described daily record distributes disk space;
By the log store that includes described daily record data to described disk space in.
2. method according to claim 1, is characterized in that, determines that Log Source sends the transmission rate of daily record data, comprising:
By monitoring described log store request, determine that Log Source sends the transmission rate of daily record data.
3. method according to claim 1, is characterized in that, according to the size of described daily record, adopts preallocated method, is that described daily record distributes disk space in the disk corresponding with described memory address, comprising:
According to the size of described daily record, determine the grouping of free space in the required disk of this daily record;
The big or small interval that is more than or equal to described daily record size between seek area in described grouping;
Using the disk space as this daily record being more than or equal in the interval of described daily record size.
4. a log storage, is characterized in that, comprising:
Storage resource request receiver module, the log store request sending for receiving Log Source, described log store request comprises: daily record data, the mark of daily record that includes described daily record data and the IP address of Log Source;
Transmission rate determination module, for determining that Log Source sends the transmission rate of daily record data;
Daily record size determination module, for according to the transmission rate of daily record data, determines the size of the daily record that includes described daily record data;
Memory address determination module, for according to the IP address of the mark of described daily record and Log Source, determines the memory address of daily record;
Disk space distribution module, for according to the size of described daily record, adopts preallocated method, is that described daily record distributes disk space in the disk corresponding with described memory address;
Log store module, for by the log store that includes described daily record data to described disk space.
5. device according to claim 1, is characterized in that, transmission rate determination module, specifically for by the described log store request of monitoring, determines that Log Source sends the transmission rate of daily record data.
6. device according to claim 1, is characterized in that, disk space distribution module comprises:
Grouping determining unit, for according to the size of described daily record, determines the grouping of free space in the required disk of this daily record;
Range lookup unit, in described grouping, between seek area the big or small interval that is more than or equal to described daily record size;
Disk space determining unit, for being more than or equal to the disk space as this daily record in interval of described daily record size.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410290387.9A CN104021088B (en) | 2014-06-24 | 2014-06-24 | log storing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410290387.9A CN104021088B (en) | 2014-06-24 | 2014-06-24 | log storing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104021088A true CN104021088A (en) | 2014-09-03 |
| CN104021088B CN104021088B (en) | 2017-11-21 |
Family
ID=51437853
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410290387.9A Active CN104021088B (en) | 2014-06-24 | 2014-06-24 | log storing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104021088B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967696A (en) * | 2015-07-29 | 2015-10-07 | 网神信息技术（北京）股份有限公司 | Log sending method and device |
| CN106529679A (en) * | 2016-10-14 | 2017-03-22 | 腾讯科技（上海）有限公司 | Machine learning method and system |
| CN110968561A (en) * | 2018-09-30 | 2020-04-07 | 北京国双科技有限公司 | Log storage method and distributed system |
| CN112241244A (en) * | 2020-10-23 | 2021-01-19 | 杭州奔浪信息技术有限公司 | Mass storage method of streaming media logs |
| CN112783849A (en) * | 2021-01-29 | 2021-05-11 | 紫光云技术有限公司 | Production environment log exporting system and method |
| US20220417383A1 (en) * | 2021-06-25 | 2022-12-29 | Canon Kabushiki Kaisha | Image forming apparatus communicable with communication apparatus, control method for same, and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020184436A1 (en) * | 2001-06-04 | 2002-12-05 | Samsung Electronics Co., Ltd. | Flash memory management method |
| CN101923559A (en) * | 2010-03-24 | 2010-12-22 | 深圳市万兴软件有限公司 | Image file management method and system |
| CN102521114A (en) * | 2011-11-30 | 2012-06-27 | 华中科技大学 | File system log storage system under virtualization environment |
| CN103140839A (en) * | 2010-08-16 | 2013-06-05 | 赛门铁克公司 | Systems and methods for efficient sequential logging on caching-enabled storage devices |
-
2014
- 2014-06-24 CN CN201410290387.9A patent/CN104021088B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020184436A1 (en) * | 2001-06-04 | 2002-12-05 | Samsung Electronics Co., Ltd. | Flash memory management method |
| CN101923559A (en) * | 2010-03-24 | 2010-12-22 | 深圳市万兴软件有限公司 | Image file management method and system |
| CN103140839A (en) * | 2010-08-16 | 2013-06-05 | 赛门铁克公司 | Systems and methods for efficient sequential logging on caching-enabled storage devices |
| CN102521114A (en) * | 2011-11-30 | 2012-06-27 | 华中科技大学 | File system log storage system under virtualization environment |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967696A (en) * | 2015-07-29 | 2015-10-07 | 网神信息技术（北京）股份有限公司 | Log sending method and device |
| CN106529679A (en) * | 2016-10-14 | 2017-03-22 | 腾讯科技（上海）有限公司 | Machine learning method and system |
| CN106529679B (en) * | 2016-10-14 | 2020-01-14 | 腾讯科技（上海）有限公司 | Machine learning method and system |
| CN110968561A (en) * | 2018-09-30 | 2020-04-07 | 北京国双科技有限公司 | Log storage method and distributed system |
| CN112241244A (en) * | 2020-10-23 | 2021-01-19 | 杭州奔浪信息技术有限公司 | Mass storage method of streaming media logs |
| CN112783849A (en) * | 2021-01-29 | 2021-05-11 | 紫光云技术有限公司 | Production environment log exporting system and method |
| US20220417383A1 (en) * | 2021-06-25 | 2022-12-29 | Canon Kabushiki Kaisha | Image forming apparatus communicable with communication apparatus, control method for same, and storage medium |
| US11930145B2 (en) * | 2021-06-25 | 2024-03-12 | Canon Kabushiki Kaisha | Image forming apparatus communicable with communication apparatus, control method for same, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104021088B (en) | 2017-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8590050B2 (en) | Security compliant data storage management | |
| US10409781B2 (en) | Multi-regime caching in a virtual file system for cloud-based shared content | |
| CA2910211C (en) | Object storage using multiple dimensions of object information | |
| CN104021088A (en) | Log storage method and device | |
| US10108672B2 (en) | Stream-based object storage solution for real-time applications | |
| US10956051B2 (en) | Data-packed storage containers for streamlined access and migration | |
| US11429630B2 (en) | Tiered storage for data processing | |
| US9438665B1 (en) | Scheduling and tracking control plane operations for distributed storage systems | |
| CN103250143A (en) | Data storage method and storage device | |
| US20140181455A1 (en) | Category based space allocation for multiple storage devices | |
| CN102130959A (en) | System and method for scheduling cloud storage resource | |
| US20180248977A1 (en) | Selective distribution of messages in a publish-subscribe system | |
| CN104050248A (en) | File storage system and storage method | |
| CN110147203A (en) | A kind of file management method, device, electronic equipment and storage medium | |
| US11494413B1 (en) | Query alerts generation for virtual warehouse | |
| US11416176B2 (en) | Function processing using storage controllers for load sharing | |
| US9380126B2 (en) | Data collection and distribution management | |
| CN105426125B (en) | A kind of date storage method and device | |
| KR101470857B1 (en) | Network distributed file system and method using iSCSI storage system | |
| JP4224279B2 (en) | File management program | |
| WO2024021470A1 (en) | Cross-region data scheduling method and apparatus, device, and storage medium | |
| CN114528576A (en) | Data asset management method, system, electronic device and storage medium | |
| CN117032596B (en) | Data access method and device, storage medium and electronic equipment | |
| KR20130068590A (en) | Method and metadata server for managing storage device of cloud storage | |
| US11755216B2 (en) | Cache memory architecture and management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 705-708, room two, No. 121, north south of the Five Ridges Avenue, Chancheng District, Guangdong, Foshan, 528000 Applicant after: GUANGDONG RUIJIANG CLOUD COMPUTING CO., LTD. Address before: Chancheng District of Guangdong city of Foshan province south of the Five Ridges 528000 Avenue North 121 East International A District Office 7-8 Applicant before: Guangdong Efly Network Co., Ltd. |
|
| COR | Change of bibliographic data | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |