CN110046135A - A kind of storage method that client log is not volatile - Google Patents
A kind of storage method that client log is not volatile Download PDFInfo
- Publication number
- CN110046135A CN110046135A CN201910298001.1A CN201910298001A CN110046135A CN 110046135 A CN110046135 A CN 110046135A CN 201910298001 A CN201910298001 A CN 201910298001A CN 110046135 A CN110046135 A CN 110046135A
- Authority
- CN
- China
- Prior art keywords
- mmap
- log
- memory
- journal file
- transition role
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of not volatile storage methods of client log, comprising the following steps: step A finds the journal file to play a transition role under Log Directory, is mapped in MMAP memory according to the journal file that MMAP rule plays a transition role this；When receiving daily record data, which is written in MMAP memory by step B；Step C, MMAP memory prints to the daily record data received in the journal file to play a transition role；Step D scans the journal file in Log Directory, the journal file scanned is renamed.The present invention solves the problems, such as that log information is lost because of client collapse；Meanwhile multiple journal files can be write without increasing memory, it is possible to reduce client writes log bring IO consumption, reduces the Caton problem writing log and generating, and improves user experience.
Description
Technical field
The invention belongs to client log storage method field, in particular to a kind of client log not volatile storage side
Method.
Background technique
In the prior art, most client (Android IOS system) log storing method is to open one
A file handle, is then directly written and read file handle, is equivalent to directly operate disk in this way, there are one
Problem is: falling if writing client collapse when log, it would be possible that the key log currently printed is not written to magnetic
Disk causes log information to be lost.In addition, there is IO waiting in direct read/write file, probably due to writing log leads to Caton.
In the prior art, the client log storage method of fraction is to write direct stack space, waits buffer areas
After having expired, property write-in disk either opens up thread write-in disk again, and this method equally exists log information because of client
End collapses and the possibility of loss.
Summary of the invention
It is an object of the present invention in view of the above shortcomings of the prior art, provide that a kind of client log is not volatile to be deposited
Method for storing solves the problems, such as that log information is lost because of client collapse；Simultaneously, it is possible to reduce client writes log bring
IO consumption, reduces the Caton problem writing log and generating, and improves user experience.
In order to solve the above technical problems, the technical scheme adopted by the invention is that:
A kind of storage method that client log is not volatile, its main feature is that the following steps are included:
Step A finds the journal file to play a transition role under Log Directory, the day that this plays a transition role according to MMAP rule
Will File Mapping is into MMAP memory；
When receiving daily record data, which is written in MMAP memory by step B；
Step C, MMAP memory prints to the daily record data received in the journal file to play a transition role；
Step D scans the journal file in Log Directory, the journal file scanned is renamed.
It is a kind of Memory Mapping File that document memory, which maps (MMAP, map files or device into memory),
Method, a file or other objects can be mapped into memory.The characteristics of due to MMAP technology, in the process of running,
The data cached in MMAP memory can be output to by force disk by system, therefore, by the above method, even if client is collapsed,
The data cached in MMAP memory can be also synchronized in disk by system, prevent the process in writing logging process from closing and loss portion
Divide log information.
Further, in the step C, first determine whether the memory of MMAP memory has expired, if it is not, then going to step
B；If so, MMAP memory prints to the daily record data received in the journal file to play a transition role.
Further, in the step A, if not finding the journal file to play a transition role under Log Directory,
Then in system initialization, calling system MMAP creates a MMAP memory, and the MMAP memory of creation is mapped to the one of system
In a text file, this article this document is named and as the journal file to play a transition role.
It further, further include step F after step D, innovation creation one plays a transition role under Log Directory
Newly created journal file is mapped in the MMAP memory by journal file, and the data trap of MMAP memory is reset, weight
Multiple step B ~ step D.
By above structure, MMAP memory is reusable, can write multiple journal files without increasing memory, reduce
Client writes log bring IO consumption, reduces the Caton problem writing log and generating, and improves user experience.
It further, further include the write head data in MMAP memory before step B.
Since the possible front and back of every part of journal file needs associated data, it is therefore desirable to save the shape of each current log file
State information, so the write head data in MMAP memory, log is loaded into the header data that can be multiplexed last time again every time.
Compared with prior art, the present invention solves the problems, such as that log information is lost because of client collapse；Meanwhile not having to
Multiple journal files can be write by increasing memory, it is possible to reduce client writes log bring IO consumption, and reduction is write log and produced
Raw Caton problem improves user experience.
Detailed description of the invention
Fig. 1 is an embodiment of the present invention schematic diagram.
Specific embodiment
The not volatile storage method of client log of the present invention the following steps are included:
Step A finds the journal file to play a transition role under Log Directory, the day that this plays a transition role according to MMAP rule
Will File Mapping is into MMAP memory.
In the step A, if the journal file to play a transition role is not found under Log Directory, at the beginning of system
Calling system MMAP creates a MMAP memory when beginningization, and the MMAP memory of creation is mapped to a text file of system
In, this article this document is named and as the journal file to play a transition role.
When receiving daily record data, which is written in MMAP memory by step B.
It further include the write head data in MMAP memory before step B.
Step C, MMAP memory prints to the daily record data received in the journal file to play a transition role.
In the step C, first determine whether the memory of MMAP memory has expired, if it is not, the B that then gos to step；If so,
MMAP memory prints to the daily record data received in the journal file to play a transition role.
Step D scans the journal file in Log Directory, the journal file scanned is renamed.
Step F, one journal file to play a transition role of innovation creation under Log Directory, by newly created journal file
It is mapped in the MMAP memory, and the data trap of MMAP memory is reset, repeat step B ~ step D.
As shown in Figure 1, specifically, the specific embodiment of the invention includes:
(1) if not finding the journal file log.0 to play a transition role under Log Directory, in Android or
Calling system MMAP creates the MMAP memory that a size is 2M when IOS system initialization, and the MMAP memory of creation is mapped
It is log.0 by text file designation, and play a transition role log.0 as described into a text file of system
Journal file, indicate the journal file of newest storage.If having found the log to play a transition role under Log Directory
Log.0 is then mapped in MMAP memory by file log.0 according to MMAP rule.
(2) the 2K header data Header structural body (struct Header) of the MMAP memory headroom of mapping is read,
Middle structural body is provided that
struct Header {
int32_t write_len_;
int32_t free_len_;
int32_t flag_;
char ext_data_[2035];
char magic_flag_;
};
In above-mentioned setting, write_len_ expression has been written into datarams space size；Free_len_ indicates that residue is not write
The size entered；Flag_ indicates current data label；Ext_data_ indicates other additional data, and wherein size is 2035 words
Section；Magic_flag_ indicates magic number, mainly does log label and uses.
If the MMAP memory headroom header data of mapping is read successfully, corresponding magic_flag_ is 0xCC value,
Otherwise it reads unsuccessful, then the data portion of remaining 2046K is first emptied into position 0, while modifying two fields of header data
Write_len_=0, free_len_=2046*2048.
(3) when receiving daily record data, which is written into the data portion in MMAP memory, every time write-in length
Degree is n, modifies Header structural body write_len_=write_len_+n, free_len_=free_len_-n.
(4) judge whether the data space of MMAP memory has expired, MMAP memory headroom is indicated if free_len_ > 0
It is less than, it can continue to write to；If free_len_≤0 indicates that MMAP memory headroom has been expired, need to do file switching: first will
The mapping data of MMAP memory are synchronized in file log.0, then scan the log text file under file directory, and preservation pair
The filename answered, it is assumed that file log.0, log.1, log.2 are arrived in scanning, then needing to rename all journal files, are repaired
Change rule by the subsequent data+1 of log., i.e., the file scanned to be corresponded to renamed as log.1, log.2, log.3.
(5) if there are also logs to continue to write to, but the presently written space MMAP memory 2M has been expired, then creating again
File log.0 is built, log.0 is mapped in original MMAP memory, and by the data trap of MMAP memory clear 0, is then proceeded to
Walk step (1) ~ (4).
To sum up, the invention has the characteristics that:
(1) the memory mapping mechanism MMAP for utilizing system, opens up the MMAP memory headroom of one piece of 2M, journal file is mapped to
In the MMAP memory headroom, when process is closed, system can synchronize the data of MMAP memory mapping into disk, thus prevent
Write the problem of process closes lost part log in logging process.
(2) space for reusing the memory that client initialization is opened up, after the space 2M of each MMAP memory has been expired,
Internal storage data is flushed in file, is then mapped to again with another journal file in the MMAP memory block of current 2M,
Solve the problems, such as that multiple journal files can be write by not having to increase memory in this way.
(3) associated data may be needed before and after every part of journal file, it is therefore desirable to save the shape of each current log file
State information, so needing to be written the data structure on a head in the MMAP memory block of 2M, by the MMAP internal storage data of 2M point
Data information with the header information+2046K for 2K, log are loaded into the header data that can be multiplexed last time again every time.
The embodiment of the present invention is described with above attached drawing, but the invention is not limited to above-mentioned specific
Embodiment, the above mentioned embodiment is only schematical, rather than limitation, those skilled in the art
Under the inspiration of the present invention, without breaking away from the scope protected by the purposes and claims of the present invention, it can also make very much
Form, within these are all belonged to the scope of protection of the present invention.
Claims (5)
1. a kind of not volatile storage method of client log, which comprises the following steps:
Step A finds the journal file to play a transition role under Log Directory, the day that this plays a transition role according to MMAP rule
Will File Mapping is into MMAP memory；
When receiving daily record data, which is written in MMAP memory by step B；
Step C, MMAP memory prints to the daily record data received in the journal file to play a transition role；
Step D scans the journal file in Log Directory, the journal file scanned is renamed.
2. the not volatile storage method of client log as described in claim 1, which is characterized in that
In the step C, first determine whether the memory of MMAP memory has expired, if it is not, the B that then gos to step；If so, MMAP
Memory prints to the daily record data received in the journal file to play a transition role.
3. the not volatile storage method of client log as described in claim 1, which is characterized in that in the step A, if
The journal file to play a transition role is not found under Log Directory, then the calling system MMAP creation one in system initialization
A MMAP memory, and the MMAP memory of creation is mapped in a text file of system, this article this document is named
And as the journal file to play a transition role.
4. the not volatile storage method of client log as claimed in claim 2, which is characterized in that
It further include step F after step D, one journal file to play a transition role of innovation creation under Log Directory will be new
The journal file of creation is mapped in the MMAP memory, and the data trap of MMAP memory is reset, and repeats step B ~ step
D。
5. such as the not volatile storage method of the described in any item client logs of Claims 1-4, which is characterized in that in step B
It before, further include the write head data in MMAP memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910298001.1A CN110046135B (en) | 2019-04-15 | 2019-04-15 | Storage method for nonvolatile client log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910298001.1A CN110046135B (en) | 2019-04-15 | 2019-04-15 | Storage method for nonvolatile client log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110046135A true CN110046135A (en) | 2019-07-23 |
| CN110046135B CN110046135B (en) | 2023-08-18 |
Family
ID=67277000
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910298001.1A Active CN110046135B (en) | 2019-04-15 | 2019-04-15 | Storage method for nonvolatile client log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110046135B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111063195A (en) * | 2019-11-19 | 2020-04-24 | 珠海市德宇辉煌信息科技有限公司 | License plate recognition device and system |
| CN111367755A (en) * | 2020-02-17 | 2020-07-03 | 上海基分文化传播有限公司 | User log writing method and system of mobile terminal |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541691A (en) * | 2011-12-27 | 2012-07-04 | 北京人大金仓信息技术股份有限公司 | Log check point recovery method applied to memory data base OLTP (online transaction processing) |
| CN103309767A (en) * | 2012-03-08 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Method and device for processing client log |
| CN104731921A (en) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | Method for storing and processing small log type files in Hadoop distributed file system |
| US20170177447A1 (en) * | 2015-12-16 | 2017-06-22 | Plexistor Ltd. | Persistent memory based distributed-journal file system |
| US20180046556A1 (en) * | 2016-08-15 | 2018-02-15 | Oracle International Corporation | Persistent memory transactions with undo logging |
-
2019
- 2019-04-15 CN CN201910298001.1A patent/CN110046135B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541691A (en) * | 2011-12-27 | 2012-07-04 | 北京人大金仓信息技术股份有限公司 | Log check point recovery method applied to memory data base OLTP (online transaction processing) |
| CN103309767A (en) * | 2012-03-08 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Method and device for processing client log |
| CN104731921A (en) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | Method for storing and processing small log type files in Hadoop distributed file system |
| US20170177447A1 (en) * | 2015-12-16 | 2017-06-22 | Plexistor Ltd. | Persistent memory based distributed-journal file system |
| US20180046556A1 (en) * | 2016-08-15 | 2018-02-15 | Oracle International Corporation | Persistent memory transactions with undo logging |
Non-Patent Citations (1)
| Title |
|---|
| 马文龙;朱妤晴;蒋德钧;熊劲;张立新;孟潇;包云岗;: "Key-Value型NoSQL本地存储系统研究", 计算机学报, no. 08 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111063195A (en) * | 2019-11-19 | 2020-04-24 | 珠海市德宇辉煌信息科技有限公司 | License plate recognition device and system |
| CN111367755A (en) * | 2020-02-17 | 2020-07-03 | 上海基分文化传播有限公司 | User log writing method and system of mobile terminal |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110046135B (en) | 2023-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104731921B (en) | Storage and processing method of the Hadoop distributed file systems for log type small documents | |
| CN107862064B (en) | High-performance and extensible lightweight file system based on NVM (non-volatile memory) | |
| EP1176523A2 (en) | System for providing extended file attributes | |
| RU2348973C2 (en) | Methods of decompression and compression for maintenance of guidance with properties of files between systems of objects | |
| US7890469B1 (en) | File change log | |
| KR101683322B1 (en) | Maintaining undo and redo capability across metadata merges | |
| CN110046135A (en) | A kind of storage method that client log is not volatile | |
| US20090313260A1 (en) | Methods and systems for assisting information processing by using storage system | |
| WO2002050684A3 (en) | Object-based storage device with improved reliability and fast crash recovery | |
| US20090113130A1 (en) | System and method for updating dirty data of designated raw device | |
| US20030078910A1 (en) | Transaction processing system using efficient file update processing and recovery processing | |
| CN104536699B (en) | A kind of stream data wiring method based on embedded file system | |
| WO2002019110A3 (en) | Manipulation of zombie files and evil-twin files | |
| US8090925B2 (en) | Storing data streams in memory based on upper and lower stream size thresholds | |
| CA2218270A1 (en) | Text index registration and retrieval method | |
| CN102053879A (en) | Self-recovery real-time file system based on FLASH | |
| WO1995017783A9 (en) | Data compression system | |
| CN109101365A (en) | A kind of data backup and resume method deleted again based on source data | |
| US20100312804A1 (en) | Anti-item for deletion of content in a distributed datastore | |
| CN108491402A (en) | A kind of implementation method of lightweight file system | |
| CN101533346A (en) | Source file comparing unit and method thereof | |
| CN108170777A (en) | A kind of distributed file system | |
| WO2001093116A3 (en) | Storage of design data | |
| CN106776942B (en) | A kind of transmission preservation system and method for network audit log | |
| CN107526840A (en) | File system snapshot querying method, device and computer-readable recording medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |