CN110046135B - Storage method for nonvolatile client log - Google Patents
Storage method for nonvolatile client log Download PDFInfo
- Publication number
- CN110046135B CN110046135B CN201910298001.1A CN201910298001A CN110046135B CN 110046135 B CN110046135 B CN 110046135B CN 201910298001 A CN201910298001 A CN 201910298001A CN 110046135 B CN110046135 B CN 110046135B
- Authority
- CN
- China
- Prior art keywords
- log
- mmap
- memory
- file
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 101000771640 Homo sapiens WD repeat and coiled-coil-containing protein Proteins 0.000 claims abstract description 69
- 102100029476 WD repeat and coiled-coil-containing protein Human genes 0.000 claims abstract description 69
- 230000006870 function Effects 0.000 claims abstract description 16
- 238000013507 mapping Methods 0.000 claims abstract description 12
- 230000000694 effects Effects 0.000 claims abstract description 10
- 230000009191 jumping Effects 0.000 claims description 2
- 230000000903 blocking effect Effects 0.000 abstract description 4
- 230000015556 catabolic process Effects 0.000 abstract 1
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method for storing a client log in a nonvolatile manner, which comprises the following steps: step A, finding a log file with a transitional effect under a log catalog, and mapping the log file with the transitional effect into an MMAP memory according to an MMAP rule; step B, when the log data is received, the log data is written into an MMAP memory; step C, the MMAP memory prints the received log data into the log file with the transitional function; and D, scanning the log files in the log directory, and renaming the scanned log files. The invention solves the problem that log information is lost due to the breakdown of the client; meanwhile, a plurality of log files can be written without increasing a memory, IO consumption caused by log writing of a client can be reduced, the problem of blocking caused by log writing is reduced, and user experience is improved.
Description
Technical Field
The invention belongs to the field of client log storage methods, and particularly relates to a storage method for a client log which is not volatile.
Background
In the prior art, most of log storage methods of clients (Android or IOS systems) open a file handle, and then directly read and write the file handle, which is equivalent to directly operating a disk, there is a problem that: if the client crashes while the log is being written, it is possible that the currently printed critical log is not written to disk, resulting in loss of log information. In addition, direct reading and writing of files has the problem of IO latency, which may be due to the fact that writing logs causes a card.
In the prior art, a small part of client log storage method is to write the log information into a disk once again or open-up thread after the stack space is directly written and the buffer area is full, and the method also has the possibility of log information loss caused by client crash.
Disclosure of Invention
The invention aims to provide a storage method for the log of the client, which solves the problem that log information is lost due to the collapse of the client; meanwhile, IO consumption caused by log writing of the client can be reduced, the problem of blocking caused by log writing is reduced, and user experience is improved.
In order to solve the technical problems, the invention adopts the following technical scheme:
the method for storing the client log in a nonvolatile manner is characterized by comprising the following steps of:
step A, finding a log file with a transitional effect under a log catalog, and mapping the log file with the transitional effect into an MMAP memory according to an MMAP rule;
step B, when the log data is received, the log data is written into an MMAP memory;
step C, the MMAP memory prints the received log data into the log file with the transitional function;
and D, scanning the log files in the log directory, and renaming the scanned log files.
File memory mapping (MMAP, map files or device into memory) is a method of memory mapping files, which maps a file or other object into memory. Due to the characteristics of the MMAP technology, in the running process, the system can forcedly output the data cached in the MMAP memory to the disk, so that by the method, even if a client crashes, the system can synchronize the data cached in the MMAP memory to the disk, and partial log information is prevented from being lost due to the fact that a process is closed in the log writing process.
Further, in the step C, firstly, whether the memory of the MMAP memory is full is judged, if not, the step B is skipped; if yes, the MMAP memory prints the received log data into the log file with the transitional function.
Further, in the step a, if the log file with the transitional function is not found in the log directory, the system MMAP is called to create an MMAP memory during system initialization, the created MMAP memory is mapped to a text file of the system, and the text file is named and used as the log file with the transitional function.
Further, step F is further included after step D, a log file playing a role in transition is innovatively created under the log directory, the newly created log file is mapped into the MMAP memory, the data trap of the MMAP memory is cleared, and step B-step D is repeated.
By means of the structure, the MMAP memory can be reused, a plurality of log files can be written without increasing the memory, IO consumption caused by log writing of a client is reduced, the problem of blocking caused by log writing is reduced, and user experience is improved.
Further, before step B, writing header data in the MMAP memory.
Since each log file may need associated data before and after, state information of each current log file needs to be saved, header data is written in an MMAP memory, and each time the log reloads the header data of the last time.
Compared with the prior art, the method solves the problem that log information is lost due to client crash; meanwhile, a plurality of log files can be written without increasing a memory, IO consumption caused by log writing of a client can be reduced, the problem of blocking caused by log writing is reduced, and user experience is improved.
Drawings
FIG. 1 is a schematic diagram of an embodiment of the present invention.
Detailed Description
The method for storing the client log in a nonvolatile manner comprises the following steps of:
and step A, finding a log file with a transitional effect under the log directory, and mapping the log file with the transitional effect into an MMAP memory according to MMAP rules.
In the step a, if the log file with the transitional function is not found in the log directory, the system MMAP is called to create an MMAP memory during system initialization, the created MMAP memory is mapped into a text file of the system, and the text file is named and used as the log file with the transitional function.
And B, when the log data is received, writing the log data into an MMAP memory.
Before step B, writing header data in the MMAP memory.
And C, printing the received log data into the log file with the transitional function by using the MMAP memory.
In the step C, firstly judging whether the memory of the MMAP memory is full, if not, jumping to the step B; if yes, the MMAP memory prints the received log data into the log file with the transitional function.
And D, scanning the log files in the log directory, and renaming the scanned log files.
And F, creatively creating a log file with a transitional effect under the log catalog, mapping the newly created log file into the MMAP memory, clearing the data trap of the MMAP memory, and repeating the steps B-D.
As shown in fig. 1, specifically, the embodiment of the present invention includes:
(1) If log.0 of the log file with the transitional function is not found in the log directory, calling a system MMAP to create an MMAP memory with the size of 2M during the initialization of the Android or IOS system, mapping the created MMAP memory into a text file of the system, naming the text file as log.0, and taking the log.0 as the log file with the transitional function to represent the latest stored log file. If the log.0 of the log file with the transitional function is found in the log catalog, the log.0 is mapped into an MMAP memory according to MMAP rules.
(2) A 2K Header data Header structure (struct Header) of the mapped MMAP memory space is read, wherein the structure is set as follows:
struct Header {
int32_t write_len_;
int32_t free_len_;
int32_t flag_;
char ext_data_[2035];
char magic_flag_;
};
in the above setting, write_len_indicates that the data memory space size has been written; free_len_indicates the size of the remaining unwritten; flag_represents the current data flag; ext_data_represents other additional data, of which size is 2035 bytes; magic_flag_represents magic number and is mainly used for log marking.
If the mapped MMAP memory space header data read is successful, the corresponding mac_flag_is a 0xCC value, otherwise, the data portion of the remaining 2046K is cleared of the space 0, and two fields write_len_ =0, free_len_ =2046×2048 of the header data are modified.
(3) When log data is received, the log data is written into a data portion in the MMAP memory, the length of each writing is n, and a Header structure body write_len_ =write_len_ +n, free_len_ =free_len_ -n is modified.
(4) Judging whether the data space of the MMAP memory is full, if the free_len_0 indicates that the MMAP memory space is not full, writing can be continued; if free_len_ <=0 indicates that the MMAP memory space is full, file switching is required: firstly synchronizing mapping data of an MMAP memory into a log.0 file, then scanning a log text file under a file directory, and storing a corresponding file name, and renaming all log files if the log.0, the log.1 and the log.2 files are scanned, wherein a modification rule is that data+1 after log.is modified, namely, the scanned file is renamed to log.1, log.2 and log.3 corresponding to the scanned file.
(5) If the log is written continuously, but the currently written MMAP memory is full in 2M space, then the log.0 of the file is recreated, the log.0 is mapped into the original MMAP memory, the data of the MMAP memory is trapped for 0, and then the steps (1) - (4) are continued.
In summary, the invention has the following characteristics:
(1) A memory mapping mechanism MMAP of the system is utilized to open up a 2M MMAP memory space, log files are mapped into the MMAP memory space, and when a process is closed, the system synchronously maps data of the MMAP memory into a disk, so that the problem that part of logs are lost when the process is closed in the log writing process is prevented.
(2) The space of the memory opened by the client is repeatedly used, after the 2M space of the MMAP memory is full, the memory data is refreshed into the file, and then another log file is mapped into the current 2M MMAP memory block, so that the problem that a plurality of log files can be written without increasing the memory is solved.
(3) Associated data may be needed before and after each log file, so that state information of each current log file needs to be saved, a data structure of a header needs to be written into an MMAP memory block of 2M, the MMAP memory data of 2M is distributed to be 2K header information+2046K data information, and the log can reload the header data of the last time each time.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are all within the scope of the present invention.
Claims (1)
1. The method for storing the client log in a nonvolatile manner is characterized by comprising the following steps of:
step A, finding a log file with a transitional effect under a log catalog, and mapping the log file with the transitional effect into an MMAP memory according to an MMAP rule; if the log file with the transitional function is not found under the log directory, calling a system MMAP to create an MMAP memory during system initialization, mapping the created MMAP memory into a text file of the system, naming the text file and taking the text file as the log file with the transitional function;
step B, when the log data is received, the log data is written into an MMAP memory;
step C, the MMAP memory prints the received log data into the log file with the transitional function;
step D, scanning the log files in the log catalog, and renaming the scanned log files;
step F, innovatively creating a log file with a transitional effect under a log catalog, mapping the newly created log file into the MMAP memory, clearing a data trap of the MMAP memory, and repeating the steps B-D;
wherein,,
in the step C, firstly judging whether the memory of the MMAP memory is full, if not, jumping to the step B; if yes, the MMAP memory prints the received log data into the log file with the transitional function;
before step B, writing header data in the MMAP memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910298001.1A CN110046135B (en) | 2019-04-15 | 2019-04-15 | Storage method for nonvolatile client log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910298001.1A CN110046135B (en) | 2019-04-15 | 2019-04-15 | Storage method for nonvolatile client log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110046135A CN110046135A (en) | 2019-07-23 |
| CN110046135B true CN110046135B (en) | 2023-08-18 |
Family
ID=67277000
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910298001.1A Active CN110046135B (en) | 2019-04-15 | 2019-04-15 | Storage method for nonvolatile client log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110046135B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111063195A (en) * | 2019-11-19 | 2020-04-24 | 珠海市德宇辉煌信息科技有限公司 | License plate recognition device and system |
| CN111367755A (en) * | 2020-02-17 | 2020-07-03 | 上海基分文化传播有限公司 | User log writing method and system of mobile terminal |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541691A (en) * | 2011-12-27 | 2012-07-04 | 北京人大金仓信息技术股份有限公司 | Log check point recovery method applied to memory data base OLTP (online transaction processing) |
| CN103309767A (en) * | 2012-03-08 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Method and device for processing client log |
| CN104731921A (en) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | Method for storing and processing small log type files in Hadoop distributed file system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10691553B2 (en) * | 2015-12-16 | 2020-06-23 | Netapp, Inc. | Persistent memory based distributed-journal file system |
| US10229012B2 (en) * | 2016-08-15 | 2019-03-12 | Oracle International Corporation | Committing copy-on-write transaction with a persist barrier for a persistent object including payload references |
-
2019
- 2019-04-15 CN CN201910298001.1A patent/CN110046135B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541691A (en) * | 2011-12-27 | 2012-07-04 | 北京人大金仓信息技术股份有限公司 | Log check point recovery method applied to memory data base OLTP (online transaction processing) |
| CN103309767A (en) * | 2012-03-08 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Method and device for processing client log |
| CN104731921A (en) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | Method for storing and processing small log type files in Hadoop distributed file system |
Non-Patent Citations (1)
| Title |
|---|
| Key-Value型NoSQL本地存储系统研究;马文龙;朱妤晴;蒋德钧;熊劲;张立新;孟潇;包云岗;;计算机学报(第08期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110046135A (en) | 2019-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104731921B (en) | Storage and processing method of the Hadoop distributed file systems for log type small documents | |
| CN106570018B (en) | Serialization and deserialization method, device and system and electronic equipment | |
| US8225029B2 (en) | Data storage processing method, data searching method and devices thereof | |
| CN110046135B (en) | Storage method for nonvolatile client log | |
| CN101707633B (en) | Message-oriented middleware persistent message storing method based on file system | |
| CN102902724B (en) | Mass raster tile map release method | |
| JP2000194590A (en) | Extension card file system | |
| CN101783740B (en) | Method and device for managing message file | |
| CN109710185A (en) | Data processing method and device | |
| CN102053879A (en) | Self-recovery real-time file system based on FLASH | |
| US8788784B2 (en) | Method and device for storing and reading/writing composite document | |
| US7603387B2 (en) | Techniques to manage media files | |
| CN112346659A (en) | Storage method, equipment and storage medium for distributed object storage metadata | |
| CN100580669C (en) | Method for realizing cache memory relates to file allocation table on Flash storage medium | |
| CN107122140A (en) | A kind of file intelligent storage method based on metadata information | |
| CN112800007B (en) | Directory entry expansion method and system suitable for FAT32 file system | |
| KR20060120674A (en) | Information recording medium data processing apparatus and data recording method | |
| CN115774699B (en) | Database shared dictionary compression method and device, electronic equipment and storage medium | |
| CN110187837B (en) | File access method, device and file system | |
| CN108958657B (en) | Data storage method, storage device and storage system | |
| CN111639076A (en) | Cross-platform efficient key value storage method | |
| US7536507B1 (en) | Methods, systems, and computer program products for preserving atomic writes using leaves and bitmaps | |
| CN107506156B (en) | Io optimization method of block device | |
| CN110245121A (en) | File management method, system and electronic equipment | |
| CN102479213B (en) | Data buffering method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |