WO2019161551A1 - Method for processing log based on logrotate technology - Google Patents
Method for processing log based on logrotate technology Download PDFInfo
- Publication number
- WO2019161551A1 WO2019161551A1 PCT/CN2018/077080 CN2018077080W WO2019161551A1 WO 2019161551 A1 WO2019161551 A1 WO 2019161551A1 CN 2018077080 W CN2018077080 W CN 2018077080W WO 2019161551 A1 WO2019161551 A1 WO 2019161551A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- log
- data
- level
- log data
- template
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
Definitions
- the present invention relates to the field of log optimization technologies, and more particularly to a log processing method based on logrotate technology.
- log data on the disk is generally deleted as follows:
- the log processing method based on the logrotate technology provided by the present invention comprises the following steps:
- Step 1 setting a memory area in the memory, separating parameters in each statement of the target log in the memory area, and recording the parameters according to a configuration information in data matching the target log Within the set, the number of the target log is recorded in the data set, and the statement after separating the parameter is generated as a log template corresponding to the data set according to the configuration information;
- Step 2 In a case where the log template is stored in a storage disk for storing logs, the number of the log template is recorded in the data set, and after all the logs are processed, the memory area is The data set in the storage is stored in the storage disk;
- Step 3 In the case that the log template is not stored in the storage disk for storing the log, the number of the log template is generated and recorded in the data set. After all the logs are processed, The data set in the memory area and the corresponding log template are stored in the storage disk;
- Step 4 View the usage rate of the disk storing the log data in a preset time period
- Step 5 When the usage rate is in the range of the first preset threshold, search for log data from the disk that is consistent with the date of the clock system, as the first log data, and The remaining log data is used as the second log data, and the second log data is deleted. When the data volume of the first log data is greater than the second preset value, the log data of the preset level in the first log data is deleted.
- the log data has five levels, namely a first level, a second level, a third level, a fourth level, and a fifth level, the first level is the lowest level, and the fifth level is The level of the level is the highest, and the log data of the preset level includes the first level of log data, the second level of log data, and the third level of log data.
- the beneficial effects of the present invention are: in the preset time period, the log data in the disk is not directly deleted, but the disk usage rate is first checked, and when the disk usage rate is in the range of the first preset threshold, The log data is deleted.
- the method of the present invention does not occur when the disk usage rate is low, that is, all the logs are deleted, thereby ensuring the log recording function.
- the present invention provides a log processing method based on the logrotate technology.
- the present invention will be described in detail below in conjunction with specific embodiments.
- the log processing method based on the logrotate technology provided by the present invention comprises the following steps:
- Step 1 setting a memory area in the memory, separating parameters in each statement of the target log in the memory area, and recording the parameters according to a configuration information in data matching the target log Within the set, the number of the target log is recorded in the data set, and the statement after separating the parameter is generated as a log template corresponding to the data set according to the configuration information;
- Step 2 In a case where the log template is stored in a storage disk for storing logs, the number of the log template is recorded in the data set, and after all the logs are processed, the memory area is The data set in the storage is stored in the storage disk;
- Step 3 In the case that the log template is not stored in the storage disk for storing the log, the number of the log template is generated and recorded in the data set. After all the logs are processed, The data set in the memory area and the corresponding log template are stored in the storage disk;
- Step 4 View the usage rate of the disk storing the log data in a preset time period
- Step 5 When the usage rate is in the range of the first preset threshold, search for log data from the disk that is consistent with the date of the clock system, as the first log data, and The remaining log data is used as the second log data, and the second log data is deleted. When the data volume of the first log data is greater than the second preset value, the log data of the preset level in the first log data is deleted.
- the log data has five levels, namely, a first level, a second level, a third level, a fourth level, and a fifth level, the first level has the lowest level, and the fifth level has the highest level, the pre- The level of log data includes the first level of log data, the second level of log data, and the third level of log data.
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
A method for processing a log based on logrotate technology, comprising the following steps: checking the usage rate of a magnetic disk for storing log data within a preset time period; when the usage rate is within the range of a first preset threshold, searching the magnetic disk for log data with the created date consistent with the date of a clock system as first log data, and using the remaining log data in the magnetic disk as second log data; deleting the second log data; and when the data volume of the first log data is greater than a second preset value, deleting log data of default level from the first log data. The usage rate of the magnetic disk is first checked, and log data is deleted only when the usage rate of the magnetic disk is within the range of the first preset threshold, rather than directly deleting all log data in the magnetic disk is not directly deleted within the preset time period, thereby ensuring the recording function of logs.
Description
本发明涉及日志优化技术领域，更具体地，涉及基于logrotate技术的日志处理方法。The present invention relates to the field of log optimization technologies, and more particularly to a log processing method based on logrotate technology.
在现有技术中，对于很多业务系统及应用服务器，均需存储日志数据，以记录其自身的运行情况；其中，在现有技术中，一般将上述日志数据存储至磁盘上；而由于磁盘的存储空间有限，因此需定时删除磁盘上所存储的日志数据；在现有技术中，一般采用如下方式删除磁盘上的日志数据：In the prior art, for many service systems and application servers, log data needs to be stored to record its own operation; wherein, in the prior art, the log data is generally stored on a disk; The storage space is limited. Therefore, you need to delete the log data stored on the disk periodically. In the prior art, log data on the disk is generally deleted as follows:
人为设定删除时间，而系统或服务器每到设定时间时，全部删除磁盘上的日志数据；但是，在实际工作中，很有可能出现当到达设定时间时，其磁盘的使用率仍很低的情况，而此时若删除全部的日志数据，无疑丧失了日志的记录功能。Manually set the deletion time, and the system or server deletes the log data on the disk every time the set time is reached; however, in actual work, it is very likely that when the set time is reached, the disk usage rate is still very high. In the low case, if all the log data is deleted at this time, the log recording function is undoubtedly lost.
发明内容Summary of the invention
本发明的目的是提供基于logrotate技术的日志处理方法，解决上述现有技术中的一个或者是多个。It is an object of the present invention to provide a log processing method based on the logrotate technology, which solves one or more of the above prior art.
本发明提供的基于logrotate技术的日志处理方法，包括以下步骤：The log processing method based on the logrotate technology provided by the present invention comprises the following steps:
步骤1：在内存中设置一内存区域，在所述内存区域中将所述目标日志的每一语句中的参数分离出来，将所述参数按一配置信息记录于与所述目标日志匹配的数据集合内，并将所述目标日志的编号记录于所述数据集合内，分离参数后的语句按所述配置信息生成为与数据集合对应的日志模板；Step 1: setting a memory area in the memory, separating parameters in each statement of the target log in the memory area, and recording the parameters according to a configuration information in data matching the target log Within the set, the number of the target log is recorded in the data set, and the statement after separating the parameter is generated as a log template corresponding to the data set according to the configuration information;
步骤2：在一用于存储日志的存储磁盘中存有所述日志模板的情况下，记录所述日志模板的编号于所述数据集合内，在所述所有日志全部处理完成后，将内存区域中的数据集合存储于存储磁盘内；Step 2: In a case where the log template is stored in a storage disk for storing logs, the number of the log template is recorded in the data set, and after all the logs are processed, the memory area is The data set in the storage is stored in the storage disk;
步骤3：在一用于存储日志的存储磁盘中未存有所述日志模板的情况下，生成所述日志模板的编号并记录于所述数据集合内，在所述所有日志全部处 理完成后，将内存区域中的数据集合以及对应的日志模板存储于存储磁盘内；Step 3: In the case that the log template is not stored in the storage disk for storing the log, the number of the log template is generated and recorded in the data set. After all the logs are processed, The data set in the memory area and the corresponding log template are stored in the storage disk;
步骤4：在预设时间周期内，查看存储日志数据的磁盘的使用率；Step 4: View the usage rate of the disk storing the log data in a preset time period;
步骤5：当所述使用率位于第一预设阈值的范围时，从所述磁盘中查找生成日期与时钟系统的日期相一致的日志数据，作为第一日志数据，且将所述磁盘中的其余日志数据作为第二日志数据，删除所述第二日志数据，当所述第一日志数据的数据量大于第二预设值时，删除所述第一日志数据中预设级别的日志数据。Step 5: When the usage rate is in the range of the first preset threshold, search for log data from the disk that is consistent with the date of the clock system, as the first log data, and The remaining log data is used as the second log data, and the second log data is deleted. When the data volume of the first log data is greater than the second preset value, the log data of the preset level in the first log data is deleted.
在一些实施方式中，所述日志数据具有五个级别，分别为第一级别、第二级别、第三级别、第四级别和第五级别，所述第一级别的级别最低，所述第五级别的级别最高，所述预设级别的日志数据包括第一级别的日志数据、第二级别的日志数据和第三级别的日志数据。In some embodiments, the log data has five levels, namely a first level, a second level, a third level, a fourth level, and a fifth level, the first level is the lowest level, and the fifth level is The level of the level is the highest, and the log data of the preset level includes the first level of log data, the second level of log data, and the third level of log data.
本发明的有益效果为：在预设时间周期内，并不直接删除磁盘中的全部日志数据，而是首先查看磁盘的使用率，当磁盘的使用率位于第一预设阈值的范围时，才删除日志数据；采用本发明的方法，不会出现在磁盘使用率很低时，即删除全部日志的情况，从而保证了日志的记录功能。The beneficial effects of the present invention are: in the preset time period, the log data in the disk is not directly deleted, but the disk usage rate is first checked, and when the disk usage rate is in the range of the first preset threshold, The log data is deleted. The method of the present invention does not occur when the disk usage rate is low, that is, all the logs are deleted, thereby ensuring the log recording function.
本发明提供基于logrotate技术的日志处理方法，下面结合具体实施方式，对本发明作详细介绍：The present invention provides a log processing method based on the logrotate technology. The present invention will be described in detail below in conjunction with specific embodiments.
本发明提供的基于logrotate技术的日志处理方法，包括以下步骤：The log processing method based on the logrotate technology provided by the present invention comprises the following steps:
步骤1：在内存中设置一内存区域，在所述内存区域中将所述目标日志的每一语句中的参数分离出来，将所述参数按一配置信息记录于与所述目标日志匹配的数据集合内，并将所述目标日志的编号记录于所述数据集合内，分离参数后的语句按所述配置信息生成为与数据集合对应的日志模板；Step 1: setting a memory area in the memory, separating parameters in each statement of the target log in the memory area, and recording the parameters according to a configuration information in data matching the target log Within the set, the number of the target log is recorded in the data set, and the statement after separating the parameter is generated as a log template corresponding to the data set according to the configuration information;
步骤2：在一用于存储日志的存储磁盘中存有所述日志模板的情况下，记录所述日志模板的编号于所述数据集合内，在所述所有日志全部处理完成后，将内存区域中的数据集合存储于存储磁盘内；Step 2: In a case where the log template is stored in a storage disk for storing logs, the number of the log template is recorded in the data set, and after all the logs are processed, the memory area is The data set in the storage is stored in the storage disk;
步骤3：在一用于存储日志的存储磁盘中未存有所述日志模板的情况下，生成所述日志模板的编号并记录于所述数据集合内，在所述所有日志全部处理完成后，将内存区域中的数据集合以及对应的日志模板存储于存储磁盘内；Step 3: In the case that the log template is not stored in the storage disk for storing the log, the number of the log template is generated and recorded in the data set. After all the logs are processed, The data set in the memory area and the corresponding log template are stored in the storage disk;
步骤4：在预设时间周期内，查看存储日志数据的磁盘的使用率；Step 4: View the usage rate of the disk storing the log data in a preset time period;
步骤5：当所述使用率位于第一预设阈值的范围时，从所述磁盘中查找生成日期与时钟系统的日期相一致的日志数据，作为第一日志数据，且将所述磁盘中的其余日志数据作为第二日志数据，删除所述第二日志数据，当所述第一日志数据的数据量大于第二预设值时，删除所述第一日志数据中预设级别的日志数据。Step 5: When the usage rate is in the range of the first preset threshold, search for log data from the disk that is consistent with the date of the clock system, as the first log data, and The remaining log data is used as the second log data, and the second log data is deleted. When the data volume of the first log data is greater than the second preset value, the log data of the preset level in the first log data is deleted.
日志数据具有五个级别，分别为第一级别、第二级别、第三级别、第四级别和第五级别，所述第一级别的级别最低，所述第五级别的级别最高，所述预设级别的日志数据包括第一级别的日志数据、第二级别的日志数据和第三级别的日志数据。The log data has five levels, namely, a first level, a second level, a third level, a fourth level, and a fifth level, the first level has the lowest level, and the fifth level has the highest level, the pre- The level of log data includes the first level of log data, the second level of log data, and the third level of log data.
以上所述的仅是本发明的一些实施方式。对于本领域的普通技术人员来说，在不脱离本发明创造构思的前提下，还可以做出若干变形和改进，这些都属于本发明的保护范围。What has been described above is only some embodiments of the invention. It will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the spirit and scope of the invention.
Claims (2)
- 基于logrotate技术的日志处理方法，其特征在于，包括以下步骤：The log processing method based on the logrotate technology is characterized in that it comprises the following steps:步骤1：在内存中设置一内存区域，在所述内存区域中将所述目标日志的每一语句中的参数分离出来，将所述参数按一配置信息记录于与所述目标日志匹配的数据集合内，并将所述目标日志的编号记录于所述数据集合内，分离参数后的语句按所述配置信息生成为与数据集合对应的日志模板；Step 1: setting a memory area in the memory, separating parameters in each statement of the target log in the memory area, and recording the parameters according to a configuration information in data matching the target log Within the set, the number of the target log is recorded in the data set, and the statement after separating the parameter is generated as a log template corresponding to the data set according to the configuration information;步骤2：在一用于存储日志的存储磁盘中存有所述日志模板的情况下，记录所述日志模板的编号于所述数据集合内，在所述所有日志全部处理完成后，将内存区域中的数据集合存储于存储磁盘内；Step 2: In a case where the log template is stored in a storage disk for storing logs, the number of the log template is recorded in the data set, and after all the logs are processed, the memory area is The data set in the storage is stored in the storage disk;步骤3：在一用于存储日志的存储磁盘中未存有所述日志模板的情况下，生成所述日志模板的编号并记录于所述数据集合内，在所述所有日志全部处理完成后，将内存区域中的数据集合以及对应的日志模板存储于存储磁盘内；Step 3: In the case that the log template is not stored in the storage disk for storing the log, the number of the log template is generated and recorded in the data set. After all the logs are processed, The data set in the memory area and the corresponding log template are stored in the storage disk;步骤4：在预设时间周期内，查看存储日志数据的磁盘的使用率；Step 4: View the usage rate of the disk storing the log data in a preset time period;步骤5：当所述使用率位于第一预设阈值的范围时，从所述磁盘中查找生成日期与时钟系统的日期相一致的日志数据，作为第一日志数据，且将所述磁盘中的其余日志数据作为第二日志数据，删除所述第二日志数据，当所述第一日志数据的数据量大于第二预设值时，删除所述第一日志数据中预设级别的日志数据。Step 5: When the usage rate is in the range of the first preset threshold, search for log data from the disk that is consistent with the date of the clock system, as the first log data, and The remaining log data is used as the second log data, and the second log data is deleted. When the data volume of the first log data is greater than the second preset value, the log data of the preset level in the first log data is deleted.
- 根据权利要求1所述的基于logrotate技术的日志处理方法，其特征在于，所述日志数据具有五个级别，分别为第一级别、第二级别、第三级别、第四级别和第五级别，所述第一级别的级别最低，所述第五级别的级别最高，所述预设级别的日志数据包括第一级别的日志数据、第二级别的日志数据和第三级别的日志数据。The log processing method of the logrotate technology according to claim 1, wherein the log data has five levels, namely, a first level, a second level, a third level, a fourth level, and a fifth level. The level of the first level is the lowest, and the level of the fifth level is the highest. The log data of the preset level includes log data of a first level, log data of a second level, and log data of a third level.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2018/077080 WO2019161551A1 (en) | 2018-02-24 | 2018-02-24 | Method for processing log based on logrotate technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2018/077080 WO2019161551A1 (en) | 2018-02-24 | 2018-02-24 | Method for processing log based on logrotate technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2019161551A1 true WO2019161551A1 (en) | 2019-08-29 |
Family
ID=67686630
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2018/077080 WO2019161551A1 (en) | 2018-02-24 | 2018-02-24 | Method for processing log based on logrotate technology |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2019161551A1 (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090222499A1 (en) * | 2008-02-28 | 2009-09-03 | Huawei Technologies Co., Ltd. | Method, apparatus and storage device for data protection |
| CN103645984A (en) * | 2013-12-24 | 2014-03-19 | 山东中创软件工程股份有限公司 | Method and device for optimizing logs |
| CN104331487A (en) * | 2014-11-13 | 2015-02-04 | 携程计算机技术（上海）有限公司 | Method and device for processing logs |
| CN105955972A (en) * | 2016-02-01 | 2016-09-21 | 上海华测导航技术股份有限公司 | Receiver log management system and method |
| CN106503138A (en) * | 2016-10-20 | 2017-03-15 | 山东中创软件商用中间件股份有限公司 | A kind of journal file processing method and processing device |
| CN107239382A (en) * | 2017-06-23 | 2017-10-10 | 深圳市冬泉谷信息技术有限公司 | The log processing method and system of a kind of container application |
-
2018
- 2018-02-24 WO PCT/CN2018/077080 patent/WO2019161551A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090222499A1 (en) * | 2008-02-28 | 2009-09-03 | Huawei Technologies Co., Ltd. | Method, apparatus and storage device for data protection |
| CN103645984A (en) * | 2013-12-24 | 2014-03-19 | 山东中创软件工程股份有限公司 | Method and device for optimizing logs |
| CN104331487A (en) * | 2014-11-13 | 2015-02-04 | 携程计算机技术（上海）有限公司 | Method and device for processing logs |
| CN105955972A (en) * | 2016-02-01 | 2016-09-21 | 上海华测导航技术股份有限公司 | Receiver log management system and method |
| CN106503138A (en) * | 2016-10-20 | 2017-03-15 | 山东中创软件商用中间件股份有限公司 | A kind of journal file processing method and processing device |
| CN107239382A (en) * | 2017-06-23 | 2017-10-10 | 深圳市冬泉谷信息技术有限公司 | The log processing method and system of a kind of container application |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11531482B2 (en) | Data deduplication method and apparatus | |
| US8863110B2 (en) | Firmware updating system and method | |
| US20150193336A1 (en) | Computing device and method for recording system event logs of computing device | |
| KR101766790B1 (en) | Method and computing apparatus for maniging main memory database | |
| US20140280387A1 (en) | System and method for expanding storage space of network device | |
| CN105786410A (en) | Method for increasing processing speed of data storage system and data storage system | |
| CN107402870B (en) | Method and device for processing log segment in metadata server | |
| US8538925B2 (en) | System and method for backing up test data | |
| CN110888837B (en) | Object storage small file merging method and device | |
| WO2016082559A1 (en) | Data writing method and storage device | |
| US10235401B2 (en) | Method and system for handling binary large objects | |
| US20120265908A1 (en) | Server and method for buffering monitored data | |
| US9870385B2 (en) | Computer system, data management method, and computer | |
| WO2019161551A1 (en) | Method for processing log based on logrotate technology | |
| US20150067192A1 (en) | System and method for adjusting sas addresses of sas expanders | |
| KR101747262B1 (en) | Method and apparatus for reducing query processing time by dynamically changing algorithms and computer readable medium therefor | |
| CN104156327A (en) | Method for recognizing object power failure in write back mode in distributed file system | |
| KR20170106626A (en) | Method and computing apparatus for maniging main memory database | |
| CN102141964A (en) | Method and device for repeatedly using magnetic tape | |
| MX2021010056A (en) | Adaptive retrieval of objects from remote storage. | |
| CN109241011B (en) | Virtual machine file processing method and device | |
| RU2007138559A (en) | DATA OBJECT MANAGEMENT METHOD | |
| US9804968B2 (en) | Storage system and data writing method | |
| CN105072372B (en) | FAT file system self-repairing method in DVR terminal | |
| CN110879762A (en) | Cloud backup deleting method and locally-stored cloud backup deleting method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18907219 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 18907219 Country of ref document: EP Kind code of ref document: A1 |