CN112783849A - Production environment log exporting system and method - Google Patents
Production environment log exporting system and method Download PDFInfo
- Publication number
- CN112783849A CN112783849A CN202110130604.8A CN202110130604A CN112783849A CN 112783849 A CN112783849 A CN 112783849A CN 202110130604 A CN202110130604 A CN 202110130604A CN 112783849 A CN112783849 A CN 112783849A
- Authority
- CN
- China
- Prior art keywords
- module
- log
- production environment
- monitoring
- logs
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004519 manufacturing process Methods 0.000 title claims abstract description 29
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000012545 processing Methods 0.000 claims abstract description 25
- 238000012544 monitoring process Methods 0.000 claims description 37
- 238000003860 storage Methods 0.000 claims description 16
- 230000000737 periodic effect Effects 0.000 claims description 4
- 238000004140 cleaning Methods 0.000 claims description 3
- 230000006835 compression Effects 0.000 claims description 3
- 238000007906 compression Methods 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 claims description 3
- 230000003313 weakening effect Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5022—Mechanisms to release resources
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a production environment log export system and a method, which can backup logs according to daily volume, compress and store the logs in order to effectively utilize space, and because the priority of log processing is not high in service, the processing of the logs cannot consume a CPU with high consumption, and the consumption of the CPU in the log processing process needs to be limited. The production environment log export system and the method do not influence the business and simultaneously carry out the backup output of the log, can recycle the space and do not cause the loss of the log due to insufficient space.
Description
Technical Field
The invention belongs to the technical field of system logs, and particularly relates to a production environment log export system and method.
Background
The system log is used for recording the running information of the software and the application program, and developers can locate the reason of the fault of the system program or the application program through the log information. In a production environment, a running program cannot be debugged directly through a computer, so that the log is the only way for analyzing problems. By analyzing the log, the problems occurring in the program can be positioned, and further the product experience can be repaired and optimized. How to save the log and export the log display is particularly important.
Disclosure of Invention
In view of the above, the present invention is directed to a production environment log exporting system and method, so as to solve the problem that a running program cannot be directly invoked through a computer in a production environment.
In order to achieve the purpose, the technical scheme of the invention is realized as follows:
a production environment log export system comprises an application log, a storage module, a monitoring module, a scheduling module and a processing module;
the application logs are stored in the storage module, the monitoring module monitors the logs of the storage module, the scheduling module calls monitoring tasks of the monitoring module, and the processing module processes, displays and exports data.
Furthermore, the application logs comprise logs produced by programs, including normal logs and error logs.
Further, the storage module is used for caching the logs produced by the application.
Further, the monitoring module: the monitoring module has periodic tasks and is responsible for configuring the produced logs; the task of monitoring the size of the log is performed, the log is cut off and compressed after a certain threshold value is exceeded, the task of monitoring the performance of the CPU is performed, the consumption of the log system to the CPU is reduced, the task of monitoring the space of the disk is performed, and early warning information is sent out and the space for releasing old data is cleaned after the space of the disk reaches the threshold value.
Further, the scheduling module is used for scheduling the implementation of the monitoring task.
Further, the processing module comprises: the system comprises a performance control module, a backup module, a space releasing module and an early warning module;
the performance control module is used for weakening the consumption of the CPU and ensuring that the main service has enough CPU resources;
the backup module is used for storing backup data and information of the production environment log;
the space releasing module is used for cleaning old data and releasing the disk space;
the early warning processing module is used for informing the state of the disk in time and carrying out active early warning operation after the capacity is exceeded.
A production environment log export method comprising the steps of:
s1, storing the log generated by the application in a log storage module;
s2, the monitoring module monitors the size of the log, and after the size of the log exceeds a threshold value, compression processing is carried out to increase the utilization rate of the disk;
s3, the scheduling module processes the event sent by the monitoring module;
s4, if the disk space is full, the scheduling module is handed to an early warning module for processing to warn operation and maintenance personnel;
and S5, the scheduling module notifies the backup module at regular time to backup the log, release the space at regular time and increase the disk space.
Further, the scheduling module in step S3 processes the time as follows: if the CPU performance consumption is too large, the performance module is handed to process the CPU performance problem, and the CPU is degraded.
Compared with the prior art, the production environment log export system and the production environment log export method have the following advantages:
the production environment log export system and the method can backup logs according to daily volume, compress and store the logs in order to effectively utilize space, and because the priority of log processing is not high in service, the logs cannot consume too high CPU, the consumption of the CPU in the log processing process needs to be limited, the backup output of the logs is carried out while the service is not influenced, the space can be recycled, and the log loss caused by insufficient space is avoided.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate an embodiment of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a diagram of a production environment log export system according to an embodiment of the present invention;
fig. 2 is a flowchart of a method for exporting a production environment log according to an embodiment of the present invention.
Detailed Description
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used only for convenience in describing the present invention and for simplicity in description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and thus, are not to be construed as limiting the present invention. Furthermore, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first," "second," etc. may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless otherwise specified.
In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art through specific situations.
The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
As shown in fig. 1 to 2, a production environment log export system includes an application log, a storage module, a monitoring module, a scheduling module, and a processing module;
the application logs are stored in the storage module, the monitoring module monitors the logs of the storage module, the scheduling module calls monitoring tasks of the monitoring module, and the processing module processes, displays and exports data.
The application logs comprise logs which are made by programs, including normal logs and error logs.
The storage module is used for caching logs produced by the application.
The monitoring module: the monitoring module has periodic tasks and is responsible for configuring the produced logs; the task of monitoring the size of the log is performed, the log is cut off and compressed after a certain threshold value is exceeded, the task of monitoring the performance of the CPU is performed, the consumption of the log system to the CPU is reduced, the task of monitoring the space of the disk is performed, and early warning information is sent out and the space for releasing old data is cleaned after the space of the disk reaches the threshold value.
The scheduling module is used for scheduling implementation of the monitoring tasks.
The processing module comprises: the system comprises a performance control module, a backup module, a space releasing module and an early warning module;
the performance control module is used for weakening the consumption of the CPU and ensuring that the main service has enough CPU resources;
the backup module is used for storing backup data and information of the production environment log;
the space releasing module is used for cleaning old data and releasing the disk space;
the early warning processing module is used for informing the state of the disk in time and carrying out active early warning operation after the capacity is exceeded.
As shown in fig. 2, a production environment log exporting method includes the following steps:
s1, storing the log generated by the application in a log storage module;
s2, the monitoring module monitors the size of the log, and after the size of the log exceeds a threshold value, compression processing is carried out to increase the utilization rate of the disk;
s3, the scheduling module processes the event sent by the monitoring module;
s4, if the disk space is full, the scheduling module is handed to an early warning module for processing to warn operation and maintenance personnel;
and S5, the scheduling module notifies the backup module at regular time to backup the log, release the space at regular time and increase the disk space.
The scheduling module processing time procedure in step S3 is as follows: if the CPU performance consumption is too large, the performance module is handed to process the CPU performance problem, and the CPU is degraded.
The monitoring module can perform periodic backup on the log of the storage module, wherein the period is one day. If the current log size is less than 200M, backup is carried out according to the day; if the size of the current log exceeds 200M, the current log is divided by 200M, meanwhile, backup is continuously carried out according to days, and a plurality of log files exist in one day.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (8)
1. A production environment log export system, characterized by: the system comprises an application log, a storage module, a monitoring module, a scheduling module and a processing module;
the application logs are stored in the storage module, the monitoring module monitors the logs of the storage module, the scheduling module calls monitoring tasks of the monitoring module, and the processing module processes, displays and exports data.
2. The production environment log export system of claim 1, wherein: the application logs comprise logs which are made by programs, including normal logs and error logs.
3. The production environment log export system of claim 1, wherein: the storage module is used for caching logs produced by the application.
4. The production environment log export system of claim 1, wherein: the monitoring module: the monitoring module has periodic tasks and is responsible for configuring the produced logs; the task of monitoring the size of the log is performed, the log is cut off and compressed after a certain threshold value is exceeded, the task of monitoring the performance of the CPU is performed, the consumption of the log system to the CPU is reduced, the task of monitoring the space of the disk is performed, and early warning information is sent out and the space for releasing old data is cleaned after the space of the disk reaches the threshold value.
5. The production environment log export system of claim 1, wherein: the scheduling module is used for scheduling implementation of the monitoring tasks.
6. The production environment log export system of claim 1, wherein: the processing module comprises: the system comprises a performance control module, a backup module, a space releasing module and an early warning module;
the performance control module is used for weakening the consumption of the CPU and ensuring that the main service has enough CPU resources;
the backup module is used for storing backup data and information of the production environment log;
the space releasing module is used for cleaning old data and releasing the disk space;
the early warning processing module is used for informing the state of the disk in time and carrying out active early warning operation after the capacity is exceeded.
7. A production environment log export method, based on a production environment log export system of claims 1 to 6, characterized by comprising the steps of:
s1, storing the log generated by the application in a log storage module;
s2, the monitoring module monitors the size of the log, and after the size of the log exceeds a threshold value, compression processing is carried out to increase the utilization rate of the disk;
s3, the scheduling module processes the event sent by the monitoring module;
s4, if the disk space is full, the scheduling module is handed to an early warning module for processing to warn operation and maintenance personnel;
and S5, the scheduling module notifies the backup module at regular time to backup the log, release the space at regular time and increase the disk space.
8. The production environment log export method of claim 7, wherein: the scheduling module processing time procedure in step S3 is as follows: if the CPU performance consumption is too large, the performance module is handed to process the CPU performance problem, and the CPU is degraded.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110130604.8A CN112783849A (en) | 2021-01-29 | 2021-01-29 | Production environment log exporting system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110130604.8A CN112783849A (en) | 2021-01-29 | 2021-01-29 | Production environment log exporting system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112783849A true CN112783849A (en) | 2021-05-11 |
Family
ID=75760032
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110130604.8A Pending CN112783849A (en) | 2021-01-29 | 2021-01-29 | Production environment log exporting system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112783849A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021088A (en) * | 2014-06-24 | 2014-09-03 | 广东睿江科技有限公司 | Log storage method and device |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN106406771A (en) * | 2016-10-20 | 2017-02-15 | 福建天泉教育科技有限公司 | Log recording method and log recorder |
| CN109189745A (en) * | 2018-09-20 | 2019-01-11 | 郑州云海信息技术有限公司 | A kind of method and device of the log management based on cloud platform |
| CN109933462A (en) * | 2019-02-25 | 2019-06-25 | 深圳小云字符信息科技有限公司 | Log information processing method |
| CN109960686A (en) * | 2019-03-26 | 2019-07-02 | 北京百度网讯科技有限公司 | The log processing method and device of database |
-
2021
- 2021-01-29 CN CN202110130604.8A patent/CN112783849A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021088A (en) * | 2014-06-24 | 2014-09-03 | 广东睿江科技有限公司 | Log storage method and device |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN106406771A (en) * | 2016-10-20 | 2017-02-15 | 福建天泉教育科技有限公司 | Log recording method and log recorder |
| CN109189745A (en) * | 2018-09-20 | 2019-01-11 | 郑州云海信息技术有限公司 | A kind of method and device of the log management based on cloud platform |
| CN109933462A (en) * | 2019-02-25 | 2019-06-25 | 深圳小云字符信息科技有限公司 | Log information processing method |
| CN109960686A (en) * | 2019-03-26 | 2019-07-02 | 北京百度网讯科技有限公司 | The log processing method and device of database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109857558A (en) | A kind of data flow processing method and system | |
| CN103092746A (en) | Positioning method and system for thread anomaly | |
| US20120072582A1 (en) | Method, apparatus and program storage device for scheduling the performance of maintenance tasks to maintain a system environment | |
| US20240020017A1 (en) | Monitoring method and apparatus for electronic device, and electronic device | |
| CN109271243B (en) | Cluster task management system | |
| CN102346725A (en) | Message notification device and method for changing equipment configuration file | |
| CN109165138A (en) | A kind of method and apparatus of monitoring equipment fault | |
| CN109408341A (en) | Distributed memory system method for monitoring performance, device, equipment and readable storage medium storing program for executing | |
| CN110399220A (en) | A kind of management method and relevant apparatus of FPGA accelerator card | |
| CN114356499A (en) | Kubernetes cluster alarm root cause analysis method and device | |
| CN114154035A (en) | Data processing system for dynamic loop monitoring | |
| CN101827120A (en) | Cluster storage method and system | |
| CN112486701A (en) | Message asynchronous processing method and equipment thereof | |
| CN103763143A (en) | Method and system for equipment abnormality alarming based on storage server | |
| CN104679623A (en) | Server hard disk maintaining method, system and server monitoring equipment | |
| CN202713533U (en) | TV diagnosis and maintenance system and TV | |
| CN112783849A (en) | Production environment log exporting system and method | |
| CN107357809A (en) | Magnanimity platform Hisoon real-time database metric data access system | |
| US7032019B1 (en) | Apparatuses for generation and collection of information based on priority and predetermined collecting conditions | |
| CN109597736A (en) | A kind of monitoring method, device, equipment and medium | |
| CN117331794A (en) | Big data-based application software monitoring analysis system and method | |
| CN112416719B (en) | Monitoring processing method, system, equipment and storage medium for database container | |
| CN115269341A (en) | Multi-dimensional monitoring method and system for GPU (graphics processing Unit) virtual resource utilization rate | |
| CN113778340B (en) | Data storage method and device, electronic equipment and storage medium | |
| CN112965793B (en) | Identification analysis data-oriented data warehouse task scheduling method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210511 |