CN111639098A - Cloud log management method - Google Patents
Cloud log management method Download PDFInfo
- Publication number
- CN111639098A CN111639098A CN202010394268.3A CN202010394268A CN111639098A CN 111639098 A CN111639098 A CN 111639098A CN 202010394268 A CN202010394268 A CN 202010394268A CN 111639098 A CN111639098 A CN 111639098A
- Authority
- CN
- China
- Prior art keywords
- log
- database
- management method
- cloud
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 18
- 238000000034 method Methods 0.000 claims abstract description 17
- 238000004140 cleaning Methods 0.000 claims description 3
- 230000002688 persistence Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 abstract description 3
- 230000006399 behavior Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a cloud log management method, which comprises the following steps: s1, the front end acquires a real-time log; s2, dynamically pushing the log to a message component by the filecut component in the database; s3, the log service module is connected with a database to backup the log; s4, the log service module is connected with a database to clean the log; and S5, the user queries, deletes and uploads the log in the object storage module through the http api. The method of the invention can dynamically realize the real-time checking of the logs of various databases; the backup log can be conveniently and quickly checked; the method analyzes the following open source components as part of the system. The lightweight filebeat log collection component can be installed on a server, and continuous collection and sending of logs are achieved. kafka is a high-throughput distributed publish-subscribe messaging system, which can store log streams with persistence and fault tolerance and can solve the problem of inconsistent log collection speed and processing speed.
Description
Technical Field
The invention belongs to the technical field of information, and particularly relates to a cloud log management method.
Background
In an information system, the log plays a crucial role, and can record all the behaviors of the system, and the use condition of the system can be analyzed through the behaviors, so that the optimizable direction can be analyzed. Particularly abnormal information and the like which occur in the use of the system, and related maintenance personnel can be reminded to respond in time through the information.
Typically, each database has its own type of log, such as error log, run log, slow log, etc. To backup and store the logs, a separate program development for each database is required, which is cumbersome and time consuming.
When the database has a long running time and a large log, the risk of full disk exists, and the user search is not very convenient.
The traditional log backup method is to store the log in a host, and the log is not convenient to use and view.
Disclosure of Invention
In view of this, the present invention is directed to a cloud log management method, which adopts an object storage method for conveniently, quickly and automatically collecting logs, for automatically collecting and performing real-time query analysis on various logs of a system, for backing up and storing logs of various database systems, for conveniently and quickly checking backed-up logs, and for avoiding the influence caused by overlarge logs.
In order to achieve the purpose, the technical scheme of the invention is realized as follows:
a cloud log management method comprises the following steps:
s1, the front end acquires a real-time log;
s2, dynamically pushing the log to a message component by the filecut component in the database;
s3, the log service module is connected with a database to backup the log;
s4, the log service module is connected with a database to clean the log;
and S5, the user queries, deletes and uploads the log in the object storage module through the http api.
Further, the front end acquires and implements the log in step S1 as follows, the front end sends out request database information, the log service module returns the database information, and the front end connects to the kafka module according to the database information to monitor the log, thereby acquiring the real-time log.
Further, the message component in step S2 is a kafka module, which is configured to receive the message, the kafka module stores the received message in the topic of topic, and the user reads the message in topic.
Further, the database information includes the type and state of the data.
Further, the process of connecting the log service module in steps S3 and S4 to the database is as follows, where the log service module obtains all database information from the database resource pool and connects the database according to the database information.
Further, the log backup process in step S3 is as follows, and the log backup is implemented by transmitting the log to the object storage module by command.
Further, the log cleaning process in step S4 is to obtain the history log of the database stored in the object by command, and clean the invalid log at regular time according to the set log saving time.
Compared with the prior art, the cloud log management method has the following advantages:
(1) the method of the invention can dynamically realize the real-time checking of the logs of various databases; the backup log can be conveniently and quickly checked.
(2) The method of the present invention analyzes the following open source components as part of the system. The lightweight filebeat log collection component can be installed on a server, and continuous collection and sending of logs are achieved. kafka is a high-throughput distributed publish-subscribe messaging system, which can store log streams with persistence and fault tolerance and can solve the problem of inconsistent log collection speed and processing speed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate an embodiment of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
fig. 1 is a structural diagram of a cloud log management method according to an embodiment of the present invention.
Detailed Description
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used only for convenience in describing the present invention and for simplicity in description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and thus, are not to be construed as limiting the present invention. Furthermore, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first," "second," etc. may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless otherwise specified.
In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art through specific situations.
The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
As shown in fig. 1, a cloud log management method includes the following steps:
s1, the front end acquires a real-time log;
s2, dynamically pushing the log to a message component by the filecut component in the database;
s3, the log service module is connected with a database to backup the log;
s4, the log service module is connected with a database to clean the log;
and S5, the user queries, deletes and uploads the log in the object storage module through the http api.
The front end acquires and implements the log in step S1 as follows, the front end sends out request database information, the log service module returns the database information, and the front end connects with the kafka module according to the database information to monitor the log, thereby acquiring the real-time log.
The message component in the step S2 is a kafka module, which is used to receive the message, the kafka module stores the message in the topic of topic after receiving the message, and the user reads the message in topic.
The database information comprises the type and the state of data.
The process of the step S3 and the log service module connecting the database in S4 is as follows, the log service module obtains all database information from the database resource pool, and connects the database according to the database information.
The log backup process in step S3 is as follows, and the log backup is implemented by transmitting the log to the object storage module by command.
The log cleaning process in step S4 is to obtain the history log of the database stored in the object by command, and clean the invalid log at regular time according to the set log saving time.
The detailed steps are as follows:
the method comprises the following steps: the front end sends out request database information, the log service module returns the database information, and the front end monitors the log according to the database information connection kafka so as to obtain a real-time log;
step two: dynamically pushing the log to a kafka module by a filecut component in the database;
step three: the log service module acquires all database information from the database resource pool, connects the database according to the database information including types, states and the like, and transmits the log to an object for storage through a command to realize the backup of the log;
step four: the log service module acquires all database information from a database resource pool, connects the database according to the database information including types, states and the like, acquires a historical log of an object stored in the database through a command, and regularly cleans invalid logs according to set log storage time;
step five: and querying and deleting the uploaded log in the object storage module by the user through the http api.
In order to conveniently, rapidly and automatically collect logs and query and analyze the logs in real time, a cloud log management method is provided, which is used for analyzing and displaying various logs in a collection and storage system and solving the problems of query and storage of the logs in the system.
In order to realize automatic collection and real-time query analysis of various logs of the system, the following open-source components are analyzed as components of the system. The lightweight filebeat log collection component can be installed on a server, and continuous collection and sending of logs are achieved. kafka is a high-throughput distributed publish-subscribe messaging system, which can store log streams with persistence and fault tolerance and can solve the problem of inconsistent log collection speed and processing speed.
In order to realize backup and storage of various database system logs, the cloud log management system acquires various database types from a database resource pool, acquires database connection information, establishes database connection, backs up and deletes logs by judging different database types.
In order to conveniently and quickly check the backed-up log and avoid the influence caused by overlarge log, an object storage method is adopted to upload the log to an object storage system, and the log can be downloaded and deleted through an http interface.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (7)
1. A cloud log management method is characterized by comprising the following steps:
s1, the front end acquires a real-time log;
s2, dynamically pushing the log to a message component by the filecut component in the database;
s3, the log service module is connected with a database to backup the log;
s4, the log service module is connected with a database to clean the log;
and S5, the user queries, deletes and uploads the log in the object storage module through the http api.
2. The cloud log management method according to claim 1, wherein: the front end acquires and implements the log in step S1 as follows, the front end sends out request database information, the log service module returns the database information, and the front end connects with the kafka module according to the database information to monitor the log, thereby acquiring the real-time log.
3. The cloud log management method according to claim 1, wherein: the message component in the step S2 is a kafka module, which is used to receive the message, the kafka module stores the message in the topic of topic after receiving the message, and the user reads the message in topic.
4. The cloud log management method according to claim 3, wherein: the database information comprises the type and the state of data.
5. The cloud log management method according to claim 1, wherein: the process of the step S3 and the log service module connecting the database in S4 is as follows, the log service module obtains all database information from the database resource pool, and connects the database according to the database information.
6. The cloud log management method according to claim 1, wherein: the log backup process in step S3 is as follows, and the log backup is implemented by transmitting the log to the object storage module by command.
7. The cloud log management method according to claim 1, wherein: the log cleaning process in step S4 is to obtain the history log of the database stored in the object by command, and clean the invalid log at regular time according to the set log saving time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010394268.3A CN111639098A (en) | 2020-05-11 | 2020-05-11 | Cloud log management method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010394268.3A CN111639098A (en) | 2020-05-11 | 2020-05-11 | Cloud log management method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111639098A true CN111639098A (en) | 2020-09-08 |
Family
ID=72329282
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010394268.3A Pending CN111639098A (en) | 2020-05-11 | 2020-05-11 | Cloud log management method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111639098A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114584464A (en) * | 2022-03-07 | 2022-06-03 | 浪潮云信息技术股份公司 | Cloud platform full-automatic management log collection method and terminal |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20140050903A (en) * | 2012-10-22 | 2014-04-30 | 주식회사 엔써티 | Real time backup system of database, system of recovering data and method of recovering data |
| CN106227877A (en) * | 2016-08-02 | 2016-12-14 | 北京集奥聚合科技有限公司 | A kind of distributed information log acquisition system based on hadoop and method |
| CN108306980A (en) * | 2018-03-06 | 2018-07-20 | 北京工业大学 | A kind of engineering flight support big data Log Analysis System |
| CN109344033A (en) * | 2018-09-27 | 2019-02-15 | 浪潮软件股份有限公司 | A kind of cloud log collection method based on distributed structure/architecture |
| CN110502581A (en) * | 2019-08-27 | 2019-11-26 | 中国联合网络通信集团有限公司 | Distributed data base system monitoring method and device |
| CN110515912A (en) * | 2019-07-18 | 2019-11-29 | 湖南星汉数智科技有限公司 | Log processing method, device, computer installation and computer readable storage medium |
-
2020
- 2020-05-11 CN CN202010394268.3A patent/CN111639098A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20140050903A (en) * | 2012-10-22 | 2014-04-30 | 주식회사 엔써티 | Real time backup system of database, system of recovering data and method of recovering data |
| CN106227877A (en) * | 2016-08-02 | 2016-12-14 | 北京集奥聚合科技有限公司 | A kind of distributed information log acquisition system based on hadoop and method |
| CN108306980A (en) * | 2018-03-06 | 2018-07-20 | 北京工业大学 | A kind of engineering flight support big data Log Analysis System |
| CN109344033A (en) * | 2018-09-27 | 2019-02-15 | 浪潮软件股份有限公司 | A kind of cloud log collection method based on distributed structure/architecture |
| CN110515912A (en) * | 2019-07-18 | 2019-11-29 | 湖南星汉数智科技有限公司 | Log processing method, device, computer installation and computer readable storage medium |
| CN110502581A (en) * | 2019-08-27 | 2019-11-26 | 中国联合网络通信集团有限公司 | Distributed data base system monitoring method and device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114584464A (en) * | 2022-03-07 | 2022-06-03 | 浪潮云信息技术股份公司 | Cloud platform full-automatic management log collection method and terminal |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109460339B (en) | Log streaming computing system | |
| CN111158983A (en) | Integrated operation and maintenance management system | |
| CN113836044B (en) | Method and system for collecting and analyzing software faults | |
| CN105302697B (en) | A kind of running state monitoring method and system of density data model database | |
| CN110231998B (en) | Detection method and device for distributed timing task and storage medium | |
| CN109462490B (en) | Video monitoring system and fault analysis method | |
| CN111104302A (en) | Improved ELK log analysis method for cluster system | |
| WO2021097919A1 (en) | Mes monitoring method, mes monitoring device, and readable storage medium | |
| CN101883009A (en) | Method, system and network management device for automatically acquiring reliability data | |
| CN111639098A (en) | Cloud log management method | |
| CN103763143A (en) | Method and system for equipment abnormality alarming based on storage server | |
| CN202150114U (en) | Oracle monitoring system | |
| CN103401722A (en) | System and method for monitoring software resources | |
| CN113254516A (en) | Method for automatically inputting server information | |
| CN107515864B (en) | Method and equipment for monitoring workflow | |
| CN112486776A (en) | Cluster member node availability monitoring equipment and method | |
| CN108228417A (en) | Car networking log processing method and processing unit | |
| CN114500178B (en) | Self-operation intelligent Internet of things gateway | |
| CN114691405A (en) | Big data based automatic information storage inspection method | |
| CN1251085C (en) | Method of monitoring machine group system operation procedure and monitoring management device | |
| CN115460072A (en) | Log processing system integrating log collection, analysis, storage and service | |
| CN116126621A (en) | Task monitoring method of big data cluster and related equipment | |
| CN109240747A (en) | A kind of information inspection method and its relevant apparatus of multistage server management system | |
| CN112533246B (en) | Monitoring system and method for frequent network requests of intelligent equipment | |
| CN112165601A (en) | State monitoring platform of distributed industrial system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200908 |