CN113138970A - Real-time statistical analysis system and method for database error logs - Google Patents
Real-time statistical analysis system and method for database error logs Download PDFInfo
- Publication number
- CN113138970A CN113138970A CN202110439699.1A CN202110439699A CN113138970A CN 113138970 A CN113138970 A CN 113138970A CN 202110439699 A CN202110439699 A CN 202110439699A CN 113138970 A CN113138970 A CN 113138970A
- Authority
- CN
- China
- Prior art keywords
- data
- error
- logs
- database
- original data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007619 statistical method Methods 0.000 title claims abstract description 30
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000004458 analytical method Methods 0.000 claims abstract description 27
- 238000004140 cleaning Methods 0.000 claims abstract description 17
- 238000011010 flushing procedure Methods 0.000 claims description 2
- 230000000007 visual effect Effects 0.000 abstract description 8
- 238000004590 computer program Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 230000003862 health status Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a real-time statistical analysis system and a method for database error logs.A database server of the system is provided with a slow log collection service unit for collecting error logs when the error logs are generated; the message middleware receives and stores the error log; the service layer cleans the error log, matches the cleaned data according to a preset rule to obtain matched data, and performs statistical analysis on the matched data to obtain analysis data; acquiring original data of the error log and cleaning the original data to obtain cleaned original data; and displaying the analysis data and the cleaned original data by the display output layer. According to the technical scheme provided by the invention, error logs are directly collected in real time without logging in a command line window of a server, the error logs of all databases can be checked through a visual interface, and the error logs can be filtered according to a self-defined rule of the database. The self-service and customization of the user are realized.
Description
Technical Field
The invention belongs to the technical field of big data, and particularly relates to a real-time statistical analysis system and a real-time statistical analysis method for database error logs.
Background
In the database synchronization system of the log analysis architecture, the source end data synchronization system is responsible for capturing operation logs of the source end database, analyzing the logs to restore and obtain corresponding database operations, and then sending the corresponding database operations to the target end data synchronization system, and the target end data synchronization system is responsible for executing the operations in the target end database, so that the data synchronization process is completed. In the process, error logs are generated, and the analysis of the error logs of the database is realized by copying the error inquired logs to a temporary directory on a server for installing the database and analyzing the logs through a plug-in installed on the server. An error log is obtained for a specified time period. The status and specific errors of the current library cannot be obtained in real time. Meanwhile, the database administrator does not pay much attention to sorting out the error logs of all the databases one by one.
In the related art, the error log analysis system needs to log in a server for installing a database to operate each time, which not only affects the performance of data, but also cannot ensure the security of the server. The analysis system only analyzes historical error logs and cannot acquire the error logs generated by the system in real time. Therefore, as the business volume of a company increases, the data volume of the databases increases, the error logs are scattered on the servers of the databases, and the problem that how to stably and efficiently collect the error logs, quickly acquire the health status of the databases and quickly locate the databases becomes more and more urgent to be solved by the company.
Disclosure of Invention
In view of the above, the present invention provides a real-time statistical analysis system for database error logs and a method thereof to solve the problem that the health status of a database and a quick location database cannot be obtained quickly in the prior art.
In order to achieve the purpose, the invention adopts the following technical scheme: a real-time statistical analysis system for database error logs, comprising:
the database server is provided with a slow log collection service unit and is used for collecting error logs when the error logs are generated;
the message middleware is used for receiving and storing the error log;
the service layer is used for cleaning the error log, matching the cleaned data according to a preset rule to obtain matched data, and performing statistical analysis on the matched data to obtain analysis data; acquiring original data of the error log and cleaning the original data to obtain cleaned original data;
and the display output layer outputs and displays the analysis data and the cleaned original data.
Further, if the database servers include a plurality of database servers, each database server is provided with a slow log collection service unit.
Further, the cleaning the error log includes:
processing unstructured data in the error log into structured data.
Further, the database server further includes:
and the alarm unit is used for giving an alarm when the error log is generated.
Further, the display output layer includes:
the display unit adopts a touch screen.
The embodiment of the application provides a real-time statistical analysis method for database error logs, which comprises the following steps:
collecting error logs when the error logs are generated;
receiving and storing the error log;
cleaning the error log, matching the cleaned data according to a preset rule to obtain matched data, and performing statistical analysis on the matched data to obtain analysis data; acquiring original data of the error log and cleaning the original data to obtain cleaned original data;
and outputting and displaying the analysis data and the cleaned original data.
Further, storing the analysis data and the cleaned raw data to a distributed search engine.
By adopting the technical scheme, the invention can achieve the following beneficial effects:
the invention provides a real-time statistical analysis system and method for database error logs, wherein the system comprises a database server, a database server and a database server, wherein the database server collects error logs when the error logs are generated; the message middleware receives and stores the error log; the service layer cleans the error log, matches the cleaned data according to a preset rule, and performs statistical analysis on the matched data; cleaning the original data; and the display output layer outputs and displays the analysis data and the cleaned original data. The system provided by the invention adopts a deployment acquisition program to all database servers to acquire error logs in real time and upload the error logs to the message queue middleware, the real-time error log processing service can acquire error log details from the message queue middleware, clean the logs according to corresponding rules, finally store the original logs and the cleaned logs to a distributed search engine, and provide real-time error log analysis for a user through a visual interface display. According to the technical scheme provided by the invention, error logs are directly collected in real time without logging in a server command line window, the error logs of all databases can be checked through a visual interface, the error logs can be filtered according to the self-defined rules of the databases, and self-help and customization of users are realized.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic diagram of a real-time database error log statistical analysis system according to the present invention;
FIG. 2 is a flow chart of a real-time statistical analysis system for database error logs according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail below. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the examples given herein without any inventive step, are within the scope of the present invention.
The following describes a specific real-time statistical analysis system for database error logs and a method thereof provided in the embodiments of the present application with reference to the accompanying drawings.
As shown in fig. 1, the system for statistically analyzing an error log of a real-time database provided in the embodiment of the present application includes:
the database server is provided with a slow log collection service unit and is used for collecting error logs when the error logs are generated;
the message middleware is used for receiving and storing the error log;
the service layer is used for cleaning the error log, matching the cleaned data according to a preset rule to obtain matched data, and performing statistical analysis on the matched data to obtain analysis data; acquiring original data of the error log and cleaning the original data to obtain cleaned original data;
and the display output layer outputs and displays the analysis data and the cleaned original data.
The working principle of the real-time statistical analysis system for the database error logs is that the system adopts a deployment acquisition program to all database servers to acquire the error logs in real time and upload the error logs to a message queue middleware, real-time error log processing service can acquire error log details from the message queue middleware, clean the logs according to corresponding rules, and store original logs and cleaned logs to a distributed search engine. And provides the user with real-time error log analysis through a visual interface display.
The rule matching engine stores some rules customized by users. It can be understood that the user can set the setting according to the needs of the user, and the application is not limited herein.
Preferably, if the database server includes a plurality of database servers, each database server is provided with a slow log collection service unit.
Specifically, according to the method, all database servers are provided with slow log collection services, the slow log collection services collect slow logs in real time and actively push the slow logs to a message queue middleware, then a slow log real-time processing program obtains original data of the slow logs from the message middleware, cleans the original data, and writes the original data into a distributed search engine. Meanwhile, matching is carried out according to a rule engine defined by a user, the slow logs of the matched rule engine are subjected to statistical analysis and stored in a distributed search engine, and the slow logs are displayed through visual display service.
Preferably, the flushing the error log includes:
processing unstructured data in the error log into structured data.
Preferably, the database server further comprises:
and the alarm unit is used for giving an alarm when the error log is generated.
Specifically, the data acquisition service is deployed on the database server, when newly-added error logs are generated in the database, the service actively pulls the error log files, real-time analysis and real-time early warning of the error logs can be achieved, the logs are matched according to user-defined rules, and then the statistical data are generated through aggregation calculation.
In some embodiments, the display output layer comprises:
the display unit adopts a touch screen.
The application can finally display the analysis data and the original data obtained after cleaning through the touch screen.
As shown in fig. 2, an embodiment of the present application provides a real-time statistical analysis method for a database error log, including:
s101, collecting error logs when the error logs are generated;
s102, receiving and storing the error log;
s103, cleaning the error log, matching the cleaned data according to a preset rule to obtain matched data, and performing statistical analysis on the matched data to obtain analysis data; acquiring original data of the error log and cleaning the original data to obtain cleaned original data;
and S104, outputting and displaying the analysis data and the cleaned original data.
Preferably, the analysis data and the cleaned raw data are stored to a distributed search engine.
The working principle of the real-time statistical analysis method for the database error logs is that data acquisition service is deployed on a database server, when newly-added error logs are generated in a database, the service actively pulls error log files and actively pushes the error log files to a message queue middleware, then a slow log real-time processing program can acquire original data of the slow logs from the message middleware and clean the original data, and the original data is written into a distributed search engine. Meanwhile, matching is carried out according to a rule engine defined by a user, the slow logs of the matched rule engine are subjected to statistical analysis and stored in a distributed search engine, and finally, the slow logs are displayed through visual display service.
The embodiment of the application provides computer equipment, which comprises a processor and a memory connected with the processor;
the memory is used for storing a computer program, and the computer program is used for executing the real-time statistical analysis method for the database error log provided by any one of the above embodiments;
the processor is used to call and execute the computer program in the memory.
In summary, the present invention provides a real-time statistical analysis system and method for database error logs, wherein the system includes a database server collecting error logs when the error logs are generated; the message middleware receives and stores the error log; the service layer cleans the error log, matches the cleaned data according to a preset rule, and performs statistical analysis on the matched data; cleaning the original data; and the display output layer outputs and displays the analysis data and the cleaned original data. The system provided by the invention adopts a deployment acquisition program to all database servers to acquire error logs in real time and upload the error logs to the message queue middleware, the real-time error log processing service can acquire error log details from the message queue middleware, clean the logs according to corresponding rules, finally store the original logs and the cleaned logs to a distributed search engine, and provide real-time error log analysis for a user through a visual interface display. According to the technical scheme provided by the invention, error logs are directly collected in real time without logging in a server command line window, the error logs of all databases can be checked through a visual interface, the error logs can be filtered according to the self-defined rules of the databases, and self-help and customization of users are realized.
It is to be understood that the system embodiments provided above correspond to the method embodiments described above, and corresponding specific contents may be referred to each other, which are not described herein again.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of systems, devices (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (7)
1. A real-time statistical analysis system for database error logs, comprising:
the database server is provided with a slow log collection service unit and is used for collecting error logs when the error logs are generated;
the message middleware is used for receiving and storing the error log;
the service layer is used for cleaning the error log, matching the cleaned data according to a preset rule to obtain matched data, and performing statistical analysis on the matched data to obtain analysis data; acquiring original data of the error log and cleaning the original data to obtain cleaned original data;
and the display output layer outputs and displays the analysis data and the cleaned original data.
2. The system of claim 1,
if the database servers comprise a plurality of database servers, each database server is provided with a slow log collection service unit.
3. The system of claim 1 or 2, wherein the flushing the error log comprises:
processing unstructured data in the error log into structured data.
4. The system of claim 2, wherein the database server further comprises:
and the alarm unit is used for giving an alarm when the error log is generated.
5. The system of claim 2, wherein the presentation output layer comprises:
the display unit adopts a touch screen.
6. A real-time statistical analysis method for database error logs is characterized by comprising the following steps:
collecting error logs when the error logs are generated;
receiving and storing the error log;
cleaning the error log, matching the cleaned data according to a preset rule to obtain matched data, and performing statistical analysis on the matched data to obtain analysis data; acquiring original data of the error log and cleaning the original data to obtain cleaned original data;
and outputting and displaying the analysis data and the cleaned original data.
7. The method of claim 6,
and storing the analysis data and the cleaned original data to a distributed search engine.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110439699.1A CN113138970A (en) | 2021-04-23 | 2021-04-23 | Real-time statistical analysis system and method for database error logs |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110439699.1A CN113138970A (en) | 2021-04-23 | 2021-04-23 | Real-time statistical analysis system and method for database error logs |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113138970A true CN113138970A (en) | 2021-07-20 |
Family
ID=76813697
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110439699.1A Pending CN113138970A (en) | 2021-04-23 | 2021-04-23 | Real-time statistical analysis system and method for database error logs |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113138970A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114610316A (en) * | 2022-03-15 | 2022-06-10 | 中国银行股份有限公司 | Method and device for analyzing and controlling production line logs |
| CN116506230A (en) * | 2023-06-28 | 2023-07-28 | 广东长盈科技股份有限公司 | Data acquisition method and system based on RSA asymmetric encryption |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20180037342A (en) * | 2016-10-04 | 2018-04-12 | 주식회사 리나소프트 | Application software error monitoring, statistics management service and solution method. |
| CN110674021A (en) * | 2019-09-09 | 2020-01-10 | 深圳供电局有限公司 | Detection method and system for login log of mobile application |
| CN111949705A (en) * | 2020-07-31 | 2020-11-17 | 上海中通吉网络技术有限公司 | Slow query statistical analysis method, device and system |
-
2021
- 2021-04-23 CN CN202110439699.1A patent/CN113138970A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20180037342A (en) * | 2016-10-04 | 2018-04-12 | 주식회사 리나소프트 | Application software error monitoring, statistics management service and solution method. |
| CN110674021A (en) * | 2019-09-09 | 2020-01-10 | 深圳供电局有限公司 | Detection method and system for login log of mobile application |
| CN111949705A (en) * | 2020-07-31 | 2020-11-17 | 上海中通吉网络技术有限公司 | Slow query statistical analysis method, device and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114610316A (en) * | 2022-03-15 | 2022-06-10 | 中国银行股份有限公司 | Method and device for analyzing and controlling production line logs |
| CN116506230A (en) * | 2023-06-28 | 2023-07-28 | 广东长盈科技股份有限公司 | Data acquisition method and system based on RSA asymmetric encryption |
| CN116506230B (en) * | 2023-06-28 | 2023-10-03 | 广东长盈科技股份有限公司 | Data acquisition method and system based on RSA asymmetric encryption |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108063699B (en) | Network performance monitoring method and device, electronic equipment and storage medium | |
| US20180114126A1 (en) | Systems and methods for identifying process flows from log files and visualizing the flow | |
| CN106815125A (en) | A kind of log audit method and platform | |
| US10007571B2 (en) | Policy based dynamic data collection for problem analysis | |
| CN108737170A (en) | A kind of batch daily record abnormal data alarm method and device | |
| CN106104496A (en) | The abnormality detection not being subjected to supervision for arbitrary sequence | |
| CN110309130A (en) | A kind of method and device for host performance monitor | |
| CN103425750A (en) | Cross-platform and cross-application log collecting system and collecting managing method thereof | |
| CN111752793A (en) | System abnormity monitoring method and device, computer equipment and storage medium | |
| JP7254975B2 (en) | Key-based logging for processing structured data items with executable logic | |
| CN106663041B (en) | Method and apparatus for recovering usability of cloud-based services from system failures | |
| Sukhija et al. | Event management and monitoring framework for HPC environments using ServiceNow and Prometheus | |
| EP4020218B1 (en) | Analyzing large-scale data processing jobs | |
| CN109407984B (en) | Method, device and equipment for monitoring performance of storage system | |
| CN113542068A (en) | Redis multi-instance monitoring system and method | |
| CN103001824A (en) | System and method for monitoring multiple servers | |
| CN113138970A (en) | Real-time statistical analysis system and method for database error logs | |
| CN101989931A (en) | Operation alarm processing method and device | |
| CN110471383B (en) | Equipment management method, device and system and electronic equipment | |
| Diotalevi et al. | Collection and harmonization of system logs and prototypal Analytics services with the Elastic (ELK) suite at the INFN-CNAF computing centre | |
| CN115039092A (en) | Providing customized integrated flow templates | |
| CN113032252A (en) | Method and device for collecting buried point data, client device and storage medium | |
| CN112445583A (en) | Task management method, task management system, electronic device, and storage medium | |
| EP3471356A1 (en) | Device and method for acquiring meter values associated with a calculation task | |
| CN111597091A (en) | Data monitoring method and system, electronic equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |