CN111240936A - Data integrity checking method and equipment - Google Patents
Data integrity checking method and equipment Download PDFInfo
- Publication number
- CN111240936A CN111240936A CN202010033832.9A CN202010033832A CN111240936A CN 111240936 A CN111240936 A CN 111240936A CN 202010033832 A CN202010033832 A CN 202010033832A CN 111240936 A CN111240936 A CN 111240936A
- Authority
- CN
- China
- Prior art keywords
- data
- storage medium
- collected
- log
- related information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
- G06F11/3093—Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method, equipment and a computer readable storage medium for verifying data integrity, wherein the method comprises the following steps: receiving data collected by a log collection system and storing the data in a storage medium; utilizing a service process to detect the related information of the received data at fixed time; summarizing and calculating the collected related information; and comparing and analyzing the calculated result and the data stored in the storage medium to determine whether the data acquired by the log acquisition system is complete. The invention adopts a system architecture of a Master-Salve mode, finally realizes the real-time monitoring of the integrity of the acquired data and the stability of the service through calculation and comparison, and solves the problems that the integrity of the data cannot be ensured and whether the service acquisition assembly is normally served cannot be monitored in the data acquisition process of the current distributed log acquisition system.
Description
Technical Field
The present invention relates generally to the field of computer applications. More particularly, the present invention relates to a method and apparatus for data integrity check based on a log collection system.
Background
With the rapid development of computer technology and network technology, cloud computing, big data and the internet + become emerging technologies at present, and great convenience is brought to people. In such a large environment, data collection, and in particular, integrity collection and storage of data, is a topical topic.
At present, a large-scale application service system or platform generates a huge amount of logs, and system log collection and analysis are the basis of system operation and maintenance and user analysis. System logs can generally be divided into three major categories: the user behavior log is a series of operation record logs of a system user in the process of using the system; a service change log, which is required by a specific service field, collects the use of a certain function by a certain user in a certain period of time, and performs a certain operation on a certain service (object, data); and the system operation log is used for carrying out log acquisition and analysis on the conditions of the system operation server resources, the network and the basic middleware at regular time. Data integrity in the log collection process refers to the accuracy and reliability of the data. It is proposed to prevent data which does not comply with semantic regulations from being present in the database and to prevent invalid operations or erroneous information from being caused by input and output of erroneous information. The verification of data integrity is especially important in the log collection process.
Because a single machine system can not finish the collection of the logs generally, the prior art adopts a distributed architecture system to collect the logs, and the system of the architecture stores the dotting logs of the service system on a plurality of machines respectively. At present, a plurality of distributed log collection systems are available, such as a flash system, a filebed system, a logstack system and the like. However, in the data acquisition process, these systems cannot guarantee the integrity of data, and cannot monitor whether the service acquisition component is normally serviced.
Disclosure of Invention
In view of the above technical problems, the present invention provides a corresponding solution, and a system architecture thereof mainly adopts a Master-salt mode, and can be used for monitoring the integrity of data acquisition and the stability of log acquisition system service in real time.
Specifically, in one aspect, the present invention provides a method for data integrity check, including:
receiving data collected by a log collection system and storing the data in a storage medium;
utilizing a service process to detect the related information of the received data at fixed time;
summarizing and calculating the collected related information; and
and comparing and analyzing the calculated result and the data stored in the storage medium to determine whether the data acquired by the log acquisition system is complete.
In one embodiment, the log collection system collects logs on multiple servers.
In another embodiment, the related information includes a size and a number of pieces of the log file.
In yet another embodiment, the storage medium is a database having a distributed file system architecture configured to store the collected data during a data integrity check process.
In one embodiment, the service process includes a monitoring service process for monitoring the stability of the operation of the log collection system.
In another embodiment, the storage medium is a database having a mySQL architecture configured to store the collected data during the process of monitoring the operational stability of the log collection system.
In yet another embodiment, in detecting log collection system stability, the method further comprises observing the collected data using a Web interface.
In another embodiment, the system architecture of the method adopts a Master-Salve mode.
In another aspect, the present invention also discloses a device for verifying data integrity, including:
a processor configured to execute program instructions; and
a memory configured to store the program instructions, which when loaded and executed by the processor, cause the apparatus to perform:
receiving data collected by a log collection system and storing the data in a storage medium;
utilizing a service process to detect the related information of the received data at fixed time;
summarizing and calculating the collected related information; and
and comparing and analyzing the calculated result and the data stored in the storage medium to determine whether the data acquired by the log acquisition system is complete.
In yet another aspect, the present invention also discloses a computer-readable storage medium storing a program for data integrity checking, which when executed by a processor performs the operations of:
receiving data collected by a log collection system and storing the data in a storage medium;
utilizing a service process to detect the related information of the received data at fixed time;
summarizing and calculating the collected related information; and
and comparing and analyzing the calculated result and the data stored in the storage medium to determine whether the data acquired by the log acquisition system is complete.
By utilizing the method and the equipment provided by the invention, the problem that the integrity of the acquired data of the current distributed log acquisition system cannot be ensured is solved, whether the service acquisition assembly of the log acquisition system is normally served or not can be monitored, and the integrity of the acquired data and the stability of the service of the log acquisition system can be finally monitored in real time through calculation and comparison.
Drawings
The above-described features of the present invention can be better understood, and its numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The drawings in the following description are merely exemplary embodiments of the invention and, to one skilled in the art, other drawings may be made on the basis of these drawings without inventive faculty, in which like reference numerals denote like elements and in which:
FIG. 1 is a flow chart illustrating a data integrity checking method based on a log collection system according to an embodiment of the present invention;
FIG. 2 is a flow diagram illustrating another data integrity checking method based on a log collection system according to an embodiment of the present invention;
FIG. 3 is a block diagram illustrating the principle of stability monitoring based on a log collection system according to an embodiment of the present invention; and
fig. 4 is a block diagram illustrating a data integrity checking apparatus based on a log collection system according to an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a method and equipment for verifying the integrity of a data file of a log collection system, which can simply, conveniently and efficiently verify the integrity of a collected target file. Wherein the log collection system is based on a distributed data collection process. It will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the described embodiments of the invention. However, it will be understood by those of ordinary skill in the art that the described embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Moreover, this description is not to be taken as limiting the scope of the embodiments described herein.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a block flow diagram illustrating a data integrity verification method 100 based on a data acquisition system in accordance with an embodiment of the present invention.
As shown in fig. 1, the present invention provides a method for verifying data integrity, which includes the following steps: first, the method 100 performs S101, where the method 100 receives collected data from a log collection system and stores it in a storage medium. Wherein the collected data may include: user behavior logs, service change logs, system operation logs and the like. The storage medium can comprise HDFS, HBase and the like. Next, at step S102, the method 100 utilizes the relevant service process to periodically detect the relevant information of the received data. Then, the method 100 performs S103, where the method 100 performs a summary calculation on the collected information. Finally, the method 100 ends in step S104, where the calculated result and the data stored in the storage medium are compared and analyzed to determine whether the data collected by the log collection system is complete.
In an embodiment, the log collection system is configured to collect log data, and the log collection system may specifically collect logs on multiple servers. The log collection system can also be a distributed system, each node of the server side supports distributed cluster deployment, and high concurrency and high availability can be realized. In one embodiment, the log collection system may be comprised of three parts: log collection, log storage and analysis and result application. The log collection is mainly responsible for providing a plurality of modes for collecting logs; the log storage and analysis mainly realizes the analysis of scene analysis logs which are uniformly stored and customized; the result application then implements a service interface or default management function that provides the log analysis results for use by the application function.
After the log is collected, the log is transmitted to a message queue on line or off line, and the log is extracted by the log consumption application and then stored. And then, customizing an automatic job task according to analysis requirements to realize log data extraction, conversion and statistics, and writing result data into a relational database to provide for a service interface or a log query default management function for use. In the micro-service framework, the micro-service administration capability and the service load need to depend on the collection and analysis of the service log, so the log collection system can be the basic support function of the micro-service framework. As a specific embodiment, the log collection system may be a Flume system. In order to better understand the technical solution of the present invention, a flash system, which is a highly available, reliable and distributed mass log collection and aggregation transmission system, will be briefly introduced below. The flash system can store data generated by an application into any centralized storage, such as HDFS ("Hadoop distributed file system") designed to fit distributed file systems running on general purpose hardware, HBase, and the like. HDFS is characterized by high fault tolerance and is designed to be deployed on inexpensive hardware. Moreover, HDFS provides high throughput access to application data, suitable for applications with very large data sets. In addition, HDFS relaxes POSIX requirements so that streaming access to data in a file system can be achieved.
When the speed of collecting data exceeds the speed of writing data, that is, when the collected information encounters a peak, the collected information is very large and even exceeds the data writing capacity of the system, and at this time, the Flume system makes an adjustment between the data producer and the data container to ensure that the Flume system can provide smooth data between the two. The Flume system supports various data senders customized in the log system for collecting data. While the flash system provides the ability to simply process and write data to various data recipients. The Flume system is logically divided into three layers: agent, Collector, Storage. There are currently two versions of Flume, with Flume version 0.9X being commonly referred to as Flume-OG and Flume version 1.X being commonly referred to as Flume-NG. Further, Flume-OG employs a multi-Master approach. In order to ensure the consistency of the configuration data, the Flume system introduces a ZooKeeper for storing the configuration data, and the ZooKeeper can ensure the consistency and high availability of the configuration data. In addition, the ZooKeeper can notify the Flume Master node when the configuration data changes. The most obvious change of the Flume-NG is to cancel Master and Zookeeper which are configured in a centralized management way and change the Flume-NG into a pure transmission tool. Another major difference with flash-NG is that the read data and the write data are handled by different worker threads.
Fig. 2 is a flow chart illustrating a data integrity checking method 200 based on a log collection system according to an embodiment of the present invention.
As shown in fig. 2, the log collection system collects the dotting logs of the client, where the dotting logs can be implemented as follows: 1) for different services, a service enumeration class is added to realize an interface, and the logs can be distinguished without configuring the logs, so that the logs are printed under a specific directory; 2) the same service enumerates sub-molecular types, which can be used for data statistics, classification and summarization, etc.; 3) the log uses uniform separators, so that the query switching is convenient due to uniform format. The dotting log of the client will land on the disk, wherein the land data is the data to be persisted, and the data is generally placed in a hard disk or other persistent storage device, for example: pictures, system logs, data displayed on pages, data stored in a relational database, and the like. Landing data usually has a fixed carrier so that they do not disappear instantaneously. While the landless data generally refers to data stored in a memory or network transmission, which is instantaneous and disappears after use, for example, data requested by a browser sent to a server and data read from a database until a page is displayed.
As a specific implementation manner, it is assumed that the client-side dotting logs respectively land on 3 servers, and at this time, the logs on the 3 servers need to be collected, and the stability of the collection system and the integrity of the log file are monitored. In the process of performing data integrity check on the log acquisition system, firstly, on the one hand, the dotting logs are respectively landed on 3 servers, as shown in steps 201 and 202 in fig. 2. Next, the system will generate a file every hour, as indicated at step 203, and the method 200 will then proceed to step 204, at which point the log collection system will monitor the status of each file to determine if there are errors in the service, performance problems or whether there are problems with a drop or surge in traffic, and will collect new data and send it to the data collection component. Method flow 200 then performs step 205, where the system deposits the collected data to a storage medium. In one embodiment, the storage medium may be, for example, a database with an HDFS architecture configured to store the collected data during a data integrity check process.
In another aspect, the method of the present invention may deploy a service process on the server, where the service process is a process for monitoring data integrity, and the related information of each file generated in step 203 is detected periodically by the service process. In one embodiment, the related information includes at least a size and a number of files. The method 200 then proceeds to step 207, where the system reports the information to the Master host. The method 200 then executes step 208, where the system at the Master service node may perform a summary calculation on the size and number of log files on the three machines, for example, may perform an averaging operation on the number of logs. After completing the above two operations, the method 200 starts to execute step 209, where the system performs a comparative analysis on the results of the summary calculation in step 208 and the data stored in the HDFS in step 205. And under a certain judgment rule, judging whether the acquired log data is complete according to the result of the comparative analysis. In one embodiment, for example, whether the collected log data is complete or not may be determined according to the result of the summary calculation being larger than the data in the HDFS. Finally, the method 200 ends in step 210, in which the final result is output.
The invention not only can verify the integrity of the data acquired by the log acquisition system, but also can monitor the stability of the log acquisition system. In one embodiment, the service process may be a monitoring service process, which is used to monitor the stability of the operation of the log collection system. FIG. 3 is a block diagram 300 illustrating a stability monitoring schematic based on a log collection system, according to an embodiment of the invention. As shown in fig. 3, the log collection system 304 collects logs on multiple servers. And the log collection system collects the dotting logs of the client, wherein the dotting logs of the client fall to a magnetic disk. As a specific implementation manner, it is assumed that the client-side dotting logs respectively land on 3 servers, and at this time, the logs on the 3 servers need to be collected, and the stability of the collection system is monitored.
In another embodiment, the system architecture of the method adopts a Master-Salve mode. In this mode, a monitoring service process is deployed on each server for collecting logs. Specifically, the service process 303 for monitoring the system stability deploys one monitoring service process for the client side to click on the log ground (for example, 301 and 302), and the monitoring service process detects the service state of the log collection system 304 at regular time, and then the system reports the detected service state information to the Master server 306, and meanwhile the log collection system stores the data in the storage medium 305, and can perform result display through the terminal, for example, can perform result display through a Web page, as shown in 307. According to the monitoring service process, network management personnel can acquire the running state of any monitored log acquisition system through the Server at any time, and timely receive alarm information of the Server under the condition that the Server is abnormal (such as unexpected down machine, insufficient storage space, high CPU utilization rate and the like), so as to monitor the running stability of the log acquisition system.
As a specific implementation, in an embodiment, for example, the storage medium may be a database with mySQL architecture, which is configured to store the collected data during the process of monitoring the operation stability of the log collection system. mySQL is a relational database management system that keeps data in different tables instead of putting all data in one large repository, which increases speed and flexibility. In another embodiment, in order to facilitate observation of the execution effect of the method of the present invention, the method further includes observing the collected data or displaying the observed result by using a Web interface.
From the foregoing, it can be seen that an aspect of the present invention provides a method for data integrity check based on a log collection system. By using the method, the scheme of the invention can realize the verification of the data integrity. On the other hand, the invention also discloses equipment for carrying out work by using the method and based on the data integrity verification of the log acquisition system.
Fig. 4 is a block diagram illustrating a data integrity checking apparatus 400 based on a log collection system according to an embodiment of the present invention. The device 400 may be embodied as a server or smart client comprising a computer system, depending on the implementation. As shown in fig. 4, the device 400 may include a processor 401, a memory 402, a user interface 403, and a communication interface 404. In an embodiment, the processor 401 may be a general-purpose processor or a special-purpose processor and may be configured to execute computer program instructions, in particular computer program instructions stored in the memory 402, in order to cause the device 400 to perform the various embodiments of the methods 100, 200 and 300 according to the present invention as shown in fig. 1, 2 and 3.
The memory 402 may be a high speed random access memory such as a Dynamic Random Access Memory (DRAM), a Static Random Access Memory (SRAM), a double data rate random access memory (DDR RAM), or other random access memory. The memory 402 may also be an access solid state storage device or non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory, or other non-volatile solid state storage devices.
The user interface 403 may include one or more output devices, which may include, for example, one or more speakers and/or one or more visual displays. The user interface 403 also includes one or more input devices including, for example, a keyboard, a mouse, voice commands, an input unit or microphone, a touch screen display, a touch sensitive tablet, a gesture capture camera, or other input buttons or controls. The user interface 403 may receive instruction inputs from a data analyst and present the results of the operations to the data analyst in accordance with the present invention.
The communication interface 404 may include a network interface, a modem, various types of communication ports, a bus, a communication link, a transceiver, or other communication devices. According to the aspect of the present invention, the communication interface may receive data from a plurality of clients and transmit various types of data to the clients.
The working principle of the device of the invention is explained in detail below with reference to fig. 2 and 4. Firstly, a log collection system collects dotting logs on a plurality of servers, wherein the dotting logs of a client side fall to a magnetic disk. As a specific embodiment, it is assumed that the client-side dotting logs respectively land on 3 servers, and at this time, the logs on the 3 servers need to be collected. And then the log collection system generates a file every hour, monitors the state of each file, collects new data and sends the new data to the data collection assembly, and then stores the collected data in a storage medium. As a specific embodiment, the storage medium may be, for example, a database with HDFS architecture, which is configured to store the collected data during the data integrity check process.
The device of the invention firstly calls a data acquisition program stored on a memory 402 through a processor 401, controls a data acquisition component to acquire data through a communication interface 404, stores the acquired data in a storage medium, then regularly detects the generated related information of each file through a process of monitoring data integrity deployed in advance on the memory 402, wherein the related information at least comprises the size and the number of data files in one embodiment, then reports the information to a Master host, then the processor 401 carries out summary calculation on the size and the number of log files on three machines collected by a Master service node through the communication interface 404, finally carries out comparative analysis on the result of the summary calculation and the data stored in the storage medium (for example, HDFS), and judges whether the acquired log data is complete according to the result of the comparative analysis, the result of the final verification of whether the data is complete is displayed through the user interface 403.
In yet another aspect, in one embodiment, the invention also discloses a computer-readable storage medium storing a program for data integrity checking, which when executed by a processor performs the following operations: receiving data collected by a log collection system and storing the data in a storage medium; detecting the received related information of the data by the deployed service process at regular time, and then reporting the information to the host; the host computer carries out summary calculation on the collected information; and comparing and analyzing the result calculated by the host computer and the data stored on the storage medium, and finally determining whether the acquired data is complete.
It should also be appreciated that any module, unit, component, server, computer, terminal, or device executing instructions of the examples of the invention may include or otherwise access a computer-readable medium, such as a storage medium, computer storage medium, or data storage device (removable) and/or non-removable) such as a magnetic disk, optical disk, or magnetic tape. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data.
Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, a module, or both. Any such computer storage media may be part of, or accessible or connectable to, a device. Any applications or modules described herein may be implemented using computer-readable/executable instructions that may be stored or otherwise maintained by such computer-readable media.
Although the embodiments of the present invention are described above, the descriptions are only examples for facilitating understanding of the present invention, and are not intended to limit the scope and application scenarios of the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A method of data integrity checking, comprising:
receiving data collected by a log collection system and storing the data in a storage medium;
utilizing a service process to detect the related information of the received data at fixed time;
summarizing and calculating the collected related information; and
and comparing and analyzing the calculated result and the data stored in the storage medium to determine whether the data acquired by the log acquisition system is complete.
2. The method of claim 1, wherein the log collection system collects logs on multiple servers.
3. The method of claim 1, wherein the related information comprises a size and number of log files.
4. The method of claim 1, wherein the storage medium is a database having a distributed file system architecture configured to store the collected data during a data integrity check.
5. The method of claim 1, wherein the service process comprises a monitoring service process for monitoring stability of operation of the log collection system.
6. The method of claim 5, wherein the storage medium is a database having a mySQL architecture configured for storing the collected data during monitoring of operational stability of the log collection system.
7. The method of claim 6, further comprising observing the collected data using a Web interface.
8. The method as claimed in any one of claims 1 to 7, wherein the system architecture of the method adopts a Master-Salve model.
9. An apparatus for data integrity checking, comprising:
a processor configured to execute program instructions; and
a memory configured to store the program instructions, which when loaded and executed by the processor, cause the apparatus to perform:
receiving data collected by a log collection system and storing the data in a storage medium;
utilizing a service process to detect the related information of the received data at fixed time;
summarizing and calculating the collected related information; and
and comparing and analyzing the calculated result and the data stored in the storage medium to determine whether the data acquired by the log acquisition system is complete.
10. A computer-readable storage medium storing a program for data integrity checking, which when executed by a processor performs the operations of:
receiving data collected by a log collection system and storing the data in a storage medium;
utilizing a service process to detect the related information of the received data at fixed time;
summarizing and calculating the collected related information; and
and comparing and analyzing the calculated result and the data stored in the storage medium to determine whether the data acquired by the log acquisition system is complete.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010033832.9A CN111240936A (en) | 2020-01-13 | 2020-01-13 | Data integrity checking method and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010033832.9A CN111240936A (en) | 2020-01-13 | 2020-01-13 | Data integrity checking method and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111240936A true CN111240936A (en) | 2020-06-05 |
Family
ID=70870938
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010033832.9A Pending CN111240936A (en) | 2020-01-13 | 2020-01-13 | Data integrity checking method and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111240936A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112463516A (en) * | 2020-10-23 | 2021-03-09 | 苏州浪潮智能科技有限公司 | Method and system for collecting and verifying integrity of BMC log |
| CN113760882A (en) * | 2021-11-10 | 2021-12-07 | 北京每日菜场科技有限公司 | Data abnormity alarm method and device, electronic equipment and computer readable medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012198796A (en) * | 2011-03-22 | 2012-10-18 | Nec Corp | Log collection system, device, method and program |
| CN106095864A (en) * | 2016-06-03 | 2016-11-09 | 中国工商银行股份有限公司 | A kind of log processing system and method |
| CN106709069A (en) * | 2017-01-25 | 2017-05-24 | 焦点科技股份有限公司 | High-reliability big data logging collection and transmission method |
| CN107391746A (en) * | 2017-08-10 | 2017-11-24 | 深圳前海微众银行股份有限公司 | Log analysis method, equipment and computer-readable recording medium |
-
2020
- 2020-01-13 CN CN202010033832.9A patent/CN111240936A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012198796A (en) * | 2011-03-22 | 2012-10-18 | Nec Corp | Log collection system, device, method and program |
| CN106095864A (en) * | 2016-06-03 | 2016-11-09 | 中国工商银行股份有限公司 | A kind of log processing system and method |
| CN106709069A (en) * | 2017-01-25 | 2017-05-24 | 焦点科技股份有限公司 | High-reliability big data logging collection and transmission method |
| CN107391746A (en) * | 2017-08-10 | 2017-11-24 | 深圳前海微众银行股份有限公司 | Log analysis method, equipment and computer-readable recording medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112463516A (en) * | 2020-10-23 | 2021-03-09 | 苏州浪潮智能科技有限公司 | Method and system for collecting and verifying integrity of BMC log |
| CN113760882A (en) * | 2021-11-10 | 2021-12-07 | 北京每日菜场科技有限公司 | Data abnormity alarm method and device, electronic equipment and computer readable medium |
| CN113760882B (en) * | 2021-11-10 | 2022-01-28 | 北京每日菜场科技有限公司 | Data abnormity alarm method and device, electronic equipment and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108874640B (en) | Cluster performance evaluation method and device | |
| CN110502494B (en) | Log processing method and device, computer equipment and storage medium | |
| WO2021129367A1 (en) | Method and apparatus for monitoring distributed storage system | |
| CN101707632A (en) | Method for dynamically monitoring performance of server cluster and alarming real-timely | |
| CN101997709A (en) | Root alarm data analysis method and system | |
| CN110266544B (en) | Device and method for positioning reason of cloud platform micro-service failure | |
| CN106911519A (en) | A kind of data acquisition monitoring method and device | |
| KR20220166760A (en) | Apparatus and method for managing trouble using big data of 5G distributed cloud system | |
| CN111240936A (en) | Data integrity checking method and equipment | |
| CN114154035A (en) | Data processing system for dynamic loop monitoring | |
| CN114356499A (en) | Kubernetes cluster alarm root cause analysis method and device | |
| CN109032904A (en) | Monitored, management server and data acquisition, analysis method and management system | |
| CN114490091B (en) | Method and device for monitoring rule engine performance in industrial data acquisition management system | |
| CN114816914A (en) | Data processing method, equipment and medium based on Kubernetes | |
| CN109766238B (en) | Session number-based operation and maintenance platform performance monitoring method and device and related equipment | |
| CN113282464A (en) | Log monitoring method and system | |
| CN113760856A (en) | Database management method and device, computer readable storage medium and electronic device | |
| CN112131077A (en) | Fault node positioning method and device and database cluster system | |
| CN117743473B (en) | Data management synchronization performance monitoring system | |
| CN114090382B (en) | Health inspection method and device for super-converged cluster | |
| Carlyle et al. | Practical support solutions for a workflow-oriented Cray environment | |
| CN116260703A (en) | Distributed message service node CPU performance fault self-recovery method and device | |
| CN114647364A (en) | End-to-end management method and system for storage capacity | |
| CN114629786A (en) | Log real-time analysis method, device, storage medium and system | |
| CN117194158A (en) | Application running state analysis method and system based on distributed structure |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |