CN111522786A - Log processing system and method - Google Patents
Log processing system and method Download PDFInfo
- Publication number
- CN111522786A CN111522786A CN202010315303.8A CN202010315303A CN111522786A CN 111522786 A CN111522786 A CN 111522786A CN 202010315303 A CN202010315303 A CN 202010315303A CN 111522786 A CN111522786 A CN 111522786A
- Authority
- CN
- China
- Prior art keywords
- log
- service
- service logs
- logs
- system types
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a log processing system and a method, wherein the log processing system comprises: the system comprises a log forwarding layer, a distributed message queue, a calculation engine and a storage module; the log forwarding layer is used for receiving the service logs collected from each application server and sending the service logs to the distributed message queue; the computing engine is used for pulling the service logs from the distributed message queue and performing log segmentation on the pulled service logs according to the system types corresponding to the service logs; and the storage module is used for receiving the service logs which are sent by the computing engine and correspond to the system types and are obtained through log segmentation, and classifying and storing the service logs according to the system types. The invention realizes the technical effect of reliably and stably classifying and storing massive service logs and is beneficial to performing log analysis on each system subsequently.
Description
Technical Field
The invention relates to the technical field of log analysis, in particular to a log processing system and method.
Background
The log is a key for recording problem information in various systems, is common mass data, and can analyze the running state of the system through the log of the system at present. At present, organizations such as banks and the like have various systems, and the systems are transported day by day based on distributed application server clusters to generate massive logs. At present, due to the limitation of computing resources, log analysis can be performed only on one application server, and in order to accurately analyze the operating state of each system, log analysis needs to be performed on each application server, and the result is finally summarized, which is time-consuming and labor-consuming. Therefore, the prior art lacks a method for aggregating logs of systems in a distributed application server cluster.
Disclosure of Invention
In order to solve at least one technical problem in the background art, the present invention provides a log processing method and device.
In order to achieve the above object, according to one aspect of the present invention, there is provided a log processing system including: the system comprises a log forwarding layer, a distributed message queue, a calculation engine and a storage module;
the log forwarding layer is used for receiving the service logs collected from each application server and sending the service logs to the distributed message queue;
the computing engine is used for pulling the service logs from the distributed message queue and performing log segmentation on the pulled service logs according to the system types corresponding to the service logs;
and the storage module is used for receiving the service logs which are sent by the computing engine and correspond to the system types and are obtained through log segmentation, and classifying and storing the service logs according to the system types.
Optionally, the storage module includes: hbase database and Elasticsearch cluster;
the Hbase database is used for storing service logs corresponding to the system types in a classified manner; the Elasticissearch cluster is used for storing log indexes corresponding to the system types.
Optionally, the log processing system further includes:
and the monitoring and counting module is used for extracting data from the service logs corresponding to the system types according to the preset monitoring and counting tasks corresponding to the system types to obtain the monitoring and counting results of the monitoring and counting tasks.
Optionally, the computing engine is further configured to monitor a service log corresponding to each system type according to a preset log alarm condition corresponding to each system type, and generate alarm information when the service log meets the corresponding log alarm condition.
Optionally, the log processing system further includes:
and the webpage service layer is used for receiving the collected business logs in the front-end browser and sending the business logs to the distributed message queue.
In order to achieve the above object, according to another aspect of the present invention, there is provided a log processing method including:
the log forwarding layer sends the received service logs collected from the application servers to a distributed message queue;
the calculation engine pulls the service logs from the distributed message queue and performs log segmentation on the pulled service logs according to the system type corresponding to each service log;
and the storage module receives the service logs which are sent by the computing engine and correspond to the system types and are obtained through log segmentation, and classifies and stores the service logs according to the system types.
Optionally, the storage module performs classified storage on the service log according to the system type, and specifically includes:
and storing service logs corresponding to the system types by classification through an Hbase database, and storing log indexes corresponding to the system types through an Elasticissearch cluster.
Optionally, the log processing method further includes:
and the monitoring and counting module extracts data from the service logs corresponding to the system types according to the preset monitoring and counting tasks corresponding to the system types to obtain the monitoring and counting results of the monitoring and counting tasks.
In order to achieve the above object, according to another aspect of the present invention, there is also provided a computer device including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the log processing method when executing the computer program.
In order to achieve the above object, according to another aspect of the present invention, there is also provided a computer-readable storage medium storing a computer program which, when executed in a computer processor, implements the steps in the log processing method described above.
The invention has the beneficial effects that: the embodiment of the invention collects the service logs of each system from each application server, and forwards the service logs to the distributed message queue through the log forwarding layer, the distributed message queue has the characteristics of mass data accumulation and high throughput reading and writing, can buffer the mass service logs, realizes the functions of peak clipping and valley filling, further calculates the service logs in the consumption distributed message queue of the engine in batches, and segments the drawn service logs according to the system type corresponding to each service log, and finally stores the segmented service logs corresponding to each system type in a classified manner, thereby realizing the technical effect of reliably and stably classifying and storing the mass service logs, and being beneficial to the subsequent log analysis of each system.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts. In the drawings:
FIG. 1 is a first block diagram of a log processing system according to an embodiment of the invention;
FIG. 2 is a second block diagram of a log processing system according to an embodiment of the present invention;
FIG. 3 is a flow chart of a log processing method according to an embodiment of the invention;
FIG. 4 is a schematic diagram of a computer apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
It should be noted that the terms "comprises" and "comprising," and any variations thereof, in the description and claims of the present invention and the above-described drawings, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
Fig. 1 is a first structural block diagram of a log processing system according to an embodiment of the present invention, and as shown in fig. 1, the log processing system according to the embodiment includes: the system comprises a log forwarding layer, a distributed message queue, a computing engine and a storage module.
And the log forwarding layer is used for receiving the service logs collected from the application servers and sending the service logs to the distributed message queue. In an optional embodiment of the present invention, the log collection plug-in is set in each application server, and the log collection plug-in can send the service log generated in the application server to the log forwarding layer at regular time or in real time. In an alternative embodiment of the present invention, a plurality of systems (or service systems), such as a Redis cache system, a Kafka, etc., are deployed in the distributed application server cluster of the present invention, each system may be deployed in one or more application servers, and service logs generated by each system will be stored in the deployed application servers.
In the optional embodiment of the invention, the distributed message queue has the characteristics of mass data accumulation and high-throughput reading and writing, can buffer mass service logs, and realizes the functions of peak clipping and valley filling. In alternative embodiments of the present invention, the distributed message queue may use various message queues in the prior art, and preferably, may use Kafka cluster. The Kafka cluster has the advantages of high performance, high availability, easiness in expansion and the like, and can decouple the whole data processing flow. The Kafka cluster is used as a buffer layer for log sending, asynchronous decoupling, peak clipping and valley filling capabilities can be provided for subsequent distributed log consumption services (computing engines), and the Kafka cluster also has the characteristics of mass data accumulation and high throughput reading and writing.
In an optional embodiment of the present invention, the computing engine may adopt various computing engines in the prior art, preferably, the present invention adopts a Spark Streaming computing engine, the computing engine of the present invention may be a deployed Spark cluster, the Spark cluster includes a plurality of computing nodes, in addition, the present invention may adopt Yarn to perform computing resource management, and in an optional embodiment of the present invention, Yarn may allocate computing resources (computing nodes) for each system type according to log levels of different system types.
In an optional embodiment of the present invention, the computing engine consumes the service logs written in the distributed message queue, and the specific computing engine may pull the service logs from the distributed message queue in batches. In an optional embodiment of the present invention, each service log is identified with a corresponding system type, and the calculation engine performs segmentation on a batch of pulled service logs according to the system type corresponding to each service log to obtain the service logs of each system type in the batch of service logs. And then the computing engine sends the service logs of all system types to the storage module for classified storage.
In an optional embodiment of the present invention, the storage module is configured to store the service logs of each system in a classified manner. In an optional embodiment of the present invention, the storage module may adopt an Elasticsearch cluster, a corresponding index is created in the Elasticsearch cluster for each system type, and the service log of each system is stored in the corresponding index. In an alternative embodiment of the present invention, each index corresponds to a plurality of storage slices (shards), and the service logs of each system are stored in the storage slices corresponding to the indexes.
The invention considers that the number of logs stored in the Elasticissearch cluster will increase greatly with time, which will result in slower query or index of the logs. In an alternative embodiment of the present invention, the present invention may employ a combination of the Elasticsearch cluster and the Hbase database for log storage and indexing, as shown in fig. 2. The Elasticsearch cluster is only used for creating and storing the log indexes corresponding to the system types, the specific service log contents corresponding to the system types are stored in the Hbase database in a classified mode, and the corresponding relation between the log indexes corresponding to the system types and the service logs corresponding to the system types stored in the Hbase database is built. The system of the invention adopts a mode of combining Hbase stored original log data and ElasticSearch index content to finish storage and index, and improves the efficiency of log query or index.
Fig. 2 is a second structural block diagram of the log processing system according to the embodiment of the present invention, and as shown in fig. 2, in an alternative embodiment of the present invention, the log processing system further includes: and a monitoring and counting module.
The monitoring and counting module is used for extracting data from the service logs corresponding to the system types according to the preset monitoring and counting tasks corresponding to the system types to obtain the monitoring and counting results of the monitoring and counting tasks. The invention sets monitoring and counting tasks for each system, for example, sets monitoring and counting tasks such as monitoring command process quantity, memory usage quantity, network input/output quantity and the like for a Redis cache system. In the embodiment of the invention, the monitoring statistics acquires data according to the service log corresponding to the monitoring statistics task access task, and further obtains the monitoring statistics result of the monitoring statistics task. In an optional embodiment of the present invention, the monitoring and statistics module may access the service logs of the systems in the storage module to obtain the required data. In an optional embodiment of the present invention, the monitoring statistics module may employ a flink cluster.
In an optional embodiment of the present invention, the log processing system of the present invention further includes a web management console, and the web management console is configured to visually display the monitoring statistics result of each monitoring statistics task.
In an optional embodiment of the present invention, the computing engine is further configured to monitor a service log corresponding to each system type according to a preset log alarm condition corresponding to each system type, and generate alarm information when the service log meets the corresponding log alarm condition.
In an optional embodiment of the present invention, the computing engine is further configured to analyze an operation state of each system in real time according to a service log corresponding to each system type, so as to analyze each system problem in real time. In an optional embodiment of the present invention, before consuming log data in the distributed message queue, the computing engine may first establish a log model for the service logs of each system type and set a filtering and matching alarm rule, each computing node of the computing engine stores one such rule, and after the count reaches a threshold value configured by the alarm rule within a time set by the rule, an alarm is sent to a designated user through a designated channel, so as to find out a problem occurring in the system in time.
As shown in fig. 2, in an alternative embodiment of the present invention, the journal processing system of the present invention further includes: web service layer (web protal). The webpage service layer is used for receiving the collected business logs in the front-end browser and sending the business logs to the distributed message queue. In an optional embodiment of the present invention, the present invention adopts logs generated by each system in each front-end browser by setting a log collection plug-in the front-end browser, and sends the collected logs to a web service layer (web portal) through Http request.
Based on the same inventive concept, the embodiment of the invention also provides a log processing method. Because the principle of the log processing method for solving the problem is similar to that of the log processing system, the embodiment of the log processing method can be referred to the embodiment of the log processing system, and repeated details are not repeated.
Fig. 3 is a flowchart of a log processing method according to an embodiment of the present invention, and as shown in fig. 3, the log processing method according to an embodiment of the present invention includes steps S101 to S103.
Step S101, the log forwarding layer sends the received service logs collected from each application server to a distributed message queue.
Step S102, a computing engine pulls service logs from the distributed message queue, and performs log segmentation on the pulled service logs according to the system type corresponding to each service log.
Step S103, a storage module receives the service logs corresponding to the system types obtained by log segmentation and sent by the computing engine, and classifies and stores the service logs according to the system types.
In an optional embodiment of the present invention, the step S103 of classifying and storing the service logs according to system types by the storage module specifically includes: and storing service logs corresponding to the system types by classification through an Hbase database, and storing log indexes corresponding to the system types through an Elasticissearch cluster.
In an optional embodiment of the present invention, the log processing method of the present invention further includes:
and the monitoring and counting module extracts data from the service logs corresponding to the system types according to the preset monitoring and counting tasks corresponding to the system types to obtain the monitoring and counting results of the monitoring and counting tasks.
In an optional embodiment of the present invention, the log processing method of the present invention further includes:
and the calculation engine monitors the service logs corresponding to the system types according to preset log alarm conditions corresponding to the system types and generates alarm information when the service logs meet the corresponding log alarm conditions.
In an optional embodiment of the present invention, the log processing method of the present invention further includes:
and the webpage service layer receives the collected business logs in the front-end browser and sends the business logs to the distributed message queue.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
To achieve the above object, according to another aspect of the present application, there is also provided a computer apparatus. As shown in fig. 4, the computer device comprises a memory, a processor, a communication interface and a communication bus, wherein a computer program that can be run on the processor is stored in the memory, and the steps of the method of the above embodiment are realized when the processor executes the computer program.
The processor may be a Central Processing Unit (CPU). The Processor may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, or a combination thereof.
The memory, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and units, such as the corresponding program units in the above-described method embodiments of the present invention. The processor executes various functional applications of the processor and the processing of the work data by executing the non-transitory software programs, instructions and modules stored in the memory, that is, the method in the above method embodiment is realized.
The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor, and the like. Further, the memory may include high speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and such remote memory may be coupled to the processor via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more units are stored in the memory and when executed by the processor perform the method of the above embodiments.
The specific details of the computer device may be understood by referring to the corresponding related descriptions and effects in the above embodiments, and are not described herein again.
In order to achieve the above object, according to another aspect of the present application, there is also provided a computer-readable storage medium storing a computer program which, when executed in a computer processor, implements the steps in the log processing method described above. It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD) or a Solid State Drive (SSD), etc.; the storage medium may also comprise a combination of memories of the kind described above.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may alternatively be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, or fabricated separately as individual integrated circuit modules, or fabricated as a single integrated circuit module from multiple modules or steps. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A log processing system, comprising: the system comprises a log forwarding layer, a distributed message queue, a calculation engine and a storage module;
the log forwarding layer is used for receiving the service logs collected from each application server and sending the service logs to the distributed message queue;
the computing engine is used for pulling the service logs from the distributed message queue and performing log segmentation on the pulled service logs according to the system types corresponding to the service logs;
and the storage module is used for receiving the service logs which are sent by the computing engine and correspond to the system types and are obtained through log segmentation, and classifying and storing the service logs according to the system types.
2. The log processing system of claim 1, wherein the storage module comprises: hbase database and Elasticsearch cluster;
the Hbase database is used for storing service logs corresponding to the system types in a classified manner; the Elasticissearch cluster is used for storing log indexes corresponding to the system types.
3. The log processing system of claim 1, further comprising:
and the monitoring and counting module is used for extracting data from the service logs corresponding to the system types according to the preset monitoring and counting tasks corresponding to the system types to obtain the monitoring and counting results of the monitoring and counting tasks.
4. The log processing system of claim 1, wherein the computing engine is further configured to monitor a service log corresponding to each system type according to a preset log alarm condition corresponding to each system type, and generate alarm information when the service log meets the corresponding log alarm condition.
5. The log processing system of claim 1, further comprising:
and the webpage service layer is used for receiving the collected business logs in the front-end browser and sending the business logs to the distributed message queue.
6. A log processing method, comprising:
the log forwarding layer sends the received service logs collected from the application servers to a distributed message queue;
the calculation engine pulls the service logs from the distributed message queue and performs log segmentation on the pulled service logs according to the system type corresponding to each service log;
and the storage module receives the service logs which are sent by the computing engine and correspond to the system types and are obtained through log segmentation, and classifies and stores the service logs according to the system types.
7. The log processing method according to claim 6, wherein the storage module performs classified storage on the service logs according to system types, and specifically comprises:
and storing service logs corresponding to the system types by classification through an Hbase database, and storing log indexes corresponding to the system types through an Elasticissearch cluster.
8. The log processing method according to claim 6, further comprising:
and the monitoring and counting module extracts data from the service logs corresponding to the system types according to the preset monitoring and counting tasks corresponding to the system types to obtain the monitoring and counting results of the monitoring and counting tasks.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 6 to 8 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when executed in a computer processor, implements the method of any one of claims 6 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010315303.8A CN111522786A (en) | 2020-04-21 | 2020-04-21 | Log processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010315303.8A CN111522786A (en) | 2020-04-21 | 2020-04-21 | Log processing system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111522786A true CN111522786A (en) | 2020-08-11 |
Family
ID=71903948
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010315303.8A Pending CN111522786A (en) | 2020-04-21 | 2020-04-21 | Log processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111522786A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112035425A (en) * | 2020-10-27 | 2020-12-04 | 苏宁金融科技(南京)有限公司 | Log storage method and device and computer system |
| CN112650599A (en) * | 2020-12-21 | 2021-04-13 | 深圳前海微众银行股份有限公司 | Log processing method, device and storage medium |
| CN113722183A (en) * | 2021-09-02 | 2021-11-30 | 北京金山云网络技术有限公司 | Log alarm method and device and electronic equipment |
| CN114244901A (en) * | 2021-11-22 | 2022-03-25 | 中国建设银行股份有限公司 | Data processing method and device and electronic equipment |
| CN114978885A (en) * | 2022-08-02 | 2022-08-30 | 深圳市华曦达科技股份有限公司 | Log management method and device, computer equipment and system |
| CN115269516A (en) * | 2022-06-30 | 2022-11-01 | 北京数美时代科技有限公司 | Log data acquisition management method, system, storage medium and electronic equipment |
| CN115733718A (en) * | 2021-08-26 | 2023-03-03 | 北京神州泰岳软件股份有限公司 | Weblog analysis method and device, electronic equipment and storage medium |
| CN116541364A (en) * | 2023-07-06 | 2023-08-04 | 中电科新型智慧城市研究院有限公司 | Log data storage method and device, terminal equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101959153A (en) * | 2009-07-20 | 2011-01-26 | 中兴通讯股份有限公司 | Method and system for monitoring receipt return condition of data traffic system |
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| US20130125139A1 (en) * | 2011-11-16 | 2013-05-16 | International Business Machines Corporation | Logging In A Computer System |
| CN107622084A (en) * | 2017-08-10 | 2018-01-23 | 深圳前海微众银行股份有限公司 | Blog management method, system and computer-readable recording medium |
| CN109726074A (en) * | 2018-08-31 | 2019-05-07 | 网联清算有限公司 | Log processing method, device, computer equipment and storage medium |
| CN110413586A (en) * | 2019-08-05 | 2019-11-05 | 山东浪潮通软信息科技有限公司 | Distributed information log management method and system |
-
2020
- 2020-04-21 CN CN202010315303.8A patent/CN111522786A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101959153A (en) * | 2009-07-20 | 2011-01-26 | 中兴通讯股份有限公司 | Method and system for monitoring receipt return condition of data traffic system |
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| US20130125139A1 (en) * | 2011-11-16 | 2013-05-16 | International Business Machines Corporation | Logging In A Computer System |
| CN107622084A (en) * | 2017-08-10 | 2018-01-23 | 深圳前海微众银行股份有限公司 | Blog management method, system and computer-readable recording medium |
| CN109726074A (en) * | 2018-08-31 | 2019-05-07 | 网联清算有限公司 | Log processing method, device, computer equipment and storage medium |
| CN110413586A (en) * | 2019-08-05 | 2019-11-05 | 山东浪潮通软信息科技有限公司 | Distributed information log management method and system |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112035425A (en) * | 2020-10-27 | 2020-12-04 | 苏宁金融科技(南京)有限公司 | Log storage method and device and computer system |
| CN112035425B (en) * | 2020-10-27 | 2021-11-09 | 南京星云数字技术有限公司 | Log storage method and device and computer system |
| CN112650599A (en) * | 2020-12-21 | 2021-04-13 | 深圳前海微众银行股份有限公司 | Log processing method, device and storage medium |
| CN115733718A (en) * | 2021-08-26 | 2023-03-03 | 北京神州泰岳软件股份有限公司 | Weblog analysis method and device, electronic equipment and storage medium |
| CN113722183A (en) * | 2021-09-02 | 2021-11-30 | 北京金山云网络技术有限公司 | Log alarm method and device and electronic equipment |
| CN114244901A (en) * | 2021-11-22 | 2022-03-25 | 中国建设银行股份有限公司 | Data processing method and device and electronic equipment |
| CN115269516A (en) * | 2022-06-30 | 2022-11-01 | 北京数美时代科技有限公司 | Log data acquisition management method, system, storage medium and electronic equipment |
| CN114978885A (en) * | 2022-08-02 | 2022-08-30 | 深圳市华曦达科技股份有限公司 | Log management method and device, computer equipment and system |
| CN116541364A (en) * | 2023-07-06 | 2023-08-04 | 中电科新型智慧城市研究院有限公司 | Log data storage method and device, terminal equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111522786A (en) | Log processing system and method | |
| CN107577805B (en) | Business service system for log big data analysis | |
| US10430111B2 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| US12008027B2 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| WO2017172947A1 (en) | Managed function execution for processing data streams in real time | |
| CN111459986B (en) | Data computing system and method | |
| CN108574645B (en) | Queue scheduling method and device | |
| CN112988679B (en) | Log acquisition control method and device, storage medium and server | |
| CN107346270B (en) | Method and system for real-time computation based radix estimation | |
| CN112231296B (en) | Distributed log processing method, device, system, equipment and medium | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| CN116661705B (en) | Data management method, system, electronic equipment and storage medium based on kafka | |
| Elagib et al. | Big data analysis solutions using MapReduce framework | |
| CN112019605A (en) | Data distribution method and system of data stream | |
| CN114461792A (en) | Alarm event correlation method, device, electronic equipment, medium and program product | |
| US9043535B1 (en) | Minimizing application response time | |
| CN114153609A (en) | Resource control method and device, electronic equipment and computer readable storage medium | |
| CN117633116A (en) | Data synchronization method, device, electronic equipment and storage medium | |
| CN110022343B (en) | Adaptive event aggregation | |
| CN117667327A (en) | Job scheduling method, scheduler and related equipment | |
| CN111858918A (en) | News classification method and device, network element and storage medium | |
| Yan et al. | Cloud city traffic state assessment system using a novel architecture of big data | |
| US20240265015A1 (en) | Systems and methods for reducing the cardinality of metrics queries | |
| CN112988805A (en) | Data processing method, device and equipment based on computing framework and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220914 Address after: 25 Financial Street, Xicheng District, Beijing 100033 Applicant after: CHINA CONSTRUCTION BANK Corp. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right |