CN110515912A - Log processing method, device, computer installation and computer readable storage medium - Google Patents
Log processing method, device, computer installation and computer readable storage medium Download PDFInfo
- Publication number
- CN110515912A CN110515912A CN201910648085.7A CN201910648085A CN110515912A CN 110515912 A CN110515912 A CN 110515912A CN 201910648085 A CN201910648085 A CN 201910648085A CN 110515912 A CN110515912 A CN 110515912A
- Authority
- CN
- China
- Prior art keywords
- log
- data
- database
- log data
- alarm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 31
- 238000009434 installation Methods 0.000 title abstract description 4
- 238000012545 processing Methods 0.000 claims abstract description 54
- 238000000034 method Methods 0.000 claims abstract description 29
- 238000013500 data storage Methods 0.000 claims abstract description 10
- 238000004458 analytical method Methods 0.000 claims description 33
- 238000004590 computer program Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 15
- 238000003780 insertion Methods 0.000 claims description 6
- 230000037431 insertion Effects 0.000 claims description 6
- 230000000694 effects Effects 0.000 abstract description 8
- 230000004048 modification Effects 0.000 description 9
- 238000012986 modification Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 7
- 230000008859 change Effects 0.000 description 6
- 238000001914 filtration Methods 0.000 description 6
- 238000007619 statistical method Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 238000007405 data analysis Methods 0.000 description 2
- 230000002085 persistent effect Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention is suitable for Internet technical field, provides a kind of log processing method, device, computer installation and computer readable storage medium, and method includes: to obtain log information by Kafka system, and log information includes the daily record data of multiple projects；By daily record data storage into topic topic identical with affiliated project name；By the data processing module by JStorm Development of Framework, the daily record data obtained from topic topic, the daily record data after being parsed are parsed according to self-defining data format；By the data memory module by JStorm Development of Framework, the daily record data after parsing is stored in ElasticSearch database in corresponding tables of data according to type of database specified by default resolution rules, database-name and purpose tables of data.Log processing method provided by the invention can parse the daily record data obtained from Kafka system according to self-defining data format, and in corresponding table, will improve daily record data treatment effect in the daily record data deposit ElasticSearch database after parsing.
Description
Technical Field
The invention belongs to the technical field of internet, and particularly relates to a log processing method and device, a computer device and a computer readable storage medium.
Background
Generally, in the case of processing log data, the log data is directly written in a file, and then the file is searched for the data needed to be used. However, in a big data and distributed environment, the log data has the characteristics of many types, large quantity and wide distribution. Generally, large-scale systems are distributed deployment architectures, that is, different data processing modules are deployed on different servers. And logs corresponding to the data processing modules are also stored on the servers. In the prior art, the problems of slow log collection, more log data, complexity, slow statistics and difficult analysis exist in the process of collecting, summarizing and uniformly analyzing log data, so that the log data processing effect is poor.
Disclosure of Invention
The embodiment of the invention provides a log processing method, a log processing device, a computer device and a computer readable storage medium, and aims to solve the problem that in the prior art, log data are collected slowly, the log data are more and complicated, statistics is slow, and analysis is difficult in the process of collecting, summarizing and uniformly analyzing the log data, so that the log data processing effect is poor.
The invention is realized in such a way that a log processing method comprises the following processes:
acquiring log information through a Kafka system, wherein the log information comprises log data of a plurality of items; storing the log data into topic with the same name as the belonged project;
analyzing the log data acquired from the topic according to a user-defined data format through a data processing module developed by a JStorm framework to obtain analyzed log data;
and storing the analyzed log data into a corresponding data table in an ElasticSearch database according to the database type, the database name and a target data table specified by a preset analysis rule through a data storage module developed by a JStorm framework.
Further, the storing the analyzed log data into a corresponding data table in an ElasticSearch database according to the database type, the database name and the destination data table specified by the preset analysis rule includes the following steps:
and when the alarm mark in the preset analysis rule is true, acquiring alarm information from the analyzed log data, and sending the alarm information to an alarm contact person through an alarm module developed by the JSTOM framework.
Further, the method further comprises the following process:
and acquiring the analyzed log data from a specified data table of the ElasticSearch database through the Kibana system, and displaying the acquired log data.
Further, before the parsed log data is stored in a corresponding data table in the ElasticSearch database according to a database type, a database name and a destination data table specified by a preset parsing rule through a data storage module developed by a jstom framework, the method further includes the following processes:
and configuring the preset analysis rule through a Redis system, wherein the preset analysis rule comprises a processing algorithm, a database type, a database, an insertion algorithm, a target data table and an alarm mark.
The present invention also provides a log processing apparatus, including:
the system comprises a first logging module, a second logging module and a third logging module, wherein the first logging module is used for acquiring log information through a Kafka system, and the log information comprises log data of a plurality of items; storing the log data into topic with the same name as the belonged project;
the analysis module is used for analyzing the log data acquired from the topic according to a user-defined data format through a data processing module developed by a JStorm framework to obtain the analyzed log data;
and the second storage module is used for storing the analyzed log data into a corresponding data table in the ElasticSearch database according to the database type, the database name and the target data table specified by the preset analysis rule through the data storage module developed by the JStorm framework.
Further, the second storage module is further configured to, when the alarm flag in the preset parsing rule is true, obtain alarm information from the parsed log data, and send the alarm information to an alarm contact through an alarm module developed by the jstom framework.
Further, the log processing apparatus further includes:
and the display module is used for acquiring the analyzed log data from the specified data table of the ElasticSearch database through the Kibana system and displaying the acquired log data.
Further, the log processing apparatus further includes:
and the configuration module is used for configuring the preset analysis rule through a Redis system, wherein the preset analysis rule comprises a processing algorithm, a database type, a database, an insertion algorithm, a target data table and an alarm mark.
The invention also provides a computer device comprising a processor for implementing the steps of the log processing method when executing the computer program in the memory.
The present invention also provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the log processing method as described above.
The log processing method provided by the invention can analyze the log data acquired from the Kafka system according to the custom data format to obtain the analyzed log data, and store the analyzed log data into different tables in the ElasticSearch database, thereby improving the efficiency of analyzing the log data, improving the orderliness of storing the log data and further improving the log data processing effect.
Drawings
FIG. 1 is a flowchart of an implementation of a log processing method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a log processing apparatus according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of another log processing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of another log processing apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Fig. 1 is a flowchart illustrating a log processing method according to an embodiment of the present invention. The log processing method comprises the following processes:
step S101, obtaining log information through a Kafka system, wherein the log information comprises log data of a plurality of items; and storing the log data into the topic with the same name as the belonged project.
In this embodiment, the log data of each module or each item may be obtained by running each module or each item of the application through different servers. The server can send the obtained log data to the Kafka system, and the Kafka system stores the log data into topic of the corresponding project after receiving the log data. In the present embodiment, topic represents a topic and the Kafka system is a distributed, partitionable, replicable messaging system. The log data are stored in topic with the same name as the belonged project, and the log data can be stored in a classified mode according to the topic of the project, so that the log data can be conveniently inquired subsequently.
And step S102, analyzing the log data acquired from the topic topci of the Kafka system according to a user-defined data format through a data processing module developed by a JStorm framework to obtain the analyzed log data.
In this embodiment, the jstom framework overwrites the native Storm model, written by clojere and Java mix, with Java. In addition, the jstom framework makes many improvements over the native Storm model. A user only needs to realize a task according to a specified interface, then the task is submitted to a JStorm system, the JStorm system can continuously run the task after receiving a task instruction, and once a certain object (Worker) fails to send due to an exception, a new Worker can be allocated to replace the exception Worker by a scheduler.
It should be noted that the content of the custom data format includes log entry, log time, log level, request ID, custom display information, input, output, log elapsed time, calling method file name, line number, calling method name, and the like.
Step S103, storing the analyzed log data into a corresponding data table in an ElasticSearch database according to a database type, a database name and a target data table specified by a preset analysis rule through a data storage module developed by the JSTORM framework.
In the embodiment, the elastic search is an open source distributed search engine, and provides three functions of collecting, analyzing and storing data.
In this embodiment, when an item or a module is added, a corresponding preset parsing rule needs to be added, where the parsing rule includes an ElasticSearch connection manner, a database type, a database name, and a destination data table, and after the parsed data is obtained, the parsed data may be inserted into the destination data table specified by the preset parsing rule.
The analyzed log data are analyzed according to a custom data format, the log data are obtained through classification, the log data in different categories are stored into corresponding data tables of an ElasticSearch database according to the database type, the database name and a target data table specified by a preset analysis rule, and the log data can be conveniently read and analyzed subsequently.
The log processing method provided by the embodiment of the invention can analyze the log data acquired from the Kafka system according to the custom data format to obtain the analyzed log data, and stores the analyzed log data into different tables in the ElasticSearch database according to the database type, the database name and the target data table specified by the preset analysis rule, so that the efficiency of analyzing the log data is improved, the orderliness of storing the log data is improved, and the log data processing effect is improved.
Optionally, the step S103 includes the following processes:
and when the alarm mark in the preset analysis rule is true, acquiring alarm information from the analyzed log data, and sending the alarm information to an alarm contact person through an alarm module developed by the JSTOM framework.
In this embodiment, when an item or a module is added, a corresponding preset parsing rule needs to be added, where the parsing rule includes an alarm flag, and if the item or the module needs to alarm, the alarm flag is set to true, otherwise, the alarm flag is set to false. After the analysis rule is obtained, the processing method is obtained through field reflection of the processing method of the analysis rule, log information is analyzed into a custom data format in the processing method, then whether an alarm mark is true or not is judged, if the alarm mark is true, alarm information is obtained from the analyzed log data, and the alarm information comprises alarm information such as alarm person information, alarm level, alarm mode and the like.
Therefore, when the log data needs to be alarmed, the alarm information can be sent to the appointed alarm person, so that the alarm person can know the state of the log data in time and maintain the data quickly.
Optionally, the method further comprises the following processes:
and acquiring the analyzed log data from a specified data table of the ElasticSearch database through the Kibana system, and displaying the acquired log data.
In this embodiment, the obtaining, by the Kibana system, the parsed log data stored in the specified data table of the ElasticSearch database includes the following steps:
determining a target project through the Kibana system, taking a self-defined data structure field as a statistical analysis dimension, and acquiring analyzed log data from topic of the target project according to a specified dimension in the statistical analysis dimension; or,
determining log information with the same request ID value, tracking the change of the log information with the same request ID value, and displaying the change condition of the log information.
In this embodiment, the Kibana system is an open-source and free tool, and the Kibana can generate a Web interface of a website with a user-friendly experience according to log data provided by the elastic search, so that a user can conveniently view log data of different projects or different modules.
In the actual operation process of a user, on one hand, the user can enter a kibana system to select items or modules, a user-defined data structure field is used as a statistical analysis dimension, the dimension is specified to perform query, for example, all logs of a certain specified java class are displayed, all logs with the log level of error are displayed, the logs are displayed in the reverse order of log time sequencing, and the like. On the other hand, the user only needs to find the specified target item on the display interface of Kibana, the filtering condition can be added under the specified target item, the filtering field is selected as the request ID, the filtering value is the ID, all log data of the specified ID can be displayed, and the change condition of the log information is tracked and displayed.
Therefore, the log data which the user wants to look up can be screened out and displayed, the accuracy of displaying the log data is improved, and the user can quickly and accurately look up the related log data conveniently.
Optionally, before the parsed log data is stored in a corresponding data table in the ElasticSearch database according to a database type, a database name, and a destination data table specified by a preset parsing rule through the data storage module developed by the jstom framework, the method further includes the following steps:
and configuring the preset analysis rule through a Redis system, wherein the preset analysis rule comprises a processing algorithm, a database type, a database, an insertion algorithm, a target data table and an alarm mark.
In this embodiment, the Redis database is an open-source log-type and Key-Value database written in ANSI C language, supporting network, and capable of being based on memory and persistent, and can provide Application Programming Interface (API) of multiple languages. It can be understood that the configuration information is directly stored in the memory, which is convenient and fast, and is convenient for modification without storing the configuration file locally.
It is further supplementary to explain that the configuration information may further include alarm information, system control information, processing rule information, and storage information, where the alarm information includes an alarm level, an alarm manner, and an alarm person contact manner, and the system control information includes a consumption log timeout time, an application switch for updating a log collection topic of a project, and a name set of a project log source topic; the storage information comprises an ElasticSearch connection ip address, an ElasticSearch connection port number, an ElasticSearch authentication user name, an ElasticSearch authentication password, an ElasticSearch database name and an ElasticSearch data table name.
In the present embodiment, the purpose of controlling the present log processing apparatus can be achieved by modifying the configuration information. For example, a new project or module needs to be added for log monitoring, a topic with the project name as the name is created in the kafka cluster, and then the new topic name is added in the name set of each project log collection topic of the system control information. Then, a processing rule with the name of topic as key and the processing rule information as value is added to the processing rule information. And finally, setting the value of an application switch of the log collection topic of the system control information updating item to true, so far that the addition of a new item or module is successful. In addition, existing items may be modified or deleted by modifying or deleting configuration information.
It is further added that, before step S101, the log processing method may further include the following process:
modifying the new application configuration of the Redis system, and changing the value of an application switch of the log collection topic into true;
adding topic of a new project collection log to a name set of project log source topic topics of the Kafka system.
In this embodiment, the Redis system is a key-value storage system, and supports more stored value types, including a string (string), a linked list (list), a set (set), a sorted set-ordered set (zset), and a hash type (hash). Additionally, Redis supports a variety of different manners of ordering. In order to ensure efficiency, data is cached in a memory, and the Redis system periodically writes updated data into a disk or writes modification operations into an additional recording file, and can realize master-slave (master-slave) synchronization on the basis. Master-slave synchronization, where data may be synchronized from a master server to any number of slave servers, a slave server may be the master server associated with other slave servers.
In this embodiment, after the new application configuration in Redis is modified, the application switch of the log collection topic is modified. In addition, after the application performs a new configuration, the value of the application switch is automatically changed to false to cope with the next configuration modification. For example, an application is newly added, and after a preset parsing rule and a corresponding topic name are added, the new application cannot acquire the log without changing the value of the switch. In this embodiment, when the value of the application switch of the log collection topic in the Redis system is true, it indicates that the addition of log monitoring on a new item is allowed.
In this embodiment, the Kafka system is sourced by Apache, is a message middleware, and is a distributed, partitionable, and replicable message system. The topic can be understood as different classifications of message sources (feessof messages) of Kafka processing resources.
The Kafka system generalizes messages in units of topic. The program that issues messages to the topic of the Kafka system is called producers (producers). The program that subscribes to topics and consumes messages is called a consumer (consumer). The Kafka system operates in a cluster and may be composed of one or more services, each called a cache broker (broker). The producers send messages over the network to the Kafka cluster, which provides the messages to the consumers. The producers generate messages in real time and send a class of messages with the same nature to a topic of the kafka system. A consumer subscribes to one or more topics, and gets one or more types of messages.
It is further added that the acquiring of the log data by the Kafka system in step S101 of the embodiment shown in fig. 1 includes the following processes:
acquiring topic of project distribution through the Kafka system, and consuming the acquired topic to acquire latest log data; or,
and configuring the link address of the kafka system and the topic of each item by configuring a configuration file of a log data collector fileteam, and submitting log information acquired from a system log file to the topic corresponding to each item of the kafka system by the fileteam.
It should be noted that the filebed is a log data collector of the local file. As an agent installation on the server, Filebeat monitors the log directory, specific log file, or file tail file, and can forward the log directory, specific log file, or file tail to the Elasticsearch for indexing, etc.
Therefore, the Kafka system can quickly collect a large amount of log data, and is convenient for subsequent log data analysis.
The log processing method provided by the embodiment of the invention can analyze the log data acquired from the Kafka system according to the custom data format to obtain the analyzed log data, and store the analyzed log data into different tables in the ElasticSearch database, so that the efficiency of analyzing the log data is improved, the orderliness of storing the log data is improved, and the log data processing effect is improved.
Fig. 2 is a schematic structural diagram of a log processing apparatus 200 according to an embodiment of the present invention, and for convenience of description, only the relevant parts for implementing the present invention are shown. The log processing apparatus 200 includes:
a first logging module 201, configured to obtain log information through a Kafka system, where the log information includes log data of a plurality of items; and storing the log data into the topic with the same name as the belonged project.
In this embodiment, different servers may run and apply each module or each item to obtain log data of each module or each item. The server can send the obtained log data to the Kafka system, and the Kafka system stores the log data into topic of the corresponding project after receiving the log data. In the present embodiment, topic represents a topic and the Kafka system is a distributed, partitionable, replicable messaging system. The log data are stored in topic with the same name as the belonged project, and the log data can be stored in a classified mode according to the topic of the project, so that the log data can be conveniently inquired subsequently.
And the analysis module 202 is configured to analyze, through a data processing module developed by the jstom framework, the log data obtained from the topic topci of the Kafka system according to a custom data format, so as to obtain analyzed log data.
In this embodiment, the jstom framework overwrites the native Storm model, written by clojere and Java mix, with Java. In addition, the jstom framework makes many improvements over the native Storm model. A user only needs to realize a task according to a specified interface, then the task is submitted to a JStorm system, the JStorm system can continuously run the task after receiving a task instruction, and once a certain object (Worker) fails to send due to an exception, a new Worker can be allocated to replace the exception Worker by a scheduler.
It should be noted that the content of the custom data format includes log entry, log time, log level, request ID, custom display information, input, output, log elapsed time, calling method file name, line number, calling method name, and the like.
And a second storing module 203, configured to store the analyzed log data into a corresponding data table in the ElasticSearch database according to the database type, the database name, and the destination data table specified by the preset analysis rule through a data storage module developed by the jstom framework.
In the embodiment, the elastic search is an open source distributed search engine, and provides three functions of collecting, analyzing and storing data.
In this embodiment, when an item or a module is added, a corresponding preset parsing rule needs to be added, where the parsing rule includes an ElasticSearch connection manner, a database type, a database name, and a destination data table, and after the parsed data is obtained, the parsed data may be inserted into the destination data table specified by the preset parsing rule.
The analyzed log data are analyzed according to a custom data format, the log data are obtained through classification, the log data in different categories are stored into corresponding data tables of an ElasticSearch database according to the database type, the database name and a target data table specified by a preset analysis rule, and the log data can be conveniently read and analyzed subsequently.
The log processing device provided by the embodiment of the invention can analyze the log data acquired from the Kafka system according to the custom data format to obtain the analyzed log data, and stores the analyzed log data into different tables in the ElasticSearch database according to the database type, the database name and the target data table specified by the preset analysis rule, so that the efficiency of analyzing the log data is improved, the orderliness of storing the log data is improved, and the log data processing effect is improved.
Optionally, the second storing module 203 is further configured to, when the alarm flag in the preset parsing rule is true, obtain alarm information from the parsed log data, and send the alarm information to an alarm contact through an alarm module developed by the jstom framework.
In this embodiment, when an item or a module is added, a corresponding preset parsing rule needs to be added, where the parsing rule includes an alarm flag, and if the item or the module needs to alarm, the alarm flag is set to true, otherwise, the alarm flag is set to false. After the analysis rule is obtained, the processing method is obtained through field reflection of the processing method of the analysis rule, log information is analyzed into a custom data format in the processing method, then whether an alarm mark is true or not is judged, if the alarm mark is true, alarm information is obtained from the analyzed log data, and the alarm information comprises alarm information such as alarm person information, alarm level, alarm mode and the like.
Therefore, when the log data needs to be alarmed, the alarm information can be sent to the appointed alarm person, so that the alarm person can know the state of the log data in time and maintain the data quickly.
As shown in fig. 3, the log processing apparatus 200 further includes:
a display module 204, configured to obtain, by using the Kibana system, the analyzed log data from the specified data table of the ElasticSearch database, and display the obtained log data.
Optionally, the display module 204 is further configured to determine a target item through the Kibana system, use a self-defined data structure field as a statistical analysis dimension, and obtain analyzed log data from topic of the target item according to a specified dimension in the statistical analysis dimension; or,
determining log information with the same request ID value, tracking the change of the log information with the same request ID value, and displaying the change condition of the log information.
In this embodiment, the Kibana system is an open-source and free tool, and the Kibana can generate a Web interface of a website with a user-friendly experience according to log data provided by the elastic search, so that a user can conveniently view log data of different projects or different modules.
In the actual operation process of a user, on one hand, the user can enter a kibana system to select items or modules, a user-defined data structure field is used as a statistical analysis dimension, the dimension is specified to perform query, for example, all logs of a certain specified java class are displayed, all logs with the log level of error are displayed, the logs are displayed in the reverse order of log time sequencing, and the like. On the other hand, the user only needs to find the specified target item on the display interface of Kibana, the filtering condition can be added under the specified target item, the filtering field is selected as the request ID, the filtering value is the ID, all log data of the specified ID can be displayed, and the change condition of the log information is tracked and displayed.
Therefore, the log data which the user wants to look up can be screened out and displayed, the accuracy of displaying the log data is improved, and the user can quickly and accurately look up the related log data conveniently.
As shown in fig. 4, the log processing apparatus 200 further includes:
the configuration module 205 is configured to configure the preset parsing rule through a Redis system, where the preset parsing rule includes a processing algorithm, a database type, a database, an insertion algorithm, a destination data table, and an alarm flag.
In this embodiment, the Redis database is an open-source log-type and Key-Value database written in ANSI C language, supporting network, and capable of being based on memory and persistent, and can provide Application Programming Interface (API) of multiple languages. It can be understood that the configuration information is directly stored in the memory, which is convenient and fast, and is convenient for modification without storing the configuration file locally.
It is further supplementary to explain that the configuration information may further include alarm information, system control information, processing rule information, and storage information, where the alarm information includes an alarm level, an alarm manner, and an alarm person contact manner, and the system control information includes a consumption log timeout time, an application switch for updating a log collection topic of a project, and a name set of a project log source topic; the storage information comprises an ElasticSearch connection ip address, an ElasticSearch connection port number, an ElasticSearch authentication user name, an ElasticSearch authentication password, an ElasticSearch database name and an ElasticSearch data table name.
In the present embodiment, the purpose of controlling the present log processing apparatus can be achieved by modifying the configuration information. For example, a new project or module needs to be added for log monitoring, a topic with the project name as the name is created in the kafka cluster, and then the new topic name is added in the name set of each project log collection topic of the system control information. Then, a processing rule with the name of topic as key and the processing rule information as value is added to the processing rule information. And finally, setting the value of an application switch of the log collection topic of the system control information updating item to true, so far that the addition of a new item or module is successful. In addition, existing items may be modified or deleted by modifying or deleting configuration information.
It is further added that the log processing 200 may further include:
the first modification module is used for modifying the new application configuration of the Redis system and changing the value of an application switch of the log collection topic into true;
a second modification module for adding topic of a new project collection log to a name set of project log source topic topics of the Kafka system.
In this embodiment, the Redis system is a key-value storage system, and supports more stored value types, including a string (string), a linked list (list), a set (set), a sortedset-ordered set (zset), and a hash type (hash). Additionally, Redis supports a variety of different manners of ordering. In order to ensure efficiency, data is cached in a memory, and the Redis system periodically writes updated data into a disk or writes modification operations into an additional recording file, and can realize master-slave (master-slave) synchronization on the basis. Master-slave synchronization, where data may be synchronized from a master server to any number of slave servers, a slave server may be the master server associated with other slave servers.
In this embodiment, after the new application configuration in Redis is modified, the application switch of the log collection topic is modified. In addition, after the application performs a new configuration, the value of the application switch is automatically changed to false to cope with the next configuration modification. For example, an application is newly added, and after a preset parsing rule and a corresponding topic name are added, the new application cannot acquire the log without changing the value of the switch. In this embodiment, when the value of the application switch of the log collection topic in the Redis system is true, it indicates that the addition of log monitoring on a new item is allowed.
In this embodiment, the Kafka system is sourced by Apache, is a message middleware, and is a distributed, partitionable, and replicable message system. The topic can be understood as different classifications of message sources (feessof messages) of Kafka processing resources.
The Kafka system generalizes messages in units of topic. The program that issues messages to the topic of the Kafka system is called producers (producers). The program that subscribes to topics and consumes messages is called a consumer (consumer). The Kafka system operates in a cluster and may be composed of one or more services, each called a cache broker (broker). The producers send messages over the network to the Kafka cluster, which provides the messages to the consumers. The producers generate messages in real time and send a class of messages with the same nature to a topic of the kafka system. A consumer subscribes to one or more topics, and gets one or more types of messages.
It is further added that the first logging module 201 is further configured to obtain topic of project allocation through the Kafka system, and consume the obtained topic to obtain the latest log data; or,
and configuring the link address of the kafka system and the topic of each item by configuring a configuration file of a log data collector fileteam, and submitting log information acquired from a system log file to the topic corresponding to each item of the kafka system by the fileteam.
It should be noted that the filebed is a log data collector of the local file. As an agent installation on the server, Filebeat monitors the log directory, specific log file, or file tail file, and can forward the log directory, specific log file, or file tail to the Elasticsearch for indexing, etc.
Therefore, the Kafka system can quickly collect a large amount of log data, and is convenient for subsequent log data analysis.
The log processing device provided by the embodiment of the invention can analyze the log data acquired from the Kafka system according to the custom data format to obtain the analyzed log data, and store the analyzed log data into different tables in the ElasticSearch database, so that the efficiency of analyzing the log data is improved, the orderliness of storing the log data is improved, and the log data processing effect is improved.
An embodiment of the present invention provides a computer apparatus, where the computer apparatus includes a processor, and the processor is configured to implement the steps of the log processing method provided in each of the above method embodiments when executing a computer program in a memory.
Illustratively, a computer program can be partitioned into one or more modules, which are stored in memory and executed by a processor to implement the present invention. One or more of the modules may be a sequence of computer program instruction segments for describing the execution of a computer program in a computer device that is capable of performing certain functions. For example, the computer program may be divided into the steps of the log processing method provided by the various method embodiments described above.
Those skilled in the art will appreciate that the above description of a computer apparatus is by way of example only and is not intended to be limiting of computer apparatus, and that the apparatus may include more or less components than those described, or some of the components may be combined, or different components may be included, such as input output devices, network access devices, buses, etc.
The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like which is the control center for the computer device and which connects the various parts of the overall computer device using various interfaces and lines.
The memory may be used to store the computer programs and/or modules, and the processor may implement various functions of the computer device by running or executing the computer programs and/or modules stored in the memory and invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
The modules/units integrated by the computer device may be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as separate products. Based on such understanding, all or part of the flow in the method according to the above embodiments may be implemented by a computer program, which may be stored in a computer readable storage medium and used by a processor to implement the steps of the log processing method embodiments. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, electrical signals, software distribution medium, and the like.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (10)
1. A log processing method, comprising:
acquiring log information through a Kafka system, wherein the log information comprises log data of a plurality of items; storing the log data into topic with the same name as the belonged project;
analyzing the log data acquired from the topic according to a user-defined data format through a data processing module developed by a JStorm framework to obtain analyzed log data;
and storing the analyzed log data into a corresponding data table in an ElasticSearch database according to the database type, the database name and a target data table specified by a preset analysis rule through a data storage module developed by a JStorm framework.
2. The log processing method according to claim 1, wherein the storing the parsed log data into a corresponding data table in an ElasticSearch database according to the database type, the database name and the destination data table specified by the preset parsing rule comprises the following processes:
and when the alarm mark in the preset analysis rule is true, acquiring alarm information from the analyzed log data, and sending the alarm information to an alarm contact person through an alarm module developed by the JSTOM framework.
3. The log processing method of claim 1, wherein the method further comprises the process of:
and acquiring the analyzed log data from a specified data table of the ElasticSearch database through the Kibana system, and displaying the acquired log data.
4. The log processing method according to claim 1, wherein before storing the parsed log data into the corresponding data table in the ElasticSearch database according to the database type, the database name and the destination data table specified by the preset parsing rule through the data storage module developed by the jstom framework, the method further comprises the following processes:
and configuring the preset analysis rule through a Redis system, wherein the preset analysis rule comprises a processing algorithm, a database type, a database, an insertion algorithm, a target data table and an alarm mark.
5. A log processing apparatus, comprising:
the system comprises a first logging module, a second logging module and a third logging module, wherein the first logging module is used for acquiring log information through a Kafka system, and the log information comprises log data of a plurality of items; storing the log data into topic with the same name as the belonged project;
the analysis module is used for analyzing the log data acquired from the topic according to a user-defined data format through a data processing module developed by a JStorm framework to obtain the analyzed log data;
and the second storage module is used for storing the analyzed log data into a corresponding data table in the ElasticSearch database according to the database type, the database name and the target data table specified by the preset analysis rule through the data storage module developed by the JStorm framework.
6. The log processing apparatus of claim 5, wherein the second logging module is further configured to, when an alarm flag in the preset parsing rule is true, obtain alarm information from the parsed log data, and send the alarm information to an alarm contact through an alarm module developed by the jstom framework.
7. The log processing apparatus of claim 6, further comprising:
and the display module is used for acquiring the analyzed log data from the specified data table of the ElasticSearch database through the Kibana system and displaying the acquired log data.
8. The log processing apparatus of claim 7, further comprising:
and the configuration module is used for configuring the preset analysis rule through a Redis system, wherein the preset analysis rule comprises a processing algorithm, a database type, a database, an insertion algorithm, a target data table and an alarm mark.
9. A computer arrangement, characterized in that the computer arrangement comprises a processor for implementing the steps of the log processing method according to any of claims 1-4 when executing a computer program in a memory.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when being executed by a processor, realizes the steps of the log processing method as claimed in any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910648085.7A CN110515912A (en) | 2019-07-18 | 2019-07-18 | Log processing method, device, computer installation and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910648085.7A CN110515912A (en) | 2019-07-18 | 2019-07-18 | Log processing method, device, computer installation and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110515912A true CN110515912A (en) | 2019-11-29 |
Family
ID=68623648
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910648085.7A Pending CN110515912A (en) | 2019-07-18 | 2019-07-18 | Log processing method, device, computer installation and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110515912A (en) |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111049846A (en) * | 2019-12-20 | 2020-04-21 | 北京明略软件系统有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111177237A (en) * | 2019-12-10 | 2020-05-19 | 腾讯科技（深圳）有限公司 | Data processing system, method and device |
| CN111241044A (en) * | 2020-01-08 | 2020-06-05 | 中国联合网络通信集团有限公司 | Method, device and equipment for building heterogeneous database and readable storage medium |
| CN111414269A (en) * | 2020-03-17 | 2020-07-14 | 深圳市朱墨科技有限公司 | Log alarm method, device, storage medium and equipment |
| CN111552719A (en) * | 2020-04-23 | 2020-08-18 | 深圳前海微众银行股份有限公司 | Data management method, device and system, big data platform and readable storage medium |
| CN111611127A (en) * | 2020-04-26 | 2020-09-01 | 第四范式（北京）技术有限公司 | Processing method, device and equipment for task running log and storage medium |
| CN111639098A (en) * | 2020-05-11 | 2020-09-08 | 紫光云技术有限公司 | Cloud log management method |
| CN112115112A (en) * | 2020-08-10 | 2020-12-22 | 上海金仕达软件科技有限公司 | Log information processing method and device and electronic equipment |
| CN112269825A (en) * | 2020-11-13 | 2021-01-26 | 中盈优创资讯科技有限公司 | Method and device for ETL (extract transform and load) analysis abnormal data retention |
| CN112506735A (en) * | 2020-11-26 | 2021-03-16 | 中移（杭州）信息技术有限公司 | Service quality monitoring method, system, server and storage medium |
| CN112948334A (en) * | 2021-03-31 | 2021-06-11 | 建信金融科技有限责任公司 | Log processing method and device |
| CN113297240A (en) * | 2021-06-01 | 2021-08-24 | 杭州每刻科技有限公司 | PostgreSQL database synchronization method and system |
| CN113391973A (en) * | 2021-05-24 | 2021-09-14 | 青岛海信智慧生活科技股份有限公司 | Internet of things cloud container log collection method and device |
| CN113778810A (en) * | 2021-09-27 | 2021-12-10 | 杭州安恒信息技术股份有限公司 | Log collection method, device and system |
| CN113886199A (en) * | 2021-08-31 | 2022-01-04 | 联想(北京)有限公司 | Data processing method and device |
| CN114153823A (en) * | 2022-02-09 | 2022-03-08 | 北京华品博睿网络技术有限公司 | Distributed computing job log data processing method and system |
| CN114386035A (en) * | 2021-12-31 | 2022-04-22 | 航天信息股份有限公司 | Method and device for detecting threat data and electronic equipment |
| CN114936195A (en) * | 2022-03-08 | 2022-08-23 | 天津光电通信技术有限公司 | Interface system based on Spring Boot framework |
| CN116578655A (en) * | 2023-07-06 | 2023-08-11 | 舟谱数据技术南京有限公司 | Data transmission system and control method thereof |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105631026A (en) * | 2015-12-30 | 2016-06-01 | 北京奇艺世纪科技有限公司 | Security data analysis system |
| US20160306817A1 (en) * | 2015-04-14 | 2016-10-20 | Et International, Inc. | Systems and methods for key-value stores |
| CN106850258A (en) * | 2016-12-22 | 2017-06-13 | 北京锐安科技有限公司 | A kind of Log Administration System, method and device |
| CN108519942A (en) * | 2018-04-11 | 2018-09-11 | 车巴达（苏州）网络科技有限公司 | A kind of Log Analysis System |
-
2019
- 2019-07-18 CN CN201910648085.7A patent/CN110515912A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160306817A1 (en) * | 2015-04-14 | 2016-10-20 | Et International, Inc. | Systems and methods for key-value stores |
| CN105631026A (en) * | 2015-12-30 | 2016-06-01 | 北京奇艺世纪科技有限公司 | Security data analysis system |
| CN106850258A (en) * | 2016-12-22 | 2017-06-13 | 北京锐安科技有限公司 | A kind of Log Administration System, method and device |
| CN108519942A (en) * | 2018-04-11 | 2018-09-11 | 车巴达（苏州）网络科技有限公司 | A kind of Log Analysis System |
Non-Patent Citations (2)
| Title |
|---|
| 蒋理 等: "《中国制造2025智能制造企业信息系统》", 31 May 2018 * |
| 高登: "《云计算与Hadoop应用技术研究》", 31 December 2017 * |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111177237B (en) * | 2019-12-10 | 2024-02-13 | 腾讯科技（深圳）有限公司 | Data processing system, method and device |
| CN111177237A (en) * | 2019-12-10 | 2020-05-19 | 腾讯科技（深圳）有限公司 | Data processing system, method and device |
| CN111049846A (en) * | 2019-12-20 | 2020-04-21 | 北京明略软件系统有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111241044A (en) * | 2020-01-08 | 2020-06-05 | 中国联合网络通信集团有限公司 | Method, device and equipment for building heterogeneous database and readable storage medium |
| CN111241044B (en) * | 2020-01-08 | 2023-09-19 | 中国联合网络通信集团有限公司 | Method, device, equipment and readable storage medium for constructing heterogeneous database |
| CN111414269A (en) * | 2020-03-17 | 2020-07-14 | 深圳市朱墨科技有限公司 | Log alarm method, device, storage medium and equipment |
| CN111552719A (en) * | 2020-04-23 | 2020-08-18 | 深圳前海微众银行股份有限公司 | Data management method, device and system, big data platform and readable storage medium |
| CN111611127A (en) * | 2020-04-26 | 2020-09-01 | 第四范式（北京）技术有限公司 | Processing method, device and equipment for task running log and storage medium |
| CN111611127B (en) * | 2020-04-26 | 2023-10-31 | 第四范式（北京）技术有限公司 | Task running log processing method, device, equipment and storage medium |
| CN111639098A (en) * | 2020-05-11 | 2020-09-08 | 紫光云技术有限公司 | Cloud log management method |
| CN112115112A (en) * | 2020-08-10 | 2020-12-22 | 上海金仕达软件科技有限公司 | Log information processing method and device and electronic equipment |
| CN112269825A (en) * | 2020-11-13 | 2021-01-26 | 中盈优创资讯科技有限公司 | Method and device for ETL (extract transform and load) analysis abnormal data retention |
| CN112506735A (en) * | 2020-11-26 | 2021-03-16 | 中移（杭州）信息技术有限公司 | Service quality monitoring method, system, server and storage medium |
| CN112948334A (en) * | 2021-03-31 | 2021-06-11 | 建信金融科技有限责任公司 | Log processing method and device |
| CN113391973A (en) * | 2021-05-24 | 2021-09-14 | 青岛海信智慧生活科技股份有限公司 | Internet of things cloud container log collection method and device |
| CN113391973B (en) * | 2021-05-24 | 2022-11-25 | 青岛海信智慧生活科技股份有限公司 | Internet of things cloud container log collection method and device |
| CN113297240A (en) * | 2021-06-01 | 2021-08-24 | 杭州每刻科技有限公司 | PostgreSQL database synchronization method and system |
| CN113886199A (en) * | 2021-08-31 | 2022-01-04 | 联想(北京)有限公司 | Data processing method and device |
| CN113886199B (en) * | 2021-08-31 | 2024-04-19 | 联想(北京)有限公司 | Data processing method and device |
| CN113778810A (en) * | 2021-09-27 | 2021-12-10 | 杭州安恒信息技术股份有限公司 | Log collection method, device and system |
| CN114386035A (en) * | 2021-12-31 | 2022-04-22 | 航天信息股份有限公司 | Method and device for detecting threat data and electronic equipment |
| CN114153823A (en) * | 2022-02-09 | 2022-03-08 | 北京华品博睿网络技术有限公司 | Distributed computing job log data processing method and system |
| CN114936195A (en) * | 2022-03-08 | 2022-08-23 | 天津光电通信技术有限公司 | Interface system based on Spring Boot framework |
| CN116578655A (en) * | 2023-07-06 | 2023-08-11 | 舟谱数据技术南京有限公司 | Data transmission system and control method thereof |
| CN116578655B (en) * | 2023-07-06 | 2023-09-15 | 舟谱数据技术南京有限公司 | Data transmission system and control method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110515912A (en) | Log processing method, device, computer installation and computer readable storage medium | |
| CN108874558B (en) | Message subscription method of distributed transaction, electronic device and readable storage medium | |
| CN109800207B (en) | Log analysis method, device and equipment and computer readable storage medium | |
| CN111143382B (en) | Data processing method, system and computer readable storage medium | |
| US11593357B2 (en) | Databases and methods of storing, retrieving, and processing data | |
| CN111209352A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN109840298B (en) | Multi-information-source acquisition method and system for large-scale network data | |
| CN113448837B (en) | Development and test environment deployment method, system, electronic equipment and medium | |
| CN110955578A (en) | Log collection method and device based on host machine, computer equipment and storage medium | |
| CN109753596B (en) | Information source management and configuration method and system for large-scale network data acquisition | |
| CN111177237B (en) | Data processing system, method and device | |
| US7783743B1 (en) | Methods and apparatus for processing electronic mail-related data | |
| CN111198976A (en) | On-cloud asset association analysis system, method, electronic device and medium | |
| CN115934855A (en) | Full-link field level blood margin analysis method, system, equipment and storage medium | |
| CN109614088B (en) | Form component generation method and device | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| CN113391973A (en) | Internet of things cloud container log collection method and device | |
| CN112187509A (en) | Multi-architecture cloud platform execution log management method, system, terminal and storage medium | |
| CN116204540A (en) | Operation log recording method, device, equipment and storage medium | |
| CN113885860A (en) | Method and equipment for automatically configuring management page to generate interface service | |
| CN109582347B (en) | Method and device for acquiring front-end codes | |
| CN111913996B (en) | Data processing method, device, equipment and storage medium | |
| CN111459411B (en) | Data migration method, device, equipment and storage medium | |
| CN114020565A (en) | Intelligent log collection processing method and device, electronic equipment and storage medium | |
| CN113434585A (en) | Resource saving method and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information |
Inventor after: Zeng Yan Inventor after: Deng Dawei Inventor before: Zeng Yan Inventor before: Deng Dawei Inventor before: Zhang Shengdong |
|
| CB03 | Change of inventor or designer information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191129 |
|
| RJ01 | Rejection of invention patent application after publication |