CN117591376A - Log aggregation analysis method, device, network equipment and medium - Google Patents
Log aggregation analysis method, device, network equipment and medium Download PDFInfo
- Publication number
- CN117591376A CN117591376A CN202311753883.9A CN202311753883A CN117591376A CN 117591376 A CN117591376 A CN 117591376A CN 202311753883 A CN202311753883 A CN 202311753883A CN 117591376 A CN117591376 A CN 117591376A
- Authority
- CN
- China
- Prior art keywords
- log data
- log
- data
- metadata information
- application service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000002776 aggregation Effects 0.000 title claims abstract description 31
- 238000004220 aggregation Methods 0.000 title claims abstract description 31
- 238000004458 analytical method Methods 0.000 title claims abstract description 29
- 238000003860 storage Methods 0.000 claims abstract description 58
- 238000000034 method Methods 0.000 claims abstract description 56
- 238000012545 processing Methods 0.000 claims abstract description 31
- 230000008569 process Effects 0.000 claims abstract description 11
- 230000006870 function Effects 0.000 claims description 7
- 238000012423 maintenance Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 6
- 238000007726 management method Methods 0.000 description 5
- 238000013500 data storage Methods 0.000 description 4
- 230000006399 behavior Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Human Computer Interaction (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a log aggregation analysis method, a device, network equipment and a medium, wherein the method is applied to log processing service and comprises the following steps: under the condition that the application service completes authentication, acquiring a plurality of first log data sent by the application service, wherein the first log data comprises corresponding metadata information; judging that the first target log data needs to be encrypted according to metadata information corresponding to the first target log data, encrypting the first target log data through an encryption flow of the link, and obtaining second log data; classifying the second log data according to the metadata information, and adding real-time state information corresponding to the classified second log data to acquire a plurality of third log data; and respectively storing the plurality of third log data in different storage structures according to the service requirements of the third log data. The log data is encrypted according to the requirement, so that the problem that the log aggregation method in the prior art is low in safety and cannot process large-scale log data in real time is solved.
Description
Technical Field
The present invention relates to the field of data management technologies, and in particular, to a method, an apparatus, a network device, and a medium for log aggregation analysis.
Background
A Data Warehouse (Data Warehouse) is a large-scale, centralized database system whose main responsibility is to store, manage and analyze massive amounts of Data from various sources. In a large data warehouse system, there may be tens or even hundreds of services running, each of which generates a large number of journals. The size and complexity of these log data are very large. How to effectively analyze the logs and how to monitor the operation condition of the data warehouse through the logs becomes an important work of operation and maintenance personnel of the data warehouse. The log information not only contains information of system operation, but also possibly contains information of business activities, such as behavior log of the user. These logs can help us to understand the behavior patterns of users, optimizing products and services.
Implementing an efficient and reliable unified log aggregation analysis system is important for modern IT operation and maintenance work. The main technologies for realizing the log architecture at present are an ELK (Elasticsearch, logstash, kibana) technology stack and a PLG (Promtail, loki, grafana) technology stack. These technical stacks provide powerful log collection, storage, query and visualization functions that can help the operations and maintenance team to efficiently process and analyze large-scale log data.
The use of ELK has several problems:
1. the data storage structure is based on inverted index, and for a large amount of log data, the storage structure can cause the index file to be oversized, and the maintenance cost of the index file can also be increased.
2. Deployment and management are complex, and special operation and maintenance personnel are required for maintenance.
3. Long-term storage and archiving are difficult and require special storage systems for storage.
4. The security is not enough, the data is not encrypted, and the security of the data cannot be ensured.
The use of PLG has several problems:
prometheus is mainly used for index monitoring and is not suitable for storing and analyzing logs.
Loki, while a lightweight log aggregation system, suffers from reduced performance in the face of large amounts of log data.
Plg is more suitable for cloud-native scenarios, which are complex to deploy and manage for traditional data warehouse systems.
Plg is more concerned with monitoring and visualization, and support for log storage and analysis is inadequate.
The existing log aggregation method is not enough in safety, data are not encrypted, the safety of the data cannot be guaranteed, and performance of the existing log aggregation method is reduced when a large amount of log data are faced.
Disclosure of Invention
The invention aims to provide a log aggregation analysis method, a device, network equipment and a medium, which are used for solving the problems that the log aggregation method in the prior art is low in safety and cannot process large-scale log data in real time.
In order to solve the above technical problems, an embodiment of the present invention provides a log aggregation analysis method, where the method is applied to a log processing service, and the method includes:
acquiring a plurality of first log data sent by an application service under the condition that the application service completes authentication; wherein the first log data includes corresponding metadata information;
under the condition that the first target log data is judged to need to be encrypted according to the metadata information corresponding to the first target log data in the first log data, encrypting the first target log data through a Flink encryption process to obtain second log data;
classifying the second log data according to the metadata information, and adding real-time state information corresponding to the classified second log data to acquire a plurality of third log data;
and respectively storing a plurality of third log data in different storage structures according to the service requirements of the third log data.
Optionally, the method further includes, after the acquiring the plurality of first log data sent by the application service:
transmitting the first log data to a plurality of topics of a Kafka message queue according to the metadata information; the theme is obtained by dividing the metadata information in advance;
first log data is periodically obtained from the subject.
Optionally, the method, before the authentication of the application service is completed, comprises:
acquiring the metadata information sent by the application service; wherein the metadata information includes at least one of: an application service name, an application service version number, a server address of an application service;
confirming the version and the function of the application service according to the metadata information, and completing the registration of the application service;
and sending authentication success information to the application service.
Optionally, the method, wherein classifying the second log data according to the metadata information, adding real-time status information corresponding to the classified second log data, and obtaining a plurality of third log data includes:
classifying the second log data according to the metadata information to obtain a plurality of classified data; wherein the metadata information includes at least one of: the source of the log data, the type of the log data, the grade of the log data;
acquiring the real-time state information according to the classification data;
and acquiring the third log data according to the classification data, the metadata information corresponding to the classification data and the real-time state information.
Optionally, the method, wherein the storage structure includes at least one of:
ClickHouse；
Elasticsearch；
Doris。
optionally, the method, wherein storing the plurality of third log data in different storage structures according to the service requirement of the third log data includes:
under the condition that the service requirement of the third log data is high in load requirement, the third log data is sent to a ClickHouse for storage;
and sending the third log data to an elastic search storage under the condition that the service requirement of the third log data is that the dynamic combination requirement is required for the query condition.
Optionally, the method, after storing the plurality of third log data in different storage structures according to the service requirement of the third log data, the method further includes:
and reading the third log data in the storage structure through Grafana.
In order to achieve the above object, the present invention also provides a log aggregation analysis device, wherein the device is applied to a log processing service, the device comprising:
the first acquisition module is used for acquiring a plurality of first log data sent by the application service under the condition that the authentication of the application service is completed; wherein the first log data includes corresponding metadata information;
the second acquisition module is used for encrypting the first target log data through an encryption flow of the link under the condition that the first target log data is judged to need to be encrypted according to the metadata information corresponding to the first target log data in the first log data, so as to acquire second log data;
the third acquisition module is used for classifying the second log data according to the metadata information, adding real-time state information corresponding to the second log data and acquiring a plurality of third log data;
and the first storage module is used for respectively storing a plurality of the third log data in different storage structures according to the service requirements of the third log data.
In order to achieve the above object, the present invention further provides a network device, including: a transceiver, a processor, a memory, and a program or instructions stored on the memory and executable on the processor; the processor executes the program or the instruction to implement the log aggregation analysis method.
In order to achieve the above object, the present invention also provides a readable storage medium having stored thereon a program or instructions which, when executed by a processor, implement the steps in the log aggregation analysis method as described above.
The technical scheme of the invention has the following beneficial effects:
in the above scheme, the log processing service acquires the first log data sent by the application service when the application service completes authentication, encrypts to acquire the second log data under the condition that the first target log data is judged to need to be encrypted according to the metadata information included in the first log data, classifies the second log data, adds corresponding real-time state information to obtain the third log data, and stores the third log data in different storage structures according to the service requirements of the third log data, thereby solving the problems that the security of the log aggregation method in the prior art is not high and large-scale log data cannot be processed in real time.
Drawings
FIG. 1 is a schematic diagram of a log aggregation analysis method according to an embodiment of the present invention;
FIG. 2 is a flowchart of a log aggregation analysis method according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a log aggregation analysis device according to an embodiment of the invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Aiming at the problems that the security of a log aggregation method in the prior art is not high and large-scale log data cannot be processed in real time, the invention provides the log aggregation analysis method, the device, the network equipment and the medium.
A Data Warehouse (Data wire) is a large-scale, centralized Data storage system specifically designed to store, manage, and analyze large amounts of Data from various business systems. These data may contain information about various business transactions, operation logs, user behavior, etc., all of which are important bases for organizing operations and decisions. The primary goal of the data warehouse is to support the decision making process of an organization, which provides a comprehensive view of the organization so that users can conduct in-depth data analysis and reporting to drive data-based decisions.
In conventional journal management schemes, archiving and storing of journals is an important but rather difficult problem. First, the size of log data is typically very large, requiring a specialized storage system to store. Second, the maintenance and management of storage systems is very complex, including data backup, restore, migration, cleaning, etc., which require significant human and financial effort.
In order to solve the above problems, an embodiment of the present invention proposes a log aggregation analysis method, which is based on stream computing. Streaming computing is a computational model that processes real-time data streams, processes and analyzes the data in real-time, and then stores the results of the processing in other storage structures. The model can effectively process large-scale real-time data and meet the real-time and large-scale requirements of log management. In the method, apache Flink is used as a streaming computing engine, clickHouse and elastic search are used as data storage systems, and Apache Kafka is used as a data transmission middleware. The flank is a high-performance, scalable streaming computing framework that can process and analyze data in real-time. ClickHouse and elastic search are two high performance data storage systems that can efficiently store and query large-scale data. Kafka is a high throughput distributed message queue system that can efficiently transfer large-scale data.
As shown in fig. 1, an embodiment of the present invention provides a log aggregation analysis method, where the method is applied to a log processing service, and the method includes:
s10, under the condition that authentication is completed by an application service, acquiring a plurality of first log data sent by the application service; wherein the first log data includes corresponding metadata information;
when the application service is started, metadata information is registered with the log processing service, and the application service can only send the first log data under the condition that authentication is ensured to be completed. This not only ensures the security of the data, but also makes it more convenient to manage and track the application services.
S20, under the condition that the first target log data is judged to need to be encrypted according to the metadata information corresponding to the first target log data in the first log data, encrypting the first target log data through an encryption flow of a link to obtain second log data;
in the authentication stage, if the application service declares that the first target log data needs to be encrypted in metadata information sent to the log processing service, public key or private key information needed for encryption is confirmed and is delivered to the log processing service for decryption.
S30, classifying the second log data according to the metadata information, and adding real-time state information corresponding to the classified second log data to acquire a plurality of third log data;
and S40, respectively storing a plurality of third log data in different storage structures according to the service requirements of the third log data.
In this embodiment, the log processing service obtains the first log data sent by the application service when the authentication is completed by the application service, encrypts to obtain the second log data when it is determined that the first target log data needs to be encrypted according to the metadata information included in the first log data, classifies the second log data, adds the corresponding real-time status information, and obtains the third log data, and stores the third log data in different storage structures according to service requirements of the third log data, thereby solving the problem that in the prior art, the security of the log aggregation method is not high and the large-scale log data cannot be processed in real time.
Optionally, the method further includes, after the step S10:
transmitting the first log data to a plurality of topics of a Kafka message queue according to the metadata information; the theme is obtained by dividing the metadata information in advance;
first log data is periodically obtained from the subject.
In this embodiment, as shown in fig. 2, the application service pushes the first log data to the topic corresponding to the Kafka message queue through an asynchronous thread. The asynchronous mode can avoid blocking the normal operation of the application service, and can ensure the real-time performance of the log data. Kafka acts as a high throughput message queue, efficiently handling large amounts of log data, and periodically reading log data from the subject matter of Kafka to buffer it for subsequent processing and analysis.
Optionally, the method, before the step S10, includes:
acquiring the metadata information sent by the application service; wherein the metadata information includes at least one of: an application service name, an application service version number, a server address of an application service;
confirming the version and the function of the application service according to the metadata information, and completing the registration of the application service;
and sending authentication success information to the application service.
In this embodiment, as shown in fig. 2, when the application service is started, metadata information is registered with the log processing service, where the metadata information includes, but is not limited to: application service name, application service version number, server address of application service, application service running environment, etc. After registration is completed, the application service waits for authentication of the log processing service, and the application service can only send the first log data under the condition that authentication is ensured to be completed. This not only ensures the security of the data, but also makes it more convenient to manage and track the application services. While the initialization service will determine the storage structure in the storage space for the final stored log. The registration may be regarded as that after the application service and the log processing service confirm the basic attribute information of the first log data, only the log content of the core is received and transmitted at the time of communication, and the incidental information is not transmitted, so that the throughput of the log is reduced, and the log processing service is handed to the log processing service to reassemble and analyze the basic attribute information. The log processing service confirms whether the log pushing function of the application service is matched with the current version and function of the log processing service through an authentication process. The authentication step ensures the consistency and accuracy of the data and prevents data loss or errors due to version mismatch or functional incompatibility. The log processing service collects the first log data through a Kafka message queue.
Optionally, the method, wherein the step S30 includes:
classifying the second log data according to the metadata information to obtain a plurality of classified data; wherein the metadata information includes at least one of: the source of the log data, the type of the log data, the grade of the log data;
acquiring the real-time state information according to the classification data;
and acquiring the third log data according to the classification data, the metadata information corresponding to the classification data and the real-time state information.
In this embodiment, the log processing service splits the second log data according to the source, type, level, and other information of the second log data, and combines the second log data with the metadata information and other data (for example, the real-time status information corresponding to the classification data) provided when the application service is registered into the third log data. Finally, the third log data is modified according to other logic to facilitate subsequent analysis and querying.
The flank is used as the core technology for processing services because flank itself is a stream processing architecture that separates definition tasks from execution tasks. After the submitted task is modified, the node which actually executes the task again; the nodes for executing the tasks are reduced or increased in a cutting dynamic mode, and the execution flow of the tasks cannot be influenced. This ensures an efficient and robust capability of log analysis, providing effective support for maintenance of the application services. And the Flink provides a powerful state management and fault tolerance mechanism, so that the accuracy and consistency of data processing are ensured, and the data can be kept not to be lost even under the condition of node faults. The Flink also provides an interface method easy to operate, and the capability of analyzing and analyzing the log according to the definition can be realized more conveniently.
An example of an embodiment of the present invention is as follows:
and the operation log sensitive to certain user data in the database is sent to the log processing service after the application service is encrypted according to a public key. The log processing service combines the metadata information during other registration, the name of the log, the host server and application information (ip, port, etc.) of the application service, and then reclassifies according to the information such as read or write operation, successful or failed execution result, operator, etc.
Optionally, the method, wherein the storage structure includes at least one of:
ClickHouse；
Elasticsearch；
Doris。
optionally, the method, wherein the step S40 includes:
under the condition that the service requirement of the third log data is high in load requirement, the third log data is sent to a ClickHouse for storage;
and sending the third log data to an elastic search storage under the condition that the service requirement of the third log data is that the dynamic combination requirement is required for the query condition.
In this embodiment, the storage service may obtain a log storage structure during the application registration phase, and open up storage space for the log storage structure. The third log data is matched to the appropriate storage structure at the time of storage. If the service requirement of the third log data is high in load requirement, pushing the third log data to ClickHouse storage; and if the service requirement of the third log data is that the dynamic combination requirement is provided for the query condition, pushing the third log data to an elastic search and the like. The flexible storage mode can meet different business requirements, and meanwhile, the storage and query efficiency of data is optimized.
An example provided by the embodiment of the invention is as follows:
the read-write service load of the database is generally high and can be stored in the clickHouse. The application initialization stage creates a database table for it and sets fields according to the category of the metadata information category, and the log can be queried through the fields during query.
Optionally, the method further includes, after the step S40:
and reading the third log data in the storage structure through Grafana.
In this embodiment, grafana or other program may read the third log data in storage through the log processing service. The method can conveniently analyze and visualize the data, can form an OPEN API, is provided for other services, and improves the availability and value of the data. Because the storage adopts the services such as a database or an Elastic Stack distributed search analysis engine, the method can find the position of the target log to be queried through classified category query faster than reading the log from the file.
It should be noted that, the log aggregation analysis method in this embodiment provides a high-efficiency and real-time log processing manner for the user by integrating Kafka, flink, clickHouse, elasticsearch and Grafana technologies. At the same time, this approach also takes into account the security, consistency and availability of the data so that users can better understand and optimize their IT systems.
As shown in fig. 3, to achieve the above object, the present invention also provides a log aggregation analysis device, wherein the device is applied to a log processing service, and the device includes:
a first obtaining module 301, configured to obtain, when authentication is completed by an application service, a plurality of first log data sent by the application service; wherein the first log data includes corresponding metadata information;
a second obtaining module 302, configured to encrypt, according to an encryption flow of the link, the first target log data to obtain second log data when it is determined that the first target log data needs to be encrypted according to the metadata information corresponding to the first target log data in the first log data;
a third obtaining module 303, configured to classify the second log data according to the metadata information, and add real-time status information corresponding to the second log data to obtain a plurality of third log data;
the first storage module 304 is configured to store a plurality of the third log data in different storage structures according to the service requirements of the third log data.
Optionally, the apparatus further comprises:
the first sending module is used for sending the first log data to a plurality of topics of a Kafka message queue according to the metadata information; the theme is obtained by dividing the metadata information in advance;
and the fourth acquisition module is used for periodically acquiring the first log data from the theme.
Optionally, the apparatus further comprises:
a fifth obtaining module, configured to obtain the metadata information sent by the application service; wherein the metadata information includes at least one of: an application service name, an application service version number, a server address of an application service;
the first processing module is used for confirming the version and the function of the application service according to the metadata information and finishing registration of the application service;
and the second sending module is used for sending authentication success information to the application service.
Optionally, the apparatus, wherein the third obtaining module 303 includes:
the first acquisition unit is used for classifying the second log data according to the metadata information to acquire a plurality of classified data; wherein the metadata information includes at least one of: the source of the log data, the type of the log data, the grade of the log data;
the second acquisition unit is used for acquiring the real-time state information according to the classification data;
and a third obtaining unit, configured to obtain the third log data according to the classification data, the metadata information corresponding to the classification data, and the real-time status information.
Optionally, the apparatus, wherein the storage structure includes at least one of:
ClickHouse；
Elasticsearch；
Doris。
optionally, the apparatus, wherein the first storage module 304 includes:
the first sending unit is used for sending the third log data to the ClickHouse storage under the condition that the service requirement of the third log data is high in load requirement;
and the second sending unit is used for sending the third log data to an elastic search storage under the condition that the service requirement of the third log data is a dynamic combination requirement on the query condition.
Optionally, the apparatus further comprises:
and the first reading module is used for reading the third log data in the storage structure through Grafana.
It should be noted that, the above device provided in the embodiment of the present invention can implement all the method steps implemented in the method embodiment and achieve the same technical effects, and detailed descriptions of the same parts and beneficial effects as those in the method embodiment in this embodiment are omitted.
In order to achieve the above object, the present invention further provides a network device, including: a transceiver, a processor, a memory, and a program or instructions stored on the memory and executable on the processor; the processor executes the program or the instruction to implement the log aggregation analysis method.
In order to achieve the above object, the present invention also provides a readable storage medium having stored thereon a program or instructions which, when executed by a processor, implement the steps in the log aggregation analysis method as described above.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (10)
1. A method of aggregate analysis of logs, for use in a log processing service, the method comprising:
acquiring a plurality of first log data sent by an application service under the condition that the application service completes authentication; wherein the first log data includes corresponding metadata information;
under the condition that the first target log data is judged to need to be encrypted according to the metadata information corresponding to the first target log data in the first log data, encrypting the first target log data through a Flink encryption process to obtain second log data;
classifying the second log data according to the metadata information, and adding real-time state information corresponding to the classified second log data to acquire a plurality of third log data;
and respectively storing a plurality of third log data in different storage structures according to the service requirements of the third log data.
2. The method of claim 1, wherein after the obtaining the plurality of first log data sent by the application service, the method further comprises:
transmitting the first log data to a plurality of topics of a Kafka message queue according to the metadata information; the theme is obtained by dividing the metadata information in advance;
first log data is periodically obtained from the subject.
3. The method of claim 1, wherein before the application service completes authentication, the method comprises:
acquiring the metadata information sent by the application service; wherein the metadata information includes at least one of: an application service name, an application service version number, a server address of an application service;
confirming the version and the function of the application service according to the metadata information, and completing the registration of the application service;
and sending authentication success information to the application service.
4. The method of claim 1, wherein classifying the second log data according to the metadata information and adding real-time status information corresponding to the classified second log data, and obtaining a plurality of third log data includes:
classifying the second log data according to the metadata information to obtain a plurality of classified data; wherein the metadata information includes at least one of: the source of the log data, the type of the log data, the grade of the log data;
acquiring the real-time state information according to the classification data;
and acquiring the third log data according to the classification data, the metadata information corresponding to the classification data and the real-time state information.
5. The method of claim 1, wherein the storage structure comprises at least one of:
ClickHouse；
Elasticsearch；
Doris。
6. the method according to claim 5, wherein storing the plurality of third log data in different storage structures according to the service requirement of the third log data, respectively, comprises:
under the condition that the service requirement of the third log data is high in load requirement, the third log data is sent to a ClickHouse for storage;
and sending the third log data to an elastic search storage under the condition that the service requirement of the third log data is that the dynamic combination requirement is required for the query condition.
7. The method of claim 1, wherein after storing a plurality of the third log data in different storage structures, respectively, according to the traffic demand of the third log data, the method further comprises:
and reading the third log data in the storage structure through Grafana.
8. A log aggregation analysis device, for application to a log processing service, the device comprising:
the first acquisition module is used for acquiring a plurality of first log data sent by the application service under the condition that the authentication of the application service is completed; wherein the first log data includes corresponding metadata information;
the second acquisition module is used for encrypting the first target log data through an encryption flow of the link under the condition that the first target log data is judged to need to be encrypted according to the metadata information corresponding to the first target log data in the first log data, so as to acquire second log data;
the third acquisition module is used for classifying the second log data according to the metadata information, adding real-time state information corresponding to the second log data and acquiring a plurality of third log data;
and the first storage module is used for respectively storing a plurality of the third log data in different storage structures according to the service requirements of the third log data.
9. A network device, comprising: a transceiver, a processor, a memory, and a program or instructions stored on the memory and executable on the processor; the method according to any one of claims 1 to 7, wherein the processor, when executing the program or instructions, implements a log aggregation analysis method.
10. A readable storage medium having stored thereon a program or instructions which when executed by a processor performs the steps in the log aggregation analysis method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311753883.9A CN117591376A (en) | 2023-12-19 | 2023-12-19 | Log aggregation analysis method, device, network equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311753883.9A CN117591376A (en) | 2023-12-19 | 2023-12-19 | Log aggregation analysis method, device, network equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117591376A true CN117591376A (en) | 2024-02-23 |
Family
ID=89915198
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311753883.9A Pending CN117591376A (en) | 2023-12-19 | 2023-12-19 | Log aggregation analysis method, device, network equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117591376A (en) |
-
2023
- 2023-12-19 CN CN202311753883.9A patent/CN117591376A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10824525B2 (en) | Distributed data monitoring device | |
| CN112313916B (en) | Method and system for pseudo-storage of anti-tampering logs by fusing block chain technology | |
| US11228570B2 (en) | Safe-transfer exchange protocol based on trigger-ready envelopes among distributed nodes | |
| US11966797B2 (en) | Indexing data at a data intake and query system based on a node capacity threshold | |
| US12019634B1 (en) | Reassigning a processing node from downloading to searching a data group | |
| US8745179B2 (en) | Managing multiple application configuration versions in a heterogeneous network | |
| US20110099273A1 (en) | Monitoring apparatus, monitoring method, and a computer-readable recording medium storing a monitoring program | |
| US11892976B2 (en) | Enhanced search performance using data model summaries stored in a remote data store | |
| Sanjappa et al. | Analysis of logs by using logstash | |
| Essid et al. | Combining intrusion detection datasets using MapReduce | |
| US8073938B2 (en) | Information processing apparatus and method of operating the same | |
| US8510473B1 (en) | Converting message character sets for a queue manager | |
| CN103399943A (en) | Communication method and communication device for parallel query of clustered databases | |
| CN117591376A (en) | Log aggregation analysis method, device, network equipment and medium | |
| US11841827B2 (en) | Facilitating generation of data model summaries | |
| Silalahi et al. | A survey on logging in distributed system | |
| US8332537B1 (en) | Establishing a bypass channel between queue managers | |
| US8244746B2 (en) | Parallel linking system and parallel linking method | |
| CN105095248A (en) | Database cluster system, restoring method for same and management node of same | |
| US12013970B2 (en) | System and method for detecting and obfuscating confidential information in task logs | |
| US20230367887A1 (en) | System and method for updating a distributed ledger of a blockchain based on detecting anomalies in blockchain transactions | |
| US20230367636A1 (en) | System and method for determining memory resource configuration for network nodes to operate in a distributed computing network | |
| US11886229B1 (en) | System and method for generating a global dictionary and performing similarity search queries in a network | |
| CN115023925B (en) | Metric and event infrastructure | |
| Pakdaman | Real-time Video Analytics at Scale: Design, Deployment and Evaluation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |