CN112527618A - Log collection method and log collection system - Google Patents
Log collection method and log collection system Download PDFInfo
- Publication number
- CN112527618A CN112527618A CN202011499997.1A CN202011499997A CN112527618A CN 112527618 A CN112527618 A CN 112527618A CN 202011499997 A CN202011499997 A CN 202011499997A CN 112527618 A CN112527618 A CN 112527618A
- Authority
- CN
- China
- Prior art keywords
- log
- target service
- service
- target
- server node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000013473 artificial intelligence Methods 0.000 claims description 14
- 238000012544 monitoring process Methods 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 5
- 230000000977 initiatory effect Effects 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 7
- 230000000875 corresponding effect Effects 0.000 description 35
- 238000005516 engineering process Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 210000001503 joint Anatomy 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a log collection method and a log collection system, wherein the method comprises the following steps: receiving a starting request of a target service, and starting the target service according to a log theme corresponding to the target service; outputting a log generated in the running process of the target service to a preset log storage directory corresponding to the target service; each log comprises service operation unit information, and the service operation unit information comprises an identifier of a log theme corresponding to a target service; analyzing the identification and log data of the log theme of the newly added log according to the service operation unit information of the newly added log in the preset log storage directory; and sending the analyzed log data and the identification of the log subject to a Kafka message system. The log is collected by using the identification of the log subject, so that the subsequent use and processing complexity of the log can be reduced, and the application difficulty of the log can be reduced. In addition, the PaaS platform can directly interface with the Kafka message system, so that the log collection method has high practicability.
Description
Technical Field
The present application relates to the field of operation and maintenance technologies, and in particular, to a log collection method and a log collection system.
Background
The log refers to some information related to the running of the application program recorded in the running process of the application program. In the function service, the application program is only responsible for business logic, and does not consider the server and the related operation and maintenance part. Thus, the PaaS platform needs to collect logs for applications. In the existing technology, logs generated by running different application programs are mixed and collected together, so that the subsequent use and processing of the logs are very complicated. The method brings great difficulty to applications such as log retrieval, query and analysis, and is difficult to achieve clear and definite log authority management.
In the existing log collection technology, a Knative log system uses an ElasticSearch plug-in at a sending end, and the Knative log system can only send logs to an ElasticSearch engine. That is, the Knative journal system can only interface to the ElasticSearch engine. However, in practical application, logs are often collected by the Kafka message system, and the existing Knative log system cannot be interfaced with the Kafka message system, so that the collection scheme is poor in practicability.
Disclosure of Invention
Based on the above problems, the present application provides a log collection method and a log collection system, so as to reduce the complexity of subsequent use and processing of logs, reduce the application difficulty of logs, and improve the practicability of log collection.
The embodiment of the application discloses the following technical scheme:
in a first aspect, the present application provides a log collection method, which is applied to a PaaS platform and includes:
receiving a starting request of a target service, and starting the target service according to a log theme corresponding to the target service;
outputting a log generated in the running process of the target service to a preset log storage directory corresponding to the target service; each log comprises service operation unit information, and the service operation unit information comprises an identifier of a log theme corresponding to a target service;
analyzing the identification and log data of the log theme of the newly added log according to the service operation unit information of the newly added log in the preset log storage directory;
and sending the analyzed log data and the identification of the log subject to a Kafka message system.
Optionally, the target service is started on the target server node; a log agent module is deployed on the target server node;
analyzing the identification and the log data of the log theme of the newly added log according to the service operation unit information of the newly added log in the preset log storage directory, and specifically comprising the following steps:
monitoring a preset log storage directory through a log agent module, and when the condition that a newly added log exists in the preset log storage directory is monitored, analyzing an identifier of a log theme of the newly added log and log data by the log agent module according to service operation unit information of the newly added log;
sending the analyzed log data and the log subject identifier to a Kafka message system, which specifically comprises the following steps:
and sending the analyzed log data and the identification of the log subject to a Kafka message system through a log agent module.
Optionally, the log agent module is implemented by a fluent program; the fluent program uses the Kafka message system as the log output.
Optionally, the PaaS platform is implemented by a kubernets engine; the PaaS platform comprises at least one server node, and the at least one server node comprises a target server node; a copy service operation unit corresponding to the DemeonSet resource is respectively deployed on each server node in at least one server node; the replica service operation unit is used as a log proxy module on the server node where the replica service operation unit is located, and is used for monitoring logs of all service operation units on the server node.
Optionally, before receiving the start request for the target service, the method further includes:
receiving configuration information of a log theme corresponding to a target service; the configuration information of the log subject comprises an identifier of the log subject; identifying a target log topic;
configuring the target log theme as a log theme corresponding to the target service according to the configuration information;
starting the target service according to the log theme corresponding to the target service, which specifically comprises the following steps:
the target service is started with the target log topic.
Optionally, before receiving the start request for the target service, the method further includes:
receiving deployment parameters for a target service; the deployment parameters include: first identification information of the artificial intelligence model;
storing a mirror image of the artificial intelligence model; the mirror image of the artificial intelligence model is in one-to-one correspondence with the first identification information;
the initiation request includes: second identification information; after receiving the starting request of the target service, the method further comprises the following steps:
judging whether the second identification information is consistent with the first identification information, and if so, operating a mirror image of the artificial intelligence model; the mirror image is used for providing various services of the PaaS platform at the running time.
Optionally, the method further comprises:
a service request for a target service is received.
Optionally, the Kafka message system receives the log topic as a basis for performing any one of the following operations on the log data:
storage, querying, analysis, or rights control.
In a second aspect, the present application provides a log collection system, comprising: the PaaS platform is in communication connection with the Kafka message system;
the PaaS platform is used for receiving a starting request of the target service and starting the target service according to a log theme corresponding to the target service; outputting a log generated in the running process of the target service to a preset log storage directory corresponding to the target service; each log comprises service operation unit information, and the service operation unit information comprises an identifier of a log theme corresponding to a target service; analyzing the identification and log data of the log theme of the newly added log according to the service operation unit information of the newly added log in the preset log storage directory; sending the analyzed log data and the mark of the log subject to a Kafka message system;
the Kafka message system is used for receiving log data and log subject identification sent by the PaaS platform.
Optionally, the PaaS platform includes at least one server node; each server node in the at least one server node is respectively provided with a log agent module; the at least one server node comprises a target server node; the target service is started on the target server node;
and the log agent module is specifically used for monitoring the preset log storage directory, when the condition that the newly added log exists in the preset log storage directory is monitored, resolving the identification of the log subject and the log data of the newly added log according to the service operation unit information of the newly added log, and sending the resolved log data and the resolved identification of the log subject to the Kafka message system.
Compared with the prior art, the method has the following beneficial effects:
the application provides a log collection method and a log collection system, wherein the method is applied to a PaaS platform and comprises the following steps: receiving a starting request of a target service, and starting the target service according to a log theme corresponding to the target service; outputting a log generated in the running process of the target service to a preset log storage directory corresponding to the target service; each log comprises service operation unit information, and the service operation unit information comprises an identifier of a log theme corresponding to a target service; analyzing the identification and log data of the log theme of the newly added log according to the service operation unit information of the newly added log in the preset log storage directory; and sending the analyzed log data and the identification of the log subject to a Kafka message system. In the application, because the target service is started with the corresponding log theme, the log generated in the running process of the target service can carry the identifier of the log theme corresponding to the target service, and the log data and the identifier of the log theme can be obtained by analyzing the service running unit information of the log. And sending the log data and the representation of the log subject to a Kafka message system for collection, so that the Kafka can collect the subject identifier of the log generated by the target service, and constructing the association relation between the log of the target service and the subject identifier of the log. The log is collected by using the identification of the log subject, so that the subsequent use and processing complexity of the log can be reduced, and the application difficulty of the log can be reduced. In addition, the PaaS platform can be directly connected with the Kafka message system, so that the log collection method has high practicability.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic view of an application scenario of a log collection method according to an embodiment of the present application;
fig. 2 is a flowchart of a log collection method according to an embodiment of the present application;
FIG. 3 is a flowchart of another log collection method provided in an embodiment of the present application;
fig. 4 is a schematic structural diagram of a log collection system according to an embodiment of the present application.
Detailed Description
To facilitate understanding of the technical solutions of the present application, the meanings of some terms in the art related to the present application will be described first.
PaaS: refers to a platform, i.e., a service, that provides infrastructure (including networks, servers, operating systems, or storage) as a service to users, who do not need to manage and control the cloud infrastructure.
Kubernetes: a container orchestration engine for Google open sources.
Pod, Node and Kubernetes: kubernetes is a container arrangement engine for Google open source, Pod is a unit used by Kubernetes to run containers, Pod always runs on Node, Node is a working machine in Kubernetes, usually a virtual machine or a physical machine.
DemeonSet: kubernetes is a resource type. Kubernetes guarantees that only one copy Pod corresponding to the DemeonSet resource is run on each server node. In this patent, the replica is responsible for collecting a log of all Pod on this node.
Fluent: an open source data collector for a unified journal layer. The Fluentd adopts a pluggable framework, and the community provides dozens of data sources and data output plug-ins, so that different data sources and data storage systems are connected.
Kafka: is a distributed messaging system developed by Linked in that is widely used with horizontal scalability and high throughput.
Elastic search: a distributed, highly scalable, highly real-time search and data analysis engine.
As described above, the current Knative log system used in the log collection solution can only interface with the elastic search, and cannot interface with the Kafka message system, so that there is a problem of insufficient practicability. In addition, in the current technical scheme of log collection, the problems of log collection post-processing and inconvenient use exist.
Based on the above problems, the inventors have studied and provided a log collection method and a log collection system in the present application. By configuring the log theme, log data generated in the service operation process and the log theme are correlated, so that the complexity of processing and using after log collection is reduced. In addition, the PaaS platform for realizing the log collection method can be in butt joint with a Kafka message system, and the practicability of the technical scheme is improved.
In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Method embodiment
Fig. 1 is a schematic view of an application scenario of the log collection method according to the embodiment of the present application. As shown in fig. 1, the application scenario includes: developer users, end users, PaaS platforms, and Kafka systems. The PaaS platform deploys at least one server node, wherein the server node comprises a target server node supporting target service starting and running.
The PaaS platform can provide an interactive interface for the convenience of the developer user. A developer user can configure a respective log theme corresponding to each service on the PaaS platform. In one possible implementation, the same type of journal topic is configured for each service of an application. In practical application, the service log theme can be configured according to the experience of a developer user in log data classification, and can also be set by an application program for providing the service. Here with respect to log topic type restrictions for configuration.
After the log theme configuration of the target service is completed, the developer user may initiate a start request to the PaaS platform to request to start the target service. After receiving a starting request for the target service, the PaaS platform can start the target service according to the log theme corresponding to the target service.
The terminal user can be in communication connection with the PaaS platform through own terminal equipment. The terminal user can send a service request for the target service to the PaaS platform through the terminal equipment of the terminal user. And the target server node of the PaaS platform operates the target service according to the service request. The target service runtime generates a log. The target server node may output the log generated by the operation of the target service to a preset log storage directory corresponding to the target service in the target server node. In one possible implementation, each service has a corresponding preset log storage directory; in another possible implementation, a server node has only one preset log storage directory, and logs of all services running on the server node are output to the preset log storage directory.
Each server node on the PaaS platform is provided with a log agent module. The log agent module is used for monitoring a preset log storage directory on the server node, and when a log is newly added in the directory, the log agent module analyzes the identification of the log subject and the log data. And finally, the log agent module sends the analyzed identifier and the log data to the Kafka message system together, so that the Kafka message system collects the log data according to the identifier classification.
Fig. 2 is a flowchart of a log collection method according to an embodiment of the present application. The method takes a PaaS platform as an execution subject.
As shown in fig. 2, the log collection method includes:
s201: receiving a starting request of a target service, and starting the target service according to a log theme corresponding to the target service.
The developer user can input or select deployment parameters of the target service on the PaaS platform, and input or select configuration information of a log theme of the target service. The configuration information includes an identification of a log topic, the representation indicating a target log topic. For example, the configuration information includes a log topic identifier a, which indicates that the selected target log topic is topic a, and topic a is the target log topic. And the subsequent logs generated by starting the target service and running the target service take the topic A as the log topic.
The deployment parameters may include first identification information of the artificial intelligence model. A mirror image of the artificial intelligence model is stored in advance. The first identification information also corresponds to the mirror images of the artificial intelligence model one to one. If the second identification information included in the start request is consistent with the first identification information, the second identification information can be determined to correspond to the stored mirror image of the artificial intelligence model. That is, when identification information (e.g., ID) matching the image of the artificial intelligence model is provided in the boot request, the image of the artificial intelligence model may be run on the PaaS platform. And when the mirror image is operated, various services of the PaaS platform can be provided.
After receiving a starting request of a developer user for a target service, the PaaS platform can start the target service which the developer user requires to start according to a log theme configured by the developer user for the target service before and after the model mirror runs.
S202: and outputting the logs generated in the running process of the target service to a preset log storage directory corresponding to the target service.
Each generated log comprises service operation unit information (Pod information), and the service operation unit information comprises an identifier of a log subject corresponding to the target service. Since the log is output to the preset log storage directory, Pod information of the log is also located in the preset log storage directory.
S203: and analyzing the identification of the log theme and the log data of the newly added log according to the service operation unit information of the newly added log in the preset log storage directory.
This step may be performed in real time, for example, every time a log is added, S203 is performed. Of course, S203 may be executed periodically, for example, every 30 min. S203 may also be executed every new N pieces, where N is a positive integer. The execution frequency of S203 is not limited here.
Because the Pod information includes the identifier of the log topic corresponding to the target service, the identifier of the log topic can be analyzed according to the Pod information, and the log data can be analyzed.
S204: and sending the analyzed log data and the identification of the log subject to a Kafka message system.
In a possible implementation manner, the parsed log data and the log subject identifier of the log may be sent to the Kafka message system in a package or after being bound.
The above is a log collection method provided by the embodiment of the present application. In the log collection method provided by the application, because the target service is started by the corresponding log theme, the log generated in the running process of the target service can carry the identifier of the log theme corresponding to the target service, and the log data and the identifier of the log theme can be obtained by analyzing the service running unit information of the log. And sending the log data and the representation of the log subject to a Kafka message system for collection, so that the Kafka can collect the subject identifier of the log generated by the target service, and constructing the association relation between the log of the target service and the subject identifier of the log. The log is collected by using the identification of the log subject, so that the subsequent use and processing complexity of the log can be reduced, and the application difficulty of the log can be reduced. In addition, the PaaS platform can be directly connected with the Kafka message system, so that the log collection method has high practicability.
In a possible implementation manner of the application, a log proxy module is deployed on each server node of the PaaS platform. The log agent module is used for monitoring, analyzing and sending the newly added logs in the directory.
Therefore, the foregoing S203 specifically includes: monitoring a preset log storage directory through a log agent module, and when the condition that a newly added log exists in the preset log storage directory is monitored, analyzing an identifier of a log theme of the newly added log and log data by the log agent module according to service operation unit information of the newly added log;
the S204 specifically includes: and sending the analyzed log data and the identification of the log subject to a Kafka message system through a log agent module.
The log agent module may be implemented by a fluent program. Namely, a log agent module is built through a fluent program. The fluent program takes the Kafka message system as a log output end, namely the log agent module takes the Kafka message system as the log output end to output the analyzed log data and the log subject identification.
In a possible implementation, the PaaS platform is implemented by a kubernets engine, and is not limited to the kubernets engine. Each server node of the PaaS platform is respectively provided with a copy service operation unit Pod corresponding to the DemeonSet resource. The duplicate Pod is used as a log agent module on the server node where the duplicate Pod is located, and is used for monitoring logs of all service operation units on the server node.
Because the Kafka message system receives the log data analyzed by the PaaS platform and the identification of the log theme corresponding to the target service, the basis for storing, querying, analyzing or authority control can be carried out on the log data according to the log theme indicated by the identification.
For example, the log data is classified according to the identification A, B and C of the log topic, namely a log data A set, a log data B set and a log data C set. The log data A set, the log data B set and the log data C set are respectively stored and collected without mutual interference. The log data may be queried according to the identifier A, B and C when querying. When the log data needs to be analyzed, the analysis may be performed according to the subject classification of the log data to be analyzed. When a user only has the right to acquire the log data with the subject identification A, the log data B set and the log data C set are not open to the user.
FIG. 3 illustrates the flow of another log collection method. May be understood by reference to the description of the foregoing embodiments.
Based on the log collection method provided by the foregoing embodiment, correspondingly, the present application further provides a log collection system. The implementation of the system is described below with reference to the drawings and the embodiments.
System embodiment
Fig. 4 is a schematic structural diagram of a log collection system according to an embodiment of the present application. As shown in fig. 4, the log collection system includes: the PaaS platform is in communication connection with the Kafka message system;
the PaaS platform is used for receiving a starting request of the target service and starting the target service according to a log theme corresponding to the target service; outputting a log generated in the running process of the target service to a preset log storage directory corresponding to the target service; each log comprises service operation unit information, and the service operation unit information comprises an identifier of a log theme corresponding to a target service; analyzing the identification and log data of the log theme of the newly added log according to the service operation unit information of the newly added log in the preset log storage directory; sending the analyzed log data and the mark of the log subject to a Kafka message system;
the Kafka message system is used for receiving log data and log subject identification sent by the PaaS platform.
In the application, because the target service is started with the corresponding log theme, the log generated in the running process of the target service can carry the identifier of the log theme corresponding to the target service, and the log data and the identifier of the log theme can be obtained by analyzing the service running unit information of the log. And sending the log data and the representation of the log subject to a Kafka message system for collection, so that the Kafka can collect the subject identifier of the log generated by the target service, and constructing the association relation between the log of the target service and the subject identifier of the log. The log is collected by using the identification of the log subject, so that the subsequent use and processing complexity of the log can be reduced, and the application difficulty of the log can be reduced. In addition, the PaaS platform can directly interface with the Kafka message system, and the log collection system has high practicability.
Optionally, the PaaS platform includes at least one server node; each server node in the at least one server node is respectively provided with a log agent module; the at least one server node comprises a target server node; the target service is started on the target server node;
and the log agent module is specifically used for monitoring the preset log storage directory, when the condition that the newly added log exists in the preset log storage directory is monitored, resolving the identification of the log subject and the log data of the newly added log according to the service operation unit information of the newly added log, and sending the resolved log data and the resolved identification of the log subject to the Kafka message system.
Optionally, the log agent module is implemented by a fluent program; the fluent program uses the Kafka message system as the log output.
Optionally, the PaaS platform is further configured to receive configuration information of a log topic corresponding to the target service; the configuration information of the log subject comprises an identifier of the log subject; identifying a target log topic; configuring the target log theme as a log theme corresponding to the target service according to the configuration information;
the PaaS platform is specifically configured to start a target service with a target log topic.
Optionally, the PaaS platform is in communication connection with the user terminal equipment; the PaaS platform is also used for receiving a business request for the target service from the user terminal equipment.
Optionally, the Kafka message system receives the log topic as a basis for performing any one of the following operations on the log data:
storage, querying, analysis, or rights control.
The above description is only one specific embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A log collection method is applied to a PaaS platform and comprises the following steps:
receiving a starting request of a target service, and starting the target service according to a log theme corresponding to the target service;
outputting the logs generated in the running process of the target service to a preset log storage directory corresponding to the target service; each log comprises service operation unit information, and the service operation unit information comprises an identifier of a log theme corresponding to the target service;
analyzing the identification of the log theme and the log data of the newly added log according to the service operation unit information of the newly added log in the preset log storage directory;
and sending the analyzed log data and the identification of the log subject to a Kafka message system.
2. The log collection method of claim 1, wherein the target service is initiated on a target server node; a log agent module is deployed on the target server node;
the analyzing the identification and the log data of the log subject of the newly added log according to the service operation unit information of the newly added log in the preset log storage directory specifically comprises:
monitoring the preset log storage directory through the log agent module, and when monitoring that a newly added log exists in the preset log storage directory, analyzing the identification and log data of the log theme of the newly added log by the log agent module according to the service operation unit information of the newly added log;
the sending of the analyzed log data and the log subject identifier to the Kafka message system specifically includes:
and sending the analyzed log data and the log subject identification to a Kafka message system through the log agent module.
3. The log collection method of claim 2, wherein the log agent module is implemented by a fluent program; the fluent program takes the Kafka message system as a log output.
4. The log collection method of claim 2, wherein the PaaS platform is implemented by a kubernets engine; the PaaS platform comprises at least one server node, and the at least one server node comprises the target server node; a copy service operation unit corresponding to the DemeonSet resource is respectively deployed on each server node in the at least one server node; the replica service operation unit is used as a log proxy module on the server node where the replica service operation unit is located and is used for monitoring logs of all service operation units on the server node.
5. The log collection method of claim 1, further comprising, prior to said receiving a start request for a target service:
receiving configuration information of a log theme corresponding to the target service; the configuration information of the log topic comprises an identification of the log topic; the identification indicates a target log topic;
configuring the target log theme as a log theme corresponding to the target service according to the configuration information;
the starting the target service by using the log theme corresponding to the target service specifically includes:
and starting the target service by using the target log subject.
6. The log collection method of claim 5, further comprising, prior to said receiving a start request for a target service:
receiving deployment parameters for the target service; the deployment parameters include: first identification information of the artificial intelligence model;
storing a mirror image of the artificial intelligence model; the mirror image of the artificial intelligence model is in one-to-one correspondence with the first identification information;
the initiation request includes: second identification information; after the receiving the start request for the target service, the method further comprises:
judging whether the second identification information is consistent with the first identification information, if so, operating a mirror image of the artificial intelligence model; the mirror image is used for providing various services of the PaaS platform during operation.
7. The log collection method of claim 1, wherein the method further comprises:
and receiving a service request for the target service.
8. The log collection method of any one of claims 1 to 7, wherein the Kafka message system receives the log topic as a basis for performing any one of the following operations on the log data:
storage, querying, analysis, or rights control.
9. A log collection system, comprising: the system comprises a PaaS platform and a Kafka message system, wherein the PaaS platform is in communication connection with the Kafka message system;
the PaaS platform is used for receiving a starting request of a target service and starting the target service according to a log theme corresponding to the target service; outputting the logs generated in the running process of the target service to a preset log storage directory corresponding to the target service; each log comprises service operation unit information, and the service operation unit information comprises an identifier of a log theme corresponding to the target service; analyzing the identification of the log theme and the log data of the newly added log according to the service operation unit information of the newly added log in the preset log storage directory; sending the analyzed log data and the mark of the log subject to a Kafka message system;
the Kafka message system is used for receiving the log data and the identification of the log subject sent by the PaaS platform.
10. The log collection system of claim 9, wherein the PaaS platform comprises at least one server node; each server node in the at least one server node is respectively provided with a log agent module; the at least one server node comprises a target server node; the target service is started on a target server node;
the log agent module deployed on the target server node is specifically configured to monitor the preset log storage directory, when it is monitored that a newly added log exists in the preset log storage directory, analyze an identifier of a log topic and log data of the newly added log according to the service operation unit information of the newly added log, and send the analyzed log data and the identifier of the log topic to a Kafka message system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011499997.1A CN112527618A (en) | 2020-12-17 | 2020-12-17 | Log collection method and log collection system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011499997.1A CN112527618A (en) | 2020-12-17 | 2020-12-17 | Log collection method and log collection system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112527618A true CN112527618A (en) | 2021-03-19 |
Family
ID=75001228
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011499997.1A Pending CN112527618A (en) | 2020-12-17 | 2020-12-17 | Log collection method and log collection system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112527618A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113382071A (en) * | 2021-06-09 | 2021-09-10 | 北京猿力未来科技有限公司 | Link creation method and device based on hybrid cloud architecture |
| CN113419773A (en) * | 2021-05-21 | 2021-09-21 | 北京达佳互联信息技术有限公司 | Log file generation method and device, electronic equipment, storage medium and product |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108197233A (en) * | 2017-12-29 | 2018-06-22 | 飞狐信息技术（天津）有限公司 | A kind of data managing method, middleware and data management system |
| CN109525448A (en) * | 2019-01-10 | 2019-03-26 | 北京智信未来信息技术有限公司 | Log data acquisition system and method |
| CN111046011A (en) * | 2019-11-27 | 2020-04-21 | 中科曙光国际信息产业有限公司 | Log collection method, system, node, electronic device and readable storage medium |
| CN111124830A (en) * | 2019-12-24 | 2020-05-08 | 个体化细胞治疗技术国家地方联合工程实验室（深圳） | Monitoring method and device for micro-service |
| CN111930700A (en) * | 2020-07-13 | 2020-11-13 | 车智互联（北京）科技有限公司 | Distributed log processing method, server, system and computing equipment |
-
2020
- 2020-12-17 CN CN202011499997.1A patent/CN112527618A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108197233A (en) * | 2017-12-29 | 2018-06-22 | 飞狐信息技术（天津）有限公司 | A kind of data managing method, middleware and data management system |
| CN109525448A (en) * | 2019-01-10 | 2019-03-26 | 北京智信未来信息技术有限公司 | Log data acquisition system and method |
| CN111046011A (en) * | 2019-11-27 | 2020-04-21 | 中科曙光国际信息产业有限公司 | Log collection method, system, node, electronic device and readable storage medium |
| CN111124830A (en) * | 2019-12-24 | 2020-05-08 | 个体化细胞治疗技术国家地方联合工程实验室（深圳） | Monitoring method and device for micro-service |
| CN111930700A (en) * | 2020-07-13 | 2020-11-13 | 车智互联（北京）科技有限公司 | Distributed log processing method, server, system and computing equipment |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113419773A (en) * | 2021-05-21 | 2021-09-21 | 北京达佳互联信息技术有限公司 | Log file generation method and device, electronic equipment, storage medium and product |
| CN113419773B (en) * | 2021-05-21 | 2023-11-28 | 北京达佳互联信息技术有限公司 | Log file generation method and device, electronic equipment, storage medium and product |
| CN113382071A (en) * | 2021-06-09 | 2021-09-10 | 北京猿力未来科技有限公司 | Link creation method and device based on hybrid cloud architecture |
| CN113382071B (en) * | 2021-06-09 | 2022-09-06 | 北京猿力未来科技有限公司 | Link creation method and device based on hybrid cloud architecture |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107431651B (en) | Life cycle management method and equipment for network service | |
| CN107704360B (en) | Monitoring data processing method, equipment, server and storage medium | |
| US11539590B2 (en) | Detect impact of network maintenance in software defined infrastructure | |
| CN105052076B (en) | Network element management system and network element management method based on cloud computing | |
| US20220052923A1 (en) | Data processing method and device, storage medium and electronic device | |
| CN114741060B (en) | Business system development method and device based on middle platform | |
| CN112350854B (en) | Flow fault positioning method, device, equipment and storage medium | |
| CN102387043B (en) | Alarm analysis method, workstation and system based on simple network management protocol | |
| CN112527618A (en) | Log collection method and log collection system | |
| CN109684038B (en) | Docker service container log processing method and device and electronic equipment | |
| CN102457390A (en) | Fault location method and system based on quality of experience (QoE) | |
| US11601354B2 (en) | Distributed packet capture | |
| CN113055421A (en) | Service grid management method and system | |
| CN113626002A (en) | Service execution method and device | |
| CN112330519A (en) | Data processing method and device | |
| CN113595832A (en) | Network data acquisition system and method | |
| CN112612802A (en) | Real-time data middlebox processing method, device and platform | |
| GB2598544A (en) | Network slice analytics | |
| Spillner et al. | Co-transformation to cloud-native applications: development experiences and experimental evaluation | |
| CN114500530B (en) | Automatic adjustment method for civil edge information system | |
| Coppolino et al. | Effective QoS monitoring in large scale social networks | |
| KR100358156B1 (en) | Converting Method of Managing Operation from service management system to Switching Command in a Integrated Network | |
| CN109167826B (en) | Method, device and system for putting WEB application on shelf | |
| CN112650815A (en) | Method and device for synchronizing environmental data, storage medium and electronic device | |
| CN113760836B (en) | Wide table calculation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |