CN117978633A

CN117978633A - Log acquisition method, system, equipment and storage medium based on registry

Info

Publication number: CN117978633A
Application number: CN202410142912.6A
Authority: CN
Inventors: 张庆国; 朱浩; 李文塔; 李如先; 许佳
Original assignee: Shenzhen Qianhai Huanrong Lianyi Information Technology Service Co Ltd
Current assignee: Shenzhen Qianhai Huanrong Lianyi Information Technology Service Co Ltd
Priority date: 2024-01-31
Filing date: 2024-01-31
Publication date: 2024-05-03

Abstract

The application provides a log acquisition method, a system, equipment and a storage medium based on a log, wherein the method is applied to a log service in a log acquisition system, and the log acquisition system also comprises a Fluent Bit service instance and a plurality of Fluentd service instances; the registry service maintains a registered Fluentd service instance set, including: receiving Fluentd service address inquiry requests sent by Fluent Bit service examples; and responding to Fluentd service address inquiry requests, acquiring service address information in the registered Fluentd service instance set and sending the service address information to the Fluent Bit service instance, so that the Fluent Bit service instance determines first target service address information and performs address configuration so as to enable the corresponding Fluentd service instance to be responsible for aggregation processing of target log data. The application improves the fault resistance of the log acquisition system.

Description

Log acquisition method, system, equipment and storage medium based on registry

Technical Field

The present application relates to the field of log collection technologies, and in particular, to a log collection method, system, device and storage medium based on a registry.

Background

With the popularity of cloud-native technology and containerization, applications are often deployed and run in the form of containers. A typical cloud native application is typically made up of a plurality of micro-services, each of which may contain a plurality of containers, each of which may generate a large number of logs, the collection of container logs may assist the developing operations and maintenance personnel in troubleshooting, monitoring and performance analysis, security and compliance auditing, and log analysis and visualization, among other things.

In the related log collection scheme, normally, a Fluent Bit is deployed and operated on a log data source such as a container, a host, an edge device and the like as a log collection Agent to collect logs, and a Fluentd is deployed and operated as a log aggregation Aggregator node to receive log data sent by the Fluent Bit and aggregate and perform subsequent processing. In this implementation Fluentd is running as a centralized log aggregation and processing tool and single point, making the log collection system less resistant to Fluentd node failures.

Disclosure of Invention

The embodiment of the application provides a log acquisition method, a system, equipment and a storage medium based on a registry, which can improve the resistance of a log acquisition system to the occurrence of faults of Fluentd nodes and is beneficial to ensuring the normal operation of a log acquisition flow.

In a first aspect, an embodiment of the present application provides a registry-based log collection method, which is applied to a registry service in a log collection system, where the log collection system further includes a Fluent Bit service instance and a plurality of Fluentd service instances; a registered Fluentd service instance set is maintained in the registry service, wherein the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is currently in a normal running state and is registered in the registry service in advance when being started; the method comprises the following steps:

Receiving Fluentd service address inquiry requests sent by the Fluent Bit service instance; the Fluentd service address query request is sent by the Fluent Bit service instance before log acquisition is carried out;

responding to the Fluentd service address inquiry request, and acquiring one or more pieces of service address information in the registered Fluentd service instance set;

And sending the one or more pieces of service address information to the Fluent Bit service instance, so that the Fluent Bit service instance determines first target service address information in the one or more pieces of service address information, and configures the first target service address, so that the Fluentd service instance corresponding to the first target service address receives target log data from the Fluent Bit service instance and carries out log aggregation processing when the Fluent Bit service instance carries out log acquisition.

In a second aspect, the embodiment of the application provides another log collection method based on a registry, which is applied to a Fluent Bit service instance in a log collection system, wherein the log collection system further comprises a registry service and a plurality of Fluentd service instances; a registered Fluentd service instance set is maintained in the registry service, wherein the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is currently in a normal running state and is registered in the registry service in advance when being started; the method comprises the following steps:

Before log collection is carried out on a log data source, a Fluentd service address query request is sent to the registry service, so that the registry service obtains one or more service address information in the registered Fluentd service instance set;

receiving the one or more service address information sent by the registry service, and determining first target service address information from the one or more service address information;

address configuration is carried out in a configuration file by utilizing the first target service address information;

and carrying out log acquisition on a log data source, and sending acquired target log data to the Fluentd service instance corresponding to the first target service address information according to the first target service address information in the configuration file so as to carry out log aggregation processing on the target log data by the Fluentd service instance.

In a third aspect, an embodiment of the present application provides a log collection device based on a registry, where the device includes a unit for executing the log collection method based on a registry in the first aspect, or includes a unit for executing the log collection method based on a registry in the second aspect.

In a fourth aspect, the embodiment of the application also provides another log collection method based on a registry, which is applied to a log collection system, wherein the log collection system comprises a registry service, a Fluent Bit service instance and a plurality of Fluentd service instances; a registered Fluentd service instance set is maintained in the registry service, wherein the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is currently in a normal running state and is registered in the registry service in advance when being started; the method comprises the following steps:

The Fluentd service instance sends registration request information to the registry service when starting; the registration request information carries the service address information corresponding to the Fluentd service instance;

The registry service receives and responds to the registration request information, adds the service address information into the registered Fluentd service instance set, and monitors the running state of the Fluentd service instance;

Before log acquisition is carried out on a log data source, the Fluent Bit service instance sends Fluentd a service address query request to the registry service;

The registry service receives and responds to the Fluentd service address query request, acquires one or more service address information in the registered Fluentd service instance set and sends the service address information to the Fluent Bit service instance;

The Fluent Bit service instance receives the one or more service address information sent by the registry service, determines first target service address information from the one or more service address information, and performs address configuration in a configuration file by using the first target service address information;

The Fluent Bit service instance performs log acquisition on a log data source, and sends acquired target log data to the Fluentd service instance corresponding to the first target service address information according to the first target service address information in the configuration file;

The Fluentd service instance receives the target log data from the Fluent Bit service instance and carries out log aggregation processing on the target log data.

In a fifth aspect, an embodiment of the present application provides a log collection system, where the log collection system includes a registry service, a Fluent Bit service instance, and a plurality of Fluentd service instances; the registry service maintains a registered Fluentd service instance set, wherein the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is currently in a normal running state and is registered in the registry service in advance when being started; wherein,

The Fluentd service instance is configured to send registration request information to the registry service when starting, where the registration request information carries the service address information corresponding to the Fluentd service instance;

the registry service is used for receiving and responding to the registration request information, adding the service address information into the registered Fluentd service instance set, and monitoring the running state of the Fluentd service instance;

the Fluent Bit service instance is used for sending Fluentd service address inquiry requests to the registry service before log acquisition is carried out on a log data source;

The registry service is further configured to receive and respond to the Fluentd service address query request, obtain one or more service address information in the registered Fluentd service instance set, and send the service address information to the Fluent Bit service instance;

The Fluent Bit service instance is further configured to receive the one or more service address information sent by the registry service, determine first target service address information from the one or more service address information, and perform address configuration in a configuration file by using the first target service address information;

The Fluent Bit service instance is further configured to perform log collection on a log data source, and send collected target log data to the Fluentd service instance corresponding to the first target service address information according to the first target service address information in the configuration file;

The Fluentd service instance is further configured to receive the target log data from the Fluent Bit service instance, and perform log aggregation processing on the target log data.

In a sixth aspect, an embodiment of the present application further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the registry-based log collection method described in the first aspect when executing the computer program, or implements the registry-based log collection method described in the second aspect when executing the computer program.

In a seventh aspect, an embodiment of the present application further provides a computer readable storage medium, where the computer readable storage medium stores a computer program, where the computer program when executed by a processor causes the processor to perform the registry-based log collection method described in the first aspect, or where the computer program when executed by the processor causes the processor to perform the registry-based log collection method described in the second aspect.

Compared with the prior art, on one hand, the log acquisition system provided by the application is provided with a plurality of Fluentd service instances for log aggregation, so that the problem that the log acquisition processing of the whole system is affected due to the fact that only a single Fluentd service instance breaks down during operation is avoided, the aggregation processing of a large amount of log acquisition data is shared, the influence of the faults of a single Fluentd service instance on the system is dispersed, and the resistance of the log acquisition system to the faults of Fluentd nodes is improved; on the other hand, the registration center service for registering Fluentd service instances is introduced into the log collection system, and maintains a registered Fluentd service instance set comprising service address information corresponding to the Fluentd service instances which are currently in a normal running state and are registered in advance at the starting time, so that before log collection is carried out on the Fluent Bit service instances, a Fluentd service address query request is sent to the registration center service to acquire first target service address information in the registered Fluentd service instance set, configuration of the first target service address is carried out, and then the Fluentd service instance corresponding to the first target service address is responsible for receiving target log data from the Fluent Bit service instances and carrying out log aggregation processing, thereby realizing dynamic management on Fluentd service instances available in the log collection system, ensuring that the Fluent Bit service instances can acquire Fluentd service instances in the normal running state to be responsible for subsequent log collection processing, and being beneficial to ensuring normal operation of log collection.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of an application scenario of a log acquisition system according to an embodiment of the present application;

fig. 2 is a first interaction schematic diagram of a log collection method based on a registry according to an embodiment of the present application;

FIG. 3 is another interactive schematic diagram of a log collection method based on a registry according to an embodiment of the present application;

FIG. 4 is a schematic block diagram of a first registry-based log collection device according to an embodiment of the present application;

FIG. 5 is a schematic block diagram of a second registry-based log collection device provided by an embodiment of the present application;

Fig. 6 is a schematic block diagram of a computer device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

It should be understood that the terms "comprises" and "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

The application provides a log acquisition method, a related system, equipment and a storage medium based on a registry, which can improve the resistance of a log acquisition system to the occurrence of faults of Fluentd nodes and are beneficial to ensuring the normal operation of a log acquisition flow. The execution main body of the log collection method based on the registry can be the registry service provided by the embodiment of the application, or the Fluent Bit service instance provided by the embodiment of the application, or the log collection system provided by the application; the registry service may be integrated with a computer device, or the registry service may be integrated with a Fluent Bit service instance. The registry service, fluent Bit service instance and the log acquisition system can be realized in a hardware or software mode, the computer equipment can be a terminal or a server, and the terminal can be a smart phone, a tablet computer, a palm computer, a notebook computer or the like.

For example, referring to fig. 1, fig. 1 is a schematic view of an application scenario of a log collection system according to an embodiment of the present application. The log collection method based on the registry provided by the application can be particularly applied to the log collection system shown in fig. 1. Specifically, the log collection system adopts a micro-service architecture, and specifically includes Fluent Bit service instances (in the figure, 2 Fluent Bit service instances A1 and A2 are taken as examples) for collecting logs of log data sources (for example, containers, hosts and edge devices), and a plurality of Fluentd service instances (in the figure, 2 Fluentd service instances B1 and B2 are taken as examples) for carrying out log aggregation processing on log data (target result data described below) from the Fluent Bit service instances.

It can be seen that the log collection system provided by the present application further includes a registry service in which a registered Fluentd service instance set is maintained. Specifically, when the Fluentd service instance is started, a registration request is sent to a registry service, and the registry service adds the service address information of the Fluentd service instance into a registered Fluentd service instance set, and continuously monitors the running state of Fluentd service instances corresponding to the service address information in the registered Fluentd service instance set, so as to ensure that the service address information of the Fluentd service instance in the normal running state is stored in the registered Fluentd service instance set. Before log collection is performed on the log data source, the Fluent Bit service instance firstly acquires an available Fluentd service address from the registered Fluentd service instance set from the registry service, and then configures by utilizing the Fluentd service address, so that when the Fluent Bit service instance performs log collection on the log data source, the Fluentd service instance corresponding to the Fluentd service address is responsible for performing log aggregation processing on the collected log data.

The Fluent Bit is lightweight, quick and reliable log collection and processing software, supports various input and output plugins, and is suitable for application programs in various cloud primary environments. The method adopts the C language and the Lua script for development, and can be integrated with a storage and processing system such as ELASTICSEARCH, INFLUXDB, KAFKA and the like to realize efficient log data processing and visual analysis.

Fluentd is an open source data collection, stream processing and data delivery software that can be used to collect and process a variety of structured and unstructured data. The design concept is to collect data from different sources, such as log files, message queues, network data and the like, into a central data pool, and transmit the data to different destinations after processing and processing.

The registry refers to a centralized store or service responsible for managing and storing service instance information that provides APIs and interfaces for service providers to register, unregister, and service consumers to discover available service instances.

Specifically, referring to fig. 2, fig. 2 is an interactive flow diagram of a log collection method based on a registry according to an embodiment of the present application, where the log collection method based on the registry may be applied to the log collection system, and the log collection system includes a registry service, a Fluent Bit service instance, and a plurality of Fluentd service instances; the registry service maintains a registered Fluentd service instance set, wherein the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is currently in a normal running state and is registered in the registry service in advance when being started; the log collection method based on the registry specifically comprises the following steps S101 to S108.

Step S101, a target Fluentd service instance sends registration request information to the registry service when starting; the registration request information carries the service address information corresponding to the service instance of the target Fluentd.

Wherein the target Fluentd service instance is any one or more of the Fluentd service instances. In this embodiment, each target Fluentd service instance sends the above-mentioned registration request information to the registry service when it is started.

In this embodiment, the registry service is configured to provide a registry service to Fluentd service instances (corresponding to step S101 to step S102 described below), and provide a service discovery function to Fluent Bit service instances (corresponding to step S103 to step S105 described below). In implementations, the registry service may be implemented based on etcd, consul, zooKeeper or the like.

Specifically, the above registration request information sent by the target Fluent Bit service instance is used for the target Fluentd service instance to register itself in the registry service, which indicates that the target Fluentd service instance starts to enter into an available state, and may provide a log aggregation service. The registration request information carries service address information corresponding to the target Fluentd service instance, and the service address information can be in communication connection with the target Fluentd service instance, that is, the Fluent Bit service instance can send the collected log data to the target Fluentd service instance according to the service address information. In a specific embodiment, the service address information may specifically include an IP address and a port number corresponding to the service instance of the destination Fluentd.

Step S102, the registry service receives and responds to the registration request information, adds the service address information to the registered Fluentd service instance set, and monitors the running state of the target Fluentd service instance.

In this embodiment, after receiving the target registration request information from the Fluentd service instance, the registry service adds the service address information to the registered Fluentd service instance set in response to the registration request information, and starts to monitor the running state of the target Fluentd service instance.

The set of registered Fluentd service instances maintained by the registry service includes service address information corresponding to a target Fluentd service instance that is registered and still currently in a normal running state. In implementation, for each service address information in the registered Fluentd service instance set, the registry service continuously monitors whether the target Fluentd service instance corresponding to the service address information is in a normal running state. If the target Fluentd service instance is in a normal running state, it may indicate that the target Fluentd service instance can normally provide the log aggregation service, that is, is in an available state, and if the target Fluentd service instance cannot normally provide the log aggregation service, that is, is in an unavailable state, then the target Fluentd service instance is considered to be in an abnormal running state.

In the specific monitoring of the target Fluentd service instance, the registry service may send a network connection request to the target Fluentd service instance through the service address information, and if no response data sent by the target Fluentd service instance for the network connection request is received, determine that the target Fluentd service instance is in an abnormal running state.

To ensure that the registered Fluentd service instance set includes service address information corresponding to a target Fluentd service instance in a normal operating state, in one embodiment, when a target Fluentd service instance with an abnormal operating state is detected, the service address information corresponding to the target Fluentd service instance is deleted from the registered Fluentd service instance set.

Step S103, before log collection is carried out on the log data source, the Fluent Bit service instance sends Fluentd service address query requests to the registry service.

The log data source is a host, a device or a container which needs to perform log collection and generates log data, and the Fluent Bit service instance can be deployed in the log data source when the method is implemented; the Fluentd service address query request is used for acquiring service address information corresponding to the available Fluentd service instance in the registry service. In this embodiment, before log collection is performed on the log data source, the Fluent Bit service instance sends Fluentd a service address query request to the registry service.

Step S104, the registry service receives and responds to the Fluentd service address query request, acquires one or more service address information in the registered Fluentd service instance set, and sends the service address information to the Fluent Bit service instance.

The one or more service address information may be all service address information in the registered Fluentd service instance set, or one or more service address information determined by the registration service center after screening based on a preset service address screening policy. For example, to achieve load balancing, in some embodiments, the service address filtering policy may determine one or more service address information for a respective corresponding current number of uses for each service address information in the registered Fluentd service instance set.

Step 105, the Fluent Bit service instance receives the one or more service address information sent by the registry service, determines first target service address information from the one or more service address information, and performs address configuration in a configuration file by using the first target service address information.

In specific implementation, the Fluent Bit service instance may determine the first target service address information from the one or more service address information based on a preset service address selection policy. For example, the Fluent Bit service instance may perform network testing on the target Fluentd service instances corresponding to one or more service address information respectively, including delay time testing, packet loss rate testing, and the like, and perform comprehensive ranking according to the test result, so as to determine the service address information corresponding to the highest-ranking target Fluentd service instance as the first target service address information.

In this embodiment, the Fluent Bit service instance uses the first target service address information to perform address configuration in the configuration file, which represents that log data (target log data described below) generated after the subsequent collection and processing by the Fluent Bit service instance is all sent to the target Fluentd service instance corresponding to the first target service address information, and then the target Fluentd service instance performs processing such as log aggregation on the log data, after the address configuration in the configuration file is completed, when the Fluent Bit service instance needs to send the log data to the Fluentd service instance, the Fluent Bit service instance reads its configuration file, finds the service address of the Fluentd service instance, and then sends the log data to the service address, and then the target Fluentd service instance corresponding to the service address can receive the log data.

In this embodiment, the Fluent Bit service instance is used for log collection, the Fluentd service instance is used for log aggregation, and in order to avoid the log aggregation process of the Fluentd service instance, in an embodiment, the Fluentd service instance may specifically include a Fluentd service instance main thread and a SideCar service, and the Fluentd service instance main thread is used for executing a specific operation of log aggregation processing, that is, for executing the following step S108, and the SideCar service is used for executing the rest of auxiliary operations, that is, the step S101.

Wherein SideCar is an architecture model, which is commonly used in a micro-service architecture, and is used for expanding and enhancing functions of services, and has the following functions:

1. the function expansion and disaggregation, by extracting the specific function into SideCar, can keep the main service application simple, and can conveniently add, update or delete the function without affecting the stability of the main service.

2. Monitoring and tracking: sideCar may be responsible for collecting monitoring data, distributed trace data, and logs, providing real-time performance monitoring and troubleshooting capabilities for the entire micro-service architecture.

3. The safety is enhanced: security functions such as authentication, authorization, encryption, etc. can be separated through SideCar, ensuring that the primary service only needs to pay attention to the core business logic.

4. A communication agent: sideCar can be used as a communication proxy to handle inter-service communication, load balancing, retry, and failover, thereby alleviating the communication burden of the primary service.

5. Data conversion and adaptation: sideCar can handle data format conversion, protocol adaptation and the like, and ensure seamless cooperation between different services.

In order to avoid affecting the log collection process of the Fluent Bit service instance, in an embodiment, the Fluent Bit service instance may specifically include a Fluent Bit service instance main thread and SideCar services, where the Fluent Bit service instance main thread is used to perform specific operations of log collection and sending, that is, to perform the following steps S106 and S107, sideCar services are used to perform the remaining auxiliary operations, that is, the above steps S103 and S105.

Further, in order to ensure that the configured Fluentd service instance is in a normal running state, in an embodiment, after the step S105, the method may further include:

sending a network connection request to the Fluentd service instance corresponding to the first target service address information at preset time intervals through the SideCar service;

if the response data of the Fluentd service instance corresponding to the first target service address information for the network connection request is not received, sending service address unavailable information to the registry service through the SideCar service, and acquiring third target service address information from the registered Fluentd service instance set;

And performing address updating configuration in the configuration file by utilizing the third target service address information through the SideCar service.

The service address unavailable information indicates that the Fluentd service instance corresponding to the first target service address information configured by the Fluent Bit service instance cannot normally process log data; the third target service address information is service address information of the registered Fluentd service instance set different from the first target service address information. In this embodiment, after receiving the service address unavailable information, the registry service may delete the first target service address information in the registered Fluentd service instance set, and acquire one or more service address information from the first target service address information and send the service address information to the Fluent Bit service instance, where the Fluent Bit service instance further determines third target service address information from the one or more service address information and performs address configuration.

In this embodiment, when the Fluentd service instance is in a normal running state, if a network connection request is received, response data is sent to a network connection request sender, and by using SideCar service in a Fluent Bit service instance to monitor the running state of the Fluentd service instance corresponding to the first target service address information by sending the network connection request at preset time intervals, when no response data is received, service address unavailable information is sent to a registry service, further third target service address information from the registered Fluentd service instance set is obtained, and address update configuration is performed by using the third target service address information in the configuration file through the SideCar service, so that active state monitoring of the Fluentd service instance for log aggregation is realized, fluentd service instances in a normal running state are conveniently replaced in time when the Fluentd service instance is in an abnormal running state, and after log aggregation of log data collected by the Fluent Bit service instance is ensured, normal process of log acquisition of the Fluent Bit service instance is ensured.

And S106, the Fluent Bit service instance performs log acquisition on the log data source to obtain target log data.

The target log data is data obtained by performing log collection and log processing operations on a log data source by using a Fluent Bit service instance. In particular implementations, log processing may include log filtering, log marking, and the like.

In a specific implementation, the Fluent Bit service instance may be configured with an input plug-in, a filter plug-in and an output plug-in, where the input plug-in is used to collect log data, the filter plug-in is used to filter, mark and the like the collected log data, and the output plug-in is used to send result data obtained after the filter plug-in is processed to the Fluentd service instance.

Step S107, sending the collected target log data to the Fluentd service instance corresponding to the first target service address information according to the first target service address information in the configuration file.

In this embodiment, after the Fluent Bit service instance finishes processing to obtain the target log data, the target log data is sent to the Fluentd service instance (i.e., the target Fluentd service instance) corresponding to the first target service address information according to the first target service address information in the configuration file.

Step S108, the target Fluentd service instance receives the target log data from the Fluent Bit service instance, and performs log aggregation processing on the target log data.

Note that Fluentd service instance is configured to receive target log data from multiple Fluent Bit service instances, where the above-mentioned processing of log aggregation on the target log data refers to that Fluentd service instance aggregates all the target log data together and performs related processing.

In a specific implementation, an input plug-in, a filter plug-in and an output plug-in may be configured in the target Fluentd service instance, where the input plug-in is configured to receive target log data from each Fluent Bit service instance, and the filter plug-in is configured to perform further processing such as conversion and structuring on each target log data; the output plug-in is used for sending and storing result data obtained after processing each target log data to a preset log storage position, such as Kafka, elasticsearch, S, mongoDB and the like.

In order to update the registered Fluentd service instance set in time, to ensure that the Fluent Bit service instance obtains the Fluentd service instance in the normal running state (available state), in an embodiment, after the step S102, the method may further include the step that if the registration service center monitors that the target Fluentd service instance is in the abnormal running state, deleting the service address information of the target Fluentd service instance from the registered Fluentd service instance set.

In this embodiment, the service address information of the target Fluentd service instance is collected from the registered Fluentd service instance, which may indicate that the registration state of the target Fluentd service instance in the registry service is changed to an unregistered state, and when the target Fluentd service instance resumes the normal running state, the registry service needs to send registration request information to the registry service again for registration, and then the registry service adds the service address information of the target Fluentd service instance into the registered Fluentd service instance collection again, and continuously monitors the running state of the target Fluentd service instance.

In a specific implementation, after the service address information of the target Fluentd service instance is deleted from the registered Fluentd service instance set, the registration service center may notify that the registration state of the target Fluentd service instance is changed to unregistered, so that the target Fluentd service instance knows the registration state change, and registers with the registry service again when the normal running state is restored or restarted.

In order to timely notify the Fluent Bit service instance of the target Fluentd service instance configured with the abnormal running state, the influence on the log collection flow of the Fluent Bit service instance is reduced, in an embodiment, after the registry service deletes the service address information of the target Fluentd service instance from the registered Fluentd service instance set, the following steps S201 to S204 may be further included.

Step 201, the registry service determines whether there is a target Fluent Bit service instance configured with the service address information of the target Fluentd service instance.

The target Fluent Bit service instance is a Fluent Bit service instance configured with the service address information of the target Fluentd service instance with abnormal running state. In a specific implementation, the registry service may record configuration status information corresponding to each service address information in the registered Fluentd service instance set, where the configuration status information indicates each Fluent Bit service instance in which the corresponding service address information is configured, and after deleting the service address information of the target Fluentd service instance from the registered Fluentd service instance set, it may determine whether one or more target Fluent Bit service instances exist based on the configuration status information corresponding to the deleted service address information.

To illustrate this process, please refer to fig. 3. Taking the example of the log collection system including two target Fluentd service instances in fig. 3, the first target Fluentd service instance and the second target Fluentd service instance register with the registry service in sequence, and after the registry service adds the service address information of the corresponding target Fluentd service instance to the registered Fluentd service instance set, the registry service starts to monitor the registered target Fluentd service instances. After the registration of the first target Fluentd service instance is completed, the Fluent Bit service instance determines a service address corresponding to the first target Fluentd service instance, namely first target service address information, by sending Fluentd service address query requests to the registry service, and performs address configuration, after that, the Fluent Bit service instance performs log acquisition on a log data source, sends the obtained target log data to the first target Fluentd service instance, and the first target Fluentd service instance performs log aggregation processing on the target log data from the Fluent Bit service instance.

After that, the registry service monitors that the running state of the service instance of the first target Fluentd is abnormal, deletes the service address information of the service instance of the first target Fluentd, that is, the first target service address information, from the registered Fluentd service instance set, and performs the above step S201, to determine whether there is a target Fluent Bit service instance in which the service address information of the service instance of the first target Fluentd has been configured.

Step S202, when the target Fluent Bit service instance exists, the registry service sends Fluentd service instance replacement notification information to the target Fluent Bit service instance.

The service instance replacement notification information of Fluentd indicates that the target Fluent Bit service instance changes Fluentd service instance. In a specific implementation, the service instance replacement notification information Fluentd may specifically instruct the target Fluent Bit service instance to send the Fluentd service address query request to the registry service again, and re-acquire new service address information (the second target service address information described below) from the registered Fluentd service instance set based on the response of the registry service, and perform address configuration (the same as the above steps S103 to S105); in some embodiments, the service instance replacement notification information Fluentd may also be one or more pieces of service address information that carry the registry service to retrieve from the registered Fluentd service instance set, so that the target Fluent Bit service instance may redetermine new service address information for configuration (the second target service address information described below) from the one or more pieces of service address information.

Step S203, the target Fluent Bit service instance receives and responds to the Fluentd service instance replacement notification information, and obtains second target service address information from the registered Fluentd service instance set.

Wherein the second target service address information is service address information different from the first target service address information in the registered Fluentd service instance set.

In a specific implementation, if Fluentd service instance replacement notification information indicates that the target Fluent Bit service instance sends a Fluentd service address query request to the registry service again to retrieve new service address information for configuration, after receiving the Fluentd service instance replacement notification information, the target Fluent Bit service instance may acquire second target service address information from the registered Fluentd service instance set by sending Fluentd service address query request to the registry service (the process of specifically acquiring the second target service address information may refer to the corresponding process in steps S103 to S105); if Fluentd service instance replacement notification information is one or more service address information that is acquired by the registry service from the registered Fluentd service instance set, the target Fluent Bit service instance may acquire second target service address information from the one or more service address information (for implementation, refer to the corresponding description in step S105).

As shown in fig. 3, the target Fluent Bit service instance responds to the Fluentd service instance replacement notification information to obtain second target service address information from the registered Fluentd service instance set, where the second target service address information is service address information corresponding to the second target Fluentd service instance.

Step S204, the target Fluent Bit service instance performs address update configuration in the configuration file by using the second target service address information, so that the target Fluentd service instance corresponding to the second target service address information receives the target log data from the target Fluent Bit service instance and performs log aggregation processing.

In this embodiment, after the target Fluent Bit service instance obtains the latest service address information, that is, the second target service address information may be set to the configuration address (corresponding to Fluentd service instances) in the configuration file, so that the target Fluentd service instance corresponding to the second target service address information receives the target log data from the target Fluent Bit service instance and performs log aggregation processing.

As shown in fig. 3, the target Fluent Bit service instance uses the second target service address information to perform address update configuration in the configuration file, and then, the second target Fluentd service instance corresponding to the second target service address information performs aggregation processing on the target log data from the target Fluent Bit service instance.

In a specific implementation, in order to validate the address update configuration, after the address update configuration is performed, the target log data of the target Fluent Bit service instance may be implemented by restarting the target Fluent Bit service instance, and then the target log data of the target Fluent Bit service instance is sent to the target Fluentd service instance corresponding to the second target service address information according to the second target service address information configured recently.

In order to avoid affecting the log collection process of the Fluent Bit service instance, in an embodiment, the Fluent Bit service instance may specifically include a Fluent Bit service instance main thread and SideCar services, and the above steps S203 and S204 may specifically be executed by the SideCar service.

In this embodiment, after determining that the target Fluentd service instance with an abnormal running state exists, the registry service deletes the service address information corresponding to the target Fluentd service instance from the registered Fluentd service instance set, further determines whether the target Fluent Bit service instance with the deleted service address information is existed, when the target Fluent Bit service instance exists, the target Fluent Bit service instance is enabled to redetermine the second target service address information from the registered Fluentd service instance set and configure the second target service address information by sending the Fluentd service instance replacement notification information to the target Fluent Bit service instance, which is favorable for timely switching of the target Fluent Bit service instance to the abnormal target Fluentd service instance, reduces the influence of the Fluentd service instance abnormality to the journal collection flow of the Fluent Bit service instance, ensures that the normal running Fluentd service instance carries out subsequent aggregation and other processes on the target journal data from the Fluent Bit service instance, and improves the resistance and fault tolerance of the Fluentd service instance in the journal collection process, and improves the stability and robustness of the system.

The embodiment of the application also provides a first log collection device based on a registry, which can be configured in a registry service in a log collection system, wherein the log collection system can further comprise a Fluent Bit service instance and a plurality of Fluentd service instances, a registered Fluentd service instance set is maintained in the registry service, and the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is in a normal running state at present and is registered in the registry service in advance when being started; the first log collection device based on the registry is used for executing the step that the execution subject serves the registry in the log collection method based on the registry. Specifically, referring to fig. 4, fig. 4 shows a schematic structural diagram of a first registry-based log collection device 300 according to an embodiment of the present application, where the first registry-based log collection device 300 specifically includes a first receiving unit 301, an obtaining unit 302, and a first transmitting unit 303; wherein,

A first receiving unit 301, configured to receive a Fluentd service address query request sent by the Fluent Bit service instance; the Fluentd service address query request is sent by the Fluent Bit service instance before log acquisition is carried out;

an obtaining unit 302, configured to obtain, in response to the Fluentd service address query request, one or more service address information in the registered Fluentd service instance set;

And a first sending unit 303, configured to send the one or more service address information to the Fluent Bit service instance, so that the Fluent Bit service instance determines first target service address information in the one or more service address information, and configures the first target service address, so that the Fluentd service instance corresponding to the first target service address receives target log data from the Fluent Bit service instance and performs log aggregation processing when the Fluent Bit service instance performs log acquisition.

In some embodiments of the present application, the first receiving unit 301 may be further configured to receive registration request information of a service instance of the target Fluentd; the registration request information carries the service address information of the target Fluentd service instance; the registration request information is sent to the registry service when the registration request information is started by the target Fluentd service instance; the first registry-based log collection device 300 may further include Fluentd service instance management unit, configured to add the service address information of the target Fluentd service instance to the registered Fluentd service instance set in response to the registration request information, and perform operation status monitoring on the target Fluentd service instance; if the target Fluentd service instance is monitored to be in an abnormal running state, deleting the service address information of the target Fluentd service instance from the registered Fluentd service instance set.

In some embodiments of the present application, the first registry-based log collection device 300 may further include a determining unit, configured to determine whether there is a target Fluent Bit service instance in which the service address information of the target Fluentd service instance is configured; the first sending unit 303 may be further configured to send Fluentd service instance replacement notification information to the target Fluent Bit service instance if the target Fluent Bit service instance exists, so that the target Fluent Bit service instance obtains second target service address information from the registered Fluentd service instance set, and performs replacement configuration of the second target service address information, so that the Fluentd service instance corresponding to the second target service address information receives the target log data from the target Fluent Bit service instance and performs log aggregation processing.

The embodiment of the application also provides a second log acquisition device based on the registry, which can be configured in a Fluent Bit service instance in a log acquisition system, wherein the log acquisition system also comprises a registry service and a plurality of Fluentd service instances; a registered Fluentd service instance set is maintained in the registry service, wherein the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is currently in a normal running state and is registered in the registry service in advance when being started; the second log collection device based on the registry is used for executing the step of executing the service instance with the main body being Fluent Bit in the log collection method based on the registry. Specifically, referring to fig. 5, fig. 5 shows a schematic structural diagram of a second log collection device 400 based on a registry, where the second log collection device 400 based on the registry specifically includes a second sending unit 401, a second receiving unit 402, a configuration unit 403 and a log collection unit 404, and in this embodiment,

A second sending unit 401, configured to send Fluentd a service address query request to the registry service before log collection is performed on a log data source, so that the registry service obtains one or more service address information in the registered Fluentd service instance set;

A second receiving unit 402, configured to receive the one or more service address information sent by the registry service, and determine first target service address information from the one or more service address information;

a configuration unit 403, configured to perform address configuration in a configuration file by using the first target service address information;

The log collection unit 404 is configured to collect logs of a log data source, and send collected target log data to the Fluentd service instance corresponding to the first target service address information according to the first target service address information in the configuration file, so that the Fluentd service instance performs log aggregation processing on the target log data.

In some embodiments of the present application, the Fluent Bit service instance includes a Fluent Bit service instance main thread and SideCar services; the second sending unit 401 may be specifically configured to send Fluentd a service address query request to the registry service through the SideCar service; the second receiving unit 402 may be specifically configured to receive, by using the SideCar service, the one or more service address information sent by the registry service, and determine first target service address information from the one or more service address information; the configuration unit 403 may be specifically configured to perform address configuration in a configuration file by using the first target service address information through the SideCar service; the log collection unit 404 may be specifically configured to collect a log of a log data source by using the Fluent Bit service instance main thread, and send, according to the first target service address information in the configuration file, the collected target log data to the Fluentd service instance corresponding to the first target service address information.

In some embodiments of the present application, the second receiving unit 402 may be further configured to receive Fluentd service instance replacement notification information sent by the registry service; acquiring second target service address information from the registered Fluentd service instance set in response to the Fluentd service instance change notification information; the configuration unit 403 may be further configured to perform address update configuration in the configuration file by using the second target service address information.

In some embodiments of the present application, the second sending unit 401 may be further configured to send, by the SideCar service, a network connection request to the Fluentd service instance corresponding to the first target service address information at a preset time interval; if the response data of the Fluentd service instance corresponding to the first target service address information for the network connection request is not received, sending service address unavailable information to the registry service through the SideCar service, and acquiring third target service address information from the registered Fluentd service instance set; the configuration unit 403 may be further configured to perform address update configuration in the configuration file by using the third target service address information through the SideCar service.

It should be noted that, as those skilled in the art can clearly understand, the specific implementation processes of the first log collecting device based on the registry, the second log collecting device based on the registry and each unit may refer to the corresponding descriptions in the foregoing method embodiments, and for convenience and brevity of description, no further description is provided herein.

The first registry-based log collection device and the second registry-based log collection device described above may be implemented in the form of a computer program that may be run on a computer device as shown in fig. 6.

Referring to fig. 6, fig. 6 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a terminal device such as a smart phone, a tablet computer, a Personal Computer (PC), a learning machine, a smart wearable device, etc. Referring to fig. 6, the computer apparatus 500 includes a processor 502, a memory, and a network interface 505, which are connected by a device bus 501, wherein the memory may include a storage medium 503 and an internal memory 504.

The storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, may cause the processor 502 to perform a registry-based log collection method.

The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.

The internal memory 504 provides an environment for the execution of a computer program 5032 in the storage medium 503, which computer program 5032, when executed by the processor 502, causes the processor 502 to perform a registry-based log collection method.

The network interface 505 is used for network communication, such as providing for transmission of data information, etc. It will be appreciated by those skilled in the art that the architecture shown in fig. 6 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting of the computer device 500 to which the present inventive arrangements may be implemented, as a particular computer device 500 may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

The processor 502 is configured to execute a computer program 5032 stored in a memory, so as to implement the registry-based log collection method disclosed in the embodiment of the present application.

Those skilled in the art will appreciate that the embodiment of the computer device shown in fig. 6 is not limiting of the specific construction of the computer device, and in other embodiments, the computer device may include more or less components than those shown, or certain components may be combined, or a different arrangement of components. For example, in some embodiments, the computer device may include only a memory and a processor, and in such embodiments, the structure and function of the memory and the processor are consistent with the embodiment shown in fig. 6, and will not be described again.

It should be appreciated that in embodiments of the present application, the Processor 502 may be a central processing unit (Central Processing Unit, CPU), the Processor 502 may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application SPECIFIC INTEGRATED Circuits (ASICs), off-the-shelf Programmable gate arrays (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

In another embodiment of the application, a computer-readable storage medium is provided. The computer readable storage medium may be a nonvolatile computer readable storage medium or a volatile computer readable storage medium. The computer readable storage medium stores a computer program, wherein the computer program when executed by a processor implements the registry-based log collection method disclosed in the embodiments of the present application.

It will be clearly understood by those skilled in the art that, for convenience and brevity of description, specific working procedures of the apparatus, device and unit described above may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein. Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the several embodiments provided by the present application, it should be understood that the disclosed apparatus, device and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of units is merely a logical function division, there may be another division manner in actual implementation, or units having the same function may be integrated into one unit, for example, multiple units or components may be combined or may be integrated into another apparatus, or some features may be omitted, or not performed. In addition, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices, or elements, or may be an electrical, mechanical, or other form of connection.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units may be stored in a storage medium if implemented in the form of software functional units and sold or used as stand-alone products. Based on such understanding, the technical solution of the present application may be essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a background server, or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, an optical disk, or other various media capable of storing program codes.

The present application is not limited to the above embodiments, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the present application, and these modifications and substitutions are intended to be included in the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.

Claims

1. The log acquisition method based on the registry is characterized by being applied to registry service in a log acquisition system, wherein the log acquisition system further comprises a Fluent Bit service instance and a plurality of Fluentd service instances; a registered Fluentd service instance set is maintained in the registry service, wherein the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is currently in a normal running state and is registered in the registry service in advance when being started; the method comprises the following steps:

2. The method of claim 1, wherein prior to receiving Fluentd service address query requests sent by the Fluent Bit service instance, the method further comprises:

Receiving registration request information of a target Fluentd service instance; the registration request information carries the service address information of the target Fluentd service instance; the registration request information is sent to the registry service when the registration request information is started by the target Fluentd service instance;

Adding the service address information of the target Fluentd service instance into the registered Fluentd service instance set in response to the registration request information, and monitoring the running state of the target Fluentd service instance;

If the target Fluentd service instance is monitored to be in an abnormal running state, deleting the service address information of the target Fluentd service instance from the registered Fluentd service instance set.

3. The method of claim 2, wherein after the deleting the service address information of the target Fluentd service instance from the registered Fluentd service instance set, the method further comprises:

judging whether a target Fluent Bit service instance of the service address information of the target Fluentd service instance is configured or not;

If the target Fluent Bit service instance exists, fluentd service instance replacement notification information is sent to the target Fluent Bit service instance, so that the target Fluent Bit service instance obtains second target service address information from the registered Fluentd service instance set, replacement configuration of the second target service address information is conducted, and the Fluentd service instance corresponding to the second target service address information receives the target log data from the target Fluent Bit service instance and conducts log aggregation processing.

4. The log acquisition method based on the registry is characterized by being applied to Fluent Bit service examples in a log acquisition system, wherein the log acquisition system also comprises a registry service and a plurality of Fluentd service examples; a registered Fluentd service instance set is maintained in the registry service, wherein the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is currently in a normal running state and is registered in the registry service in advance when being started; the method comprises the following steps:

5. The method according to claim 4, wherein the Fluent Bit service instance includes a Fluent Bit service instance main thread and SideCar services; the sending Fluentd a service address query request to the registry service includes:

sending Fluentd a service address query request to the registry service through the SideCar service;

the receiving the one or more service address information sent by the registry service, and determining first target service address information from the one or more service address information, includes:

Receiving, by the SideCar service, the one or more service address information sent by the registry service, and determining first target service address information from the one or more service address information;

the address configuration in the configuration file by using the first target service address information comprises the following steps:

Performing address configuration in a configuration file by using the first target service address information through the SideCar service;

The log data source is subjected to log acquisition, and the acquired target log data is sent to the Fluentd service instance corresponding to the first target service address information according to the first target service address information in the configuration file, including:

And carrying out log acquisition on a log data source through the Fluent Bit service instance main thread, and sending acquired target log data to the Fluentd service instance corresponding to the first target service address information according to the first target service address information in the configuration file.

6. The method of claim 4, wherein after the address configuration in the configuration file using the first destination service address information, the method further comprises:

receiving Fluentd service instance replacement notification information sent by the registry service;

Acquiring second target service address information from the registered Fluentd service instance set in response to the Fluentd service instance change notification information;

And performing address updating configuration in the configuration file by utilizing the second target service address information.

7. The method of claim 5, wherein after said address configuration in the configuration file using said first destination service address information by said SideCar service, said method further comprises:

8. A registry-based log collection device comprising means for performing the method of any one of claims 1-3 or means for performing the method of any one of claims 4-7.

9. The log acquisition method based on the registry is characterized by being applied to a log acquisition system, wherein the log acquisition system comprises a registry service, a Fluent Bit service instance and a plurality of Fluentd service instances; a registered Fluentd service instance set is maintained in the registry service, wherein the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is currently in a normal running state and is registered in the registry service in advance when being started; the method comprises the following steps:

10. The log acquisition system is characterized by comprising a registry service, a Fluent Bit service instance and a plurality of Fluentd service instances; the registry service maintains a registered Fluentd service instance set, wherein the registered Fluentd service instance set comprises service address information corresponding to the Fluentd service instance which is currently in a normal running state and is registered in the registry service in advance when being started; wherein,