CN111930700A - Distributed log processing method, server, system and computing equipment - Google Patents
Distributed log processing method, server, system and computing equipment Download PDFInfo
- Publication number
- CN111930700A CN111930700A CN202010669517.5A CN202010669517A CN111930700A CN 111930700 A CN111930700 A CN 111930700A CN 202010669517 A CN202010669517 A CN 202010669517A CN 111930700 A CN111930700 A CN 111930700A
- Authority
- CN
- China
- Prior art keywords
- log
- software
- server
- data storage
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 56
- 238000003860 storage Methods 0.000 claims abstract description 22
- 238000013500 data storage Methods 0.000 claims description 99
- 238000000034 method Methods 0.000 claims description 41
- 238000004891 communication Methods 0.000 claims description 26
- 238000009826 distribution Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 7
- 230000004044 response Effects 0.000 claims description 5
- YCKRFDGAMUMZLT-UHFFFAOYSA-N Fluorine atom Chemical compound [F] YCKRFDGAMUMZLT-UHFFFAOYSA-N 0.000 claims description 2
- 229910052731 fluorine Inorganic materials 0.000 claims description 2
- 239000011737 fluorine Substances 0.000 claims description 2
- 238000012544 monitoring process Methods 0.000 claims description 2
- 239000000126 substance Substances 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 9
- 238000004519 manufacturing process Methods 0.000 description 6
- 238000012384 transportation and delivery Methods 0.000 description 6
- 238000011161 development Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 230000009467 reduction Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 238000005096 rolling process Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000004134 energy conservation Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000007723 transport mechanism Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The invention discloses a distributed log processing method which is suitable for being executed in a configuration server, and comprises the following steps: receiving registration software parameters generated in a browser at a release end by a user, wherein the registration software parameters at least comprise a software unique identifier, a software type, a deployment type, a log server IP address and a log storage path; determining the type of a log collector corresponding to the registered software parameter, calculating a registered hash value corresponding to the registered software parameter, and generating a configuration file after determining a message queue cluster and a message subject corresponding to the registered hash value by adopting a consistent hash algorithm; and sending the configuration file to the IP address of the log server so as to install a log collector of the determined type in the corresponding log server, wherein the log collector collects the log message of the registered software and then sends the log message to the determined message queue cluster and the message subject. The invention also discloses a corresponding configuration server and a distributed log processing system.
Description
Technical Field
The present invention relates to the field of log processing, and in particular, to a distributed log processing method, server, system, and computing device.
Background
With the rapid growth of internal and external services of an IT enterprise, subscribing software real-time rolling logs is a cloud tool which is most intuitive and urgently needed by system operation and maintenance engineers and software development engineers for checking software running conditions, troubleshooting and solving problems in real time. At this time, an engineer generally needs to enter a virtual machine or a container through a bastion machine and then check a software log by using a Linux command, so that great potential safety hazards exist. Moreover, the system operation and maintenance and the service development in the large internet enterprise belong to different teams, the software deployment architecture is complex, the two teams are oblivious to the storage positions of the specific log files, so that the development engineer is difficult to view the system log and the middleware log, and the system operation and maintenance engineer is difficult to view the application log. In addition, the scheme has high requirements on the Linux command use threshold of an engineer, and particularly when a large amount of logs are searched, a large amount of time is consumed to locate and search the required information in the logs.
The other mode is that the collector pushes the collected logs to a Kafka cluster in batches, then the logs are distributed to a plurality of Kafka topics according to service requirements through Kafka Stream technology, and then the user subscribes the corresponding Kafka topics through a Kafka client according to the service requirements. However, the software deployment architecture of the internet enterprise is extremely complex, the installation positions of all software are different, and the software log paths are also different. The scheme cannot solve the problem of log collection management and log path configuration management of various software under the mixed deployment condition. Software installation and deployment and log collection processes are complex and prone to errors, and therefore work efficiency is low. Moreover, the daily log amount in medium and large internet enterprises is billions and billions, and the scale of log storage amount of hundreds of T is continuously increased. The method is in front of massive logs, and cannot realize instant subscription of second-level delay of software logs. And although the Kafka cluster can process massive logs for users to consume and look up, when subscribers increase steeply, topic fragmentation and network bandwidth fullness can be caused, and the communication of the internal network of an enterprise is blocked. Finally, the scheme occupies a large amount of storage resources, causes the waste of the storage resources, and does not meet the current targets of energy conservation, emission reduction and cost conservation development of enterprises.
Disclosure of Invention
In view of the above, the present invention proposes a distributed log processing method, server, system and computing device in an attempt to solve, or at least solve, the problems presented above.
According to an aspect of the present invention, there is provided a distributed log processing method, adapted to be executed in a configuration server, the configuration server being respectively connected in communication with a software distribution terminal and a plurality of log servers, the log servers being connected in communication with a plurality of message queue clusters, the method comprising the steps of: receiving registered software parameters generated in a browser at a release end by a user, wherein the registered software parameters at least comprise a software unique identifier, a software type, a deployment type, a log server IP address and a log storage path, the deployment type comprises a virtual machine server and a container server, and the IP address is a virtual machine IP input by the user in the browser at the release end or a container instance IP acquired through a container IP interface; determining the type of a log collector corresponding to the registered software parameters, calculating a registered hash value corresponding to the character string combination of the registered software parameters, and generating a configuration file after determining a message queue cluster and a message subject corresponding to the registered hash value by adopting a consistent hash algorithm; and sending the configuration file to the IP address of the log server so as to install a log collector of the determined type in the corresponding log server, wherein the log collector collects the log message of the registered software and then sends the log message to the determined message queue cluster and the message subject.
Optionally, in the distributed log processing method according to the present invention, a subscription listener step in the subscription listener is further included: acquiring a subscription combination condition generated in a browser of a subscription end by a user, wherein the subscription combination condition comprises a software unique identifier, a software type, a deployment type, a log server IP address and a log storage path; and calculating a subscription hash value corresponding to the subscription combination condition, determining a data storage cluster and a data storage channel corresponding to the subscription hash value by adopting a consistent hash algorithm, and marking the data storage cluster and the data storage channel as subscribed.
Optionally, in the distributed log processing method according to the present invention, a log processing distribution step in the log processing engine is further included: reading log information from a message subject of the message queue cluster and analyzing software registration parameters and a log text in the log information; calculating an analytic hash value corresponding to the software registration parameter, and determining a data storage cluster and a data storage channel corresponding to the analytic hash value by adopting a consistent hash algorithm; and querying whether the data storage channel in the data storage cluster is subscribed from the subscription listener, if so, sending the analyzed log message to the data storage channel in the data storage cluster, otherwise, discarding the log message.
Optionally, in the distributed log processing method according to the present invention, the log subscribing and unsubscribing step in the subscribing browser further includes: responding to a subscription combination condition generated in a subscriber H5 page by a user, establishing a WebSocket long connection with a data storage channel corresponding to the subscription combination condition, and displaying a log message transmitted from the data storage channel in real time in the H5 page; and in response to a unsubscribe request initiated by the user in the subscriber H5 page, disconnecting the long connection to the data storage channel.
Optionally, in the distributed log processing method according to the present invention, the software types include system software, application software, and middleware software; the journal collector type includes at least one of an Rsyslog collector, a fluent collector, a Filebeat collector, and a fluorine collector.
Optionally, in the distributed log processing method according to the present invention, the log saving path of the application software is obtained by manually adding the log saving path to the browser at the publishing end by the user; the log saving path of the system software and the middleware software is a pre-configured log path, and is automatically displayed in the browser at the release end according to other registration parameters input by a user.
Optionally, in the distributed log processing method according to the present invention, the log collector type of the system software is an Rsyslog collector; the log collector type of the middleware software is a fluent collector; the log collector type of the application software on the container server is a Filebeat collector; the log collector type of the application software on the virtual machine server is a flash collector.
Optionally, in the distributed log processing method according to the present invention, the software registration parameter and the subscription combination condition further include an environment level and/or an environment name; the environmental levels include a production environment, a test environment, and a pre-production environment.
Optionally, in the distributed log processing method according to the present invention, the step of sending the configuration file to the IP address of the log server includes: if the deployment type in the configuration file is the virtual machine server, the configuration file is sent to a first acquisition configurator so that the first acquisition configurator can send the configuration file to a corresponding virtual server IP; and if the deployment type in the configuration file is the container server, sending the configuration file to a second acquisition configurator so that the second acquisition configurator can send the configuration file to the corresponding container server IP.
Optionally, in the distributed log processing method according to the present invention, the virtual machine server is an OpenStack server, and the container server is a kubernets server; the first acquisition configurator is a SaltStack acquisition configurator, and the second acquisition configurator is a Pod acquisition configurator; the message queue Cluster is a Kafaka Cluster, the log processing engine is an Apach Flink real-time processing engine, and the data storage Cluster is a Redis Cluster Cluster.
According to another aspect of the present invention, there is provided a configuration server, which is respectively connected to a software distribution terminal and a plurality of log servers in communication, wherein the log servers are connected to a plurality of message queue clusters in communication, and the configuration server includes: the receiving module is suitable for receiving registered software parameters of a user in a browser at a release end, wherein the registered software parameters at least comprise a software unique identifier, a deployment type, a software type, a server IP address and a log storage path, the deployment type comprises a virtual machine server and a container server, and the IP address is a virtual machine IP input by the user in the browser at the release end or a container instance IP acquired through a container IP interface; the calculation module is suitable for determining the type of the log collector corresponding to the registered software parameters, calculating the registered hash value corresponding to the character string combination of the registered software parameters, and generating a configuration file after determining the message queue cluster and the message subject corresponding to the registered hash value by adopting a consistent hash algorithm; and the sending module is suitable for sending the configuration file to the IP address of the log server so as to install the log collector of the determined type in the corresponding log server, and the log collector sends the log message of the registered software to the determined message queue cluster and the message theme after collecting the log message.
According to another aspect of the present invention, there is provided a distributed log processing system including: a configuration server as described above, a plurality of log servers communicatively coupled to the configuration server, and a plurality of message queue clusters communicatively coupled to the log servers; the configuration server is suitable for generating a configuration file based on software registration parameters input by a user, and installing a log collector of a corresponding type in the determined server IP address according to the configuration file; the journal collector is suitable for sending the collected journal information to the corresponding information queue cluster and the information subject.
Optionally, in the distributed log processing system according to the present invention, the distributed log processing system further includes a log processing engine, a plurality of data storage clusters, a plurality of subscribing terminals, and a subscription listener, where the log processing engine is respectively communicatively connected to the plurality of message queue clusters and the plurality of data storage clusters, and is adapted to send the log message parsed from the message queue cluster to the corresponding data storage cluster; the subscription monitor is respectively in communication connection with the plurality of data storage clusters and the plurality of subscribers and is suitable for monitoring data storage channels subscribed by the subscribers so that the data storage clusters can send the log information in the corresponding channels to the subscribers.
According to yet another aspect of the present invention, there is provided a computing device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs when executed by the processors implement the steps of the distributed log processing method as described above.
According to a further aspect of the present invention, there is provided a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, implement the steps of the distributed log processing method as described above.
According to the distributed log processing scheme, the software mass logs which are mixed and deployed on the private cloud can be automatically collected uniformly, and the mass logs can be subscribed and checked in real time. According to the scheme, manual intervention is not needed for management of the acquisition configurator, the log collector and the software deployment are decoupled, the deployment efficiency of the collector is improved, and the log acquisition problem of the software in mixed deployment is solved. The invention can still provide the software real-time rolling log subscription experience of second-level delay for the user on the premise of billion-level mass logs. And the distributed system can automatically scale, automatically add services, or reduce services as the traffic and log size change. The invention reduces a large amount of server storage and use, memory use and bandwidth occupation, and accords with the development direction of energy conservation and emission reduction and cost saving of enterprises. Finally, the scheme greatly saves the time for engineers to check the software real-time logs and locate the software problems through the software logs, and also provides higher safety guarantee for enterprise information systems.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.
FIGS. 1A-1C each show a block diagram of a distributed log processing system 100, according to one embodiment of the invention;
FIG. 2 shows a block diagram of a computing device 200, according to one embodiment of the invention;
FIG. 3 illustrates a flow diagram of a distributed log processing method 300 according to one embodiment of the invention;
FIG. 4 illustrates a schematic diagram of a publisher software registration page, in accordance with one embodiment of the present invention;
FIG. 5 illustrates a flow diagram of log data processing and publishing in accordance with one embodiment of the invention;
FIG. 6 illustrates a flow diagram of log subscribing and unsubscribing according to one embodiment of the invention;
FIG. 7 illustrates a schematic diagram of a real-time log subscription interface, according to one embodiment of the invention; and
fig. 8 shows a block diagram of a configuration server 800 according to an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
FIGS. 1A-C each show a schematic diagram of a distributed log processing system 100, according to one embodiment of the invention. As shown, system 100 includes a configuration server 110, a plurality of log servers 120, a plurality of message queue clusters 130, a log processing engine 140, and a plurality of data storage clusters 150. The configuration server 110 is communicatively connected to a plurality of log servers 120, the log servers 120 are communicatively connected to a plurality of message queue clusters 130, and the log processing engine 140 is communicatively connected to the plurality of message queue clusters 130 and a plurality of data storage clusters 150, respectively.
Specifically, the configuration server 110 receives software registration parameters from the software publisher. The software publishing end is used for registering software, registering the log path configuration information of the software and the related information of the software into the configuration server 110, so that the configuration server 110 automatically installs and updates the collector and provides a data source for the log acquisition path of the automatic configuration collector. As shown in fig. 4, when registering software, a user may fill in a plurality of software registration parameters, where the software registration parameters include at least one of a software unique identifier, a software type, a deployment type, an environment level, a log server IP address, and a log saving path. The publishing browser provides an input box or an option box of each parameter, so that the user can select the required content from various pull-down options or fill in parameter values by manually adding input.
The deployment type includes a virtual machine server and a container server, the virtual machine server is, for example, an OpenStack server, and the container server is, for example, a kubernets server, but is not limited thereto. The IP address of the log server is the IP address of the log server deployed by software, and corresponding log server IPs exist in different deployment types. When the deployment type is the virtual machine server, a user is required to manually fill in the IP list of the software deployment. At this time, an IP input box is provided in the browser at the release end for the user to manually add and input the virtual machine IP, and the log server IP is the virtual machine IP input by the user in the browser at the release end. When the deployment type is the container server, the container of the container server does not need to provide an IP list, and the container instance IP (pod IP) of the container server can be automatically obtained through a container IP interface. At this time, the log server IP is the container instance IP acquired through the container IP interface, and the issuing-end browser can automatically acquire and display the container instance IP.
The environmental levels include a production environment, a test environment, and a pre-production environment. The software types include system software, application software, and middleware software located therebetween. When software registration is carried out, different types of software have different log saving path filling modes. The log saving path of the application software is obtained by manually adding the log saving path in the browser of the issuing end by the user, so that when the software issuing end detects that the software type input by the user is the application software, the adding option of the log saving path is displayed for the user to manually add the log saving path. For the system software and the middleware software, the system can preset the log saving paths of the two types of software, so that the log saving paths of the two types of software are automatically displayed in the browser at the release end according to other registration parameters (such as software types, deployment environments and the like) input by a user. Generally, when the software publishing terminal 110 detects that the software type selected by the user is the two types of software, a preset log saving path is automatically obtained for the user to check. Meanwhile, the software publishing terminal provides log path modification and addition options so that the user can modify and add the log path when checking that the log path is wrong.
After generating various software registration parameters in the software distribution end, the software distribution end sends the parameters to the configuration server 110. The configuration server 110 determines the corresponding journal logger type according to the software registration parameters. Generally, the log collector type of the system software is the Rsyslog collector; the log collector type of the middleware software is a fluent collector; the log collector type of the application software on the container server is a Filebeat collector; the log collector type of the application software on the virtual machine server is a flash collector. On the other hand, the configuration server 110 calculates the corresponding registration hash value by using the software registration parameters as a combined character string, and determines the message queue cluster and the message Topic (Topic) corresponding to the registration hash value by using a consistent hash algorithm.
Here, each log server 120 may be connected to a plurality of message queue clusters 130, each of which in turn includes a plurality of message topics. In order to enable the system to have strong flexibility, expansibility and robustness, the invention uses a consistent hash algorithm to calculate the message queue cluster and the message subject to be delivered by the software log. The main idea of the consistent hash algorithm is to associate each node with one or more hash value domain intervals, map each object to a point on the edge of the hash ring, and then the system maps the available node machines to different positions of the ring. When a machine corresponding to an object is searched, the position on the ring edge corresponding to the object is obtained by calculation through a consistent hash algorithm, and the machine is the position where the object should be stored until a certain node machine is encountered.
In the invention, firstly, the message queue cluster is divided into a plurality of message queue clusters according to different software types, and each cluster is composed of a plurality of sub-clusters, thereby logically forming a plurality of message queue cluster delivery rings. And then, character string identification formed by combining software registration parameters is used as an entry parameter, and a corresponding address is acquired from a subordinate message queue cluster delivery ring through a consistent hash algorithm, so that a cluster and a message theme to be delivered by the software are obtained. Here, the hash rings of the plurality of message queue clusters and the message topics may be constructed, respectively, so as to determine the target message queue cluster and the target message topic to which the registered hash value belongs. Moreover, when the target message topic is judged, a hash ring of a plurality of message topics in the message queue cluster can be constructed, so that the target message topic to which the registered hash value belongs can be determined. Therefore, the log data in the same log file can be delivered to a certain fixed message subject in the same message queue cluster, and the log is ensured to be still ordered when being read.
The configuration server 110 generates a configuration file based on the determined information items, where the configuration file includes the registered software parameters (software unique identifier, software type, deployment type, log server IP address, log saving path, and the like), and the determined collection parameters such as log collector type, message queue cluster, and message topic. The configuration server 110 then sends the configuration file to the corresponding log server 120, so as to install the determined type of log collector in the log server 120. The journal logger collects journal messages for the registered software and sends them to the determined message queue cluster 130 and message topic.
According to an embodiment of the present invention, a collection configurator (not shown in the figure) may be further disposed between the configuration server 110 and the log server 120, such that the configuration server 110 sends the configuration file to the collection configurator and the collection configurator sends the configuration file to the corresponding log server. An acquisition configurator interface may be invoked here to implement acquisition configuration of the configuration file. The acquisition configurator may be located within the configuration server 110 or may be located outside the configuration server 110 as another device and communicatively connected to the configuration server 110. The acquisition configurator installs a corresponding log collector for the software according to the transmitted configuration file (including acquisition parameters and a unique software identifier), starts log acquisition service, and pushes logs of the software to a corresponding message theme in batch and quickly.
Different deployment types in the configuration file are sent to different acquisition configurators. And if the deployment type is the virtual machine server, sending the configuration file to a first acquisition configurator so that the first acquisition configurator sends the configuration file to a corresponding virtual server IP to install a corresponding log collector, configure a log storage path and start the log collector on the IP virtual machine. And if the deployment type is the container server, sending the configuration file to a second acquisition configurator so that the second acquisition configurator sends the configuration file to a corresponding container server IP (Internet protocol), so as to configure a corresponding log collector container in the Pod and collect logs of a main container in which the software in the Pod is positioned. Specifically, the second acquisition configurator finds the corresponding Pod according to the transmitted parameters, and can configure the collector container according to the SideCar mode, restart the Pod where the container is located, and start the main container where the software is located and the log collector container. According to an embodiment, the first acquisition configurator is a SaltStack acquisition configurator and the second acquisition configurator is a Pod acquisition configurator, although not limited thereto.
The log processing engine 140 is connected to a plurality of data storage clusters 150, each having a plurality of data storage channels. The log processing engine 140 performs a process and a distribution flow of log data, and a flowchart thereof is shown in fig. 5. As shown in fig. 5, the log processing engine 140 continuously reads each log data from the message topic of the message queue cluster, and extracts the software registration parameters and the log text according to the type identifier of each log or the specific type of the software (such as Cron log of system software, Nginx log of middleware, and spingbot log of application software) according to the corresponding parsing rule. Different types of log file analysis rules are different, for example, a text type log file can be analyzed by adopting the analysis rule of a regular expression; json type log files (for example, the logs collected by the Filebeat collector are in a Json format) can be analyzed by adopting Json regularized analysis rules. In addition, after the log processing engine 140 parses the log original text, it can also determine whether multi-line merging is needed by regular rules, if so, merging is performed, otherwise, parsing is performed according to lines.
Then, the log processing engine 140 calculates an analytic hash value corresponding to the software registration parameter, and determines a data storage cluster and a data storage channel corresponding to the analytic hash value by using a consistent hash algorithm. Similar to the message queue cluster 130 and the message topic, in order to make the system have stronger flexibility, expansibility and robustness, the invention adopts a consistent hash algorithm to construct a log router to calculate the data storage cluster to which the software log is to be distributed. Firstly, dividing a data storage cluster into a plurality of data storage clusters according to different software types, wherein each data storage cluster is composed of a plurality of sub-clusters, and therefore a plurality of data storage cluster delivery rings are formed logically. And then, taking the character string identification of each software registration parameter combination as a reference, and acquiring a corresponding bucket address from a subordinate data storage cluster delivery ring through a consistent hash algorithm, thereby finally obtaining a data storage cluster address and a data storage channel to which the software needs to be distributed. The log processing engine 10 then publishes the log onto the target data storage channel for real-time subscription by the user.
Optionally, the message queue cluster 130 is a Kafaka cluster, and the corresponding message Topic is Kafaka Topic. The log processing engine 140 is an Apach Flink real-time processing engine, and performs log processing distribution work by Flink Job. The data storage Cluster 150 is a Redis Cluster Cluster, and the corresponding data storage Channel is a Redis Cluster Channel. Of course, these clusters and engines may be configured as other common devices, and the present invention is not limited thereto.
The software subscription end is used for subscribing and unsubscribing logs, the process of subscribing and unsubscribing users is shown in fig. 6, and the log subscription and log display interface is shown in fig. 7. Generally, a user initiates a network interface request through a subscription combination condition of an H5 interface, and establishes a long link with a data storage channel corresponding to the subscription combination condition, so as to acquire log messages in the channel in real time and display the log messages in a browser page in real time. The subscription combination condition comprises at least one of a software unique identifier, a software type, a deployment type, a log server IP address and a log saving path. Of course, the environment name and/or environment level may also be included. It should be noted that, in order to ensure the consistency of the hash value calculation, consistent software parameters are adopted for calculation in the present invention for the registered hash value, the analytic hash value and the subscription hash value, that is, if five parameters are adopted for the registered hash value, the five parameters are also adopted for the analytic hash value and the subscription hash value, and so on.
In one aspect, in response to a subscription combination condition generated in a subscriber H5 page by a user, a subscriber calculates a subscription hash value corresponding to the subscription combination condition, where the subscription hash value has a corresponding data storage cluster and a corresponding data storage channel. And then, establishing a WebSocket long connection between the subscription end and the data storage channel corresponding to the subscription combination condition so as to display the log message transmitted from the data storage channel in real time in the H5 page. On the other hand, in response to a request for unsubscribing initiated by the user in the page of the subscriber H5, the subscriber disconnects the long connection with the data storage channel, so that the real-time log is no longer acquired and displayed.
According to an embodiment of the present invention, the system 100 may further include a subscription listener 160, and the subscription listener 160 is communicatively connected to the log processing engine 140 and the plurality of subscribers respectively, and is configured to listen to the data storage clusters 150 and the data storage channels subscribed by the subscribers. Thus, the log processing engine 140 queries the subscription listener 160 to determine whether the data storage channel in the data storage cluster 150 is subscribed, and if so, sends the parsed log message to the data storage channel in the data storage cluster 150, otherwise, discards the log message. That is, when the subscription listener 160 listens to a log of a user subscribing to software, the log processing engine 140 will distribute the log to channels of the data storage cluster 150 for the user to subscribe to. No corresponding subscriber in the subscription listener is not distributed.
Meanwhile, when the subscription end determines that a certain data storage channel is subscribed by a certain subscriber, the subscription monitor is informed to update the number of the log subscribers of the software, namely the number of the log subscribers of the software is added by 1; if the subscription is cancelled, the subscription listener is notified to decrement the number of log subscribers for the software by 1. When the number of log subscribers for the software is 0, the log processing engine 140 stops publishing logs to the data storage channel of the software. Of course, if the number of log subscribers is not 0, the log is normally published to the data storage channel.
According to an embodiment of the present invention, the system 100 may further include a micro-service web interface 170(WebSockets-micro service), and the micro-service web interface 170 is communicatively connected to the subscription side, the data storage cluster 150, and the subscription listener 160, respectively. The microservice web page interface 170 may be located in the subscriber, or may be located outside the subscriber as another device and communicatively connected to the subscriber, which is not limited in the present invention. When log subscription is performed, a subscription end initiates a network interface request, the micro service web interface 170 establishes long connection after receiving the request, and a router adopting a consistent hash algorithm directly acquires the target data storage cluster 150 and the data storage channel to be subscribed. Then, the micro-service web interface 170 initiates a subscription to the target data storage channel, and continuously pushes the acquired log information to an H5 page of the subscriber end through the long connection of the web interface, thereby realizing the effect of viewing the log in real time by scrolling. On the other hand, when canceling the subscription, when the user initiates a cancel subscription operation on the H5 page, the microservice web page interface 170 where the long connection is located initiates a cancel subscription to the target data storage channel and disconnects the long connection.
It should be noted that, after the system 100 sets the micro service web page interface 170, the subscription listener 160 can directly communicate with the micro service web page interface 170, and is not directly connected to a plurality of subscribers. At this point, the microservice web interface 170 determines which data storage channels are subscribed to and notifies the subscription listener 160, causing the subscription listener 160 to update the log subscriber number for the data storage channels in real-time. When the number is 0, the log processing engine 140 does not perform log distribution any more, and when the number is not 0, the log distribution is performed normally.
It should be noted that the distributed log processing system 100 of fig. 1A and 1B is merely exemplary, and in particular implementations, there may be different numbers of configuration servers 110, log servers 120, message queue clusters 130, log processing engines 140, data storage clusters 150, and subscription listeners 160 in the system 100, and the present invention does not limit the number of devices included in the system 100. Moreover, each of the servers may be an individual server, or a server cluster composed of a plurality of servers, or a cloud computing service center, and a plurality of servers used for constituting the server cluster or the cloud computing service center may reside in a plurality of geographic locations.
According to an embodiment of the present invention, the configuration server 110, the log server 120, the message queue cluster 130, the log processing engine 140, the data storage cluster 150, and the subscription listener 160 in the distributed log processing system 100 can be implemented by the computing device 200 as described below. FIG. 2 shows a block diagram of a computing device 200, according to one embodiment of the invention.
In a basic configuration 202, computing device 200 typically includes system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between the processor 204 and the system memory 206.
Depending on the desired configuration, the processor 104 may be any type of processing, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a Digital Signal Processor (DSP), or any combination thereof. The processor 204 may include one or more levels of cache, such as a level one cache 210 and a level two cache 212, a processor core 214, and registers 216. Example processor cores 214 may include Arithmetic Logic Units (ALUs), Floating Point Units (FPUs), digital signal processing cores (DSP cores), or any combination thereof. The example memory controller 218 may be used with the processor 204, or in some implementations the memory controller 218 may be an internal part of the processor 204.
Depending on the desired configuration, system memory 206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 106 may include an operating system 220, one or more applications 222, and program data 224. In some implementations, the application 222 can be arranged to operate with the program data 124 on an operating system. Program data 124 includes instructions and in computing device 200 according to the present invention, program data 224 contains instructions for performing distributed log processing method 300.
Computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus/interface controller 230. The example output device 242 includes a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 252. Example peripheral interfaces 244 can include a serial interface controller 254 and a parallel interface controller 256, which can be configured to facilitate communications with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 258. An example communication device 246 may include a network controller 260, which may be arranged to facilitate communications with one or more other computing devices 262 over a network communication link via one or more communication ports 264.
A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media.
Computing device 200 may be implemented as a server, such as a file server, a database server, an application server, a WEB server, etc., or as part of a small-sized portable (or mobile) electronic device, such as a cellular telephone, a Personal Digital Assistant (PDA), a wireless WEB-watch device, an application specific device, or a hybrid device that include any of the above functions. Computing device 200 may also be implemented as a personal computer including both desktop and notebook computer configurations. In some embodiments, the computing device 200 is configured to perform the distributed log processing method 300.
FIG. 3 shows a flow diagram of a distributed log processing method 300 according to one embodiment of the invention. The method 300 is performed in a configuration server, such as the configuration server 110, to complete the log collection, parsing, and distribution process.
As shown in fig. 3, the method begins at step S310. In step S310, the registered software parameters generated in the publisher browser by the user are received. The registered software parameters, i.e. the unique software identifier, software type, deployment type, log server IP address and log saving path described above, may also include environment level and environment name. The software registration process and the meaning of each parameter are described in detail above and will not be described further herein.
Subsequently, in step S320, the log collector type corresponding to the registered software parameter is determined, the registered hash value corresponding to the character string combination of the registered software parameter is calculated, and the configuration file is generated after the message queue cluster and the message topic corresponding to the registered hash value are determined by using a consistent hash algorithm.
According to one embodiment, system software logs are collected by a Rsyslog collector; the middleware software log is collected by a fluent collector; the logs of the application software on the container server are collected by a Filebeat collector; the logs of the application software on the virtual machine server are collected by a flash collector.
Subsequently, in step S330, the configuration file is sent to the IP address of the log server, so that a log collector of the determined type is installed in the corresponding log server, and the log collector collects the log message of the registered software and sends the log message to the determined message queue cluster and message topic.
Here, if the deployment type in the configuration file is a virtual machine server, the configuration file is sent to the first collection configurator, so that the first collection configurator sends the configuration file to the corresponding virtual server IP. And if the deployment type in the configuration file is the container server, sending the configuration file to a second acquisition configurator so that the second acquisition configurator can send the configuration file to the corresponding container server IP.
According to one embodiment, the method 300 further comprises a subscription listening step in the subscription listener: acquiring a subscription combination condition generated in a browser of a subscription end by a user; and calculating a subscription hash value corresponding to the subscription combination condition, determining a data storage cluster and a data storage channel corresponding to the subscription hash value by adopting a consistent hash algorithm, and marking the data storage cluster and the data storage channel as subscribed.
According to another embodiment, the method 300 may further comprise a log processing distribution step in the log processing engine: reading log information from a message subject of the message queue cluster and analyzing software registration parameters and a log text in the log information; calculating an analytic hash value corresponding to the software registration parameter, and determining a data storage cluster and a data storage channel corresponding to the analytic hash value by adopting a consistent hash algorithm; and inquiring whether the data storage cluster and the data storage channel are subscribed or not from the subscription listener, if so, sending the analyzed log message to the data storage cluster and the data storage channel, and otherwise, discarding the log message.
According to yet another embodiment, the method 300 may further include log subscribing and unsubscribing steps in the subscribing browser: responding to a subscription combination condition generated in a subscriber H5 page by a user, establishing a WebSocket long connection with a data storage channel corresponding to the subscription combination condition, and displaying a log message transmitted from the data storage channel in real time in the H5 page; and in response to a unsubscribe request initiated by the user in the subscriber H5 page, disconnecting the long connection to the data storage channel.
Fig. 8 shows a block diagram of a configuration server 800 according to an embodiment of the present invention, which is adapted to generate a configuration file based on software registration parameters input by a user, and install a log collector of a corresponding type in a determined IP address of the server according to the configuration file. As shown in fig. 8, the configuration server 800 includes: a receiving module 810, a calculating module 820 and a sending module 830.
The receiving module 810 receives the registered software parameters of the user in the browser at the publishing terminal, where the registered software parameters at least include a unique software identifier, a deployment type, a software type, a server IP address, and a log saving path. The receiving module 810 may perform processing corresponding to the processing described above in step S310, and a detailed description thereof will not be repeated here.
The calculation module 820 determines the log collector type corresponding to the registered software parameter, calculates the registered hash value corresponding to the character string combination of the registered software parameter, and generates a configuration file after determining the message queue cluster and the message topic corresponding to the registered hash value by using a consistent hash algorithm. The calculation module 820 may perform processing corresponding to the processing described above in step S320, and a detailed description thereof will not be repeated.
The sending module 830 sends the configuration file to the IP address of the log server, so as to install a log collector of the determined type in the corresponding log server, and the log collector sends the log message of the registered software to the determined message queue cluster and message topic after collecting the log message. The sending module 830 may perform processing corresponding to the processing described above in step S330, and details thereof will not be described herein.
According to the technical scheme of the invention, a uniform automatic collector configuration management tool without manual intervention is provided, the log collection of software and the software deployment are decoupled, the collector deployment efficiency is improved, and the log collection problem of software in mixed deployment is solved. Even if the high-performance memory subscription/unsubscribe characteristics of a real-time big data processing engine and a data storage cluster face massive logs, the software real-time rolling log subscription experience of second-level delay is still guaranteed. Moreover, the arrangement of the multi-message queue cluster, the real-time big data processing engine, the multi-data storage cluster and the micro-service webpage interface fully ensures the distributed characteristic in the log data circulation process, and resource expansion or capacity reduction, service increase or service reduction can be carried out at any time according to the traffic and the log quantity. In addition, the invention reduces the use of a large number of data storage clusters, and saves a large amount of storage for enterprises; the implementation of the intelligent listener saves a large amount of memory for the data storage cluster and reduces a large amount of bandwidth occupation. Meanwhile, the invention also greatly saves the time for engineers to check the software real-time log and locate the software problems through the software log, and also provides higher safety guarantee for enterprise information systems.
A6, the method as in A5, wherein the log saving path of the application software is manually added by the user in the browser at the publishing end; the log saving path of the system software and the middleware software is a pre-configured log path, and is automatically displayed in the browser at the release end according to other registration parameters input by a user. A7, the method of A5, wherein the journal logger type of the system software is Rsyslog logger; the log collector type of the middleware software is a fluent collector; the log collector type of the application software on the container server is a Filebeat collector; the log collector type of the application software on the virtual machine server is a flash collector. A8, the method as in any one of a1-a7, wherein the software registration parameters and subscription combination conditions further include an environment level and/or an environment name; the environment levels include a production environment, a test environment, and a pre-production environment.
A9, the method of any one of A1-A8, wherein the step of sending the configuration file to the log server IP address comprises: if the deployment type in the configuration file is a virtual machine server, sending the configuration file to a first acquisition configurator so that the first acquisition configurator can send the configuration file to a corresponding virtual server IP; and if the deployment type in the configuration file is the container server, sending the configuration file to a second acquisition configurator so that the second acquisition configurator can send the configuration file to a corresponding container server IP. A10, the method as in a9, wherein the virtual machine server is an OpenStack server, and the container server is a kubernets server; the first acquisition configurator is a SaltStack acquisition configurator, and the second acquisition configurator is a Pod acquisition configurator; the message queue Cluster is a Kafaka Cluster, the log processing engine is an Apach Flink real-time processing engine, and the data storage Cluster is a Redis Cluster Cluster.
The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard drives, U.S. disks, floppy disks, CD-ROMs, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.
In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to perform the method of the invention according to instructions in said program code stored in the memory.
By way of example, and not limitation, readable media may comprise readable storage media and communication media. Readable storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of readable media.
In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with examples of this invention. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in a device as described in this embodiment or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into multiple sub-modules.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is used to implement the functions performed by the elements for the purpose of carrying out the invention.
As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense with respect to the scope of the invention, as defined in the appended claims.
Claims (10)
1. A distributed log processing method, adapted to be executed in a configuration server, the configuration server being respectively connected in communication with a software distribution end and a plurality of log servers, the log servers being connected in communication with a plurality of message queue clusters, the method comprising the steps of:
receiving registered software parameters generated in a browser at a release end by a user, wherein the registered software parameters at least comprise a software unique identifier, a software type, a deployment type, a log server IP address and a log storage path, the deployment type comprises a virtual machine server and a container server, and the IP address is a virtual machine IP input by the user in the browser at the release end or a container instance IP acquired through a container IP interface;
determining the type of a log collector corresponding to the registered software parameters, calculating a registered hash value corresponding to a character string combination of the registered software parameters, and generating a configuration file after determining a message queue cluster and a message subject corresponding to the registered hash value by adopting a consistent hash algorithm; and
and sending the configuration file to the IP address of the log server so as to install a log collector of the determined type in the corresponding log server, wherein the log collector collects the log message of the registered software and then sends the log message to the determined message queue cluster and the message subject.
2. The method of claim 1, further comprising a subscription listening step in the subscription listener:
acquiring a subscription combination condition generated in a browser of a subscription end by a user, wherein the subscription combination condition comprises a software unique identifier, a software type, a deployment type, a log server IP address and a log storage path;
and calculating a subscription hash value corresponding to the subscription combination condition, determining a data storage cluster and a data storage channel corresponding to the subscription hash value by adopting a consistent hash algorithm, and marking the data storage cluster and the data storage channel as subscribed.
3. The method of claim 1, further comprising a log processing distribution step in the log processing engine:
reading log information from the message subject of the message queue cluster and analyzing software registration parameters and log texts in the log information;
calculating an analytic hash value corresponding to the software registration parameter, and determining a data storage cluster and a data storage channel corresponding to the analytic hash value by adopting a consistent hash algorithm; and
and inquiring whether the data storage channel in the data storage cluster is subscribed or not from the subscription listener, if so, sending the analyzed log message to the data storage channel in the data storage cluster, and otherwise, discarding the log message.
4. The method according to claim 2 or 3, further comprising log subscribing and unsubscribing steps in a subscribing browser:
responding to a subscription combination condition generated in a subscriber H5 page by a user, establishing a WebSocket long connection with a data storage channel corresponding to the subscription combination condition, and displaying a log message transmitted from the data storage channel in real time in the H5 page; and
in response to a unsubscribe request initiated by the user in the subscriber H5 page, the long connection to the data storage channel is broken.
5. The method of any one of claims 1-4,
the software types comprise system software, application software and middleware software;
the log collector type comprises at least one of an Rsyslog collector, a fluent collector, a Filebeat collector and a fluorine collector.
6. A configuration server communicatively coupled to a software distribution site and to a plurality of log servers, respectively, the log servers communicatively coupled to a plurality of message queue clusters, the configuration server comprising:
the receiving module is suitable for receiving registered software parameters of a user in a browser at a release end, wherein the registered software parameters at least comprise a software unique identifier, a deployment type, a software type, a server IP address and a log storage path, the deployment type comprises a virtual machine server and a container server, and the IP address is a virtual machine IP input by the user in the browser at the release end or a container instance IP acquired through a container IP interface;
the calculation module is suitable for determining the type of the log collector corresponding to the registered software parameters, calculating the registered hash value corresponding to the character string combination of the registered software parameters, and generating a configuration file after determining the message queue cluster and the message subject corresponding to the registered hash value by adopting a consistent hash algorithm; and
and the sending module is suitable for sending the configuration file to the IP address of the log server so as to install a log collector of the determined type in the corresponding log server, and the log collector collects the log message of the registered software and then sends the log message to the determined message queue cluster and the message theme.
7. A distributed log processing system, comprising:
the configuration server of claim 6, a plurality of log servers communicatively connected to the configuration server, and a plurality of message queue clusters communicatively connected to the log servers; wherein the content of the first and second substances,
the configuration server is suitable for generating a configuration file based on software registration parameters input by a user and installing a log collector of a corresponding type in the determined server IP address according to the configuration file;
the journal collector is suitable for sending collected journal messages to the corresponding message queue clusters and the message subjects.
8. The system of claim 7, further comprising a log processing engine, a plurality of data storage clusters, a plurality of subscribers, and a subscription listener, wherein,
the log processing engine is respectively in communication connection with the plurality of message queue clusters and the plurality of data storage clusters and is suitable for sending the log messages analyzed from the message queue clusters to the corresponding data storage clusters;
the subscription listener is respectively in communication connection with the plurality of data storage clusters and the plurality of subscribers and is suitable for monitoring data storage channels subscribed by the subscribers so that the data storage clusters can send the log information in the corresponding channels to the subscribers.
9. A computing device, comprising:
a memory;
one or more processors;
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the methods of claims 1-5.
10. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform any of the methods of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010669517.5A CN111930700A (en) | 2020-07-13 | 2020-07-13 | Distributed log processing method, server, system and computing equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010669517.5A CN111930700A (en) | 2020-07-13 | 2020-07-13 | Distributed log processing method, server, system and computing equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111930700A true CN111930700A (en) | 2020-11-13 |
Family
ID=73313894
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010669517.5A Pending CN111930700A (en) | 2020-07-13 | 2020-07-13 | Distributed log processing method, server, system and computing equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111930700A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112527618A (en) * | 2020-12-17 | 2021-03-19 | 中国农业银行股份有限公司 | Log collection method and log collection system |
| CN112866029A (en) * | 2021-02-03 | 2021-05-28 | 树根互联股份有限公司 | Log data processing method and device based on cloud platform and server side equipment |
| CN113010483A (en) * | 2020-11-20 | 2021-06-22 | 云智慧（北京）科技有限公司 | Mass log management method and system |
| CN113065785A (en) * | 2021-04-13 | 2021-07-02 | 国网江苏省电力有限公司信息通信分公司 | Dynamic resource expansion method for electric power internet of things management platform |
| CN113342552A (en) * | 2021-07-05 | 2021-09-03 | 湖南快乐阳光互动娱乐传媒有限公司 | Data processing method and device, storage medium and electronic equipment |
| CN113742112A (en) * | 2021-09-15 | 2021-12-03 | 武汉联影智融医疗科技有限公司 | Electrocardio image generation method and system and electronic device |
| CN114070879A (en) * | 2021-11-26 | 2022-02-18 | 安天科技集团股份有限公司 | Data acquisition unit control method, device and related equipment |
| CN114205333A (en) * | 2021-12-09 | 2022-03-18 | 北京奇妙时光科技有限公司 | IP configuration method, cluster construction method, computer device and storage medium |
| CN114244557A (en) * | 2021-11-09 | 2022-03-25 | 浪潮通用软件有限公司 | Development operation log isolation method and system based on user characteristics |
| CN114546668A (en) * | 2022-04-27 | 2022-05-27 | 恒生电子股份有限公司 | Log collection method and device, electronic equipment and computer readable storage medium |
| CN114756301A (en) * | 2022-04-24 | 2022-07-15 | 北京百度网讯科技有限公司 | Log processing method, device and system |
| CN114816583A (en) * | 2022-05-31 | 2022-07-29 | 以萨技术股份有限公司 | Flink-based data automatic processing method and device and electronic equipment |
| CN114879942A (en) * | 2022-05-20 | 2022-08-09 | 北京宇信科技集团股份有限公司 | Distributed time wheel packet registration verification method, device, medium and equipment |
| CN115150418A (en) * | 2022-08-26 | 2022-10-04 | 北京蔚领时代科技有限公司 | Data storage method of server cluster |
| CN116980474A (en) * | 2023-07-31 | 2023-10-31 | 深圳市亲邻科技有限公司 | Log data subscription management method of data pushing system |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080034393A1 (en) * | 2006-08-02 | 2008-02-07 | Aaron Crayford | Distribution of content and advertisement |
| US8041683B1 (en) * | 2006-09-29 | 2011-10-18 | Emc Corporation | Methods and apparatus for locating network logs |
| CN104836686A (en) * | 2015-05-11 | 2015-08-12 | 北京明略软件系统有限公司 | Method and device for generating configuration files of servers |
| CN105224445A (en) * | 2015-10-28 | 2016-01-06 | 北京汇商融通信息技术有限公司 | Distributed tracking system |
| CN105827608A (en) * | 2016-03-31 | 2016-08-03 | 微梦创科网络科技（中国）有限公司 | Distributed API service abnormal user identification analysis method and reverse agent service gateway |
| KR20170067118A (en) * | 2015-12-07 | 2017-06-15 | 한양대학교 에리카산학협력단 | Docker container security log analysis method and system based on hadoop distributed file system in cloud environment |
| CN109660397A (en) * | 2018-12-21 | 2019-04-19 | 北京百度网讯科技有限公司 | For acquiring system, the method and apparatus of log |
| CN109800589A (en) * | 2019-01-25 | 2019-05-24 | 深信服科技股份有限公司 | A kind of information security management and control method, system, device and readable storage medium storing program for executing |
| CN109815092A (en) * | 2019-01-28 | 2019-05-28 | 中国工商银行股份有限公司 | Cloud platform automatic telescopic method and system |
| CN109857713A (en) * | 2018-12-24 | 2019-06-07 | 中国电力科学研究院有限公司 | A kind of distributed information log acquisition processing system and method towards power business terminal |
| CN110427299A (en) * | 2019-07-19 | 2019-11-08 | 腾讯科技（深圳）有限公司 | Log processing method, relevant device and the system of micro services system application |
| CN110995497A (en) * | 2019-12-16 | 2020-04-10 | 厦门市美亚柏科信息股份有限公司 | Method for unified operation and maintenance in cloud computing environment, terminal device and storage medium |
| CN111046011A (en) * | 2019-11-27 | 2020-04-21 | 中科曙光国际信息产业有限公司 | Log collection method, system, node, electronic device and readable storage medium |
| CN111143286A (en) * | 2019-12-13 | 2020-05-12 | 无锡华云数据技术服务有限公司 | Cloud platform log management method and system |
| CN111258722A (en) * | 2020-02-14 | 2020-06-09 | 苏州浪潮智能科技有限公司 | Cluster log acquisition method, system, device and medium |
-
2020
- 2020-07-13 CN CN202010669517.5A patent/CN111930700A/en active Pending
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080034393A1 (en) * | 2006-08-02 | 2008-02-07 | Aaron Crayford | Distribution of content and advertisement |
| US8041683B1 (en) * | 2006-09-29 | 2011-10-18 | Emc Corporation | Methods and apparatus for locating network logs |
| CN104836686A (en) * | 2015-05-11 | 2015-08-12 | 北京明略软件系统有限公司 | Method and device for generating configuration files of servers |
| CN105224445A (en) * | 2015-10-28 | 2016-01-06 | 北京汇商融通信息技术有限公司 | Distributed tracking system |
| KR20170067118A (en) * | 2015-12-07 | 2017-06-15 | 한양대학교 에리카산학협력단 | Docker container security log analysis method and system based on hadoop distributed file system in cloud environment |
| CN105827608A (en) * | 2016-03-31 | 2016-08-03 | 微梦创科网络科技（中国）有限公司 | Distributed API service abnormal user identification analysis method and reverse agent service gateway |
| CN109660397A (en) * | 2018-12-21 | 2019-04-19 | 北京百度网讯科技有限公司 | For acquiring system, the method and apparatus of log |
| CN109857713A (en) * | 2018-12-24 | 2019-06-07 | 中国电力科学研究院有限公司 | A kind of distributed information log acquisition processing system and method towards power business terminal |
| CN109800589A (en) * | 2019-01-25 | 2019-05-24 | 深信服科技股份有限公司 | A kind of information security management and control method, system, device and readable storage medium storing program for executing |
| CN109815092A (en) * | 2019-01-28 | 2019-05-28 | 中国工商银行股份有限公司 | Cloud platform automatic telescopic method and system |
| CN110427299A (en) * | 2019-07-19 | 2019-11-08 | 腾讯科技（深圳）有限公司 | Log processing method, relevant device and the system of micro services system application |
| CN111046011A (en) * | 2019-11-27 | 2020-04-21 | 中科曙光国际信息产业有限公司 | Log collection method, system, node, electronic device and readable storage medium |
| CN111143286A (en) * | 2019-12-13 | 2020-05-12 | 无锡华云数据技术服务有限公司 | Cloud platform log management method and system |
| CN110995497A (en) * | 2019-12-16 | 2020-04-10 | 厦门市美亚柏科信息股份有限公司 | Method for unified operation and maintenance in cloud computing environment, terminal device and storage medium |
| CN111258722A (en) * | 2020-02-14 | 2020-06-09 | 苏州浪潮智能科技有限公司 | Cluster log acquisition method, system, device and medium |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113010483A (en) * | 2020-11-20 | 2021-06-22 | 云智慧（北京）科技有限公司 | Mass log management method and system |
| CN112527618A (en) * | 2020-12-17 | 2021-03-19 | 中国农业银行股份有限公司 | Log collection method and log collection system |
| CN112866029A (en) * | 2021-02-03 | 2021-05-28 | 树根互联股份有限公司 | Log data processing method and device based on cloud platform and server side equipment |
| CN112866029B (en) * | 2021-02-03 | 2023-09-29 | 树根互联股份有限公司 | Log data processing method and device based on cloud platform and server device |
| CN113065785A (en) * | 2021-04-13 | 2021-07-02 | 国网江苏省电力有限公司信息通信分公司 | Dynamic resource expansion method for electric power internet of things management platform |
| CN113342552A (en) * | 2021-07-05 | 2021-09-03 | 湖南快乐阳光互动娱乐传媒有限公司 | Data processing method and device, storage medium and electronic equipment |
| CN113742112B (en) * | 2021-09-15 | 2024-04-16 | 武汉联影智融医疗科技有限公司 | Electrocardiogram image generation method, system and electronic device |
| CN113742112A (en) * | 2021-09-15 | 2021-12-03 | 武汉联影智融医疗科技有限公司 | Electrocardio image generation method and system and electronic device |
| CN114244557A (en) * | 2021-11-09 | 2022-03-25 | 浪潮通用软件有限公司 | Development operation log isolation method and system based on user characteristics |
| CN114244557B (en) * | 2021-11-09 | 2023-06-16 | 浪潮通用软件有限公司 | Development operation log isolation method and system based on user characteristics |
| CN114070879A (en) * | 2021-11-26 | 2022-02-18 | 安天科技集团股份有限公司 | Data acquisition unit control method, device and related equipment |
| CN114070879B (en) * | 2021-11-26 | 2024-01-26 | 安天科技集团股份有限公司 | Data collector control method and device and related equipment |
| CN114205333A (en) * | 2021-12-09 | 2022-03-18 | 北京奇妙时光科技有限公司 | IP configuration method, cluster construction method, computer device and storage medium |
| CN114205333B (en) * | 2021-12-09 | 2024-02-27 | 北京奇妙时光科技有限公司 | IP configuration method, cluster construction method, computer device and storage medium |
| CN114756301A (en) * | 2022-04-24 | 2022-07-15 | 北京百度网讯科技有限公司 | Log processing method, device and system |
| CN114756301B (en) * | 2022-04-24 | 2023-09-01 | 北京百度网讯科技有限公司 | Log processing method, device and system |
| CN114546668A (en) * | 2022-04-27 | 2022-05-27 | 恒生电子股份有限公司 | Log collection method and device, electronic equipment and computer readable storage medium |
| CN114879942A (en) * | 2022-05-20 | 2022-08-09 | 北京宇信科技集团股份有限公司 | Distributed time wheel packet registration verification method, device, medium and equipment |
| CN114816583B (en) * | 2022-05-31 | 2024-03-19 | 以萨技术股份有限公司 | Flink-based data automatic processing method and device and electronic equipment |
| CN114816583A (en) * | 2022-05-31 | 2022-07-29 | 以萨技术股份有限公司 | Flink-based data automatic processing method and device and electronic equipment |
| CN115150418B (en) * | 2022-08-26 | 2024-01-26 | 北京蔚领时代科技有限公司 | Data storage method of server cluster |
| CN115150418A (en) * | 2022-08-26 | 2022-10-04 | 北京蔚领时代科技有限公司 | Data storage method of server cluster |
| CN116980474A (en) * | 2023-07-31 | 2023-10-31 | 深圳市亲邻科技有限公司 | Log data subscription management method of data pushing system |
| CN116980474B (en) * | 2023-07-31 | 2024-04-02 | 深圳市亲邻科技有限公司 | Log data subscription management method of data pushing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111930700A (en) | Distributed log processing method, server, system and computing equipment | |
| CN111427766B (en) | Request processing method and device and proxy server | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| CN110113381B (en) | Method and device for subscribing theme in block chain | |
| WO2021012568A1 (en) | Data processing method and related device | |
| US11042546B2 (en) | Normalization and extraction of log data | |
| CN112232881A (en) | Data detection method and device, electronic equipment and storage medium | |
| CN103379019A (en) | Microblog message pushing method, device and system | |
| CN103678372A (en) | Method and equipment for obtaining application performance of page | |
| WO2022007008A1 (en) | Resource request response method, redirection server and decision distribution server | |
| CN112422450B (en) | Computer equipment, and flow control method and device for service request | |
| CN109753424B (en) | AB test method and device | |
| CN114024972A (en) | Long connection communication method, system, device, equipment and storage medium | |
| CN105335313A (en) | Basic data transmission method and apparatus | |
| CN111815292A (en) | Business approval method and device and computer readable storage medium | |
| CN107508705B (en) | Resource tree construction method of HTTP element and computing equipment | |
| CN112235367B (en) | Method, system, terminal and storage medium for subscribing entity behavior relation message | |
| CN114969603A (en) | 5G message-based picture acquisition and picture generation method and system | |
| CN111416795B (en) | Data synchronization method, device, computing equipment and medium | |
| US20120075996A1 (en) | Techniques to support large numbers of subscribers to a real-time event | |
| CN112860292A (en) | Configuration management method and device based on application | |
| CN115202800A (en) | Edge cloud service data processing method and device, computer equipment and storage medium | |
| CN115373831A (en) | Data processing method, device and computer readable storage medium | |
| CN115988574B (en) | Data processing method, system, equipment and storage medium based on flow table | |
| CN112291202B (en) | Message data visualization method, device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |