CN111740884A - Log processing method, electronic equipment, server and storage medium - Google Patents
Log processing method, electronic equipment, server and storage medium Download PDFInfo
- Publication number
- CN111740884A CN111740884A CN202010861335.8A CN202010861335A CN111740884A CN 111740884 A CN111740884 A CN 111740884A CN 202010861335 A CN202010861335 A CN 202010861335A CN 111740884 A CN111740884 A CN 111740884A
- Authority
- CN
- China
- Prior art keywords
- log
- data
- edge node
- statistical data
- statistical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/026—Capturing of monitoring data using flow identification
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0813—Configuration setting characterised by the conditions triggering a change of settings
- H04L41/0816—Configuration setting characterised by the conditions triggering a change of settings the condition being an adaptation, e.g. in response to network events
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application relates to the technical field of internet, and discloses a log processing method, electronic equipment, a server and a storage medium, which can improve the efficiency of log processing. The method comprises the following steps: the edge node collects incremental logs generated in a preset time length when a service program of the edge node runs, and performs stream type analysis processing on the incremental logs according to configuration information of the edge node to obtain log statistical data. And the edge node reports the log statistical data to the central server, and the central server stores the log statistical data, analyzes the log statistical data to obtain a configuration instruction, and then sends the configuration instruction to the edge node. And the edge node receives the configuration instruction from the central service end and updates the configuration information according to the configuration instruction.
Description
Technical Field
The present application relates to the field of internet technologies, and in particular, to a log processing method, an electronic device, a server, and a storage medium.
Background
A monitoring system in a network mainly comprises two large components, namely an edge node and a cloud server, so that a client/server (C/S) framework is formed. The edge nodes are monitored equipment and are used for operating a service program and generating log files, and the cloud server side operates a management platform service and is used for collecting and analyzing the log files of the edge nodes. In a traditional log processing mode, each edge node uploads collected logs to a cloud server, and then the cloud server performs calculation analysis on the logs reported by a plurality of edge nodes. In practice, it is found that in this way, the cloud server needs to process log files with huge data volume, and therefore the efficiency of log processing is low.
Disclosure of Invention
The embodiment of the application discloses a log processing method, electronic equipment, a server and a storage medium, which can improve the efficiency of log processing.
A first aspect of the embodiments of the present application provides a log processing method, where the method is applied to an edge node; the method comprises the following steps:
acquiring an incremental log generated in a preset time length when a service program of the edge node operates;
performing stream analysis processing on the incremental log according to the configuration information of the edge node to obtain log statistical data;
reporting the log statistical data to a central server side, so that the central server side stores the log statistical data, analyzes the log statistical data to obtain a configuration instruction, and then sends the configuration instruction to the edge node;
and receiving the configuration instruction from the central server, and updating the configuration information according to the configuration instruction.
As an optional implementation manner, in the first aspect of this embodiment of the present application, the configuration information includes a query language instruction; the performing stream analysis processing on the incremental log according to the configuration information of the edge node to obtain log statistical data includes:
formatting the incremental log to obtain a log data stream meeting a preset format corresponding to the query language instruction;
and performing stream analysis processing on the log data stream according to the query language instruction to obtain log statistical data.
As an optional implementation manner, in the first aspect of the embodiment of the present application, the performing stream analysis processing on the log data stream according to the query language instruction to obtain log statistical data includes:
determining the data type inquired by the query language instruction according to the query language instruction;
dividing the log data stream according to the divider included in the log data stream to obtain a plurality of lines of log data;
acquiring target data corresponding to the data type in each row of log data;
and according to the query language instruction, performing stream type statistical analysis on target data corresponding to the plurality of lines of log data to obtain log statistical data.
As an optional implementation manner, in the first aspect of the embodiment of the present application, the log statistical data includes a log statistical table, where the log statistical table includes a first column of attributes and a second column of attributes, the first column of attributes is used to represent node information, and the second column of attributes is used to represent processing time; before reporting the log statistical data to a central server, the method further includes:
acquiring edge node information and node processing time corresponding to the edge node;
writing the edge node information into a corresponding column of the first column attribute in the log statistical table, and writing the node processing time into a corresponding column of the second column attribute in the log statistical table.
A second aspect of the embodiments of the present application provides a log processing method, where the method is applied to a central server; the method comprises the following steps:
receiving log statistical data reported by an edge node, wherein the log statistical data is data obtained after the edge node performs streaming analysis processing on an incremental log according to configuration information of the edge node, and the incremental log is a log generated within a preset time length when a service program of the edge node operates;
storing the log statistical data, analyzing the log statistical data and generating a configuration instruction;
and issuing the configuration instruction to the edge node so that the edge node updates the configuration information according to the configuration instruction.
As an optional implementation manner, in the second aspect of this embodiment of the present application, after the storing the log statistics, the method further includes:
acquiring full log data stored by the central server, wherein the full log data comprises log statistical data reported by all edge nodes managed by the central server;
analyzing the log statistical data to generate a configuration instruction, wherein the configuration instruction comprises the following steps:
performing stream analysis on the log statistical data to obtain real-time statistical data;
performing statistical analysis on the full log data to obtain historical statistical data;
determining a summary result according to the real-time statistical data and the historical statistical data;
analyzing by combining the summary result and the configuration information of the edge node to obtain an analysis result;
and generating a configuration instruction according to the analysis result and/or the user configuration information.
As an optional implementation manner, in the second aspect of this embodiment of the present application, after obtaining the analysis result, the method further includes:
if the analysis result contains the alarm item reaching the alarm condition, acquiring a data item relation chain of each storage table stored by the central server;
determining a target data item matched with the alarm item in the data item relation chain, and determining a monitoring target corresponding to the target data item;
and outputting alarm information of the monitoring target.
A third aspect of the embodiments of the present application provides an electronic device, including:
one or more memories;
one or more processors for executing one or more computer programs stored in the one or more memories for performing the method according to the first aspect of the application.
A fourth aspect of the embodiments of the present application provides a server, including:
one or more memories;
one or more processors for executing one or more computer programs stored in the one or more memories to perform the method according to the second aspect of the present application.
A fifth aspect of embodiments of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, cause the computer to perform the method according to the first or second aspect of the present application.
A sixth aspect of embodiments of the present application provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method according to the first or second aspect of the present application.
Compared with the prior art, the embodiment of the application has the following beneficial effects:
in the embodiment of the application, the edge node collects the incremental logs generated within a preset time length during the operation of the service program, performs streaming analysis processing on the incremental logs according to the configuration information of the edge node to obtain the log statistical data, and can perform real-time statistical analysis on the collected log data by using the computing capacity of the edge node, so that the reported data volume of the edge node is reduced, the network congestion possibility is reduced, the data processing pressure of a central server is reduced, and the real-time performance and efficiency of log processing are improved. And then, the edge node reports the log statistical data to the central server, so that the central server can store the log statistical data reported by the edge node, perform secondary analysis on the log statistical data to obtain a configuration instruction issued to the edge node, and enable the edge node to update the configuration information of the edge node according to the configuration instruction received from the central server, thereby realizing real-time monitoring and configuration updating of each edge node through log analysis and improving the self-adaptive adjustment capability of the system.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a system architecture diagram of a log processing system according to an embodiment of the present disclosure;
FIG. 2 is a schematic flowchart of a log processing method disclosed in an embodiment of the present application;
FIG. 3 is a schematic flow chart diagram illustrating another log processing method disclosed in an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application;
FIG. 5 is a schematic structural diagram of a server disclosed in an embodiment of the present application;
fig. 6 is a schematic structural diagram of another electronic device disclosed in an embodiment of the present application;
fig. 7 is a schematic structural diagram of another server disclosed in the embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first", "second", "third", "fourth", and the like in the description and claims of the present application are used for distinguishing different objects, and are not used for describing a specific order. The terms "comprises," "comprising," and "having," and any variations thereof, of the embodiments of the present application, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The embodiment of the application discloses a log processing method, electronic equipment and a server, which can improve the efficiency of log processing. The following detailed description is made with reference to the accompanying drawings.
In order to better understand the log processing method disclosed in the embodiment of the present application, a log processing system disclosed in the embodiment of the present application is described below.
Referring to fig. 1, fig. 1 is a schematic diagram of a system architecture of a log processing system according to an embodiment of the present disclosure. As shown in fig. 1, the log processing system may include a central server 10 and N edge nodes, where N is a positive integer. For ease of understanding, the following description will be given by taking the edge node 12 as an example, and the description of the edge node 12 is also applicable to other edge nodes in the system.
In the embodiment of the present application, the central server 10 is mainly used for running a management platform service, and collecting and analyzing data reported by each edge node. The central server 10 may be a single server running an operating system (including Linux, Unix, Windows, and the like), or may be a cluster formed by multiple servers, which is not limited in this respect.
In the embodiment of the present application, the edge node 12 may correspond to a monitored device managed by the central server 10, and the monitored device may be an electronic device running an operating system (including Linux, Unix, Windows, and the like). The electronic device may include a smart phone, a wearable device, a vehicle-mounted terminal, a portable terminal, a Personal Digital Assistant (PDA), a Portable Multimedia Player (PMP) device, a notebook pad, a wireless broadband (WiBro) terminal, a tablet PC, and a smart PC, which are not limited in particular. When the edge node 12 runs the service program, the edge node 12 may generate a log of the running of the service program, where the log is a system operation event record, and each row of the log may record descriptions of related operations such as date, time, user, and action, and has functions of processing historical data, tracing diagnostic problems, and understanding system activities.
In some alternative implementations, the central server 10 is communicatively coupled and data exchanged with the edge nodes 12 via a transport network. The transmission network may include, but is not limited to, a wireless computer network, a wired computer network, a fourth generation mobile communication technology network (4G network), a fifth generation mobile communication technology network (5G network), a frequency modulation transmission network, a medium wave transmission network, a short wave transmission network, or a satellite transmission network.
In some alternative implementations, the log processing system may employ a MapReduce computing framework. MapReduce is a distributed programming model and can automatically complete the parallelization processing of computing tasks. In the embodiment of the application, tasks can be automatically distributed and executed on the edge nodes, incremental logs can be collected, pre-analysis processing can be carried out on the incremental logs, complex parallel computing such as data distribution storage, data communication and secondary analysis processing is given charge to the central server, and therefore programming and computing processing of large-scale data are achieved.
It should be understood that the log processing system described above is applicable to the log processing method disclosed in the embodiment of the present application. The following describes in detail a log processing method disclosed in an embodiment of the present application.
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a log processing method according to an embodiment of the present disclosure. As shown in fig. 2, the method includes:
201. the edge node collects incremental logs generated in a preset time length when a service program of the edge node runs.
In this embodiment, the service program of the edge node may include a computer program running on an operating system, such as a system service and an application program installed on the edge node, which is not particularly limited. Different business programs are used to perform specific business functions, such as document processing programs, image processing programs, streaming media applications, Web applications, driver applications, and Web applications.
In this embodiment of the application, the incremental log is a log that is newly added within a preset time length when the service program of the edge node runs, and the preset time length may be set manually, for example, 5 minutes or 10 minutes, and is not particularly limited.
In some optional implementation manners, the edge node may insert a listener into a source code corresponding to the service program, and collect, by the listener, an incremental log generated when the service program runs. The insertion listener may include, but is not limited to, the following three ways: 1. writing a check code in a calling interface corresponding to the service program; 2. intercepting an appointed interface corresponding to a service program by using an interceptor function of Spring, and realizing a signature verification algorithm by a monitor, wherein the Spring is a container frame of lightweight control inversion (IoC) and Aspect Oriented Programming (AOP); 3. and adding annotations to the calling method corresponding to the business program by using the self-defined annotation function of the Spring AOP.
202. And the edge node performs stream analysis processing on the incremental log according to the configuration information of the edge node to obtain log statistical data.
In the embodiment of the application, the stream analysis processing may refer to that the edge node collects the data stream of the incremental log and performs analysis processing on the collected data stream in real time, so that the edge node does not need to report the incremental log to the central server side for batch processing after uniformly collecting the incremental log, and is suitable for a dynamic and real-time big data processing scenario.
In this embodiment of the application, the configuration information of the edge node may include a processing instruction and threshold information corresponding to the edge node when the edge node collects the incremental log, analyzes the incremental log, and subsequently reports the log statistical data, which is not specifically limited. Wherein the threshold information may be used to screen out data that meets a particular threshold condition.
Accordingly, the format of the log statistics may include, but is not limited to, a log format, a flog format, a squid format, a nginx format, an apache format, a lighttp format, an fms format, a wms format, a custom list format, and the like. Optionally, the custom list format refers to table attributes defined by a user in a table storing log data according to a requirement, for example, the table attributes may include column attributes, row attributes and the like manually preset by the user, and the custom list format may also be flexibly adjusted according to configuration parameters indicated by the processing instruction, which is not specifically limited. Illustratively, if the configuration parameters include date, timestamp, belonging service, node number, log content, etc., the custom list format may also include five-column attributes of date, timestamp, belonging service, node number, and log content.
As an optional implementation manner, the threshold information may include a reporting threshold condition. After the edge node obtains the log statistical data, whether the log statistical data meets a reporting threshold condition can be detected. If not, the edge node can continue to collect the incremental logs generated during the operation of the service program of the edge node; if so, step 203 may be performed. The reporting threshold condition may include a data amount threshold satisfying the reporting condition, and a specific value thereof may be set manually and is not limited specifically. Therefore, the reasonable control of the data volume reported by the edge node is realized through the reporting threshold condition, and the effective utilization rate of the data reported by the edge node every time can be improved.
203. And the edge node reports the log statistical data to the central server.
204. And the central server receives the log statistical data reported by the edge node and stores the log statistical data.
In this embodiment of the application, the central server may store the log statistical data in a database, where the database may be a local database or a cloud database, and is not limited.
Optionally, the central server may further store the log statistical data of the edge node into a database storage address corresponding to the edge node, so that data of different edge nodes are independently stored. Thus, in one implementation, the log statistics may also include edge node information and node processing time corresponding to the edge node. The node processing time may be a processing time for obtaining the log statistical data by the edge node. The edge node information may include identification information of the edge node (such as a device number or a node number), Linux Virtual Server (LVS)/remote dictionary service (REDIS)/MySQL configuration check information, hypertext transfer protocol (HTTP)/Transmission Control Protocol (TCP)/Domain Name System (DNS) test information, and Central Processing Unit (CPU)/memory/disk/network test information, which are not particularly limited.
Therefore, the edge node information and the node processing time are added into the log statistical data and reported to the central server, so that the reference effect can be achieved for correctly storing the log statistical data, and the central server can conveniently make a more comprehensive analysis decision by combining various configuration detection information and test information of the edge node, so that the accuracy of adjusting the configuration of the edge node is further improved.
205. And the central server analyzes the log statistical data to generate a configuration instruction.
In the embodiment of the present application, the configuration instruction is used to determine the update content in the configuration information, and instruct the edge node to update the existing configuration information according to the update content. For example, if the configuration instruction is to modify the acquired data amount threshold of the incremental log to be 100, the edge node may update the threshold information when the edge acquires the incremental log to be 100 according to the configuration instruction.
As an optional implementation manner, the central server may analyze the log statistical data according to a preset analysis algorithm to obtain an analysis result. The predetermined analysis algorithm may include, but is not limited to, a statistical analysis algorithm, a classification pattern mining algorithm, a clustering pattern mining algorithm, a sequence pattern mining algorithm, and an association rule algorithm. The statistical analysis algorithm can be used for performing statistical analysis on frequency, average value and the like of log data items such as browsing time, browsing path and the like, the classification pattern mining algorithm can be used for finding out whether a classification rule of a specific subset or class exists among the data items in the log statistical data, the clustering pattern mining algorithm can be used for classifying the data items with similar characteristics in the log statistical data, the sequence pattern mining algorithm can be used for mining the sequence of high-frequency occurrence in the log statistical data, and the association rule algorithm can be used for analyzing the association among different data items in the log statistical data.
Based on this, in an implementation manner, the central server may directly obtain, according to the analysis result, configuration adjustment information corresponding to the analysis result, so as to generate the configuration instruction according to the configuration adjustment information. For example, if it is known that the number of queries on a certain query field in the configuration information is smaller than the query threshold according to the analysis result, the configuration adjustment information for removing the query field may be determined, so as to improve the query efficiency.
In another implementation, the central server may output the analysis result to a User Interface (UI) for display. And then, the central server receives user configuration information input by an administrator through a user interface, and generates a configuration instruction for the edge node according to the user configuration information. For example, the administrator manually deletes a certain parameter in the configuration information, or the administrator adjusts the size of threshold information in the configuration information.
In another implementation manner, the central server may further combine the analysis result and the user configuration information to obtain a comprehensive evaluation result, and generate a configuration instruction for the edge node according to the comprehensive evaluation result.
Therefore, by implementing the optional implementation manner, the configuration adjustment information can be automatically determined for the analysis result through the central server, or the configuration can be manually adjusted, so that the real-time monitoring and the configuration updating of each edge node are realized.
206. And the central server side issues the configuration instruction to the edge node.
207. And the edge node receives the configuration instruction from the central service end and updates the configuration information according to the configuration instruction.
Therefore, by implementing the method embodiment, the collected log data can be subjected to real-time statistical analysis by utilizing the computing power of the edge node, so that the reported data volume of the edge node is reduced, the network congestion possibility is reduced, the data processing pressure of a central server is reduced, and the real-time performance and efficiency of log processing are improved. In addition, real-time monitoring and configuration updating of each edge node can be realized through log analysis, and the self-adaptive adjustment capability of the system is improved.
Referring to fig. 3, fig. 3 is a schematic flowchart illustrating another log processing method according to an embodiment of the present disclosure. In the embodiment shown in FIG. 3, the configuration information for the edge node may include query language instructions. As shown in fig. 3, the method includes:
301. the edge node collects incremental logs generated in a preset time length when a service program of the edge node runs.
302. And the edge node formats the incremental log to obtain a log data stream meeting the preset format corresponding to the query language instruction.
In the embodiment of the present application, the query language instruction is a query instruction generated based on a preset query language, and the query language instruction is used for determining a query target from the log data stream, and performing statistics and analysis on the query target. The query language may include a Structured Query Language (SQL), an Object Query Language (OQL), or the like, and is not particularly limited. Different query languages can process query targets in different formats, so that the incremental log can be converted into a log data stream meeting the preset format by formatting the incremental log according to the preset format corresponding to the query language instruction. For example, if the query language is SQL, the corresponding predefined format may be a data table, which is a collection of data items consisting of columns and rows. The log data stream is a log data sequence that is read in a prescribed order. Therefore, the log data stream is obtained by formatting the incremental log, and the universality of the log data stream to different query languages can be improved, so that the feasibility and the accuracy of analysis processing are ensured.
As an alternative, the configuration information may also include an increase data volume threshold. After step 302, the edge node may also detect whether the data volume of the log data stream is greater than or equal to the incremental data volume threshold. If the data volume of the log data stream is less than the incremental data volume threshold, then execution continues with step 303. If the data volume of the log data stream is greater than or equal to the increase data volume threshold, in one implementation, the edge node may randomly select data from the log data stream to discard, obtain the log data stream after discarding the data, and make the data volume of the log data stream after discarding the data less than the increase data volume threshold. In another implementation, the edge node may further select data discard from the log data stream according to a preset discard percentage, for example, the discard percentage is 5% of the log data stream. In another implementation manner, the edge node may further determine a data type with the lowest processing priority in the log data stream, and then select data corresponding to the data type from the log data stream to discard, so as to obtain the log data stream with discarded data. The processing priority may be artificially set, or the processing priority may be related to the processing time (for example, the earlier the processing time, the lower the processing priority), which is not particularly limited.
Correspondingly, the edge node can perform stream analysis processing on the log data stream after discarding the data according to the query language instruction to obtain log statistical data. Therefore, when the log data stream is increased too fast, partial data of the log data stream is lost, the log data stream can be controlled to keep reasonable increase, and the stability of subsequent processing data stream is improved.
303. And the edge node performs stream analysis processing on the log data stream according to the query language instruction to obtain log statistical data.
As an optional implementation manner for step 303, the edge node may specifically determine, according to the query language instruction, a data type queried by the query language instruction. And then, the edge node divides the log data stream according to the dividers included in the log data stream to obtain a plurality of lines of log data. And finally, the edge node acquires target data corresponding to the data type in each row of log data, and performs stream type statistical analysis on the target data corresponding to the plurality of rows of log data according to the query language instruction to acquire log statistical data.
The separator is used to divide the log data stream into rows, and the separator may include uniquotation marks, double quotation marks, commas, blanks, semicolons, colons, and the like, which is not particularly limited. And the query language instructions include the data type of the query and the computation function. When the query language instruction is an SQL instruction, the data type of the query may refer to the column attributes in the table, and each row of log data may refer to the tuple information (i.e., row) in the table. The calculation function may include a summation function, an average calculation function, a counting function, a maximum search function, a minimum search function, a ratio calculation function, and the like, and is not particularly limited. For example, assume that the type of data queried by the query language instruction is an IP address and the computation function of the query language instruction is a count function. The edge node can count and count the target data corresponding to the IP address in each row of log data according to the query language instruction, and finally obtain the counting times corresponding to different IP addresses, so that the distribution and access frequency of the source IP addresses can be conveniently analyzed.
Therefore, target data to be analyzed is screened from the log data stream through the query language instruction, irrelevant data in the log data stream can be filtered, pertinence of data analysis is improved, and accordingly data processing and analyzing efficiency is improved. In addition, the query language instruction can also specify various calculation functions, so that diversified statistical analysis requirements can be met, effective information can be screened in advance, and reduction of the reported data volume of the edge nodes is facilitated.
Further, as an optional implementation manner, the log statistic data may include a log statistic table, in this case, the log statistic table may further include a first column of attributes and a second column of attributes, where the first column of attributes is used to represent the node information, and the second column of attributes is used to represent the processing time. Based on the above, before reporting the log statistical data to the central server, the edge node can also obtain the edge node information and the node processing time of the edge node, and directly write the edge node information into the corresponding column of the first column attribute in the log statistical table, and write the node processing time into the corresponding column of the second column attribute in the log statistical table, so as to quickly add the matched edge node information and node processing time to each row of log data in the log statistical table.
304. And the edge node reports the log statistical data to the central server.
305. And the central server receives the log statistical data reported by the edge node and stores the log statistical data.
As an optional implementation manner, the central server may specifically construct a newly added storage table of the edge node according to the column attribute information and the tuple information included in the log statistical data, and write each data item into the newly added storage table according to the column attribute and the tuple corresponding to each data item in the log statistical table. For example, if the log statistics table data includes 20 rows of log data, and each row of log data corresponds to four data types of access time, user IP address, access URL, and request method, the newly added storage table constructed may include four column attributes of access time, user IP address, access URL, and request method, and 20 rows of tuples. The central server side can write each data item of the log statistical data into the corresponding position of the newly added storage table according to the corresponding relation between the column attributes and the tuples. Therefore, the central server side does not need to build a table in advance, can automatically generate a table structure according to the content of the log statistical data and store the data, and improves the flexibility of data storage and the effective utilization rate of storage resources.
306. The central server side obtains the total log data stored by the central server side, wherein the total log data comprises log statistical data reported by all edge nodes managed by the central server side.
In the embodiment of the application, the central server can obtain the stored full log data from the database. The total log data may be log statistical data reported by all edge nodes managed by the central server within the same node processing time period, or may be all log statistical data stored in the database by each node, which is not limited specifically.
307. And the central server performs stream analysis on the log statistical data to obtain real-time statistical data, and performs statistical analysis on the full log data to obtain historical statistical data.
308. And the central server side determines a summary result according to the real-time statistical data and the historical statistical data.
In this embodiment of the application, optionally, the central server may directly superimpose the same data items in the real-time statistical data and the historical statistical data, and determine a summary result. Or, optionally, the central server may superimpose the real-time statistical data and the historical statistical data according to the configured superimposition proportion to obtain a summary result. The superposition proportion is the ratio of the weights of the real-time statistical data and the historical statistical data when data are superposed, such as 1: 2 or 2: 3. illustratively, if the superposition ratio is 1: 2, for data item a in the real-time statistical data and data item b in the historical statistical data, assuming that a and b are the same data type, the summary result = a +2 b.
309. And the central server analyzes the summary result and the configuration information of the edge nodes to obtain an analysis result.
As an optional implementation manner, if the analysis result contains an alarm item reaching the alarm condition, alarm information is output, so that a timely alarm feedback effect is achieved. The alarm condition may include a threshold condition corresponding to each data type, and the alarm item may be a data item that reaches the alarm condition in the analysis result. The manner of outputting the alarm information by the central server may include, but is not limited to: the central server side can display the alarm items in the analysis result in a marking way (such as highlighting, color changing or flashing) when the analysis result is output by the user interface; or, the central service end may output an alarm popup, and display alarm information related to the alarm item in the alarm popup.
Further, as an alternative implementation, the stored storage table in the database of the central server may be a relational data table. If the analysis result contains the alarm item meeting the alarm condition, the central server side can specifically acquire the data item relation chain of each storage table stored by the central server side. And then, the central server determines a target data item matched with the alarm item in the data item relation chain, and determines a monitoring target corresponding to the target data item, so that alarm information of the monitoring target is output.
The target data item may be a data item of which the column attribute in the tuple to which the alarm item of the target storage table belongs corresponds to the edge node information, and the target storage table may be a storage table containing the alarm item. Accordingly, the monitoring target may be an edge node in the log processing system that matches the target data item. Therefore, based on the data item relationship among the storage tables, the monitoring target related to the alarm item can be quickly tracked, and the effect of positioning the alarm source layer by layer is realized.
Still further, optionally, the central server may further send the alarm information to the monitoring target, so as to output an alarm at the corresponding edge node at the same time.
310. And the central server generates a configuration instruction according to the analysis result and/or the user configuration information.
In this embodiment of the application, the central server performs stream analysis on the log statistical data and the full log data, and in steps 309 to 310, reference may be made to the description of step 205 in the foregoing embodiment, which is not described herein again. Obviously, by implementing the steps 306 to 309, the log statistical data reported by a single edge node and the log statistical data of all edge nodes can be combined to perform comprehensive statistical analysis, so that the comprehensiveness of log analysis is further improved, more accurate analysis results can be obtained, and the reasonability of configuration adjustment decision of each edge node in the system is further improved.
311. And the central server side issues the configuration instruction to the edge node.
312. And the edge node receives the configuration instruction from the central service end and updates the configuration information according to the configuration instruction.
Therefore, by implementing the method embodiment, the collected log data can be subjected to real-time statistical analysis by utilizing the computing power of the edge node, so that the reported data volume of the edge node is reduced, the network congestion possibility is reduced, the data processing pressure of a central server is reduced, and the real-time performance and efficiency of log processing are improved. In addition, real-time monitoring and configuration updating of each edge node can be realized through log analysis, and the self-adaptive adjustment capability of the system is improved. Furthermore, the universality of the log data stream to different query languages can be improved, so that the feasibility and the accuracy of analysis processing are ensured. Furthermore, the comprehensiveness of log analysis can be further improved, more accurate analysis results can be obtained, and the reasonability of configuration adjustment decisions of each edge node in the system can be further improved.
The log processing method in the embodiment of the present application is described above, and the electronic device and the server in the embodiment of the present application are described below.
Referring to fig. 4, fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 4, the electronic device includes a log collection module 401, a log preprocessing module 402, a data reporting module 403, and a configuration module 404, where:
the log collection module 401 is configured to collect an incremental log generated within a preset time length when a service program of the electronic device runs.
And the log preprocessing module 402 is configured to perform streaming analysis processing on the incremental log according to the configuration information of the electronic device, so as to obtain log statistical data.
The data reporting module 403 is configured to report the log statistical data to the central server, so that the central server stores the log statistical data, analyzes the log statistical data, obtains a configuration instruction, and issues the configuration instruction to the electronic device.
The configuration module 404 is configured to receive a configuration instruction from the central service end, and update configuration information according to the configuration instruction.
In the embodiment of the present application, as an optional implementation manner, the configuration information includes a query language instruction. The log preprocessing module 402 may include a log formatting sub-module and a log processing sub-module, where the log formatting sub-module is configured to format the incremental log to obtain a log data stream meeting a preset format corresponding to the query language instruction. And the log processing submodule is used for performing stream analysis processing on the log data stream according to the query language instruction to obtain log statistical data.
Further, as an optional implementation manner, the log processing sub-module is further configured to determine, according to the query language instruction, a data type queried by the query language instruction; segmenting the log data stream according to the segmenters included in the log data stream to obtain a plurality of lines of log data; acquiring target data corresponding to the data type in each row of log data; and according to the query language instruction, performing stream type statistical analysis on target data corresponding to the plurality of lines of log data to obtain log statistical data.
In this embodiment, as an optional implementation manner, the log statistical data includes a log statistical table, where the log statistical table includes a first column of attributes and a second column of attributes, the first column of attributes is used to represent node information, and the second column of attributes is used to represent processing time.
In this embodiment, as an optional implementation manner, the log preprocessing module 402 is further configured to obtain edge node information and node processing time corresponding to the electrical device before the data reporting module 403 reports the log statistical data to the central server; and writing the edge node information into a corresponding column of a first column of attributes in the log statistical table, and writing the node processing time into a corresponding column of a second column of attributes in the log statistical table.
It should be noted that the electronic device in this embodiment is suitable for the edge node, and the specific implementation process of this embodiment may refer to the specific implementation process described in the method embodiment, which is not described herein again.
Therefore, by implementing the embodiment, the collected log data can be subjected to real-time statistical analysis by utilizing the computing power of the edge node, so that the reported data volume of the edge node is reduced, the network congestion possibility is reduced, the data processing pressure of a central server is reduced, and the real-time performance and efficiency of log processing are improved. In addition, real-time monitoring and configuration updating of each edge node can be realized through log analysis, and the self-adaptive adjustment capability of the system is improved.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a server according to an embodiment of the present disclosure. As shown in fig. 5, the server includes a data receiving module 501, a data storage module 502, a configuration updating module 503, and a notification module 504, wherein:
the data receiving module 501 is configured to receive log statistical data reported by an edge node, where the log statistical data is obtained after the edge node performs streaming processing on an incremental log according to configuration information of the edge node, and the incremental log is a log generated within a preset time length when a service program of the edge node runs;
a data storage module 502, configured to store log statistical data;
a configuration update module 503, configured to analyze the log statistical data and generate a configuration instruction;
the notification module 504 is configured to issue a configuration instruction to the edge node, so that the edge node updates the configuration information according to the configuration instruction.
In this embodiment, as an optional implementation manner, the server may further include an obtaining module, where: the obtaining module is configured to obtain full log data stored by the server after the log statistical data is stored by the data storage module 502, where the full log data includes log statistical data reported by all edge nodes managed by the server. The configuration updating module 503 is further configured to perform streaming analysis on the log statistical data to obtain real-time statistical data; carrying out statistical analysis on the full log data to obtain historical statistical data; determining a summary result according to the real-time statistical data and the historical statistical data; analyzing by combining the summary result and the configuration information of the edge nodes to obtain an analysis result; and generating a configuration instruction according to the analysis result and/or the user configuration information.
In this embodiment, as an optional implementation manner, the server may further include an alarm module, where the alarm module is configured to, after the configuration update module 503 obtains the analysis result, obtain a data item relationship chain of each storage table stored by the central server if the analysis result includes an alarm item meeting an alarm condition; determining a target data item matched with the alarm item in the data item relation chain, and determining a monitoring target corresponding to the target data item; and outputting alarm information of the monitoring target.
It should be noted that the server in this embodiment is applicable to the central server, and the specific implementation process of this embodiment may refer to the specific implementation process described in the method embodiment, which is not described herein again.
Therefore, by implementing the embodiment, the collected log data can be subjected to real-time statistical analysis by utilizing the computing power of the edge node, so that the reported data volume of the edge node is reduced, the network congestion possibility is reduced, the data processing pressure of a central server is reduced, and the real-time performance and efficiency of log processing are improved. In addition, real-time monitoring and configuration updating of each edge node can be realized through log analysis, and the self-adaptive adjustment capability of the system is improved.
Referring to fig. 6, fig. 6 is a schematic structural diagram of another electronic device disclosed in the embodiment of the present application. The electronic device includes:
one or more memories 601;
one or more processors 602 for executing one or more computer programs stored in the one or more memories 601 to perform the methods described in the embodiments above.
It should be noted that the electronic device in this embodiment may be the edge node, so that the specific implementation process of this embodiment may refer to the specific implementation process described in the method embodiment, and is not described here.
Referring to fig. 7, fig. 7 is a schematic structural diagram of another server disclosed in the embodiment of the present application. The server includes:
one or more memories 701;
one or more processors 702 to execute one or more computer programs stored in the one or more memories 701 to perform the methods described in the embodiments above.
It should be noted that the server in this embodiment may be the central server, so that the specific implementation process of this embodiment may refer to the specific implementation process described in the method embodiment, and is not described here.
The embodiment of the present application provides a computer-readable storage medium, on which computer instructions are stored, and when the computer instructions are executed, the computer instructions cause a computer to execute the log processing method described in the above method embodiment.
The embodiments of the present application also disclose a computer program product, wherein, when the computer program product runs on a computer, the computer is caused to execute part or all of the steps of the method as in the above method embodiments.
It will be understood by those of ordinary skill in the art that all or part of the steps in the methods of the above embodiments may be performed by associated hardware instructed by a program, and the program may be stored in a computer-readable storage medium, where the storage medium includes read-only memory (ROM), Random Access Memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), or other memory, a magnetic disk, a magnetic tape, or a combination thereof, Or any other medium which can be used to carry or store data and which can be read by a computer.
The above detailed description is given to a log processing method, an electronic device, a server, and a storage medium disclosed in the embodiments of the present application, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiments is only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A log processing method is characterized in that the method is applied to an edge node; the method comprises the following steps:
acquiring an incremental log generated in a preset time length when a service program of the edge node operates;
performing stream analysis processing on the incremental log according to the configuration information of the edge node to obtain log statistical data;
reporting the log statistical data to a central server side, so that the central server side stores the log statistical data, analyzes the log statistical data to obtain a configuration instruction, and then sends the configuration instruction to the edge node;
and receiving the configuration instruction from the central server, and updating the configuration information according to the configuration instruction.
2. The method of claim 1, wherein the configuration information comprises query language instructions; the performing stream analysis processing on the incremental log according to the configuration information of the edge node to obtain log statistical data includes:
formatting the incremental log to obtain a log data stream meeting a preset format corresponding to the query language instruction;
and performing stream analysis processing on the log data stream according to the query language instruction to obtain log statistical data.
3. The method according to claim 2, wherein the performing stream analysis processing on the log data stream according to the query language instruction to obtain log statistics data comprises:
determining the data type inquired by the query language instruction according to the query language instruction;
dividing the log data stream according to the divider included in the log data stream to obtain a plurality of lines of log data;
acquiring target data corresponding to the data type in each row of log data;
and according to the query language instruction, performing stream type statistical analysis on target data corresponding to the plurality of lines of log data to obtain log statistical data.
4. The method according to any one of claims 1 to 3, wherein the log statistical data comprises a log statistical table, the log statistical table comprises a first column of attributes and a second column of attributes, the first column of attributes is used for representing node information, and the second column of attributes is used for representing processing time; before reporting the log statistical data to a central server, the method further includes:
acquiring edge node information and node processing time corresponding to the edge node;
writing the edge node information into a corresponding column of the first column attribute in the log statistical table, and writing the node processing time into a corresponding column of the second column attribute in the log statistical table.
5. A log processing method is characterized in that the method is applied to a central server; the method comprises the following steps:
receiving log statistical data reported by an edge node, wherein the log statistical data is data obtained after the edge node performs streaming analysis processing on an incremental log according to configuration information of the edge node, and the incremental log is a log generated within a preset time length when a service program of the edge node operates;
storing the log statistical data, analyzing the log statistical data and generating a configuration instruction;
and issuing the configuration instruction to the edge node so that the edge node updates the configuration information according to the configuration instruction.
6. The method of claim 5, wherein after storing the log statistics, the method further comprises:
acquiring full log data stored by the central server, wherein the full log data comprises log statistical data reported by all edge nodes managed by the central server;
analyzing the log statistical data to generate a configuration instruction, wherein the configuration instruction comprises the following steps:
performing stream analysis on the log statistical data to obtain real-time statistical data;
performing statistical analysis on the full log data to obtain historical statistical data;
determining a summary result according to the real-time statistical data and the historical statistical data;
analyzing by combining the summary result and the configuration information of the edge node to obtain an analysis result;
and generating a configuration instruction according to the analysis result and/or the user configuration information.
7. The method of claim 6, wherein after obtaining the analysis results, the method further comprises:
if the analysis result contains the alarm item reaching the alarm condition, acquiring a data item relation chain of each storage table stored by the central server;
determining a target data item matched with the alarm item in the data item relation chain, and determining a monitoring target corresponding to the target data item;
and outputting alarm information of the monitoring target.
8. An electronic device, characterized in that the electronic device comprises:
one or more memories;
one or more processors to execute one or more computer programs stored in the one or more memories to perform the method of any of claims 1-4.
9. A server, wherein the service comprises:
one or more memories;
one or more processors to execute one or more computer programs stored in the one or more memories to perform the method of any of claims 5-7.
10. A computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any of claims 1 to 4 or to perform the method of any of claims 5 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010861335.8A CN111740884B (en) | 2020-08-25 | 2020-08-25 | Log processing method, electronic equipment, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010861335.8A CN111740884B (en) | 2020-08-25 | 2020-08-25 | Log processing method, electronic equipment, server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111740884A true CN111740884A (en) | 2020-10-02 |
| CN111740884B CN111740884B (en) | 2021-06-25 |
Family
ID=72658736
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010861335.8A Active CN111740884B (en) | 2020-08-25 | 2020-08-25 | Log processing method, electronic equipment, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111740884B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112291085A (en) * | 2020-10-10 | 2021-01-29 | 北京金山云网络技术有限公司 | Fault positioning method, device, equipment and medium |
| CN112511456A (en) * | 2020-12-21 | 2021-03-16 | 北京百度网讯科技有限公司 | Flow control method, flow control device, flow control equipment, storage medium and computer program product |
| CN112615742A (en) * | 2020-12-18 | 2021-04-06 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for early warning |
| CN113010891A (en) * | 2021-02-26 | 2021-06-22 | 中科天齐(山西)软件安全技术研究院有限公司 | Application program safety detection method and device, electronic equipment and storage medium |
| CN113055368A (en) * | 2021-03-08 | 2021-06-29 | 云盾智慧安全科技有限公司 | Web scanning identification method and device and computer storage medium |
| CN113190522A (en) * | 2021-06-29 | 2021-07-30 | 阿里云计算有限公司 | Log processing method and device, computing equipment and storage medium |
| CN114090644A (en) * | 2022-01-20 | 2022-02-25 | 飞狐信息技术（天津）有限公司 | Data processing method and device |
| WO2024109569A1 (en) * | 2022-11-21 | 2024-05-30 | 中兴通讯股份有限公司 | Data processing method and apparatus, and electronic device and computer-readable storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020046273A1 (en) * | 2000-01-28 | 2002-04-18 | Lahr Nils B. | Method and system for real-time distributed data mining and analysis for network |
| CN103942210A (en) * | 2013-01-21 | 2014-07-23 | 中国移动通信集团上海有限公司 | Processing method, device and system of mass log information |
| CN105357041A (en) * | 2015-10-30 | 2016-02-24 | 上海帝联信息科技股份有限公司 | Edge node server, and log file uploading method and system |
| CN106534174A (en) * | 2016-12-07 | 2017-03-22 | 北京奇虎科技有限公司 | Cloud protection method, apparatus and system of sensitive data |
| US20170093888A1 (en) * | 2015-09-28 | 2017-03-30 | International Business Machines Corporation | Autonomic exclusion in a tiered delivery network |
| CN108234233A (en) * | 2016-12-22 | 2018-06-29 | 阿里巴巴集团控股有限公司 | Log processing method and device |
| CN109408330A (en) * | 2018-10-15 | 2019-03-01 | 东软集团股份有限公司 | Log analysis method, device, terminal device and readable storage medium storing program for executing |
| CN109951323A (en) * | 2019-02-27 | 2019-06-28 | 网宿科技股份有限公司 | A kind of log analysis method and system |
| CN110225091A (en) * | 2019-05-10 | 2019-09-10 | 宁波安信数控技术有限公司 | A kind of DTU and its data transmission method of belt edge computing capability |
| CN111416875A (en) * | 2020-04-10 | 2020-07-14 | 中国人民解放军国防科技大学 | Service directory synchronization method and system oriented to cloud edge coordination |
-
2020
- 2020-08-25 CN CN202010861335.8A patent/CN111740884B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020046273A1 (en) * | 2000-01-28 | 2002-04-18 | Lahr Nils B. | Method and system for real-time distributed data mining and analysis for network |
| CN103942210A (en) * | 2013-01-21 | 2014-07-23 | 中国移动通信集团上海有限公司 | Processing method, device and system of mass log information |
| US20170093888A1 (en) * | 2015-09-28 | 2017-03-30 | International Business Machines Corporation | Autonomic exclusion in a tiered delivery network |
| CN105357041A (en) * | 2015-10-30 | 2016-02-24 | 上海帝联信息科技股份有限公司 | Edge node server, and log file uploading method and system |
| CN106534174A (en) * | 2016-12-07 | 2017-03-22 | 北京奇虎科技有限公司 | Cloud protection method, apparatus and system of sensitive data |
| CN108234233A (en) * | 2016-12-22 | 2018-06-29 | 阿里巴巴集团控股有限公司 | Log processing method and device |
| CN109408330A (en) * | 2018-10-15 | 2019-03-01 | 东软集团股份有限公司 | Log analysis method, device, terminal device and readable storage medium storing program for executing |
| CN109951323A (en) * | 2019-02-27 | 2019-06-28 | 网宿科技股份有限公司 | A kind of log analysis method and system |
| CN110225091A (en) * | 2019-05-10 | 2019-09-10 | 宁波安信数控技术有限公司 | A kind of DTU and its data transmission method of belt edge computing capability |
| CN111416875A (en) * | 2020-04-10 | 2020-07-14 | 中国人民解放军国防科技大学 | Service directory synchronization method and system oriented to cloud edge coordination |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112291085A (en) * | 2020-10-10 | 2021-01-29 | 北京金山云网络技术有限公司 | Fault positioning method, device, equipment and medium |
| CN112291085B (en) * | 2020-10-10 | 2023-01-20 | 北京金山云网络技术有限公司 | Fault positioning method, device, equipment and medium |
| CN112615742A (en) * | 2020-12-18 | 2021-04-06 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for early warning |
| CN112511456A (en) * | 2020-12-21 | 2021-03-16 | 北京百度网讯科技有限公司 | Flow control method, flow control device, flow control equipment, storage medium and computer program product |
| CN112511456B (en) * | 2020-12-21 | 2024-03-22 | 北京百度网讯科技有限公司 | Flow control method, apparatus, device, storage medium, and computer program product |
| CN113010891A (en) * | 2021-02-26 | 2021-06-22 | 中科天齐(山西)软件安全技术研究院有限公司 | Application program safety detection method and device, electronic equipment and storage medium |
| CN113055368A (en) * | 2021-03-08 | 2021-06-29 | 云盾智慧安全科技有限公司 | Web scanning identification method and device and computer storage medium |
| CN113055368B (en) * | 2021-03-08 | 2022-12-13 | 云盾智慧安全科技有限公司 | Web scanning identification method and device and computer storage medium |
| CN113190522A (en) * | 2021-06-29 | 2021-07-30 | 阿里云计算有限公司 | Log processing method and device, computing equipment and storage medium |
| CN114090644A (en) * | 2022-01-20 | 2022-02-25 | 飞狐信息技术（天津）有限公司 | Data processing method and device |
| CN114090644B (en) * | 2022-01-20 | 2022-04-26 | 飞狐信息技术（天津）有限公司 | Data processing method and device |
| WO2024109569A1 (en) * | 2022-11-21 | 2024-05-30 | 中兴通讯股份有限公司 | Data processing method and apparatus, and electronic device and computer-readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111740884B (en) | 2021-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111740884B (en) | Log processing method, electronic equipment, server and storage medium | |
| US11379475B2 (en) | Analyzing tags associated with high-latency and error spans for instrumented software | |
| CN109697456B (en) | Service analysis method, device, equipment and storage medium | |
| CN108009236B (en) | Big data query method, system, computer and storage medium | |
| US10191962B2 (en) | System for continuous monitoring of data quality in a dynamic feed environment | |
| US20130332423A1 (en) | Data lineage tracking | |
| US11042525B2 (en) | Extracting and labeling custom information from log messages | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN109359026A (en) | Log reporting method, device, electronic equipment and computer readable storage medium | |
| CN112181931A (en) | Big data system link tracking method and electronic equipment | |
| US20140337274A1 (en) | System and method for analyzing big data in a network environment | |
| CN112084224A (en) | Data management method, system, device and medium | |
| CN113220756A (en) | Logistics data real-time processing method, device, equipment and storage medium | |
| CN111708774A (en) | Industry analytic system based on big data | |
| CN110377576A (en) | Create method and apparatus, the log analysis method of log template | |
| CN115033876A (en) | Log processing method, log processing device, computer device and storage medium | |
| CN112395333A (en) | Method and device for checking data exception, electronic equipment and storage medium | |
| CN111800292A (en) | Early warning method and device based on historical flow, computer equipment and storage medium | |
| CN106250397B (en) | User behavior characteristic analysis method and device | |
| CN106557483B (en) | Data processing method, data query method, data processing equipment and data query equipment | |
| CN112711614B (en) | Service data management method and device | |
| CN114116872A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN113297253A (en) | Equipment identification method, device, equipment and readable storage medium | |
| CN112800016A (en) | Log data classification and sorting method and device | |
| CN111917848A (en) | Data processing method based on edge computing and cloud computing cooperation and cloud server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |