CN111352903A - Log management platform, log management method, medium, and electronic device - Google Patents
Log management platform, log management method, medium, and electronic device Download PDFInfo
- Publication number
- CN111352903A CN111352903A CN202010176909.8A CN202010176909A CN111352903A CN 111352903 A CN111352903 A CN 111352903A CN 202010176909 A CN202010176909 A CN 202010176909A CN 111352903 A CN111352903 A CN 111352903A
- Authority
- CN
- China
- Prior art keywords
- log
- data
- module
- computing
- log data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Abstract
The disclosure relates to the technical field of computers, and provides a log management platform, a log management method, a log management medium and log management equipment. Wherein, this platform includes: a log collection module to: acquiring data streams in real time to collect log data about a plurality of business systems; a log splitting module to: carrying out multi-dimensional splitting on the log data; the log calculation module is used for: calling and processing the split log data with corresponding quantity according to the quantity of the computing units; the control module is used for: dynamically optimizing the number of computing units in the log computing module so that the log computing module processes split log data through the optimized computing units; a log screening module to: screening the stored log data according to the log retrieval request; a target log processing module to: and displaying the key items of the retrieved target log in a hyperlink mode so that the user can obtain the corresponding key item details through the hyperlink. According to the technical scheme, the efficiency and convenience of log query can be improved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a log management platform, a log management method, and a computer storage medium and an electronic device for implementing the log management method.
Background
In the big data era, the importance of data is more and more highlighted. The log data can be used for recording user behaviors and system operation states, and therefore can be used as a data reference source for problem investigation and system diagnosis in a service development process, a test process and an operation and maintenance process. With the increasing abundance of service types and the increasing complexity of log services, a series of management tasks need to be performed on log data to better serve users.
In a conventional enterprise application system, log data is stored in a plurality of nodes in a scattered manner. If the target Bug needs to be located through the log, the traditional command mode is needed to be used for inquiring on each node, and the efficiency of log inquiring is very low. Therefore, the log management scheme provided by the related art cannot meet the real-time log query requirement.
It is to be noted that the information disclosed in the background section above is only used to enhance understanding of the background of the present disclosure.
Disclosure of Invention
The disclosure aims to provide a log management platform, a log management method, a computer storage medium and an electronic device, so that the efficiency and convenience of log query are improved at least to a certain extent.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to an aspect of the present disclosure, there is provided a log management platform, including:
a log collection module to: acquiring data streams in real time to collect log data about a plurality of business systems;
a log splitting module to: carrying out multi-dimensional splitting on the log data to obtain split log data of each dimension;
the device comprises a control module and a plurality of log calculation modules, wherein each log calculation module comprises a plurality of calculation units;
the log calculation module is configured to: calling and processing the split log data through the computing unit, and storing the processed log data; the control module is used for: dynamically optimizing the number of computing units participating in calling and processing the split log data in the log computing module;
a log screening module to: receiving and analyzing a log retrieval request, and screening stored log data according to the log retrieval request;
a target log processing module to: displaying the key items of the retrieved target log in a hyperlink mode so that a user can obtain the corresponding key item details through the hyperlink.
In an exemplary embodiment of the present disclosure, based on the foregoing scheme, the control module is specifically configured to:
acquiring the processing speed of each computing unit in the ith log computing module in the M log computing modules, and optimizing the number of computing units participating in calling and processing the split log data in the ith log computing module according to the processing speed;
wherein M is an integer greater than or equal to 2, and i is a positive integer less than or equal to M.
In an exemplary embodiment of the present disclosure, based on the foregoing scheme, the control module is further specifically configured to:
and acquiring target log calculation modules corresponding to the log data with a logical relation with the target split log data according to the target split log data corresponding to the ith log calculation module, and acquiring data flow information among the target log calculation modules so as to optimize the number of calculation units participating in calling and processing the split log data in the ith log calculation module according to the processing speed and the data flow information.
In an exemplary embodiment of the present disclosure, based on the foregoing scheme, the log collection module includes a data transmission pipeline with a throughput greater than a preset value, so as to implement real-time writing of mass log data into the log management platform.
In an exemplary embodiment of the present disclosure, based on the foregoing solution, the log management platform further includes: and the log display module is used for displaying the log data in a list mode.
In an exemplary embodiment of the present disclosure, based on the foregoing solution, the log retrieval request includes: one or more of a request address, a user, interface parameters, and an interface response status.
In an exemplary embodiment of the present disclosure, based on the foregoing scheme, the key items include: a request header, a request parameter, and a response parameter for the target log.
According to another aspect of the present disclosure, there is provided a log management method including:
acquiring data streams in real time to collect log data about a plurality of business systems;
carrying out multi-dimensional splitting on the log data to obtain split log data of each dimension;
calling and processing the split log data through a plurality of computing modules, and dynamically optimizing the number of computing units contained in each computing module, so that the plurality of computing modules process the split log data based on the optimized number of computing units, and store the processed log data;
receiving and analyzing a log retrieval request, and screening stored log data according to the log retrieval request;
displaying the screened key items of the target log in a hyperlink mode so that a user can obtain corresponding key item details through the hyperlink.
According to an aspect of the present disclosure, there is provided a computer storage medium having stored thereon a computer program which, when executed by a processor, implements the log management method of the second aspect.
According to an aspect of the present disclosure, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the log management method of the second aspect via execution of the executable instructions.
According to the technical scheme, the log management platform in the exemplary embodiment of the disclosure has at least the following advantages and positive effects:
in the technical solutions provided by some embodiments of the present disclosure, a log collection module acquires a data stream in real time to collect log data about a plurality of service systems, and splits the log data in multiple dimensions to further call the split log according to the number of computing units in the log computing module, where the number of computing units in each log computing module is dynamically optimized. Therefore, the number of the computing units contained in each log computing module can meet the requirement of real-time log query, and the waste of computing resources is avoided.
And finally, displaying the key items of the screened target log in a hyperlink mode through a target log processing module so that a user can obtain the details of the corresponding key items through the hyperlink, thereby enabling the log to be quickly reproduced and further improving the efficiency and convenience of log query.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.
In the drawings:
FIG. 1 is a schematic diagram illustrating a structure of a log management platform to which embodiments of the present disclosure may be applied;
FIG. 2 is a schematic diagram illustrating a log processing flow to which embodiments of the present disclosure may be applied;
FIG. 3 shows a schematic structural diagram of a log computation module and a control module to which embodiments of the present disclosure may be applied;
FIG. 4 is a page presentation diagram of a log search term to which embodiments of the present disclosure may be applied;
FIG. 5 shows a page presentation diagram of a log list to which embodiments of the present disclosure may be applied;
FIG. 6 is a flow chart illustrating a log management method to which embodiments of the present disclosure may be applied; and the number of the first and second groups,
fig. 7 shows a schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
The technical scheme aims to provide a uniform log service solution to complete a log service platform and place the system on the ground of enterprise deployment by combining the daily problems of enterprises in log collection on the basis of extensive research and analysis of the log collection system. Specifically, a massive log real-time collection, retrieval and calculation platform is established, and rapid collection, consumption, extraction and the like of log data are realized. The user can enjoy the convenient, fast, stable, reliable and low-cost log service.
Specific embodiments of the log management platform, the log management method, and the electronic device according to the present technical solution are described below. Fig. 1 shows a schematic structural diagram of a log management platform to which an embodiment of the present disclosure may be applied.
Referring to fig. 1, the log management system 100 provided in the present embodiment includes: the log processing system comprises a log collection module 110, a log splitting module 120, M log calculation modules 131, a control module 132 for controlling the log calculation modules, a log storage module 140 and a target log processing module 150, wherein M is an integer greater than 2.
In the technical solution provided in this embodiment, the log collection module 110 obtains the data stream in real time to collect log data about a plurality of service systems, and the log splitting module 120 splits the log data in multiple dimensions, so as to further invoke and process the split log according to the computing unit in the log computing module 131. Wherein, the number of computing units (which may be referred to as "execution units") participating in the calling and processing of the split log in each log computing module is dynamically optimized based on the control module 132. Therefore, the number of execution units contained in each log calculation module can meet the requirement of real-time log query, and the waste of calculation resources is avoided. Further, the log screening module 140 receives and analyzes the log retrieval request, screens the stored log data according to the log retrieval request, and finally, the target log processing module 150 displays the screened key items of the target log in a hyperlink manner so that the user can obtain the details of the corresponding key items through the hyperlink, so that the log can be quickly reproduced, and the efficiency and convenience of log query are further improved.
The following describes in detail a specific implementation of each module of the log management platform with reference to a schematic diagram of a log processing flow shown in fig. 2, to which an embodiment of the present disclosure may be applied.
In an exemplary embodiment, the log collection module 110 collects log data generated by a plurality of business systems 200 (e.g., business system 1, business system 2, and … … business system N) based on Kafka clusters. Kafka is a high-throughput distributed messaging system, which is mainly used for processing real-time streaming data or action stream data. Illustratively, the collected log data may be data of web browsing, user access, search, and other user behaviors within different business systems. Due to the lower limit of throughput, the conventional processing method is to record the log data in a file form and then process the log data periodically. However, the conventional log processing and data analysis method is an off-line processing scheme, and tends to generate a relatively large delay. The Kafka cluster message queue can well process real-time or near-real-time streaming data, so that the real-time requirement of log data processing can be met. As can be seen, the log collection module 110 can implement real-time writing of massive log data into the log management platform 100 based on the data transmission pipeline with throughput greater than the preset value.
Further, referring to fig. 2, in an exemplary embodiment, the log management platform further includes: the log preprocessing module 300 is configured to preprocess the log data acquired by the log collecting module 110. Illustratively, the log preprocessing module 300 preprocesses the log data including log parsing, such as: and analyzing the data in different data formats collected by the plurality of business systems to unify the data formats of the business systems, thereby facilitating further log data processing. The preprocessing operations include log filtering, such as: repeated data is filtered to avoid log data redundancy or data volume complication. The preprocessing operations include feature set creation, log mining and classification, log statistics, and the like.
In an exemplary embodiment, the log data in the log preprocessing process may be stored in a Hadoop Distributed File System (HDFS) 400. Illustratively, the Hadoop can perform batch processing on the log data in a MapReduce operation mode and other modes, so that the preprocessing speed of the log data is effectively accelerated.
In an exemplary embodiment, with continued reference to fig. 1, for the preprocessed log data, the log splitting module 120 performs multi-dimensional splitting on the preprocessed log data, so as to obtain split log data of each dimension. Illustratively, the log data may be split according to the service system from which the log data comes, for example, the log data from the mall service system is labeled with the mall service system identifier, the log data from the medical service system is labeled with the medical service system identifier, and the like. The splitting can also be realized according to the interface type corresponding to the log data, such as marking the log data from the intranet interface with an intranet interface identifier, marking the log data from the extranet interface with an extranet interface identifier, and the like. Splitting (e.g., tagging) the data is done to facilitate summary statistics on different types of data.
Illustratively, referring to fig. 2, the splitting of log data and the aggregated statistics and storage after splitting can be completed by the Elasticsearch cluster 500. The ElasticSearch is a distributed search engine based on Lucene, and specifically, after the log data is imported into an ElasticSearch cluster, the system log is subjected to multi-dimensional splitting by using JSON as a document serialization format and is stored in a document form.
Specifically, the Elasticsearch cluster comprises a master node, a client node and a data node. Specifically, the master node may be configured to maintain and store log data before aggregation statistics and/or log data after aggregation statistics, and is responsible for managing states of other nodes of the cluster; the data node can be used for receiving log retrieval request data of a user and inquiring the log data stored in the master node based on the log retrieval request data; and the client node receives the query result gathered on the data node and returns the query result to the user.
In an exemplary embodiment, before receiving and parsing the log retrieval request, the log calculation module 131 processes the split log data as follows: in view of the fact that the average processing time for calculating different types of log splitting data by the execution unit in each log calculation module is different, the technical scheme facilitates obtaining the average processing time of the log data later by splitting the log data in advance. Referring to fig. 3, an ith log calculation module 131 (where i is a positive integer less than or equal to M) is shown, which includes a plurality of calculation units after being split.
Referring to FIG. 2, the number of execution units in the log calculation module involved in invoking the split log data is dynamically optimized by the control module 132. Illustratively, the ith log calculation module 131 currently contains NiA number of computing units based on the number N of current computing unitsiAnd calling and processing the split log data with the corresponding quantity. Meanwhile, the control module 132 acquires the ith dayCalculating the processing speed of each computing unit in the module, and selecting the number of computing units from N according to the processing speediIs optimized to be Ni' one of the two components. Then, N is passedi' the computing units process the split log data and store the processed log data. Wherein N isiAnd Ni' are also positive integers.
Specifically, when N isiLess than Ni' then, the number of execution units in the ith log calculation module is increased, or, when N is reachediGreater than Ni' then, the number of execution units in the ith log computation module is reduced.
In an exemplary embodiment, there is a continuous relationship between split log data processed in view of different log computation modules. Therefore, the control module 132 considers the logical relationship between split logs while taking into account the processing speed of each computing unit in obtaining each log computing module. Specifically, target splitting log data (for example, the sales volume of the target product in the first quarter) corresponding to the ith log calculation module is obtained, then, log data having a logical relationship with the target splitting log data (for example, the sales volume of the target product in each month in the first quarter) is obtained, and further, a log calculation module (denoted as "target log calculation module") for processing the log data (that is, the sales volume of the target product in each month in the first quarter) is obtained. And further acquiring data flow information among target log computing modules, and optimizing the number of computing modules in the ith log computing module according to the processing speed and the data flow information. Therefore, log data logically having an association relation are processed intensively and efficiently, and when a user queries an associated problem of a target problem, a query result can be fed back to the user timely and accurately.
In an exemplary embodiment, the processing operation performed by the execution unit in the log calculation module on the split log data includes one or more of the following operations: analysis processing, filtering processing, mining processing, classification processing and statistic processing. The execution unit may perform different processing procedures according to different query requirements, and this embodiment is not particularly limited.
It should be noted that the optimization process of the number of computing units in each log computing module is repeatedly performed every certain time interval. Thus, the number of computing units in each log computation module is dynamically optimized. Therefore, the number of the computing units contained in each log computing module can meet the requirement of real-time log query, and the waste of computing resources is avoided.
In an exemplary embodiment, with continued reference to fig. 1, the log filtering module 140 described above is configured to: receiving and analyzing a log retrieval request, and screening the log data stored in a centralized manner according to the log retrieval request. Wherein the log retrieval request comprises: one or more of request address, user, interface parameter and interface response state, so as to flexibly, conveniently and quickly screen log data according to the log retrieval request.
Fig. 4 shows a page display diagram of a log search term to which an embodiment of the present disclosure may be applied. Referring to fig. 4, a user or user client determines a log retrieval request based on one or more log retrieval terms exposed by the page. For example, by filling a URL of a target log in an interactive box corresponding to a "Uniform Resource Locator (URL)" search term, a log search request of the target log can be determined. For another example, the interface type of the target log is filled in the interactive frame corresponding to the "interface type" search item, and the user identifier is filled in the interactive frame corresponding to the "user ID" search item, so that the log search request of the target log is determined. Of course, relevant content may also be filled in the interactive frame corresponding to other retrieval items (e.g., the system of the target log, the requester of the target log, the response of the target log, etc.), and a retrieval request for the target log may be determined.
Illustratively, referring to fig. 4, after the related search term is filled out, the log filtering process may be triggered by "query", and the content filled out in the related search term of this time may be cancelled by "reset".
In an exemplary embodiment, after receiving a log retrieval request of a user or a user client, the log screening module 140 parses a retrieval item included in the log retrieval request, screens log data stored (for example, stored in an elasticsearch cluster) based on the retrieval item, and finally determines a target log corresponding to the retrieval item.
Illustratively, the parameters obtained by the analyzed log retrieval request are:
Request:{“verifyCode”:”807275”,”purpose”:”L”,”phone”:”186124”,”language”:”zh”,”sys”:”hmp”}。
illustratively, the corresponding resulting response may be expressed as:
{ "code": success "," data ": regFrom": IG "," accName ": 13439731745", "acid": 6002327283 "," token ": eyJhbGciOiJIUz" }, "message": success "," reqSeq ": IG 1566281347653" }
In an exemplary embodiment, after the target log is acquired, the target log processing module 150 displays the key items of the screened target log in a hyperlink manner, so that the user can acquire the details of the corresponding key items through the hyperlink.
In an exemplary embodiment, fig. 5 shows a page display diagram of a log list to which an embodiment of the present disclosure may be applied. Specifically, the log management platform 100 provided in the present technical solution further includes: the log presentation module 600.
Specifically, the log data can be presented in a list form through the log presentation module 600. The target logs can also be displayed in a list form through the log display module 600. Compared with the method for displaying the log messages in the related art, the method and the device for displaying the log messages display the log data in the list mode, so that the interface display is clearer, the user can look up the log messages more visually, and the log messages are convenient for the user to look up.
In an exemplary embodiment, referring to fig. 5, for each piece of log information in the list, not only the system to which it belongs, the URL, the request time, and the like are shown. In addition, the presentation list also comprises operation items for each log, namely, various key items of the target log can be viewed through operation.
Illustratively, key items of the target log include: a request header, a request parameter, and a response parameter for the target log. In the embodiment, each key item of the target log is displayed in a hyperlink mode (for example, a hyperlink related to a request header, a hyperlink related to a request parameter, and a hyperlink related to a response parameter) so that a user can obtain details of the corresponding key item through the hyperlink, and therefore, the log can be reproduced and a problem can be found more quickly and conveniently.
In an exemplary embodiment, the log management platform further includes a rights management module. And providing the log content corresponding to the authority of the user for the user according to the authority of the retrieval request user. The log data which is personalized and customized is provided for users with different authorities, and the normalization of log data management is facilitated.
The log management platform provided by the technical scheme is a one-stop service system for real-time collection, centralized storage, retrieval, analysis, calculation and display of mass logs, and a user can quickly complete functions of data acquisition, consumption, delivery, query, analysis and the like, so that the operation and maintenance efficiency of an enterprise can be improved, and the mass log processing capacity in a big data era can be realized.
FIG. 6 schematically shows a flow diagram of a log management method according to an embodiment of the present disclosure. The method provided by this embodiment is implemented based on the log management platform provided by the foregoing embodiment, and with reference to fig. 6, the method includes:
step S610, acquiring data flow in real time to collect log data about a plurality of service systems;
step S620, carrying out multi-dimensional splitting on the log data to obtain split log data of each dimension;
step S630, the split log data is called and processed by a plurality of log computing modules, and the number of computing units included in each log computing module is dynamically optimized, so that the split log data is processed by the plurality of log computing modules based on the optimized number of computing units, and the processed log data is stored;
step S640, receiving and analyzing a log retrieval request, and screening stored log data according to the log retrieval request; and the number of the first and second groups,
and S650, displaying the screened key items of the target log in a hyperlink mode so that a user can obtain corresponding key item details through the hyperlink.
In an exemplary embodiment, based on the foregoing scheme, the processing speed of each computing unit in the log computing module is obtained, and the number of computing units participating in invoking and processing the split log data in the log computing module is optimized according to the processing speed.
In an exemplary embodiment, based on the foregoing scheme, according to target split log data corresponding to the ith log computing module, a target log computing module corresponding to log data having a logical relationship with the target split log data is obtained, and data traffic information between the target log computing modules is obtained, so as to optimize the number of computing units participating in invoking and processing the split log data in the log computing module according to the processing speed and the data traffic information.
In an exemplary embodiment, based on the foregoing scheme, the data stream is acquired in real time through the data transmission pipeline with the throughput greater than the preset value to collect log data about a plurality of service systems, so as to implement real-time writing of massive log data into the log management platform.
In an exemplary embodiment, based on the foregoing solution, the log management method further includes: and displaying the log data in a list mode.
In an exemplary embodiment, based on the foregoing solution, the log retrieval request includes: one or more of a request address, a user, interface parameters, and an interface response status.
In an exemplary embodiment, based on the foregoing scheme, the key items include: a request header, a request parameter, and a response parameter for the target log.
Since the method is implemented based on the log management platform provided in the foregoing embodiment, and the specific details have been described in detail in the log management platform embodiment section of the specification, the details are not repeated here.
FIG. 7 illustrates a schematic structural diagram of a computer system suitable for use with the electronic device to implement embodiments of the present disclosure.
It should be noted that the computer system 700 of the electronic device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of the application of the embodiments of the present disclosure.
As shown in fig. 7, computer system 700 includes a processor 701 (including a Graphics Processing Unit (GPU), a Central Processing Unit (CPU)), which can perform various appropriate actions and processes according to a program stored in a Read-Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data necessary for system operation are also stored. The CPU 701, the ROM 702, and the RAM703 are connected to each other via a bus 704. An Input/Output (I/O) interface 705 is also connected to the bus 704.
The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a Network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.
In particular, the processes described below with reference to the flowcharts may be implemented as computer software programs, according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer storage medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program executes various functions defined in the system of the present application when executed by a Central Processing Unit (CPU) 701.
It should be noted that the computer storage media shown in the embodiments of the present disclosure may be computer readable signal media or computer readable storage media or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
In contrast, in the present disclosure, a computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer storage medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures.
For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.
As another aspect, the present application also provides a computer storage medium, which may be included in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to implement the method described in the embodiments. For example, the electronic device may implement the method embodiment as shown in fig. 7.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (10)
1. A log management platform, comprising:
a log collection module to: acquiring data streams in real time to collect log data about a plurality of business systems;
a log splitting module to: carrying out multi-dimensional splitting on the log data to obtain split log data of each dimension;
the device comprises a control module and a plurality of log calculation modules, wherein each log calculation module comprises a plurality of calculation units;
the log calculation module is configured to: calling and processing the split log data through the computing unit, and storing the processed log data; the control module is used for: dynamically optimizing the number of computing units participating in calling and processing the split log data in the log computing module;
a log screening module to: receiving and analyzing a log retrieval request, and screening stored log data according to the log retrieval request;
a target log processing module to: displaying the key items of the retrieved target log in a hyperlink mode so that a user can obtain the corresponding key item details through the hyperlink.
2. The log management platform of claim 1, wherein the control module is specifically configured to:
acquiring the processing speed of each computing unit in the ith log computing module in the M log computing modules, and optimizing the number of computing units participating in calling and processing the split log data in the ith log computing module according to the processing speed;
wherein M is an integer greater than or equal to 2, and i is a positive integer less than or equal to M.
3. The log management platform of claim 2, wherein the control module is further specifically configured to:
and acquiring target log calculation modules corresponding to the log data with a logical relation with the target split log data according to the target split log data corresponding to the ith log calculation module, and acquiring data flow information among the target log calculation modules so as to optimize the number of calculation units participating in calling and processing the split log data in the ith log calculation module according to the processing speed and the data flow information.
4. The log management platform according to any one of claims 1 to 3, wherein the log collection module comprises a data transmission pipeline with a throughput greater than a preset value, so as to realize real-time writing of mass log data into the log management platform.
5. The log management platform of any of claims 1 to 3, further comprising: and the log display module is used for displaying the log data in a list mode.
6. The log management platform of any of claims 1 to 3, wherein the log retrieval request comprises: one or more of a request address, a user, interface parameters, and an interface response status.
7. A log management platform as claimed in any one of claims 1 to 3, wherein the key items comprise: a request header, a request parameter, and a response parameter for the target log.
8. A log management method, comprising:
acquiring data streams in real time to collect log data about a plurality of business systems;
carrying out multi-dimensional splitting on the log data to obtain split log data of each dimension;
the split log data is called and processed through a plurality of log computing modules, the number of computing units contained in each log computing module is dynamically optimized, so that the split log data is processed by the plurality of log computing modules based on the optimized number of computing units, and the processed log data is stored;
receiving and analyzing a log retrieval request, and screening stored log data according to the log retrieval request;
displaying the screened key items of the target log in a hyperlink mode so that a user can obtain corresponding key item details through the hyperlink.
9. A computer storage medium having a computer program stored thereon;
the computer program, when executed by a processor, implements the log management method of claim 8.
10. An electronic device, characterized in that the electronic device comprises:
one or more processors;
storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the log management method of claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010176909.8A CN111352903A (en) | 2020-03-13 | 2020-03-13 | Log management platform, log management method, medium, and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010176909.8A CN111352903A (en) | 2020-03-13 | 2020-03-13 | Log management platform, log management method, medium, and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111352903A true CN111352903A (en) | 2020-06-30 |
Family
ID=71194489
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010176909.8A Pending CN111352903A (en) | 2020-03-13 | 2020-03-13 | Log management platform, log management method, medium, and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111352903A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112069048A (en) * | 2020-09-09 | 2020-12-11 | 北京明略昭辉科技有限公司 | Log processing method, device and storage medium |
| CN112286605A (en) * | 2020-09-29 | 2021-01-29 | 北京直真科技股份有限公司 | Mass condition screening control implementation method suitable for enterprise management system |
| CN112463527A (en) * | 2020-11-13 | 2021-03-09 | 完美世界控股集团有限公司 | Data processing method, device, equipment, system and storage medium |
| CN113065129A (en) * | 2021-04-09 | 2021-07-02 | 湖南汽车工程职业学院 | Computer information safety real-time monitoring system |
| CN113111367A (en) * | 2021-04-28 | 2021-07-13 | 北京天空卫士网络安全技术有限公司 | Security information management method, device and system |
| CN115514779A (en) * | 2022-09-30 | 2022-12-23 | 湖北大学 | Method and system for recording weblog |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017166644A1 (en) * | 2016-03-31 | 2017-10-05 | 乐视控股（北京）有限公司 | Data acquisition method and system |
| CN109144836A (en) * | 2018-08-06 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of processing method of operation log, device and electronic equipment |
| CN109831316A (en) * | 2018-12-17 | 2019-05-31 | 国网浙江省电力有限公司 | Massive logs real-time analyzer, real-time analysis method and readable storage medium storing program for executing |
| CN109933505A (en) * | 2019-03-14 | 2019-06-25 | 深圳市珍爱捷云信息技术有限公司 | Log processing method, device, computer equipment and storage medium |
-
2020
- 2020-03-13 CN CN202010176909.8A patent/CN111352903A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017166644A1 (en) * | 2016-03-31 | 2017-10-05 | 乐视控股（北京）有限公司 | Data acquisition method and system |
| CN109144836A (en) * | 2018-08-06 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of processing method of operation log, device and electronic equipment |
| CN109831316A (en) * | 2018-12-17 | 2019-05-31 | 国网浙江省电力有限公司 | Massive logs real-time analyzer, real-time analysis method and readable storage medium storing program for executing |
| CN109933505A (en) * | 2019-03-14 | 2019-06-25 | 深圳市珍爱捷云信息技术有限公司 | Log processing method, device, computer equipment and storage medium |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112069048A (en) * | 2020-09-09 | 2020-12-11 | 北京明略昭辉科技有限公司 | Log processing method, device and storage medium |
| CN112286605A (en) * | 2020-09-29 | 2021-01-29 | 北京直真科技股份有限公司 | Mass condition screening control implementation method suitable for enterprise management system |
| CN112463527A (en) * | 2020-11-13 | 2021-03-09 | 完美世界控股集团有限公司 | Data processing method, device, equipment, system and storage medium |
| CN113065129A (en) * | 2021-04-09 | 2021-07-02 | 湖南汽车工程职业学院 | Computer information safety real-time monitoring system |
| CN113111367A (en) * | 2021-04-28 | 2021-07-13 | 北京天空卫士网络安全技术有限公司 | Security information management method, device and system |
| CN115514779A (en) * | 2022-09-30 | 2022-12-23 | 湖北大学 | Method and system for recording weblog |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111352903A (en) | Log management platform, log management method, medium, and electronic device | |
| US11755452B2 (en) | Log data collection method based on log data generated by container in application container environment, log data collection device, storage medium, and log data collection system | |
| US11288142B2 (en) | Recovery strategy for a stream processing system | |
| US10277696B2 (en) | Method and system for processing data used by creative users to create media content | |
| US20070047439A1 (en) | Method and apparatus of supporting business performance management with active shared data spaces | |
| CN111143286B (en) | Cloud platform log management method and system | |
| CN112527848B (en) | Report data query method, device and system based on multiple data sources and storage medium | |
| CN113094136A (en) | Page display control method and device, storage medium and electronic equipment | |
| CN112256954A (en) | Message pushing processing method and related system | |
| US20170078361A1 (en) | Method and System for Collecting Digital Media Data and Metadata and Audience Data | |
| EP0953923A2 (en) | Method and apparatus for forming page map to present internet data meaningful to management and business operation | |
| US11625254B1 (en) | Interface for customizing dashboards based on parallel edges | |
| CN111476015A (en) | Document processing method and device, electronic equipment and storage medium | |
| CN111274256A (en) | Resource control method, device, equipment and storage medium based on time sequence database | |
| US10489179B1 (en) | Virtual machine instance data aggregation based on work definition metadata | |
| CN110866040A (en) | User portrait generation method, device and system | |
| CN114265680A (en) | Mass data processing method and device, electronic equipment and storage medium | |
| CN111444158A (en) | Long-short term user portrait generation method, device, equipment and readable storage medium | |
| CN111680799A (en) | Method and apparatus for processing model parameters | |
| US10691653B1 (en) | Intelligent data backfill and migration operations utilizing event processing architecture | |
| CN111177237B (en) | Data processing system, method and device | |
| CN107480189A (en) | A kind of various dimensions real-time analyzer and method | |
| CN115391361A (en) | Real-time data processing method and device based on distributed database | |
| CN113076254A (en) | Test case set generation method and device | |
| CN112035256A (en) | Resource allocation method, device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |