CN110597777A - Log processing method and device - Google Patents
Log processing method and device Download PDFInfo
- Publication number
- CN110597777A CN110597777A CN201910884467.XA CN201910884467A CN110597777A CN 110597777 A CN110597777 A CN 110597777A CN 201910884467 A CN201910884467 A CN 201910884467A CN 110597777 A CN110597777 A CN 110597777A
- Authority
- CN
- China
- Prior art keywords
- log
- name
- kafka
- request
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/243—Natural language query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
Abstract
The invention provides a log processing method and a log processing device, wherein the method comprises the following steps: obtaining an access request for accessing the log stored in the Kafka by a user through a human-computer interaction interface (UI), wherein the access request comprises: log name and log processing content; and when determining that the user can access the log stored in the Kafka, sending the access request to the Kafka, so as to process the log corresponding to the log name in the Kafka according to the log processing content in the access request. By the log processing method and the log processing device, logs stored in Kafka can be accessed without inputting command lines or writing programs, operation is simple, and errors are not prone to occurring.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a log processing method and device.
Background
Currently, KSQL is an open source stream processing platform developed by the Apache software foundation, written in Scala and Java. The goal of KSQL is to provide a uniform, high-throughput, low-latency platform for Kafka to process real-time data. Kafka stores a large number of KSQL message logs (logs for short), and in order to view and edit the contents of the logs in Kafka, the logs in Kafka need to be accessed.
In the related art, the log in Kafka can be accessed by inputting a Kafka command line or by writing a program.
The access mode of the log in the Kafka is complex and easy to make errors.
Disclosure of Invention
In order to solve the above problem, an object of the embodiments of the present invention is to provide a log processing method and apparatus.
In a first aspect, an embodiment of the present invention provides a log processing method, including:
obtaining an access request for accessing the log stored in the Kafka by a User through a human-computer Interface (UI), wherein the access request comprises: log name and log processing content;
and when determining that the user can access the log stored in the Kafka, sending the access request to the Kafka, so as to process the log corresponding to the log name in the Kafka according to the log processing content in the access request.
In a second aspect, an embodiment of the present invention further provides a log processing apparatus, including:
an obtaining module, configured to obtain, through a human-computer interaction interface UI, an access request for a user to access a log stored in Kafka, where the access request includes: log name and log processing content;
and the processing module is used for sending the access request to the Kafka when the user is determined to be capable of accessing the log stored in the Kafka, so that the log corresponding to the log name in the Kafka is processed according to the log processing content in the access request.
In a third aspect, the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program performs the steps of the method in the first aspect.
In a fourth aspect, an embodiment of the present invention further provides a log processing apparatus, which includes a memory, a processor, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor to perform the steps of the method according to the first aspect.
In the solutions provided in the foregoing first aspect to the fourth aspect of the embodiments of the present invention, an access request for accessing a log stored in Kafka by a user is obtained by setting a UI, and when it is determined that the user can access the log stored in Kafka, the access request is sent to Kafka, so that the log corresponding to the log name in Kafka is processed according to the log processing content in the access request.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a block diagram showing the structure of the above-described server applicable to the embodiment of the present invention;
FIG. 2 illustrates a schematic diagram of an application scenario of an embodiment of the present invention;
fig. 3 is a flowchart illustrating a log processing method according to embodiment 1 of the present invention;
fig. 4 shows a topological diagram of a log corresponding to a log name in the log processing method provided in embodiment 1 of the present invention;
fig. 5 is a schematic structural diagram illustrating a log processing apparatus according to embodiment 2 of the present invention;
fig. 6 is a schematic structural diagram illustrating another log processing apparatus provided in embodiment 3 of the present invention.
Detailed Description
Currently, KSQL is an open source stream processing platform developed by the Apache software foundation, written in Scala and Java. The goal of KSQL is to provide a uniform, high-throughput, low-latency platform for Kafka to process real-time data. Kafka stores a large number of logs, and access to the logs in Kafka is required in order to view and edit the contents of the logs in Kafka. In the related art, the log in Kafka can be accessed by inputting a Kafka command line or by writing a program. This way of accessing the logs in Kafka is complex and error prone. Based on this, the embodiment provides a log processing method and device, an access request for a user to access a log stored in Kafka is acquired by setting a UI, and when it is determined that the user can access the log stored in Kafka, the access request is sent to Kafka, so that the log corresponding to the log name in Kafka is processed according to the log processing content in the access request, and the user can access the log stored in Kafka only by inputting the access request in the UI without inputting a command line or writing a program, which is simple in operation and is not prone to errors.
Fig. 1 shows a block diagram of the server applicable to the embodiment of the present invention. As shown in fig. 1, the server 200 includes: memory 201, processor 202, and network module 203.
The memory 201 may be used to store software programs and modules, such as program instructions/modules corresponding to the log processing method and apparatus in the embodiment of the present invention, and the processor 202 executes various functional applications and data processing by running the software programs and modules stored in the memory 201, so as to implement the log processing method in the embodiment of the present invention. Memory 201 may include high speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. Further, the software programs and modules may further include: an operating system 221 and a service module 222. The operating system 221, which may be LINUX, UNIX, WINDOWS, for example, may include various software components and/or drivers for managing system tasks (e.g., memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components. The service module 222 runs on the basis of the operating system 221, and monitors a request from the network through the network service of the operating system 221, completes corresponding data processing according to the request, and returns a processing result to the client. That is, the service module 222 is used to provide network services to clients.
The network module 203 is used for receiving and transmitting network signals. The network signal may include a wireless signal or a wired signal.
It will be appreciated that the configuration shown in fig. 1 is merely illustrative and that server 200 may include more or fewer components than shown in fig. 1 or have a different configuration than shown in fig. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination thereof. In addition, the server in the embodiment of the present invention may further include a plurality of servers with different specific functions.
Referring to fig. 2, an application scenario diagram of the embodiment of the present invention is shown, where the application scenario includes: UI100 cached in the Server, access Proxy module (Rest Proxy)102, Kafka104, and KSQL Server (KSQL Server) 106; the UI100, the Rest Proxy102 and the Kafka104 are connected in sequence; the UI100 is also connected to KSQLServer 106.
The UI is used for being displayed to the user when the user logs in the Server, so that the user can access the log in the Kafka through the UI and process the log in the Kafka through the KSQL Server.
The Rest Proxy is an agent program which runs in the server and enables the user to access, query and view the log in Kafka.
The Kafka is a file system provided in the server and used for storing the log.
The KSQL Server is a service process which runs in a Server and is used for processing the logs in the Kafka.
The access, including but not limited to: the logs in Kafka are reviewed and edited.
In order to make the user access to the Kafka through the UI, the UI can be made to encapsulate the logic of the user access Kafka needing programming through the Rest Proxy, make a layer of Proxy and provide an HTTP interface in the UI to receive the access request of the user. And sends the access request sent by the user to Kafka through Rest Proxy.
For checking the log data in the Kafka, a Websocket interface is provided in the UI, and the log in the Kafka, which needs to be checked by the user, is returned to the user; of course, the logs in Kafka that the user needs to view can also be returned to the user by means of long TCP connections in the browser used by the user login server. Meanwhile, a log search bar is arranged in the UI to acquire a query request of a user, and then a log with the log name can be queried from the log of the Kafka according to the log name in the query request input by the user in the UI search bar, so that the query function of the log in the Kafka is realized.
Correspondingly, in order to realize the query function of the logs in the Kafka, a query button is further arranged on the UI, and after a user inputs a name of a log to be queried in the log search field, the user clicks the query button to trigger a query request for querying the logs of the Kafka.
Also, on the UI, how many pieces of log data (Limit) are viewed, the consumption location (OFFSET), and the filtering and exporting functions for the data can be set.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, the present application is described in further detail with reference to the accompanying drawings and the detailed description.
Example 1
The present embodiment proposes a log processing method, and the execution subject is the server shown in fig. 1.
Referring to a flowchart of a log processing method shown in fig. 3, the present embodiment proposes a log processing method, including:
step 300, obtaining an access request for accessing the log stored in Kafka by a user through a UI, wherein the access request comprises: log name and log processing content.
In step 300, after the user logs in the server, the server presents the UI to the user, and if the user wants to access the log stored in Kafka, the user inputs the name of the log and/or the log processing content to be accessed in the HTTP interface provided by the UI. The UI will send an access request into the Rest Proxy.
The log processing content may be, but is not limited to: the data structure content and parameters that need to be added, deleted, or changed in the log corresponding to the log name.
The access request comprises: log name, log processing content, source IP address, and destination IP address.
The source IP address is the IP address of the user login server.
The destination IP address is the IP address of the server.
The log name is the name of the log that the user wants to access.
Step 302, obtaining the source IP address in the access request.
In step 302, after the Rest Proxy acquires the access request, the source IP address in the access request may be acquired from the access request.
Step 304, when the source IP address is within the allowed access network segment, determining that the user can access the log stored in Kafka.
In step 304, the allowed access network segment is the IP address range that can access the log stored in Kafka. Generally, the network segment allowing access should be set to the intranet IP address of the company where the user is located, and the intranet of the company is isolated from the internet, so as to ensure the security of access.
Therefore, when the source IP address is in the access-allowed network segment, which indicates that the user accesses the log stored in Kafka in the company, and the Rest Proxy allows such operation, the Rest Proxy sends the acquired access request to Kafka. However, if the user is on the internet, the Rest Proxy prohibits the access request from the user from accessing the log stored in Kafka, and presents the access failure information to the user through the UI.
Step 306, sending the access request to Kafka, so as to process the log corresponding to the log name in Kafka according to the log processing content in the access request.
In step 306, after receiving the access request sent by the Rest Proxy, Kafka queries a log corresponding to the log name according to the log name in the access request, and then performs a processing operation on the queried log according to the log processing content in the access request. Therefore, the log stored in the Kafka can be accessed without inputting a command line or writing a program, the operation is simple, and errors are not easy to occur.
In addition to the function of accessing the log stored in Kafka, the user may also query the log that the user is unfamiliar with by using the UI, and in order to query the log stored in Kafka through the UI, the log processing method provided in this embodiment further includes the following steps (1) to (3):
(1) obtaining a query request triggered by a user through a log search bar arranged on a UI, wherein the query request comprises: a log name;
(2) according to the log name in the query request, querying a log with the log name in Kafka;
(3) and displaying the inquired log with the log name to a user.
In the step (1), when a user inputs a log name in a log search field set on the UI and clicks a query button set on the UI, the UI generates a query request according to the log name input by the user, and sends the query request to the Rest Proxy, and the Rest Proxy sends the query request to Kafka after acquiring the query request.
In the step (2), based on the log name in the search instruction, Kafka queries the log with the log name in Kafka.
Wherein, the log with the log name includes but is not limited to: the name is the log of the log name and the log whose name contains the log name.
In one embodiment, if the log name in the query request is CC, then Kafka will search out as the query result the log whose log name is CC, AACC, CCBB, etc. with CC.
When a user selects one log name from at least one log name as a query result through the UI, the UI acquires the log name of the log which the user wants to view, generates a viewing request according to the log name of the log selected by the user, and sends the viewing request to the Rest Proxy, the Rest Proxy sends the viewing request to the Kafka after acquiring the viewing request, and the Kafka returns the log corresponding to the log name carried in the viewing request to the UI after receiving the viewing request, and displays the log to the user for viewing through the UI.
It can be seen from the contents of the steps (1) to (3) that the query request triggered by the user is obtained through the log search bar arranged on the UI, and the log with the log name in Kafka is queried according to the log name in the query request and returned to the user for viewing, and the log with the log name in Kafka can be viewed without inputting a command line or writing a program, so that the operation is simple and convenient.
In the related technology, when the KSQL processes the logs in the Kafka, based on the processing logic, there is a text topology result formed by context dependency, that is, the dependency among the logs is presented in a text form, and the viewing is not intuitive and the positioning is poor. In order to more vividly show the topological result of the log, the log processing method provided by the embodiment further comprises the following steps (1) to (3):
(1) generating a topological graph generation request through a UI, wherein the topological graph generation request comprises: the log name of the log of the topological graph needs to be generated;
(2) inquiring a first log generated by a log corresponding to the log name carried by the topological graph generation request, and inquiring a second log used for generating the log corresponding to the log name carried by the topological graph generation request, wherein the first log has a first log name, and the second log has a second log name;
(3) and taking the second log name of the second log as a root node of a topological graph, taking the log name carried by the topological graph generation request as a child node of the root node, taking the first log name of the first log as a child node of the node corresponding to the log name carried by the topological graph generation request, and generating the topological graph of the log corresponding to the log name.
In the step (1), when the user views the log through the UI, a log toolbar is displayed on the UI; in the journal toolbar, there may be set, but not limited to: a topology generation button, a log association button, and a data send button.
Therefore, when the user clicks the topology generation button in the log toolbar, the UI generates a topology map generation request according to the previously acquired log name of the log viewed by the user, and sends the generated topology map generation request to the KSQLServer.
In the step (2), the KSQL Server stores: and generating a first association relationship between a first log generated by the log corresponding to the log name carried by the topological graph generation request and the log corresponding to the log name, and generating a second association relationship between a second log generated by the log corresponding to the log name carried by the topological graph generation request and the log corresponding to the log name.
The first log generated by the log corresponding to the log name carried in the topological graph generation request is the first log, and is obtained by processing data (and other logs) in the log corresponding to the log name.
For example: the data in the log corresponding to the log name carried by the topological graph generation request is as follows: the price of the vehicle sold in the last week can be obtained on the basis of the data in the log corresponding to the log name, and the log recording the average price of the vehicle sold in the last week is the first log generated by the log corresponding to the log name carried by the topological graph generation request. And obtaining a first association relation between a first log generated by the log corresponding to the log name and the log corresponding to the log name, which is carried by the topological graph generation request, and storing the generated first association relation in the KSQL Server.
The generating of the second log of the log corresponding to the log name carried by the topological graph generation request means that the data in the log corresponding to the log name carried by the topological graph generation request is obtained after the data recorded in the second log is processed.
For example: the data in the log corresponding to the log name carried by the topological graph generation request is as follows: in order to obtain the log corresponding to the log name, the log with the price data of the vehicles sold each day in the last week can be selected as a second log, and then the log corresponding to the log name carried by the topological graph generation request can be obtained by merging the data in the second log; and generating a second association relation between a second log of the log corresponding to the log name carried by the topological graph generation request and the log corresponding to the log name, and storing the generated second association relation in the KSQL Server.
Then, the KSQL Server may query and obtain a first log generated by the log corresponding to the log name carried by the topological graph generation request according to a first association relationship between the first log generated by the log corresponding to the log name carried by the topological graph generation request and the log corresponding to the log name; and querying according to a second association relationship between a second log of the log corresponding to the log name carried in the topological graph generation request and the log corresponding to the log name to obtain the second log of the log corresponding to the log name carried in the topological graph generation request.
In the step (3), by operating the example provided in the step (2) in the step (3), the KSQL Server may obtain a topological diagram of the log corresponding to the log name as shown in fig. 4. In fig. 4, the second log is a log having price data of vehicles sold every day in the last week; the log corresponding to the log name carried by the topological graph generation request is a log for recording the price of the vehicle sold in the last week; the first log is a log that records the average price of vehicles sold in the last week.
Of course, in the topology map, a plurality of first logs may also be obtained according to different processing of the logs corresponding to the log names, which is not described in detail herein.
As can be seen from the content described in the above steps (1) to (3), by querying out the first log generated by the log corresponding to the log name carried by the topology map generation request, and generating the second log of the log corresponding to the log name carried by the topology map generation request, then taking the second log name of the second log as the root node of the topology map, taking the log name carried by the topology map generation request as the child node of the root node, and taking the first log name of the first log as the child node of the log corresponding to the log name carried by the topology map generation request, generating the topology map of the log corresponding to the log name, so as to visually, vividly and vividly display the topology result of the log corresponding to the log name, so that the user can quickly locate the log which can have a problem through the topology map, the user experience is improved.
In summary, according to the log processing method provided in this embodiment, an access request for a user to access a log stored in Kafka is acquired by setting a UI, and when it is determined that the user can access the log stored in Kafka, the access request is sent to Kafka, so that the log corresponding to the log name in Kafka is processed according to the log processing content in the access request.
Based on the same inventive concept, the embodiment of the present application further provides a log processing method and a corresponding log processing apparatus, and as the principle of solving the problem of the apparatus in the embodiment of the present application is similar to the log processing method described in embodiment 1 of the present application, the implementation of the apparatus may refer to the implementation of the log processing method, and repeated details are omitted.
Example 2
The present embodiment proposes a log processing apparatus for executing the log processing method described in embodiment 1 above.
Referring to a schematic structural diagram of a log processing apparatus shown in fig. 5, the present embodiment provides a log processing apparatus, including:
an obtaining module 500, configured to obtain, through a human-computer interaction interface UI, an access request for a user to access a log stored in Kafka, where the access request includes: log name and log processing content;
a processing module 502, configured to send the access request to Kafka when it is determined that the user can access the log stored in Kafka, so as to process the log corresponding to the log name in Kafka according to the log processing content in the access request.
The access request further comprises: a source IP address;
in order to determine whether the user can access the log stored in Kafka, the apparatus further includes:
an address obtaining module, configured to obtain a source IP address in the access request;
and the determining module is used for determining that the user can access the log stored in the Kafka when the source IP address is in the network access permission segment.
In order to query the log stored in Kafka through the UI, the log processing apparatus provided in this embodiment further includes:
the query request acquisition module is used for acquiring a query request triggered by a user through a log search bar arranged on a UI, and the query request comprises: a log name;
the query module is used for querying the log with the log name in the Kafka according to the log name in the search instruction;
and the display module is used for displaying the inquired log with the log name to a user.
It can be seen from the above contents that, the query request triggered by the user is obtained through the log search bar arranged on the UI, and the log with the log name in Kafka is queried according to the log name in the query request and returned to the user for viewing, so that the log with the log name in Kafka can be viewed without inputting a command line or writing a program, and the operation is simple and convenient.
In the related art, topological results for the logs in the Kafka in the KSQL are presented in a text form, so that the checking is not intuitive enough and the positioning is poor. In order to more vividly show the topological result of the log, the log processing apparatus provided in this embodiment further includes:
the topology request module is used for generating a topology map generation request through a UI, and the topology map generation request comprises: the log name of the log of the topological graph needs to be generated;
a relationship query module, configured to query a first log generated by a log corresponding to the log name carried in the topological graph generation request, and query a second log used for generating a log corresponding to the log name carried in the topological graph generation request, where the first log has a first log name and the second log has a second log name;
and the topology generation module is used for taking the second log name of the second log as a root node of a topology graph, taking the log name carried by the topology graph generation request as a child node of the root node, taking the first log name of the first log as a child node of the node corresponding to the log name carried by the topology graph generation request, and generating the topology graph of the log corresponding to the log name.
From the above, it can be seen that, by querying a first log generated by a log corresponding to a log name carried by a topology map generation request, and generating a second log of the log corresponding to the log name carried by the topology map generation request, then taking the second log name of the second log as a root node of the topology map, taking the log name carried by the topology map generation request as a child node of the root node, and taking the first log name of the first log as a child node of the log name corresponding to the log name carried by the topology map generation request, a topology map of the log corresponding to the log name is generated, so that a topology result of the log corresponding to the log name is visually, vividly and vividly displayed, and a user can quickly locate a log which can have a problem through the topology map, the user experience is improved.
In summary, in the log processing apparatus provided in this embodiment, the UI is set to obtain an access request for a user to access the log stored in Kafka, and when it is determined that the user can access the log stored in Kafka, the access request is sent to Kafka, so that the log corresponding to the log name in Kafka is processed according to the log processing content in the access request.
Based on the same inventive concept, embodiments of the present application further provide a log processing method and a corresponding computer storage medium and a log processing apparatus, and as the principles of solving the problems of the computer storage medium and the apparatus in the embodiments of the present application are similar to the log processing method described in embodiment 1 of the present application, the apparatus may be implemented by referring to the implementation of the log processing method, and repeated details are not repeated.
Example 3
The present embodiment proposes a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the log processing method described in embodiment 1 above. For specific implementation, refer to method embodiment 1, which is not described herein again.
In addition, referring to another schematic structural diagram of the log processing device shown in fig. 6, the present embodiment further proposes a log processing device, which includes a bus 51, a processor 52, a transceiver 53, a bus interface 54, a memory 55, and a user interface 56. The log processing means includes a memory 55.
In this embodiment, the log processing apparatus further includes: one or more programs stored on the memory 55 and executable on the processor 52, configured to be executed by the processor for performing the following steps (1) to (2):
(1) obtaining an access request for accessing the log stored in the Kafka by a user through a human-computer interaction interface (UI), wherein the access request comprises: log name and log processing content;
(2) and when determining that the user can access the log stored in the Kafka, sending the access request to the Kafka, so as to process the log corresponding to the log name in the Kafka according to the log processing content in the access request.
A transceiver 53 for receiving and transmitting data under the control of the processor 52.
In fig. 6, a bus architecture (represented by bus 51), bus 51 may include any number of interconnected buses and bridges, with bus 51 linking together various circuits including one or more processors, represented by general purpose processor 52, and memory, represented by memory 55. The bus 51 may also link various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further in this embodiment. A bus interface 54 provides an interface between the bus 51 and the transceiver 53. The transceiver 53 may be one element or may be multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. For example: the transceiver 53 receives external data from other devices. The transceiver 53 is used for transmitting data processed by the processor 52 to other devices. Depending on the nature of the computing system, a user interface 56, such as a keypad, display, speaker, microphone, joystick, may also be provided.
The processor 52 is responsible for managing the bus 51 and the usual processing, running a general-purpose operating system as described above. And memory 55 may be used to store data used by processor 52 in performing operations.
Alternatively, processor 52 may be, but is not limited to: a central processing unit, a singlechip, a microprocessor or a programmable logic device.
It will be appreciated that the memory 55 in embodiments of the invention may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of example, but not limitation, many forms of RAM are available, such as Static random access memory (Static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic random access memory (Synchronous DRAM, SDRAM), Double Data rate Synchronous Dynamic random access memory (ddr SDRAM ), Enhanced Synchronous SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct memory bus RAM (DRRAM). The memory 55 of the systems and methods described in this embodiment is intended to comprise, without being limited to, these and any other suitable types of memory.
In some embodiments, memory 55 stores the following elements, executable modules or data structures, or a subset thereof, or an expanded set thereof: an operating system 551 and application programs 552.
The operating system 551 includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks. The application 552 includes various applications, such as a Media Player (Media Player), a Browser (Browser), and the like, for implementing various application services. A program implementing the method of an embodiment of the present invention may be included in the application 552.
In summary, the computer-readable storage medium and the log processing apparatus provided in this embodiment acquire, by setting the UI, an access request for a user to access the log stored in the Kafka, and send the access request to the Kafka when it is determined that the user can access the log stored in the Kafka, so as to process the log corresponding to the log name in the Kafka according to the log processing content in the access request.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A log processing method, comprising:
obtaining an access request for accessing the log stored in the Kafka by a user through a human-computer interaction interface (UI), wherein the access request comprises: log name and log processing content;
and when determining that the user can access the log stored in the Kafka, sending the access request to the Kafka, so as to process the log corresponding to the log name in the Kafka according to the log processing content in the access request.
2. The method of claim 1, wherein the access request further comprises: a source IP address;
after the step of obtaining an access request for accessing the log stored in Kafka by a user through a human-computer interaction interface UI, the method further comprises the following steps:
acquiring a source IP address in the access request;
when the source IP address is in the allowed access network segment, determining that the user can access the log stored in the Kafka.
3. The method of claim 1, further comprising:
obtaining a query request triggered by a user through a log search bar arranged on a UI, wherein the query request comprises: a log name;
according to the log name in the search instruction, inquiring the log with the log name in Kafka;
and displaying the inquired log with the log name to a user.
4. The method of claim 1, further comprising:
generating a topological graph generation request through a UI, wherein the topological graph generation request comprises: the log name of the log of the topological graph needs to be generated;
inquiring a first log generated by a log corresponding to the log name carried by the topological graph generation request, and inquiring a second log used for generating the log corresponding to the log name carried by the topological graph generation request, wherein the first log has a first log name, and the second log has a second log name;
and taking the second log name of the second log as a root node of a topological graph, taking the log name carried by the topological graph generation request as a child node of the root node, taking the first log name of the first log as a child node of the node corresponding to the log name carried by the topological graph generation request, and generating the topological graph of the log corresponding to the log name.
5. A log processing apparatus, comprising:
an obtaining module, configured to obtain, through a human-computer interaction interface UI, an access request for a user to access a log stored in Kafka, where the access request includes: log name and log processing content;
and the processing module is used for sending the access request to the Kafka when the user is determined to be capable of accessing the log stored in the Kafka, so that the log corresponding to the log name in the Kafka is processed according to the log processing content in the access request.
6. The apparatus of claim 5, wherein the access request further comprises: a source IP address;
the device, still include:
an address obtaining module, configured to obtain a source IP address in the access request;
and the determining module is used for determining that the user can access the log stored in the Kafka when the source IP address is in the network access permission segment.
7. The apparatus of claim 5, further comprising:
the query request acquisition module is used for acquiring a query request triggered by a user through a log search bar arranged on a UI, and the query request comprises: a log name;
the query module is used for querying the log with the log name in the Kafka according to the log name in the search instruction;
and the display module is used for displaying the inquired log with the log name to a user.
8. The apparatus of claim 5, further comprising:
the topology request module is used for generating a topology map generation request through a UI, and the topology map generation request comprises: the log name of the log of the topological graph needs to be generated;
a relationship query module, configured to query a first log generated by a log corresponding to the log name carried in the topological graph generation request, and query a second log used for generating a log corresponding to the log name carried in the topological graph generation request, where the first log has a first log name and the second log has a second log name;
and the topology generation module is used for taking the second log name of the second log as a root node of a topology graph, taking the log name carried by the topology graph generation request as a child node of the root node, taking the first log name of the first log as a child node of the node corresponding to the log name carried by the topology graph generation request, and generating the topology graph of the log corresponding to the log name.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of the claims 1 to 4.
10. A log processing device comprising a memory, a processor and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor to perform the steps of the method of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910884467.XA CN110597777A (en) | 2019-09-18 | 2019-09-18 | Log processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910884467.XA CN110597777A (en) | 2019-09-18 | 2019-09-18 | Log processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110597777A true CN110597777A (en) | 2019-12-20 |
Family
ID=68861053
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910884467.XA Pending CN110597777A (en) | 2019-09-18 | 2019-09-18 | Log processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110597777A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000405A (en) * | 2020-07-29 | 2020-11-27 | 深圳市智微智能软件开发有限公司 | Method, equipment and storage medium for conveniently acquiring system logs |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106168909A (en) * | 2016-06-30 | 2016-11-30 | 北京奇虎科技有限公司 | A kind for the treatment of method and apparatus of daily record |
| CN106484618A (en) * | 2016-10-10 | 2017-03-08 | 中国电力科学研究院 | A kind of based on memory access rely on to parallel playback method and device |
| CN108108285A (en) * | 2017-12-26 | 2018-06-01 | 广东欧珀移动通信有限公司 | Log processing method, device, storage medium and terminal device |
| CN108572914A (en) * | 2018-03-14 | 2018-09-25 | 口碑(上海)信息技术有限公司 | A kind of method and device generating the daily record for pressure test |
| EP3425524A1 (en) * | 2016-03-01 | 2019-01-09 | Alibaba Group Holding Limited | Cloud platform-based client application data calculation method and device |
| CN109189736A (en) * | 2018-08-01 | 2019-01-11 | 中国联合网络通信集团有限公司 | A kind of generation method and device of alarm association rule |
| US20190236194A1 (en) * | 2018-01-31 | 2019-08-01 | Splunk Inc. | Dynamic query processor for streaming and batch queries |
-
2019
- 2019-09-18 CN CN201910884467.XA patent/CN110597777A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3425524A1 (en) * | 2016-03-01 | 2019-01-09 | Alibaba Group Holding Limited | Cloud platform-based client application data calculation method and device |
| CN106168909A (en) * | 2016-06-30 | 2016-11-30 | 北京奇虎科技有限公司 | A kind for the treatment of method and apparatus of daily record |
| CN106484618A (en) * | 2016-10-10 | 2017-03-08 | 中国电力科学研究院 | A kind of based on memory access rely on to parallel playback method and device |
| CN108108285A (en) * | 2017-12-26 | 2018-06-01 | 广东欧珀移动通信有限公司 | Log processing method, device, storage medium and terminal device |
| US20190236194A1 (en) * | 2018-01-31 | 2019-08-01 | Splunk Inc. | Dynamic query processor for streaming and batch queries |
| CN108572914A (en) * | 2018-03-14 | 2018-09-25 | 口碑(上海)信息技术有限公司 | A kind of method and device generating the daily record for pressure test |
| CN109189736A (en) * | 2018-08-01 | 2019-01-11 | 中国联合网络通信集团有限公司 | A kind of generation method and device of alarm association rule |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000405A (en) * | 2020-07-29 | 2020-11-27 | 深圳市智微智能软件开发有限公司 | Method, equipment and storage medium for conveniently acquiring system logs |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10812566B2 (en) | Distributed steam processing | |
| US10909064B2 (en) | Application architecture supporting multiple services and caching | |
| US8423651B1 (en) | Cross-domain communications with a shared worker application | |
| US7992127B2 (en) | Method and system of encapsulating web site transactions for computer-aided generation of web services | |
| US8626865B1 (en) | Shared worker application | |
| US10574724B2 (en) | Automatic discovery of management nodes and generation of CLI using HA module | |
| US8924867B2 (en) | Web interface for remote platform build | |
| CN111177618A (en) | Website building method, device, equipment and computer readable storage medium | |
| CN111859224A (en) | Server rendering method and device based on node server and computer equipment | |
| CN110826978A (en) | Unified backlog processing method and device based on enterprise browser | |
| CN112612988A (en) | Page processing method and device, computer equipment and storage medium | |
| Rattanapoka et al. | An MQTT-based IoT cloud platform with flow design by Node-RED | |
| US10997357B2 (en) | User interface navigation management | |
| JP2004178263A (en) | Web server, web server with function of java servlet, and computer program | |
| US20130124971A1 (en) | Real time web script refresh using asynchronous polling without full web page reload | |
| CN113626795A (en) | Verification method and device of distributed system architecture, electronic equipment and storage medium | |
| CN110597777A (en) | Log processing method and device | |
| US8793326B2 (en) | System, method and computer program product for reconstructing data received by a computer in a manner that is independent of the computer | |
| EP4211562A1 (en) | Out-of-the-box telemetry for rich-client application runtime frameworks | |
| US10530709B2 (en) | Container contract for data dependencies | |
| RU2691834C1 (en) | Method of managing web site data | |
| US20220334817A1 (en) | Cross platform configuration domain translation | |
| CN116208676A (en) | Data back-source method, device, computer equipment, storage medium and program product | |
| WO2022126711A1 (en) | Webpage forensics method, apparatus and device | |
| US20210042334A1 (en) | Multi-cloud object store access |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191220 |