CN115203139A

CN115203139A - Log query method and device, electronic equipment and storage medium

Info

Publication number: CN115203139A
Application number: CN202210783421.0A
Authority: CN
Inventors: 王旭
Original assignee: Agricultural Bank of China
Current assignee: Agricultural Bank of China
Priority date: 2022-06-27
Filing date: 2022-06-27
Publication date: 2022-10-18

Abstract

The invention discloses a log query method and device, electronic equipment and a storage medium. The log query method is applied to a server, the server is in communication connection with a client through a full-duplex communication protocol, and the method comprises the following steps: receiving a log query request sent by the client in a non-blocking thread mode, and when the log query request is in a readable state, performing log query on a log storage system to obtain a log query result; and under the condition that the log query result comprises a newly added log, sending the log query result to the client through the full-duplex communication protocol. According to the technical scheme, the full-duplex communication protocol is used for communication, so that data exchange between the client and the server becomes simpler, the server is allowed to actively push data to the client, and the condition that the log query request occupies too many system resources can be avoided.

Description

Log query method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a log query method and apparatus, an electronic device, and a storage medium.

Background

The Elasticissearch is a distributed, highly-extended and highly-real-time search and data analysis engine which can conveniently enable a large amount of data to have the capability of searching, analyzing and exploring.

In a large-scale log collection, storage, query and analysis system based on the Elasticsearch, a user cannot query a system log in real time due to the fact that the Elasticsearch cannot realize an active push function, and in the prior art, real-time log query can be realized in an HTTP polling mode.

In the process of implementing the invention, the inventor finds that the following technical problems exist in the prior art: at present, in a real-time log query scheme implemented in an HTTP polling manner, repeated HTTP requests occupy too many system resources, resulting in too much system resource consumption.

Disclosure of Invention

The invention provides a log query method, a log query device, electronic equipment and a storage medium, and aims to solve the problem of overlarge resource consumption of a log query system.

According to an aspect of the present invention, a log query method is provided, which is applied to a server, where the server and a client are in communication connection through a full-duplex communication protocol, and the method includes:

receiving a log query request sent by the client in a non-blocking thread mode, and when the log query request is in a readable state, performing log query on a log storage system to obtain a log query result;

and under the condition that the log query result comprises a newly added log, sending the log query result to the client through the full-duplex communication protocol.

According to another aspect of the present invention, there is provided a log query apparatus, applied to a server, where the server and a client are in communication connection through a full-duplex communication protocol, the apparatus including:

the log query module is used for receiving a log query request sent by the client in a non-blocking thread mode, and when the log query request is in a readable state, performing log query on a log storage system to obtain a log query result;

and the log sending module is used for sending the log query result to the client through the full-duplex communication protocol under the condition that the log query result comprises a newly added log.

According to another aspect of the present invention, there is provided an electronic apparatus including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the log query method of any of the embodiments of the present invention.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the log query method according to any one of the embodiments of the present invention when the computer instructions are executed.

According to the technical scheme of the embodiment of the invention, the server and the client are in communication connection through a full-duplex communication protocol, and in a non-blocking thread mode, the server receives a log query request sent by the client, and when the log query request is in a readable state, log query is carried out on a log storage system to obtain a log query result; and under the condition that the log query result comprises a newly added log, sending the log query result to the client through a full-duplex communication protocol. According to the technical scheme, the communication is carried out through the full-duplex communication protocol, so that the data exchange between the client and the server becomes simpler, the server is allowed to actively push data to the client, the condition that the log query request occupies too many system resources can be avoided, and the system resources are saved.

It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present invention, nor are they intended to limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a flowchart of a log query method according to an embodiment of the present invention;

FIG. 2 is a flowchart of a log query method according to a second embodiment of the present invention;

fig. 3 is a schematic structural diagram of a log query apparatus according to a third embodiment of the present invention;

fig. 4 is a schematic structural diagram of an electronic device implementing the log query method according to the embodiment of the present invention.

Detailed Description

In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 1 is a flowchart of a log query method according to an embodiment of the present invention, where this embodiment is applicable to a situation where log query is performed based on a full duplex communication protocol, and the method may be executed by a log query device, where the log query device may be implemented in hardware and/or software, the log query device may be configured in a server, and the server and a client are in communication connection through the full duplex communication protocol, for example, the server may be a server that provides various services. As shown in fig. 1, the method includes:

s110, receiving a log query request sent by the client in a non-blocking thread mode, and when the log query request is in a readable state, performing log query on a log storage system to obtain a log query result.

In the embodiment of the present invention, the server may be a server providing various services, for example, a background server providing support for a page displayed on the client. It should be noted that, the log query method provided by the embodiment of the present application is executed by a server, and accordingly, the log query apparatus is disposed in the server. It should be understood that the number of clients and servers may be one or more. According to implementation needs, there may be any number of clients and servers, and the clients in this embodiment may specifically correspond to software systems in actual production. The log storage system is a database storing logs, the log storage system can be in communication connection with the server, and optionally, the log storage system can be an elastic search.

The non-Blocking thread mode refers to a synchronous non-Blocking IO (NIO) working mode, and the working mode can avoid the situation that a large number of threads are suspended, so that the waiting time of the threads is reduced.

Specifically, under the condition that the current thread is unavailable, if an available thread exists in a server thread pool, the current thread is replaced by the available thread, and the log query request is set to be in a readable state so as to process the current log query request, and the occupation of computing resources is reduced; and if no available thread exists in the thread pool, setting the log query request to a waiting readable state, which indicates that the log query request cannot be executed currently and the available thread needs to wait to appear.

And S120, under the condition that the log query result comprises a newly added log, sending the log query result to the client through the full-duplex communication protocol.

It should be noted that, in the prior art, the HTTP polling manner may send the log query request at regular time, and if the log storage system does not add a new log temporarily, because the log query request based on the HTTP protocol is a response type, the server has to respond to the log query request after receiving the log query request, which wastes unnecessary system computing resources. The full-duplex communication protocol is adopted for communication, and compared with the HTTP protocol, the full-duplex communication protocol has a simple request head, so that network bandwidth resources can be saved in the communication process, and only under the condition that the log query result comprises a newly added log, the server side sends the log query result to the client side, so that unnecessary response can be reduced.

Optionally, the full duplex communication protocol may include a Websocket protocol. The Websocket can realize real-time two-way communication and realize the active pushing of the server to the client, and compared with the use of HTTP polling, the Websocket can effectively save network communication frequency, reduce the occupation of a request head on network bandwidth and increase the operating efficiency of the server.

On the basis of the above embodiment, after the receiving the log query request sent by the client, the method further includes: receiving a heartbeat message sent by the client, and sending a feedback message to the client to verify whether the server is disconnected with the client; and under the condition that the server is disconnected with the client, reestablishing the communication connection based on the full-duplex communication protocol.

Specifically, the Websocket can support a disconnection reconnection mechanism, the client periodically sends a heartbeat message to the server, and if the client does not receive a feedback message of the server, the disconnection between the server and the client is indicated. Under the condition that the server side is disconnected with the client side, the Websocket connection can be reestablished, and compared with the method that the Websocket API is directly used for achieving real-time message pushing, the robustness is better.

On the basis of the above embodiment, the method further includes: and receiving a log query request sent by the client in an HTTP polling mode under the condition that the communication connection based on the full-duplex communication protocol fails to be established.

Specifically, if the browser of the client does not support the Websocket protocol, the log query method of the embodiment automatically backs to the real-time pushing of the log query request by using the HTTP polling mode, so that the log query method of the embodiment has better browser support.

Example two

Fig. 2 is a flowchart of a log query method according to a second embodiment of the present invention, and this embodiment further refines "perform log query on a log storage system" on the basis of the above embodiment. Optionally, the performing log query on the log storage system includes: and detecting whether the log storage system has data change, if so, reading the updated newly added log from the log storage system, and taking the newly added log as a log query result.

As shown in fig. 2, the method includes:

s210, receiving a log query request sent by the client in a non-blocking thread mode.

S220, when the log query request is in a readable state, detecting whether the log storage system has data change, if so, reading the updated newly added log from the log storage system, and taking the newly added log as a log query result.

And S230, under the condition that the log query result comprises a newly added log, sending the log query result to the client through the full-duplex communication protocol.

Specifically, the server can detect whether the log storage system has data change. And under the condition that the data of the log storage system changes, the log storage system is indicated to have a newly added log, and the updated newly added log can be read from the log storage system, so that the newly added log is sent to the client for displaying through the full-duplex communication protocol. Compared with the HTTP polling mode, the technical scheme has the advantages that the log query result is fed back only under the condition that data are changed, and unnecessary resource occupation is reduced.

On the basis of the foregoing embodiment, the detecting whether the data change occurs in the log storage system includes: and judging whether data change occurs in the log storage system or not based on the time stamp.

The time stamp may be a time stamp of the log file, and the time stamp of the log file may be changed every time the log file is changed.

Specifically, the timestamp may include a historical update timestamp and a current update timestamp, and if the current update timestamp is different from the historical update timestamp, it is determined that a data change occurs in the log storage system; and if the current update timestamp is the same as the historical update timestamp, determining that no data change occurs in the log storage system.

In some embodiments, the original websock interface is not used in the user agent, and the client may refer to the new sock.js and store.js to establish the websock protocol, which may effectively reduce the repeated global variable configuration in the code.

In some embodiments, the non-blocking thread mode may be implemented by Webflux, which has a non-blocking characteristic compared to HTTP live and RestTemplate, after the log query request is initiated, the thread does not wait for the HTTP response in the idle state, but processes other tasks, and when the log query request becomes readable, the server thread continues to process the log query request.

According to the technical scheme of the embodiment of the invention, the server side and the client side are in communication connection through a full-duplex communication protocol, and the server side can detect whether the log storage system has data change. And under the condition that the data of the log storage system is changed, the new log is indicated to exist in the log storage system, and the updated new log can be read from the log storage system, so that the new log is sent to the client for displaying through the full-duplex communication protocol. Compared with the HTTP polling mode, the technical scheme has the advantages that the log query result is fed back only under the condition that data are changed, and unnecessary resource occupation is reduced.

EXAMPLE III

Fig. 3 is a schematic structural diagram of a log query device according to a third embodiment of the present invention. The log query device is applied to the server side, and the server side is in communication connection with the client side through a full-duplex communication protocol.

As shown in fig. 3, the apparatus includes:

the log query module 310 is configured to receive a log query request sent by the client in a non-blocking thread mode, and perform log query on a log storage system to obtain a log query result when the log query request is in a readable state;

a log sending module 320, configured to send the log query result to the client through the full duplex communication protocol when the log query result includes a newly added log.

In some optional implementations of the embodiments of the present disclosure, optionally, the apparatus is further configured to:

under the condition that the current thread is unavailable, if an available thread exists in a thread pool, the current thread is replaced by the available thread, and the log query request is set to be in a readable state;

and if no available thread exists in the thread pool, setting the log query request to be in a waiting readable state.

In some optional implementations of the embodiment of the present disclosure, optionally, the log query module 310 includes:

and the data change detection unit is used for detecting whether the journal storage system has data change or not, reading the updated newly added journal from the journal storage system if the journal storage system has data change, and taking the newly added journal as a journal query result.

In some optional implementation manners of the embodiments of the present disclosure, optionally, the data change detecting unit includes:

and the time stamp judging subunit is used for judging whether the data change occurs in the log storage system or not based on the time stamp.

In some optional implementation manners of the embodiments of the present disclosure, optionally, the timestamp determining subunit is specifically configured to:

if the current updating time stamp is different from the historical updating time stamp, determining that data change occurs in the log storage system;

and if the current updating time stamp is the same as the historical updating time stamp, determining that no data change occurs in the log storage system.

receiving a heartbeat message sent by the client, and sending a feedback message to the client to verify whether the server is disconnected with the client;

and under the condition that the server is disconnected with the client, reestablishing the communication connection based on the full-duplex communication protocol.

and receiving a log query request sent by the client in an HTTP polling mode under the condition that the communication connection based on the full-duplex communication protocol fails to be established.

The log query device provided by the embodiment of the invention can execute the log query method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

Example four

FIG. 4 shows a schematic block diagram of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as the log query method, which includes:

In some embodiments, the log query method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the log query method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log query method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the Internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log query method is applied to a server side, wherein the server side and a client side are in communication connection through a full-duplex communication protocol, and the method comprises the following steps:

2. The method of claim 1, wherein after the receiving the log query request sent by the client, the method further comprises:

under the condition that the current thread is unavailable, if an available thread exists in a thread pool, replacing the current thread with the available thread, and setting the log query request to be in a readable state;

3. The method of claim 1, wherein performing a log query on a log storage system comprises:

and detecting whether the log storage system has data change, if so, reading the updated newly added log from the log storage system, and taking the newly added log as a log query result.

4. The method of claim 3, wherein the detecting whether the log storage system has changed data comprises:

and judging whether data change occurs in the log storage system or not based on the time stamp.

5. The method of claim 4, wherein the time stamps comprise a historical update time stamp and a current update time stamp; the judging whether the data change occurs in the log storage system based on the timestamp comprises the following steps:

6. The method of claim 1, wherein after the receiving the log query request sent by the client, the method further comprises:

and reestablishing the communication connection based on the full-duplex communication protocol under the condition that the server is disconnected with the client.

7. The method of claim 6, further comprising:

8. A log query device is applied to a server, wherein the server and a client are in communication connection through a full-duplex communication protocol, and the log query device comprises:

9. An electronic device, characterized in that the electronic device comprises:

at least one processor; and

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the log query method of any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions for causing a processor to implement the log query method of any one of claims 1-7 when executed.