CN115102830A - Log reduction method, apparatus, computer device and computer-readable storage medium - Google Patents
Log reduction method, apparatus, computer device and computer-readable storage medium Download PDFInfo
- Publication number
- CN115102830A CN115102830A CN202210731326.6A CN202210731326A CN115102830A CN 115102830 A CN115102830 A CN 115102830A CN 202210731326 A CN202210731326 A CN 202210731326A CN 115102830 A CN115102830 A CN 115102830A
- Authority
- CN
- China
- Prior art keywords
- preset
- log
- request message
- query
- storing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0273—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using web services for network management, e.g. simple object access protocol [SOAP]
- H04L41/0286—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using web services for network management, e.g. simple object access protocol [SOAP] for search or classification or discovery of web services providing management functionalities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0604—Management of faults, events, alarms or notifications using filtering, e.g. reduction of information by using priority, element types, position or time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a log reduction method, a log reduction device, a computer device and a computer readable storage medium. The method comprises the following steps: according to a preset category model and a preset calculation statistical formula, counting the number of received query type requests sent by a client in real time; generating a corresponding participation request message and a corresponding response message based on the query request, and sending the response message to the client; and when the number of the query class requests reaches the corresponding preset access amount threshold value within a preset time range, executing preset operation, wherein the preset operation comprises the step of storing the access request message corresponding to the query class request into a log table and not storing a response message. The invention can effectively reduce the data storage amount and the increment of the log table and the server log, avoid the overstaffed log table or the server log caused by a large amount of repeated logs, improve the running performance of the server, reduce invalid logs and greatly improve the effectiveness of log investigation and the response efficiency of production problem investigation.
Description
Technical Field
The present invention relates to the field of log file processing technologies, and in particular, to a log reduction method and apparatus, a computer device, and a computer-readable storage medium.
Background
With the rapid development of the internet, the information data is obviously accelerated, and the size of the data to be stored is larger and larger, so that a large amount of log information is generated to be recorded, and most of the records are repeated. Particularly in the financial industry, hundreds of millions of log information are generated every day, and the log information increases at the speed of tens of G. The conventional log recording system records all log information, and the log information is stored in a log table and a server without being changed, so that a large amount of data redundancy is caused, a large amount of log information quickly piles up a disk space of the server, the running speed of the system and the access speed of the server are reduced, and the condition that data writing of a database is slow due to too many log files is caused. Meanwhile, technicians are also required to frequently clear and file the logs, so that the historical logs cannot be queried in real time.
At present, in order to solve the problem that a server cannot cause system operation abnormity because a disk space is completely occupied by log information, a mode of reducing log writing standards is generally adopted, but the information is seriously lost when related technicians check log files in the later period, and the integrity and the accuracy of the log information are seriously influenced.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a log reduction method, apparatus, computer device and computer readable storage medium, which overcome the shortcomings in the prior art, and solve the problem that the access speed of the server is slow due to the excessive amount of logs.
The embodiment of the disclosure provides a log reduction method and device, computer equipment and a computer readable storage medium.
In a first aspect, an embodiment of the present disclosure provides a log reduction method, which is applied to a server, and the method includes:
according to a preset category model and a preset calculation statistical formula, counting the number of received query type requests sent by a client in real time;
generating a corresponding participation request message and a corresponding response message based on the query request, and sending the response message to the client;
and when the number of the query requests reaches a corresponding preset access amount threshold value within a preset time range, executing a preset operation, wherein the preset operation comprises storing the access request messages corresponding to the query requests into a log table and not storing the response messages.
Further, the method further comprises:
storing query requests within a preset interval time threshold to a server log;
and storing the access request message corresponding to the query class request exceeding the preset interval time threshold value into the log table.
Further, the method further comprises:
and when the size of the participation request message reaches a preset threshold value, compressing the participation request message and then storing the compressed participation request message to the log table.
Further, the method further comprises:
comparing the content in the participation request message with a preset invalid field;
and if the fields same as the preset invalid fields exist, deleting the fields same as the preset invalid fields in the entry request message.
Further, the method further comprises:
recognizing the participation request message by using a preset repeatability recognition model;
and if two access request messages with the repetition degree reaching a preset repetition degree threshold exist, deleting one access request message.
Further, the method further comprises:
judging whether a preset url address exists in the entry request message stored in the log table;
and if so, deleting the preset url address in the join request message to be stored in the log table.
In a second aspect, an embodiment of the present disclosure provides a log reduction apparatus, including:
the statistical module is used for carrying out real-time statistics on the number of received query type requests sent by the client according to a preset type model and a preset calculation statistical formula;
the sending module is used for generating a corresponding participation request message and a corresponding response message based on the query request and sending the response message to the client;
and the first storage module is used for executing a preset operation when the number of the query class requests reaches a corresponding preset access amount threshold value within a preset time range, wherein the preset operation comprises storing the participation request message corresponding to the query class request into a log table and not storing the response message.
Further, the apparatus further comprises:
the second storage module is used for storing the query request within the preset interval time threshold to the server log;
and the third storage module is used for storing the participation request message corresponding to the query class request exceeding the preset interval time threshold value into the log table.
In a third aspect, an embodiment of the present disclosure provides a computer device, including a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the log reduction method according to any one of the first aspect when executing the computer program.
In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the steps of the log reduction method of any one of the first aspects.
The embodiment of the invention has the following advantages:
according to the log reduction method provided by the embodiment of the application, the number of received query type requests sent by a client is counted in real time according to a preset type model and a preset calculation statistical formula; generating a corresponding participation request message and a corresponding response message based on the query request, and sending the response message to the client; and when the number of the query class requests reaches a corresponding preset access amount threshold value within a preset time range, executing a preset operation, wherein the preset operation comprises storing the access request message corresponding to the query class request into a log table and not storing the response message. The invention can effectively reduce the data storage amount and the increment of the log table and the server log, avoid the overlarge log table or the server log caused by a large number of repeated logs, improve the running performance of the server, reduce the invalid log, reduce the log length and greatly improve the effectiveness of log investigation and the response efficiency of production problem investigation.
In order to make the aforementioned objects, features and advantages of the present invention more comprehensible and comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a flowchart illustrating a log reduction method provided by an embodiment of the present application;
FIG. 2 is a schematic structural diagram illustrating a log reduction apparatus provided in an embodiment of the present application;
fig. 3 shows a hardware architecture diagram of a computer device according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
It will be understood that when an element is referred to as being "secured to" another element, it can be directly on the other element or intervening elements may also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. In contrast, when an element is referred to as being "directly on" another element, there are no intervening elements present. The terms "vertical," "horizontal," "left," "right," and the like as used herein are for illustrative purposes only.
In the present invention, unless otherwise expressly stated or limited, the terms "mounted," "connected," "secured," and the like are to be construed broadly and can, for example, be fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; either directly or indirectly through intervening media, either internally or in any other relationship. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the templates herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
Example 1
As shown in fig. 1, which is a flowchart of a log reduction method in an embodiment of the present application, the log reduction method provided in the embodiment of the present application is applied to a server, and the method includes the following steps:
specifically, the number of received query class requests sent by the client is counted in real time according to a preset class model and a preset calculation statistical formula, and further, the number of query class requests at a daily level, the number of query class requests at a weekly level and the number of query class requests at a monthly level can be counted respectively. The grading statistics can achieve the purpose of counting the number of the query requests according to a certain time period, so that a log manager and a querier can manage and query the requests conveniently.
102, generating a corresponding access request message and a corresponding response message based on the query request, and sending the response message to the client;
specifically, based on the counted query request, a corresponding access request message and a corresponding response message are generated, and the response message can be directly sent to the corresponding client after the corresponding response message is generated. The access request message comprises a message header and a message body, the message header of the access request message comprises attribute information of the access request message, the message body of the access request message comprises at least one log object, and the log object comprises corresponding data in the query request. And the server side packages the plurality of log objects according to a preset participation request message format to obtain a participation request message.
In this embodiment, the message header of the participation request message includes a message length, a header identifier, a version number, a compression identifier, and a log object number. The length of the entry request message is the total number of bytes of the message (excluding the length of the length value itself), that is, the number of bytes from the beginning of the header identifier of the entry request message to the end of the entry request message occupies four bytes, and a large-end mode is adopted, and the maximum length of a single entry request can be 4294967295 bytes. Head identification: the identification information of the entry request header may be a fixed character of one byte, for example, 'H' (0x 72); version number: the protocol version information of the current access request message occupies two bytes; and (3) compressing the identification: whether a message body for entering the participation request is compressed or not is one byte, a deflate algorithm and the like can be adopted as a compression algorithm, compression can be represented by ' C ' (0x67) ' 0 ' (0x30) ' is uncompressed; number of log objects: the number of the log objects contained in the entry request message body occupies four bytes, and a large-end mode is adopted, so that one log request message can contain 4294967295 log objects at maximum.
Furthermore, the message body of the access request message may include a plurality of log objects, the message body is a log object set, each log object may include information such as the length of the log, the log sequence number value and the like in addition to the content of the log itself, the content of a single log object may be composed of three parts, namely the length of the log object, the log sequence number and the log content, wherein the length of the log object is the total byte number (excluding the length of the length value itself) of the current log object, occupies four bytes, and a big-end mode is adopted; the log sequence number represents the arrangement position of the current log object in the message body of the access request message, occupies four bytes, adopts a large-end mode, and has uniqueness in the log sequence number of each log object content in the same access request message. The log sequence number may be from 1, after which each increment of the log sequence number is incremented by 1; the log content is the information in the query request.
Specifically, when the number of the query class requests reaches a corresponding preset access amount threshold within a preset time range, a preset operation is executed. If the number of the query class requests at the daily level reaches ten thousand times within 8 hours, the preset operation is executed, if the number of the query class requests at the weekly level reaches ten thousand times within 48 hours, the preset operation is executed, and if the number of the query class requests at the monthly level reaches one million times within 200 hours, the preset operation is executed. It can be understood that the preset time range and the preset access amount threshold may be set according to actual requirements, and this is not limited in this embodiment of the application.
Further, the preset operation includes storing the entry request packet corresponding to the query type request to a log table and not storing the response packet. Because the join request message is convenient for inquiring problems, only the join request message is stored in the log table, and the response message is not stored because the size of the data volume stored in the log table can be effectively reduced.
In an optional implementation, the method may further include:
step 104, storing the query request within a preset interval time threshold to a server log;
and 105, storing the participation request message corresponding to the query class request exceeding the preset interval time threshold value into the log table.
Specifically, query class requests of the same client within a preset interval time threshold are stored in a server log, wherein the server log is one or more log files automatically created and maintained by a server, and entry request messages corresponding to the query class requests exceeding the preset interval time threshold are stored in a log table. For example, the query class request of the same client in one hour is stored in the server log every minute, and the participation request message corresponding to the query class request of the same client in more than one hour is stored in the log table every minute. The preset interval time threshold may be set according to actual requirements, and this is not limited in the embodiment of the present application.
It can be understood that, storing one part of the query request into the log table and another part into the server log can effectively reduce the size of the data amount stored in the log table and relieve the storage pressure of the log table.
In an optional implementation, the method may further include:
and 106, when the size of the participation request message reaches a preset threshold value, compressing the participation request message and storing the compressed participation request message in the log table.
In this embodiment, when the size of each entry request message reaches a preset threshold, the entry request message may be compressed to a preset size and then dropped. Specifically, if the size of the current join request message is larger than the preset threshold, the current join request message is cut off from the end, so that the size of the join request message meets the preset threshold. The preset threshold value may be set according to actual requirements, which is not limited in the embodiment of the present application.
It can be understood that, by compressing the entry request message and then storing the entry request message into the log table, the size of the data storage amount of the log table can be effectively reduced, and the storage pressure of the log table is relieved.
In an optional implementation, the method may further include:
step 107, comparing the content in the reference request message with a preset invalid field;
and step 108, if the field identical to the preset invalid field exists, deleting the field identical to the preset invalid field in the participation request message.
Specifically, according to a preset invalid field, for example, a large string of place-occupying symbols such as "@ @ @ @ @ @ @ @ @ @ @ @ @," ################### "or"%%%%%%%% ", the content in the entry request message is compared with the preset invalid field, and if a field identical to the preset invalid field exists, the field identical to the preset invalid field in the entry request message is deleted. The preset invalid field may be set according to actual requirements, which is not limited in the embodiment of the present application.
It can be understood that, deleting the invalid field in the join request message can effectively eliminate invalid or unimportant contents, and only retain the key and necessary contents in the join request message, so that the size of the data amount stored in the log table can be effectively reduced by storing the contents in the log table, and the storage pressure of the log table is relieved.
In an optional implementation, the method may further include:
step 109, identifying the participation request message by using a preset repetition degree identification model;
and step 110, if two access participation request messages with the repetition degree reaching a preset repetition degree threshold exist, deleting one access participation request message.
Specifically, the repetition degree recognition model trained in advance can be used for carrying out repetition degree recognition on the entry-join request message, and if two entry-join request messages with the repetition degree reaching a preset repetition degree threshold exist, one entry-join request message is deleted. For example, if it is detected that the repetition degree of two entry request messages reaches 90%, one of the entry request messages is deleted. The preset repetition threshold may be set to 90%, 95%, or 97%, and the like, and may be specifically set according to actual requirements, which is not limited in this embodiment of the application.
It can be understood that deleting the join request message with high repetition degree can effectively prevent the message with approximate repetition degree from being stored in the log table, effectively reduce the size of the data storage amount of the log table, and relieve the storage pressure of the log table.
In an optional implementation, the method may further include:
step 111, judging whether a preset url address exists in the entry request message stored in the log table;
and step 112, if the entry request message exists, deleting the preset url address in the entry request message to be stored in the log table.
Specifically, whether the preset url addresses exist in the entry request message already saved to the log table is determined according to preset url (uniform resource locator) addresses, for example, keywords related to requests such as "/get,/fetch,/list,/query", and the like, and if the preset url addresses exist in the entry request message already saved to the log table, the preset url addresses to be stored in the log table entry request message are deleted. The preset url address may be set according to actual requirements, which is not limited in the embodiment of the present application.
It can be understood that, because these preset url addresses exist in the log table, all the preset url addresses can be directly accessed when address access is performed, deleting the preset url addresses to be stored in the log table entry request message can effectively avoid repeated storage of the stored address information, and can also effectively reduce the size of the data storage amount of the log table and relieve the storage pressure of the log table.
The embodiment of the invention has the following advantages:
according to the log reduction method provided by the embodiment of the application, the number of received query type requests sent by a client is counted in real time according to a preset type model and a preset calculation statistical formula; generating a corresponding participation request message and a corresponding response message based on the query request, and sending the response message to the client; and when the number of the query class requests reaches a corresponding preset access amount threshold value within a preset time range, executing a preset operation, wherein the preset operation comprises storing the access request message corresponding to the query class request into a log table and not storing the response message. The invention can effectively reduce the data storage amount and the increment of the log table and the server log, avoid the overlarge log table or the server log caused by a large number of repeated logs, improve the running performance of the server, reduce the invalid log, reduce the log length and greatly improve the effectiveness of log investigation and the response efficiency of production problem investigation.
Example 2
Fig. 2 is a schematic structural diagram of a log reduction apparatus 200 in an embodiment of the present application, where the log reduction apparatus is applied to a server, and the apparatus includes:
the statistical module 201 is configured to count, in real time, the number of received query type requests sent by the client according to a preset type model and a preset calculation statistical formula;
a sending module 202, configured to generate a corresponding entry request packet and a response packet based on the query type request, and send the response packet to the client;
the storage module 203 is configured to execute a preset operation when the number of the query class requests reaches a corresponding preset access amount threshold within a preset time range, where the preset operation includes storing a join request packet corresponding to the query class request in a log table and not storing the response packet.
Optionally, the log reduction apparatus may further include:
a first storage module 204, configured to store the query class request within a preset interval time threshold to a server log;
the second storage module 205 is configured to store a join request packet corresponding to a query class request exceeding a preset interval time threshold into the log table.
Optionally, the log reduction apparatus may further include:
a third storage module 206, configured to, when the size of the join request packet reaches a preset threshold, compress the join request packet and store the compressed join request packet in the log table.
Optionally, the log reduction apparatus may further include:
a comparing module 207, configured to compare the content in the refer request message with a preset invalid field;
a first deleting module 208, configured to delete a field in the join request message that is the same as a preset invalid field if the field that is the same as the preset invalid field exists.
Optionally, the log reduction apparatus may further include:
an identifying module 209, configured to identify the participation request packet by using a preset repetition degree identification model;
the second deleting module 210 is configured to delete one of the entry request messages if there are two entry request messages whose repetition degrees reach the preset repetition degree threshold.
Optionally, the log reduction apparatus may further include:
the judging module 211 is configured to judge whether a preset url address exists in the entry request message that has been stored in the log table;
a third deleting module 212, configured to delete the preset url address in the join request message to be saved in the log table.
The embodiment of the invention has the following advantages:
the log reduction device provided by the embodiment of the application can effectively reduce the data storage amount and the increase amount of the log table and the server log, avoid the phenomenon that the log table or the server log is too bloated due to the existence of a large number of repeated logs, improve the running performance of the server, reduce invalid logs, reduce the length of the log, and greatly improve the effectiveness of log investigation and the response efficiency of production problem investigation.
Example 3
Fig. 3 shows a hardware architecture diagram of a computer device provided in the present application, where the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the log reduction method according to embodiment 1 when executing the computer program.
In the present embodiment, the computer device 300 is a device capable of automatically performing numerical calculation and/or information processing in accordance with a command set or stored in advance. For example, the server may be a rack server, a blade server, a tower server, or a rack server (including an independent server or a server cluster composed of a plurality of servers). As shown in fig. 3, computer device 300 includes at least, but is not limited to: memory 310, processor 320, network interface 330 may be communicatively linked to each other via a system bus. Wherein:
the memory 310 includes at least one type of computer-readable storage medium including flash memory, hard disks, multimedia cards, card-type memory (e.g., SD or DX memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), magnetic memory, magnetic disks, optical disks, etc. In some embodiments, the storage 310 may be an internal storage module of the computer device 300, such as a hard disk or a memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the computer device 300. Of course, the memory 310 may also include both internal and external memory modules of the computer device 300. In this embodiment, the memory 310 is generally used for storing an operating system installed in the computer device 300 and various application software, such as program codes of a video playing method. In addition, the memory 310 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 320 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 320 is generally configured to control the overall operation of the computer device 300, such as performing control and processing related to data interaction or communication with the computer device 300. In this embodiment, the processor 320 is used to execute program codes stored in the memory 310 or process data.
The network interface 330 may include a wireless network interface or a wired network interface, with the network interface 330 generally being used to establish communication links between the computer device 300 and other computer devices. For example, the network interface 330 is used to connect the computer device 300 to an external terminal via a network, establish a data transmission channel and a communication link between the computer device 300 and the external terminal, and the like. The network may be an Intranet (Intranet), the Internet (Internet), a global system for Mobile communications (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), Wi-Fi, or other wireless or wired network.
It is noted that fig. 3 only illustrates a computer device having components 310-330, but it is understood that not all illustrated components are required to be implemented, and that more or fewer components may alternatively be implemented.
In this embodiment, the log reduction method stored in the memory 310 can also be divided into one or more program modules and executed by one or more processors (in this embodiment, the processor 320) to complete the present invention.
Example 4
The present embodiments also provide a computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the log reduction method in the embodiments.
In this embodiment, the computer-readable storage medium includes a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the computer readable storage medium may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium may be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device. Of course, the computer-readable storage medium may also include both internal and external storage units of the computer device. In this embodiment, the computer-readable storage medium is generally used for storing an operating system and various types of application software installed in the computer device. Further, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.
In all examples shown and described herein, any particular value should be construed as merely exemplary, and not as a limitation, and thus other examples of example embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
The above examples are merely illustrative of several embodiments of the present invention, and the description thereof is more specific and detailed, but not to be construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention.
Claims (10)
1. A log reduction method applied to a server, the method comprising:
according to a preset category model and a preset calculation statistical formula, counting the number of received query type requests sent by a client in real time;
generating a corresponding participation request message and a corresponding response message based on the query request, and sending the response message to the client;
and when the number of the query class requests reaches a corresponding preset access amount threshold value within a preset time range, executing a preset operation, wherein the preset operation comprises storing the access request message corresponding to the query class request into a log table and not storing the response message.
2. The log reduction method of claim 1, further comprising:
storing query requests within a preset interval time threshold to a server log;
and storing the access request message corresponding to the query class request exceeding the preset interval time threshold value into the log table.
3. The log reduction method of claim 1, further comprising:
and when the size of the participation request message reaches a preset threshold value, compressing the participation request message and then storing the compressed participation request message to the log table.
4. The log reduction method of claim 1, further comprising:
comparing the content in the participation request message with a preset invalid field;
and if the fields same as the preset invalid fields exist, deleting the fields same as the preset invalid fields in the participation request message.
5. The log reduction method of claim 1, further comprising:
recognizing the participation request message by using a preset repeatability recognition model;
and if two access request messages with the repetition degree reaching a preset repetition degree threshold exist, deleting one access request message.
6. The log reduction method of claim 1, the method further comprising:
judging whether a preset url address exists in the entry request message stored in the log table;
and if so, deleting the preset url address in the join request message to be stored in the log table.
7. An apparatus for log reduction, the apparatus comprising:
the statistical module is used for carrying out real-time statistics on the number of received query type requests sent by the client according to a preset type model and a preset calculation statistical formula;
the sending module is used for generating a corresponding access request message and a corresponding response message based on the query request and sending the response message to the client;
and the first storage module is used for executing a preset operation when the number of the query class requests reaches a corresponding preset access amount threshold value within a preset time range, wherein the preset operation comprises storing the participation request message corresponding to the query class request into a log table and not storing the response message.
8. The log reduction device of claim 7, wherein the device further comprises:
the second storage module is used for storing the query request within the preset interval time threshold to the server log;
and the third storage module is used for storing the participation request message corresponding to the query class request exceeding the preset interval time threshold value into the log table.
9. A computer arrangement, comprising a memory storing a computer program and a processor implementing the steps of the log reduction method of any of claims 1-6 when the processor executes the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the steps of the log reduction method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210731326.6A CN115102830B (en) | 2022-06-24 | 2022-06-24 | Log reduction method, device, computer equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210731326.6A CN115102830B (en) | 2022-06-24 | 2022-06-24 | Log reduction method, device, computer equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115102830A true CN115102830A (en) | 2022-09-23 |
| CN115102830B CN115102830B (en) | 2023-07-14 |
Family
ID=83293657
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210731326.6A Active CN115102830B (en) | 2022-06-24 | 2022-06-24 | Log reduction method, device, computer equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115102830B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190036953A1 (en) * | 2017-07-25 | 2019-01-31 | Netskope, Inc. | Compact logging of network traffic events |
| US10210162B1 (en) * | 2010-03-29 | 2019-02-19 | Carbonite, Inc. | Log file management |
| CN112650753A (en) * | 2020-12-23 | 2021-04-13 | 平安普惠企业管理有限公司 | Log management method, device, system, equipment and readable storage medium |
| CN113497723A (en) * | 2020-03-20 | 2021-10-12 | 阿里巴巴集团控股有限公司 | Log processing method, log gateway and log processing system |
| CN113535654A (en) * | 2021-06-11 | 2021-10-22 | 安徽安恒数智信息技术有限公司 | Log processing method, system, electronic device and storage medium |
| CN113760849A (en) * | 2021-11-10 | 2021-12-07 | 深圳市明源云科技有限公司 | Log processing method, system, electronic device and computer readable storage medium |
-
2022
- 2022-06-24 CN CN202210731326.6A patent/CN115102830B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10210162B1 (en) * | 2010-03-29 | 2019-02-19 | Carbonite, Inc. | Log file management |
| US20190036953A1 (en) * | 2017-07-25 | 2019-01-31 | Netskope, Inc. | Compact logging of network traffic events |
| CN113497723A (en) * | 2020-03-20 | 2021-10-12 | 阿里巴巴集团控股有限公司 | Log processing method, log gateway and log processing system |
| CN112650753A (en) * | 2020-12-23 | 2021-04-13 | 平安普惠企业管理有限公司 | Log management method, device, system, equipment and readable storage medium |
| CN113535654A (en) * | 2021-06-11 | 2021-10-22 | 安徽安恒数智信息技术有限公司 | Log processing method, system, electronic device and storage medium |
| CN113760849A (en) * | 2021-11-10 | 2021-12-07 | 深圳市明源云科技有限公司 | Log processing method, system, electronic device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115102830B (en) | 2023-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10652265B2 (en) | Method and apparatus for network forensics compression and storage | |
| CN109933630B (en) | Real-time database data synchronization method and device | |
| CN109815214B (en) | Database access method, system, device and storage medium | |
| WO2017113124A1 (en) | Server and method for compressing data by server | |
| CN110995273B (en) | Data compression method, device, equipment and medium for power database | |
| CN113238912B (en) | Aggregation processing method for network security log data | |
| CN113886193A (en) | Log data processing method and device, electronic equipment and storage medium | |
| CN113449339A (en) | Log collection method, system, computer device and computer readable storage medium | |
| CN110457255B (en) | Method, server and computer readable storage medium for archiving data | |
| CN110597461B (en) | Data storage method, device and equipment in block chain type account book | |
| CN111008183A (en) | Storage method and system for business wind control log data | |
| CN112291214B (en) | Industrial message analysis method and system based on redis cache | |
| CN112817926B (en) | File processing method and device, storage medium and electronic device | |
| CN115102830B (en) | Log reduction method, device, computer equipment and computer readable storage medium | |
| CN116126997B (en) | Document deduplication storage method, system, device and storage medium | |
| CN110032432B (en) | Example compression method and device and example decompression method and device | |
| CN115470186A (en) | Data slicing method, device and system | |
| CN114610792A (en) | Data processing method, device and system and industrial equipment | |
| CN114329253A (en) | Network operation data query method, device, equipment and storage medium | |
| CN102891732A (en) | Data sending method and device and data receiving method and device | |
| CN111753518A (en) | Autonomous file consistency checking method | |
| CN114070471B (en) | Test data packet transmission method, device, system, equipment and medium | |
| CN111274350B (en) | Data processing method, device, computer equipment and storage medium | |
| CN113489635B (en) | WeChat-based message recovery method and related equipment | |
| CN113127012B (en) | Software resource construction method based on software engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |