CN116302834A

CN116302834A - Log compression method, log search method and related devices

Info

Publication number: CN116302834A
Application number: CN202310310640.1A
Authority: CN
Inventors: 周弘懿
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2023-03-27
Filing date: 2023-03-27
Publication date: 2023-06-23

Abstract

The embodiment of the specification provides a log compression method, a log search method and a related device, which solve the problem of low log compression rate caused by the fact that the existing first log templates in a template library cannot be well matched with logs to be compressed after the logs to be compressed are compressed for the first time by using the first log templates in the template library, if repeated character strings still exist in a plurality of the first log templates, the first log templates are generated based on the repeated character strings, the template library is updated, and the first compressed logs are compressed for the second time based on the generated first log templates, so that compressed logs are obtained.

Description

Log compression method, log search method and related devices

Technical Field

The embodiments in the present specification relate to the field of computer application technologies, and in particular, to a log compression technology in the field of computer application technologies, and more particularly, to a log compression method, a log search method, and related devices.

Background

A log (log) is a file used to record system or application operational events. Journaling has an important role in handling historical data, locating problems, understanding the activities of the system, and the like. During the running process of network equipment, system, service program, etc., one log event record is produced, and one line of log data may be recorded with date, time, user, action, etc. Therefore, it is necessary to store the log to meet the requirement of log viewing when necessary.

In the conventional technology, the original log is usually directly stored or transmitted. However, the number of logs is generally huge, and the memory and network transmission resources required by the way of directly storing or transmitting the original logs are large, so that the logs are required to be stored or transmitted after being compressed. The current log compression method has a low log compression rate, and increases storage and transmission pressure.

Disclosure of Invention

Various embodiments in the present disclosure provide a log compression method, a log search method, and related devices, so as to solve the problem of low log compression rate in the related art.

In a first aspect, an embodiment of the present disclosure provides a log compression method, including:

acquiring a plurality of logs to be compressed;

performing first compression on a plurality of logs to be compressed by using a first log template to obtain a plurality of first compressed logs; the first log template comprises character string identifiers corresponding to the character strings; the first log template is stored in a template library;

generating a first log template for repeated character strings in the plurality of the first compressed logs; the repeated character string includes the same character string existing in a plurality of the first compressed logs;

And performing second compression on the first compression log based on the generated first log template to obtain a compression log.

In a second aspect, an embodiment of the present specification provides a log compressing apparatus, including:

the log acquisition module is used for acquiring a plurality of logs to be compressed;

the first compression module is used for carrying out first compression on the logs to be compressed by using a first log template so as to obtain a plurality of first compressed logs; the first log template comprises character string identifiers corresponding to the character strings; the first log template is stored in a template library;

a template generation module for generating a first log template for repeated character strings in the plurality of first compressed logs; the repeated character string includes the same character string existing in a plurality of the first compressed logs;

and the second compression module is used for performing second compression on the first compression log based on the generated first log template so as to obtain a compression log.

In a third aspect, an embodiment of the present specification provides a log searching method, including:

responding to a query instruction, and determining log content corresponding to the query instruction in the compressed log;

Returning the determined log content to a requester of the query instruction as a response result;

the query instruction comprises a full text query instruction, and the full text query instruction comprises a query character string;

an inverted index is established in advance, the inverted index comprises a first index and a second index, and the first index is used for representing the corresponding relation between words in a log and a first log template mark; the second index is used for representing the corresponding relation between words in the log and the row mark of the log content; the line identification of the log content represents the positioning identification of the line of the log content

The method comprises the steps of responding to a query instruction, and determining log content corresponding to the query instruction in the compressed log:

if the query instruction comprises the full-text query instruction, replacing words in the query string with the first log template identifier based on the first index to obtain a preprocessing string;

querying the compressed log based on the second index to obtain at least one log content row identifier corresponding to the preprocessing character string;

determining target line content in a compressed log corresponding to the line identifier of the log content according to the obtained line identifier of the log content;

And according to the first log template, restoring the first log template in the target row content into a word corresponding to the first log template identifier so as to obtain log content corresponding to the query instruction.

In a fourth aspect, one embodiment of the present specification provides an electronic device, including: a processor and a memory;

wherein the memory is connected with the processor and is used for storing a computer program;

the processor is configured to implement the log compression method or the log search method by running the computer program stored in the memory.

In a fifth aspect, an embodiment of the present specification provides a storage medium having stored thereon a computer program which, when executed by a processor, implements a log compression method as described above or a log search method as described above.

In a sixth aspect, one embodiment of the present specification provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium; the processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor when executing the computer instructions implements the steps of the log compression method or the log search method as described above.

According to the embodiments provided by the specification, after the log to be compressed is compressed for the first time by using the first log templates in the template library, if repeated character strings still exist in a plurality of the first log templates, the first log templates are generated based on the repeated character strings, the template library is updated, and the first compressed log is compressed for the second time based on the generated first log templates, so that the compressed log is obtained, and the problem of low log compression rate caused by the fact that the existing first log templates in the template library cannot be well matched with the log to be compressed is solved.

Drawings

Fig. 1 is a schematic diagram of an application scenario to which a log compression method and a log search method according to an embodiment of the present disclosure may be applied;

FIG. 2 is a schematic flow chart of a log compression method according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram illustrating an implementation process of a log compression method according to an embodiment of the present disclosure;

FIG. 4 is a flow chart of another log compression method according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a process for sliding partition of log content to be compressed based on a sliding window according to an embodiment of the present disclosure;

FIG. 6 is a schematic flow chart of a log searching method according to an embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of a log compressing device according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed Description

Unless defined otherwise, technical or scientific terms used in the embodiments of the present specification should be given the ordinary meaning as understood by one of ordinary skill in the art to which the present specification belongs. The terms "first," "second," and the like, as used in the embodiments of the present disclosure, do not denote any order, quantity, or importance, but rather are used to avoid intermixing of the components.

Throughout the specification, unless the context requires otherwise, the word "plurality" means "at least two", and the word "comprising" is to be construed as open, inclusive meaning, i.e. as "comprising, but not limited to. In the description of the present specification, the terms "one embodiment," "some embodiments," "example embodiments," "examples," "particular examples," or "some examples," etc., are intended to indicate that a particular feature, structure, material, or characteristic associated with the embodiment or example is included in at least one embodiment or example of the present specification. The schematic representations of the above terms do not necessarily refer to the same embodiment or example.

The technical solutions of the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is apparent that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.

SUMMARY

As described in the background art, the network device, system, service program, etc. generate an event record called a log during operation, and a line of log data may describe the date, time, user, action, etc. related operations. By means of the generated logs, the source of error occurrence can be traced, the cause of error occurrence can be analyzed, and the like. In a computer, an operating system generates various log files, such as an application log, a security log, a system log, a DNS (Domain Name System ) server log, and the like, every day, and then in a device cluster such as a large data platform or a data center formed by a plurality of computers, network devices, and the like, massive logs are generated every time. The massive logs occupy a large amount of storage space, so that cost is wasted, and therefore, the massive logs are required to be compressed and stored efficiently.

In the related art, most of log compression methods are based on template compression, namely, according to the corresponding relation between the stored character strings and the coding elements, the character strings corresponding to the coding elements in the log are replaced by the coding elements with shorter lengths, so that the storage space occupied by the log is reduced, and the log is compressed. However, this approach has the following drawbacks: when a new log format appears, the existing template (such as a coding element) cannot adapt to the new log format, so that the log has poor de-duplication effect and low compression rate.

In order to solve the problem, the inventor finds through long-term research that a plurality of logs generated in a period of time can be replaced by templates based on an existing template library to perform first-step compression, repeated fields of the plurality of logs after the first-step compression are extracted to generate a new template, and second-step compression is performed based on the new template to extract repeated fields which cannot be matched with the existing template in the template library, so that the log compression method can be suitable for the compression requirement of new log types, the compression rate of log compression is improved, and the storage space occupied by the compressed logs is smaller.

Further, the inventor further discovers through research that none of the existing log compression schemes consider the scenario of full text searching. Full text searching is a function of finding the position of a string of characters in a log by inputting the string of characters. Full text search functionality has become a rigid requirement for application log search today. The inventor establishes an inverted index for the compressed log content and establishes an inverted index for the log template, so that the function of full text search according to the input character string is realized.

Based on the above-described concept, the present embodiment of the present disclosure provides a log compression method and a log search method, and an application scenario and a possible implementation manner to which the log compression method provided by the present embodiment of the present disclosure may be applicable will be described in an exemplary manner with reference to the accompanying drawings.

Scene example

Referring to fig. 1, fig. 1 shows an application scenario in which a log compression method and a log search method may be applied, in which a client 12 communicates with a server 11 through a network. The storage device 13 may be integrated on the server 11 or may be placed on a cloud or other server. The server 11 may be used alone to perform the log compression method and/or the log search method provided in the embodiments of the present specification. For example, the server 11 may acquire a log generated by the server 11 or the client 12, compress the log according to the log compression method provided in the embodiment of the present specification, and store the compressed log in the storage 13. The server 11 may also obtain a query instruction transmitted by the client 12, execute the log searching method provided in the embodiment of the present disclosure according to the query instruction, obtain a search result from the compressed log stored in the storage device 13, and return the search result to the client 12.

The client 12, the server 11 and the storage device 13 may together form a large data platform 10. In some embodiments, the client 12, the server 11, and the storage device 13 may together form an intelligent transportation platform, a government service platform, and so on, and the application scenario to which the log compression method and the log search method provided in the embodiments of the present disclosure may be applied is not limited in this disclosure, and is specific to the actual situation.

The client 12 may be a desktop computer, a tablet computer, a notebook computer, a smart phone, a digital assistant, a smart wearable device, a shopping guide terminal, a television, a smart speaker, a microphone, etc. Wherein, intelligent wearable equipment includes but is not limited to intelligent bracelet, intelligent wrist-watch, intelligent glasses, intelligent helmet, intelligent necklace etc.. Alternatively, the client 12 may be software capable of running in the electronic device. The server 11 may be an electronic device with a certain arithmetic processing capability. Which may have a network communication module, a processor, memory, and the like. Of course, the server 11 may also refer to software running in the electronic device. The server 11 may also be a distributed server, and may be a system having a plurality of processors, memories, network communication modules, etc. operating in conjunction. Alternatively, the server 11 may be a server cluster formed for several servers. Alternatively, with the development of science and technology, the server 11 may be a new technical means capable of realizing the functions corresponding to the embodiments of the specification. For example, a new form of "server" based on quantum computing implementation may be possible.

Example method

One embodiment of the present specification provides a log compression method. The log compression method may be applied to the server 11 shown in fig. 1, and as shown in fig. 2, the log compression method may include the steps of:

s201: and obtaining a plurality of logs to be compressed.

The logs to be compressed may refer to various types of logs generated by a client or a server. The logs to be compressed obtained in step S201 may be logs generated in the last time window, and these logs to be compressed may be cached first for reading in the subsequent compression, where the length of the time window may be 3 minutes, 5 minutes, 10 minutes, etc., and the size of the time window may be generally determined by the caching capability of the server, and the specific value of the time window is not limited in this specification.

S202: performing first compression on a plurality of logs to be compressed by using a first log template to obtain a plurality of first compressed logs; the first log template comprises character string identifiers corresponding to the character strings; the first log template is stored in a template library.

The template library may be a database stored in the server comprising a plurality of first log templates. The first log template may comprise a template for replacing a specific character string (which may in particular be a character string identification), a large number of repeated character strings may be included in the log to be compressed, and these repeated character strings may be replaced with a shorter first log template. For example, the logs to be compressed generated on the same day may each include the same date character (e.g. 2023-03-16), and for these date characters, a unified template (e.g., $ {1} may be used to replace the date characters to achieve the purpose of compressing the number of characters of the log and reducing the memory size occupied by the log, in this example, the string identifier$ {1} corresponds to the string 2023-03-16, so that in other logs to be compressed, the string identifier$ {1} may also be used to replace the string 2023-03-16 in the log to be compressed.

S203: generating a first log template for repeated character strings in the plurality of the first compressed logs; the repeated character string includes the same character string existing in a plurality of the first compressed logs.

After the log to be compressed is compressed for the first time by using the first log templates in the template library, repeated character strings of a plurality of first compressed logs can be extracted, and a new first log template is generated based on the extracted repeated character strings, so that the new first log template can be corresponding to the repeated character strings which are not compressed in the first compression, and a foundation is laid for the second compression by using the newly generated first log template.

For example, assume that a row of log data is included in the log to be compressed: "2023-02-03:32 [ netty-worker-pool-28]INFO host 10.10.0.1has sent 40bytes message to host 10.20.0.1"). Before proceeding to step S203, the first log template in the template library may include: "1" and "$ {2}" where "$ { 1" corresponds to the string "2023-02-03", "$ { 2" corresponds to the string "[ netty-worker-pool-28]INFO host 10.10.0.1has sent"). Then after the first log compression, the line of log data is compressed to "$ {1}20:32$ {2}40bytes message to host10.20.0.1"). However, it was found by comparison with the repeated character strings in the other first compressed logs that "bytes message to host" is also present in the plurality of first compressed logs. Thus, a new first log template may be generated: "$ {3}" corresponds to the string "bytes message to host".

In one embodiment, whether the same character string existing in a plurality of the first compressed logs can be used as a repeated character string for generating the first log template depends on the rule of generating the first log template. For example, if the character string "20:32" in one line of log data in the above example exists in N first compressed logs at the same time, a new first log template is generated based on the character string when N is greater than the set threshold. When N is less than or equal to the set threshold, then a new first log template is not generated based on the string.

Naturally, in addition to the number of the character strings and the first compressed logs being factors for determining whether to generate the first log template, the weight may be set according to the number of the characters in the character strings, the number of the character strings and the first compressed logs being factors, and whether to generate the first log template may be determined according to the weighted result of each factor. The present specification is not limited thereto, and is specific to the actual situation.

S204: and performing second compression on the first compression log based on the generated first log template to obtain a compression log.

In this embodiment, after a new first log template is generated, the first compression log is compressed for the second time according to the generated first log template, so that the compression rate of the finally obtained compression log can be improved, and the problem that the compression rate of the traditional log compression method for the new log format is low is solved.

Still with the above example, as shown in FIG. 3, when a new first log template is generated: after the corresponding character string of "$ {3}" is bytes message to host ", the data" $ {1}20:32$ {2}40bytes message to host 10.20.0.1 "in the compressed first compressed log can be compressed for the second time according to the new first log template, and finally the row of data can be stored in the following manner" $ {1}20:32$ {2}40$ {3}10.20.0.1", so that the problem that the existing first log template in the template library cannot adapt to the repeated character string in the new log format can be solved, the secondary compression of the repeated character string in the first compressed log is realized, and the problem of low compression rate is solved.

In some embodiments, the first log template generated in step S203 may be stored in a template library, so as to improve the suitability of the template library for various logs to be compressed, improve the compression rate of the logs to be compressed when performing the first compression in step S202, avoid the situation that the first log template is repeatedly generated in step S203, and improve the execution efficiency of the method.

Still for the example above, when a new first log template is generated in step S203: after "$ {3}" corresponds to the string "bytes message to host", the first log template may be stored in the template library, so that after a row of data such as "2023-02-03-20:35 [ netty-worker-pool-28]INFO host 10.10.0.1has sent 80bytes message to host 10.20.0.2" is encountered, the following compression result is obtained through the first compression in step S202: "$ {1}20:35$ {2}80$ {3}10.20.0.2", the above template is not repeatedly generated in step S203, thereby improving the execution efficiency of the method.

In order to improve applicability of the generation rule of the first log template to the log compression scenario, in one embodiment of the present disclosure, the generating the first log template for the repeated character strings in the plurality of first compressed logs includes:

word segmentation is carried out on the log content of a plurality of first compressed logs so as to divide the log content of the first compressed logs into a plurality of character strings;

extracting repeated character strings in a plurality of first compressed logs after word segmentation by using a local matching algorithm, wherein the weight of the matched character strings is positively correlated with the length of the matched character strings in a penalty rule of the local matching algorithm; the matching strings include identical strings present in at least two of the first compressed logs;

and generating a first log template according to the repeated character strings.

The local matching algorithm is one of the sequence alignment algorithms, and is different from the global alignment algorithm (such as Needleman-Wunsch algorithm (nidman-man application algorithm): the local matching algorithm is an algorithm for performing local comparison, and is used for finding out repeated character strings between two or more logs when the local matching algorithm is applied to the field of log compression. Common local matching algorithms may include: smith-Waterman Algorithm (Smith-Waterman algorithm).

In this embodiment, the conventional partial matching algorithm is optimized, so that the partial matching algorithm can be more suitable for matching the character strings in the scheme, and the longer the character string is, the higher the benefit of compressing the character string as a template, for example, the benefit of compressing "[ netty-worker-pool-28]INFO host 10.10.0.1has sent" by using the first log template is significantly higher than the benefit of compressing "has" by using the first log template. If "has" is compressed using the first log template, the compression benefits may be very small, but some computing resources may be consumed to perform the compression step.

Therefore, in the present embodiment, in the penalty rule of the partial matching algorithm, the weight of the matching string and the length of the matching string are set to be positively correlated, and in particular, in one embodiment of the present specification, the weight of the matching string may be equal to the length of the matching string. In this way, the first log template generated based on the generation method of the first log template provided in the embodiment of the present disclosure tends to be generated based on a longer character string, and tends to be discarded from a shorter character string, thereby improving log compression efficiency.

To avoid that repeated strings of a smaller number of words are generated as the first log template, in one embodiment of the present specification, the generating the first log template from the repeated strings includes:

if the number of words in the repeated character string is greater than N, dividing the repeated character string into a plurality of sub-character strings, wherein the number of words in the sub-character strings is less than or equal to N; n is an integer greater than 1;

when the number of characters in the sub-character string is larger than a preset character number threshold, establishing a corresponding relation between the sub-character string and the character string identification so as to generate a first log template.

In this embodiment, in addition to the limitation that the weight of the matching string is positively correlated with the length of the matching string introduced in the partial matching algorithm, a preset character number threshold is set when the first log template is finally generated, so that the situation that repeated strings of fewer strings are generated as the first log template is avoided, and compression benefits when the log is compressed are improved.

To meet the search requirement for compressed logs, in one embodiment of the present specification, referring to fig. 4, the log compression method further includes:

s401: responding to a query instruction, and determining log content corresponding to the query instruction in the compressed log;

S402: and returning the determined log content to a requester of the query instruction as a response result.

In the embodiment, a response mechanism for the query instruction is established, so that the requirement of searching the log content of the query instruction is met, and the requirement of searching the compressed log by a user is met.

Searches (queries) for compressed logs can be categorized into direct searches and full text searches, and in one embodiment of the present description, a possible implementation of direct searches is provided, the query instructions comprising direct query instructions comprising a line identification of log content; the line identification of the log content represents the positioning identification of the line of the log content;

if the query instruction comprises the direct query instruction, determining target row content in a compressed log corresponding to the row identifier of the log content according to the row identifier of the log content;

and according to the first log template, restoring the first log template in the target line content into a word corresponding to the first log template identifier so as to obtain log content corresponding to the query instruction.

In the direct search, it is assumed that a row of log contents pointed by a row identifier included in the direct query instruction is: "$ {1} host10.10.0.1 $ {2}40$ {3}10.20.0.1", wherein the first log template comprises:

${1}->2023-02-03 20:32[netty-worker-pool-28]INFO；

${2}->has sent；

${3}->bytes message to host。

after the above-mentioned line of log content is found according to the direct query instruction, the line of log content is restored to the corresponding word according to the above-mentioned first log template, and "2023-02-03:32 [ netty-worker-pool-28]INFO host10.10.0.1has sent 40bytes message to host 10.20.0.1" is obtained.

With respect to implementation of full text query, in one embodiment of the present specification, the query instruction includes a full text query instruction including a query string;

an inverted index is established in advance, the inverted index comprises a first index and a second index, and the first index is used for representing the corresponding relation between words in a log and a first log template mark; the second index is used for representing the corresponding relation between words in the log and the row mark of the log content; the line identification of the log content represents the positioning identification of the line of the log content;

For example, at the time of the first log template generation, the content of the first log template may be segmented and an inverted index may be created, such as with the first log template: for the example "$ {3} - > bytes message to host" ("- >" indicates correspondence), assuming that the first log template of the template is identified as $ {3}, the following first index is created:

bytes->${3}；

message->${3}；

to->${3}；

host->${3}。

it will be appreciated that the first index may be referred to as a template content index for representing the template in which the word is located.

In compressing log storage, a second index may be established, taking log content "$ {1}20:32$ {2}40$ {3}10.20.0.1" as an example, assuming that the log content of the line of log content is identified as 1 in the line, the following second index may be created:

${1}->1；

20:32->1；

${2}->1；

40->1；

${3}->1；

10.20.0.1->1。

it will be appreciated from the above examples that the second index is used to indicate which line in the log the content in the log is in.

In the full text searching process, the query character string included in the full text query instruction input by the user is assumed to be INFO and bytes and 10.20.0.1, and words in the query character string are replaced by the first log template identification based on the first index, so that a preprocessing character string is obtained. As can be seen from the first index, the words "INFO" and "bytes" are respectively present in the corresponding strings of the first log templates $ {1} and $ {3}, and then the two words in the query string are replaced by the corresponding first log templates, so as to obtain a preprocessed string: {1} and $ {3} and 10.20.0.1. Finding at least one log content row identifier corresponding to the preprocessing character string through a second index, determining target row content in the compressed log corresponding to the log content row identifier according to the log content row identifier, and assuming that the target row content is: $ {1} host 10.10.0.1$ {2}40$ {3}10.20.0.1. And finally, restoring the first log template of the target line content into a character string corresponding to the first log template, and obtaining that the log content corresponding to the query instruction is 2023-02-03:32 [ netty-worker-pool-28]INFO host 10.10.0.1has sent 40bytes message to host 10.20.0.1 ], thereby realizing full-text search.

The full text search and direct search functions for the compressed log realized based on the mode have the characteristics of simple implementation and are beneficial to improving the search response efficiency.

The inventors found through the composition of the log that, in the log, a large amount of stack output content (also referred to as trace information) for characterizing the method execution sequence information in the code is generally included, the content is generally huge, the format and the content are uniform, and if the template alternative scheme is adopted for compression, the problem that more calculation power is consumed when the part of the content is compressed may be caused.

To solve this problem, in one embodiment of the present specification, the template library further includes: a second log template comprising a stack identifier corresponding to the stack output content; the stack output content is used for representing information of the method execution sequence in the code;

before the first log compressing is performed on the logs to be compressed by using the first log template, the method further includes:

s501: determining stack output content in the log to be compressed based on a preset regular expression;

s502: if the second log template comprises a stack identifier corresponding to the stack output content determined from the log to be compressed, replacing the stack output content in the log to be compressed by using the stack identifier;

S503: if the second log template does not exist, the second log template comprises a stack identifier corresponding to the stack output content determined from the log to be compressed, a new stack identifier is corresponding to the determined stack output content, a new second log template is generated according to the determined stack output content and the stack identifier, and the stack output content in the log to be compressed is replaced by the corresponding stack identifier based on the generated second log template.

In this embodiment, the stack output content in the log to be compressed is determined by a preset regular expression, which is simple, easy and has the advantage of low consumption. In addition, in step S503, when the determined stack output content does not have the second log template corresponding thereto, a new stack identifier is corresponding to the determined stack output content and is compressed, and a new second log template is generated according to the determined stack output content and the stack identifier. Therefore, the new stack output content can be compressed corresponding to the second log template, the adaptability of the log compression method to the new log format and content is improved, and the log compression rate of the log compression method is improved.

Similar to the newly generated first log template, the generated new second log template may also be stored in a template library to provide a usable second log template for stack output of subsequent identical logs to be compressed.

In order to improve the compression efficiency, in one embodiment of the present specification, a feasible compression method is provided, optionally, the number of words included in the first log template is less than or equal to N; n is an integer greater than 1;

the method for compressing the log to be compressed comprises the following steps:

the method comprises the steps of taking N words as windows, carrying out sliding segmentation on the log to be compressed, and dividing the log content of the log to be compressed into a plurality of character strings, wherein the number of words in the character strings is smaller than or equal to N;

and utilizing the first log template to be matched with a plurality of character strings in the log to be compressed respectively, and utilizing the first log template to replace the character strings in the log to be compressed matched with the first log target.

In this embodiment, as shown in fig. 5, assume that the log content to be compressed is "2023-02-03-20:32 [ netty-worker-pool-28]INFO host 10.10.0.1has sent 40bytes message to host 10.20.0.1", and the first log template is: {1} - > 2023-02-03:32 [ netty-worker-pool-28] INFO. N takes the value of 3, namely the number of words in the first log template is less than or equal to 3.

And taking N words as windows, and performing sliding segmentation on the log to be compressed to obtain the following twelve character strings:

1:2023-02-03 20:32[netty-worker-pool-28]；

2:20:32[netty-worker-pool-28]INFO；

3:[netty-worker-pool-28]INFO host；

4:INFO host 10.10.0.1；

5:host 10.10.0.1has；

6:10.10.0.1has sent；

7:has sent 40；

8:sent 40bytes；

9:40bytes message；

10:bytes message to；

11:message to host；

12:to host 10.20.0.1。

and respectively matching the first log templates with a plurality of character strings in the log to be compressed, wherein the first log templates are found as follows: the $ {1} matches the eleventh string, and therefore, after replacing the string in the log to be compressed that matches the first log object with the first log template, the result obtained is as follows: 2023-02-03:32 [ netty-worker-pool-28]INFO host 10.10.0.1has sent 40bytes $ {1}10.20.0.1.

By compressing the log to be compressed in the mode, the matching accuracy of the first log template and the corresponding character string in the log to be compressed can be effectively improved.

In order to improve log compression efficiency, in an embodiment of the present disclosure, the matching with the plurality of character strings in the log to be compressed by using the first log template includes:

the step of matching with the plurality of character strings in the log to be compressed respectively using the first log template is performed in parallel.

Since the number of the first log templates in the template library is usually plural, and each time the first log templates are matched with the log to be compressed, the matching process can be executed in parallel as an independent execution unit. Ways in which the parallel mode is performed include, but are not limited to: multithreaded execution, multi-process execution, big data service based execution, and the like.

Based on the same concept, the embodiment of the present disclosure further provides a log searching method, as shown in fig. 6, including:

s601: responding to a query instruction, and determining log content corresponding to the query instruction in the compressed log;

s602: returning the determined log content to a requester of the query instruction as a response result;

step S601 includes:

For specific acquisition procedures of compressed logs, reference may be made to the relevant definitions in the log compression method described above. For specific possible implementation procedures and beneficial effects of each step, reference may be made to the related descriptions above, and this description is not repeated here.

Example apparatus, electronic device, storage Medium, and software

One embodiment of the present disclosure further provides a log compressing apparatus, as shown in fig. 7, which may include:

a log obtaining module 701, configured to obtain a plurality of logs to be compressed;

a first compression module 702, configured to perform first compression on the logs to be compressed by using a first log template, so as to obtain a plurality of first compressed logs; the first log template comprises character string identifiers corresponding to the character strings; the first log template is stored in a template library;

A template generation module 703, configured to generate a first log template for repeated character strings in the plurality of first compressed logs; the repeated character string includes the same character string existing in a plurality of the first compressed logs;

and a second compression module 704, configured to compress the first compression log for a second time based on the generated first log template, so as to obtain a compression log.

The management device for computing resources provided in this embodiment belongs to the same application concept as the log compression method or the log search method provided in the foregoing embodiments of the present specification, and may perform the log compression method or the log search method provided in any of the foregoing embodiments of the present specification, and has a functional module and beneficial effects corresponding to the execution of the log compression method or the log search method. Technical details not described in detail in this embodiment may refer to specific processing content of the log compression method or the log search method provided in the foregoing embodiments of the present disclosure, and are not described herein again.

Another embodiment of the present specification further provides a computing device, referring to fig. 8, and an exemplary embodiment of the present specification further provides a computing device including: a memory storing a computer program, and a processor that executes steps in the log compression method or the log search method according to various embodiments of the present specification described in the above embodiments of the present specification when executing the computer program.

The internal structure of the computing device may be as shown in fig. 8, including a processor, memory, network interface, and input devices connected by a system bus. Wherein the processor of the computing device is configured to provide computing and control capabilities. The memory of the central control device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computing device is for communicating with an external terminal through a network connection. The computer program, when executed by a processor, performs the steps in the log compression method or the log search method according to the various embodiments of the present specification described in the above embodiments of the present specification.

The processor may include a host processor, and may also include a baseband chip, modem, and the like.

The memory stores the computer program for executing the technical scheme of the invention, and can also store an operating system and other key programs. In particular, the computer program may comprise program code comprising computer operating instructions. More specifically, the memory may include read-only memory (ROM), other types of static storage devices that may store static information and instructions, random access memory (random access memory, RAM), other types of dynamic storage devices that may store information and instructions, disk storage, flash, and the like.

The processor may be a general-purpose processor, such as a general-purpose processor (CPU), microprocessor, or the like, or may be an application-specific integrated circuit (ASIC), or one or more integrated circuits, that control the execution of programs in accordance with aspects of the present invention. But may also be a Digital Signal Processor (DSP), application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.

The input device may include means for receiving data and information entered by a user, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer or gravity sensor, etc.

The output device may include means, such as a display screen, printer, speakers, etc., that allow information to be output to the user.

The communication interface may include means, such as any transceiver, for communicating with other devices or communication networks, such as ethernet, radio Access Network (RAN), wireless Local Area Network (WLAN), etc.

The processor executes the computer program stored in the memory and invokes other devices, which may be used to implement any of the log compression method or the log search method steps provided in the above embodiments of the present application.

The computing device can also comprise a display component and a voice component, wherein the display component can be a liquid crystal display screen or an electronic ink display screen, and an input device of the computing device can be a touch layer covered on the display component, can also be a key, a track ball or a touch pad arranged on a shell of the computing device, and can also be an external keyboard, a touch pad or a mouse and the like.

Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of some of the architecture associated with the present description and is not limiting of the computing devices to which the present description may be applied, and that a particular computing device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In addition to the methods and apparatus described above, the log compression method or log search method provided by the embodiments of the present description may also be a computer program product comprising a computer program which, when executed by a processor, causes the processor to perform the steps in the log compression method or log search method according to the various embodiments of the present description described in the "exemplary method" section of the present description.

The computer program product may write program code for performing the operations of embodiments of the present description in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server.

Further, the present specification embodiment also provides a computer-readable storage medium having stored thereon a computer program that is executed by a processor to perform the steps in the log compression method or the log search method according to the various embodiments of the present specification described in the above-described "exemplary method" section of the present specification.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in this specification are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to comply with the related laws and regulations and standards of the related country and region, and are provided with corresponding operation entries for the user to select authorization or rejection.

It will be appreciated that the specific examples herein are intended only to assist those skilled in the art in better understanding the embodiments of the present description and are not intended to limit the scope of the present description.

It should be understood that, in various embodiments of the present disclosure, the sequence number of each process does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

It will be appreciated that the various embodiments described in this specification may be implemented either alone or in combination, and are not limited in this regard.

Unless defined otherwise, all technical and scientific terms used in the embodiments of this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this specification belongs. The terminology used in the description is for the purpose of describing particular embodiments only and is not intended to limit the scope of the description. The term "and/or" as used in this specification includes any and all combinations of one or more of the associated listed items. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It will be appreciated that the processor of the embodiments of the present description may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method embodiments may be implemented by integrated logic circuits of hardware in a processor or instructions in software form. The processor may be a general purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), an off-the-shelf programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The methods, steps and logic blocks disclosed in the embodiments of the present specification may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present specification may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.

It will be appreciated that the memory in the embodiments of this specification may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), or a flash memory, among others. The volatile memory may be Random Access Memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present specification.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described system, apparatus and unit may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.

In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, each unit may exist alone physically, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present specification may be essentially or portions contributing to the prior art or portions of the technical solutions may be embodied in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present specification. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a read-only memory (ROM), a random-access memory (RAM), a magnetic disk, or an optical disk, etc.

The foregoing is merely specific embodiments of the present disclosure, but the scope of the disclosure is not limited thereto, and any person skilled in the art who is skilled in the art can easily think about variations or substitutions within the scope of the disclosure of the present disclosure, and it is intended to cover the variations or substitutions within the scope of the disclosure. Therefore, the protection scope of the present specification shall be subject to the protection scope of the claims.

Claims

1. A log compression method, comprising:

acquiring a plurality of logs to be compressed;

2. The method of claim 1, wherein the generating a first log template for repeated strings in the plurality of the first compressed logs comprises:

3. The method of claim 2, wherein generating a first log template from the repeated string comprises:

4. The method as recited in claim 1, further comprising:

and returning the determined log content to a requester of the query instruction as a response result.

5. The method of claim 4, wherein the query instruction comprises a direct query instruction comprising an identification of the line on which the log content is located; the line identification of the log content represents the positioning identification of the line of the log content;

The determining, in the compressed log, log content corresponding to a query instruction in response to the query instruction includes:

6. The method of claim 4, wherein the query instruction comprises a full text query instruction comprising a query string;

7. The method of claim 1, wherein the template library further comprises: a second log template comprising a stack identifier corresponding to the stack output content; the stack output content is used for representing information of the method execution sequence in the code;

determining stack output content in the log to be compressed based on a preset regular expression;

If the second log template comprises a stack identifier corresponding to the stack output content determined from the log to be compressed, replacing the stack output content in the log to be compressed by using the stack identifier;

if the second log template does not exist, the second log template comprises a stack identifier corresponding to the stack output content determined from the log to be compressed, a new stack identifier is corresponding to the determined stack output content, a new second log template is generated according to the determined stack output content and the stack identifier, and the stack output content in the log to be compressed is replaced by the corresponding stack identifier based on the generated second log template.

8. The method of claim 1, wherein the first log template comprises a number of words less than or equal to N; n is an integer greater than 1;

9. The method of claim 8, wherein the utilizing the first log template to match the plurality of strings in the log to be compressed, respectively, comprises:

10. The method according to claim 1, wherein the method further comprises:

and adding the generated first log template into the template library.

11. A log search method, comprising:

12. An electronic device, comprising: a processor and a memory;

the processor is configured to implement the log compression method according to any one of claims 1 to 10 or the log search method according to claim 11 by running a computer program stored in the memory.

13. A storage medium having stored thereon a computer program which, when executed by a processor, implements the log compression method of any one of claims 1 to 10 or the log search method of claim 11.