CN116303315A - Log data management method, device, equipment and storage medium - Google Patents

Log data management method, device, equipment and storage medium Download PDF

Info

Publication number
CN116303315A
CN116303315A CN202310064209.3A CN202310064209A CN116303315A CN 116303315 A CN116303315 A CN 116303315A CN 202310064209 A CN202310064209 A CN 202310064209A CN 116303315 A CN116303315 A CN 116303315A
Authority
CN
China
Prior art keywords
log
logs
combined
merging
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310064209.3A
Other languages
Chinese (zh)
Inventor
王景熠
叶章龙
吴流丽
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Anheng Information Security Technology Co Ltd
Original Assignee
Hangzhou Anheng Information Security Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Anheng Information Security Technology Co Ltd filed Critical Hangzhou Anheng Information Security Technology Co Ltd
Priority to CN202310064209.3A priority Critical patent/CN116303315A/en
Publication of CN116303315A publication Critical patent/CN116303315A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The application discloses a log data management method, a device, equipment and a storage medium, and relates to the technical field of computer data management, wherein the method comprises the following steps: extracting the basic data structure of each log to be combined based on the log function type to obtain the basic data structure corresponding to each log to be combined; determining importance values of the logs to be combined based on basic data structures corresponding to the logs to be combined; screening a plurality of entry mark logs meeting a preset importance condition from each log to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in each log to be combined; and carrying out merging operation on each target log based on a preset log merging rule so as to obtain a corresponding log after merging. According to the method and the device, the logs with the same importance degree are combined by using the log combining rule, so that the storage space occupied by the logs can be effectively reduced.

Description

Log data management method, device, equipment and storage medium
Technical Field
The present invention relates to the field of computer data management technologies, and in particular, to a method, an apparatus, a device, and a storage medium for log data management.
Background
Modern large-scale business systems often have high complexity, and for the safety of system resource management and the efficiency of troubleshooting after problems occur, the business systems can record and store a large amount of logs generated in the running process. After half a year or more of operation, these log files typically occupy a significant amount of storage space, resulting in increased costs and increased operational difficulty.
The first is the storage space occupied by the log information. Although each log may be as small as a few KB, the space taken up by the log may reach hundreds of G or even TB levels when the log accumulates on a daily basis, in which case it is difficult to query or migrate. Secondly, the importance of the log is a problem, so that most operation and maintenance personnel of the system can automatically clean the log within half a year, and after one log is stored, the log may suddenly need to be used after a long time, for example, a virus with a latency of up to half a year, when the virus breaks, often the resource access log in the system is deleted half a year ago, and thus the propagation path of the virus is difficult to trace. But from a cost perspective, the common user access log does not have to be stored for more than half a year, and thus is also one of the risks in terms of system security at present. Third is the efficiency of log queries. The large data related component is used for greatly improving the speed of inquiring a large number of logs, but the scheme is to exchange space for time, store the logs in a hard disk in a whole amount, and the occupied space is only increased. How to solve these problems becomes a difficulty.
Disclosure of Invention
In view of the above, the present invention aims to provide a log data management method, device, equipment and storage medium, which can effectively control the storage space occupied by the log. The specific scheme is as follows:
a first aspect of the present application provides a log data management method, including:
extracting the basic data structure of each log to be combined based on the log function type to obtain the basic data structure corresponding to each log to be combined;
determining importance values of the logs to be combined based on basic data structures corresponding to the logs to be combined;
screening a plurality of entry mark logs meeting a preset importance condition from each log to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in each log to be combined;
and carrying out merging operation on each target log based on a preset log merging rule so as to obtain a corresponding log after merging.
Optionally, after performing the merging operation on each target log based on the preset log merging rule to obtain a corresponding merged log, the method further includes:
judging whether the logs to be combined which are not subjected to the combination operation exist currently or not;
if so, the step of selecting a plurality of item mark logs meeting the preset importance condition from the logs to be combined is re-skipped until the logs to be combined which are not subjected to the combining operation exist currently.
Optionally, after performing the merging operation on each target log based on the preset log merging rule to obtain a corresponding merged log, the method further includes:
and determining the importance value of the combined log, and compressing the combined log with the importance value smaller than a preset degree threshold value to obtain a compressed log.
Optionally, the method further comprises:
and acquiring a user instruction for representing the start of log data merging through a preset configuration interface, so as to trigger the step of extracting the basic data structure of each log to be merged based on the log function type based on the user instruction.
Optionally, the merging operation is performed on each target log based on a preset log merging rule to obtain a corresponding merged log, which includes:
and screening logs with the same log function type and basic data structure from each target log, and carrying out merging operation on the screened logs to obtain corresponding merged logs.
Optionally, the merging operation is performed on each target log based on a preset log merging rule to obtain a corresponding merged log, which includes:
screening logs with the same log function type from the target logs to obtain first screened target logs;
screening logs with the difference degree of the basic data structure in a preset difference degree range from the first screened target logs to obtain second screened target logs;
and merging the second screened target logs into one log to obtain the merged log.
Optionally, the screening logs with the same log function type from the target logs to obtain the first screened target log further includes:
judging whether the number of the logs corresponding to the first screened target logs is one, if so, directly marking the first screened target logs as a combined state.
A second aspect of the present application provides a log data management apparatus, comprising:
the basic data structure acquisition module is used for extracting the basic data structure of each log to be combined based on the log function type so as to obtain the basic data structure corresponding to each log to be combined;
the importance value determining module is used for determining importance values of the logs to be combined based on basic data structures corresponding to the logs to be combined;
the log to be combined selecting module is used for screening a plurality of item mark logs meeting a preset importance condition from each log to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in each log to be combined;
and the log merging module is used for merging the target logs based on a preset log merging rule so as to obtain corresponding merged logs.
A third aspect of the present application provides an electronic device comprising a processor and a memory; the memory is used for storing a computer program, and the computer program is loaded and executed by the processor to realize the log data management method.
A fourth aspect of the present application provides a computer readable storage medium, which when executed by a processor, implements the aforementioned log data management method.
In the application, extracting basic data structures of logs to be combined based on log function types to obtain basic data structures corresponding to the logs to be combined; determining importance values of the logs to be combined based on basic data structures corresponding to the logs to be combined; screening a plurality of entry mark logs meeting a preset importance condition from each log to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in each log to be combined; and carrying out merging operation on each target log based on a preset log merging rule so as to obtain a corresponding log after merging. . Therefore, the method and the device for merging the logs to be merged in the log merging mode divide and extract basic data structures of the logs to be merged, then determine importance values of the logs to be merged based on the basic data structures, select the logs with the same importance values and lower than importance values of other logs in the logs to be merged, merge the logs to be merged according to preset log merging rules, and greatly save storage space of the logs.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a log data management method disclosed in the present application;
FIG. 2 is a flowchart of a specific log data management method disclosed in the present application;
FIG. 3 is a flowchart of a specific log data management method disclosed in the present application;
fig. 4 is a schematic structural diagram of a log data management device disclosed in the present application;
fig. 5 is a block diagram of an electronic device disclosed in the present application.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In order to solve the problems of large occupied space of log storage and low query efficiency, the embodiment discloses a method for merging logs to be merged by extracting a basic data structure of the logs to be merged.
Referring to fig. 1, the embodiment of the invention discloses a log data management method, which comprises the following steps:
step S11: and extracting the basic data structure of each log to be combined based on the log function type to obtain the basic data structure corresponding to each log to be combined.
In this embodiment, the log base data structures of the different function types are different, but the service logs are typically in a fixed format. The function types of the daily log include, but are not limited to, a management type log, a security type log, a log of a data query function, and a log of a data modification type. The basic data structure of each log to be merged may be divided into the following parts according to the format of the log, including but not limited to: subjects, predicates, objects, time-words (at least to the order of seconds), data information, outcome data and success/failure, importance, etc. Wherein, the subject refers to an operator for generating logs, not only a user (person), but also an initiator of all logs, such as a system scanning file log, and the subject is a system scanning service; an operation log, such as user A logs into the system, and the subject is user A. Predicates refer to the main actions of the log, such as scanning, logging in, viewing, deleting, etc. An object refers to an object of a log operation, such as a "file" in which the system scans files, and a user views an "article" of an article. The time-wise term (at least to the order of seconds) refers to the time at which the log is generated, and is required to be recorded to the order of seconds or milliseconds, even microseconds, in normal time without setting a lower limit. Data information refers to specific content of an object, such as specific article content in a log of "user view articles". The result data is a result of data returned by the system API, which is often recorded with high reproducibility to the data information. Success/failure refers to success or failure of an operation, such as when the system fails, the user's view of the article results in a failure. The importance refers to the importance of the log and can be classified into different levels. Such as management logs, security logs, modification logs for data, etc., are of higher importance, while the types of logs for which the user views personal information, views certain business information, etc., are of lower importance. Therefore, the basic structure information of the data is extracted based on the function types of the logs to be combined, so that the format of the log is clear, the method has extremely high readability, and the working efficiency of inquiring the log later can be greatly improved by dividing and extracting the basic structure of the data.
Step S12: and determining importance values of the logs to be combined based on the basic data structures corresponding to the logs to be combined.
In this embodiment, the basic data structure corresponding to each log to be merged includes importance of each log to be merged, where the importance refers to importance of the log and may be classified into different levels. The importance degree of each log to be combined can be determined through the importance degree acquisition.
Step S13: and screening a plurality of entry mark logs meeting a preset importance condition from the logs to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in the logs to be combined.
In this embodiment, after obtaining the importance value of each log to be merged, the log merging operation may be started. And selecting a plurality of item mark logs from the logs to be combined. The importance values of the entries marked with the logs are equal and smaller than the importance values of other logs in the logs to be combined. The logs to be combined with low importance degree are selected and determined by the arrangement habit of a common user, and the logs with high importance degree can be selected for combination preferentially.
Step S14: and carrying out merging operation on each target log based on a preset log merging rule so as to obtain a corresponding log after merging.
In this embodiment, after each target log is selected, the logs may be merged based on a preset log merging rule. The merging operation may be to merge the target logs into one log and then keep the original log deleted, or to merge and compress the target logs to save the log storage space. Therefore, the storage space occupied by the log can be flexibly reduced in practical situations.
In this embodiment, the merging operation is performed on each target log based on a preset log merging rule, so as to obtain a corresponding merged log, and then the method further includes: judging whether the logs to be combined which are not subjected to the combination operation exist currently or not; if so, the step of selecting a plurality of item mark logs meeting the preset importance condition from the logs to be combined is re-skipped until the logs to be combined which are not subjected to the combining operation exist currently. After the log merging operation is finished, checking the remaining stored logs to check whether the logs to be merged which are not subjected to the merging operation exist, if yes, repeating the operation of the step S13, so that the logs with low importance can be merged preferentially, and then the logs are lifted in sequence, the effect of merging all the logs to be merged is achieved, and the storage space of the logs is saved.
In this embodiment, the basic data structure of each log to be combined is extracted based on the log function type to obtain the basic data structure corresponding to each log to be combined; determining importance values of the logs to be combined based on basic data structures corresponding to the logs to be combined; screening a plurality of entry mark logs meeting a preset importance condition from each log to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in each log to be combined; and carrying out merging operation on each target log based on a preset log merging rule so as to obtain a corresponding log after merging. Therefore, the method and the device for merging the logs to be merged in the log merging mode divide and extract basic data structures of the logs to be merged, then determine importance values of the logs to be merged based on the basic data structures, select the logs with the same importance values and lower than importance values of other logs in the logs to be merged, merge the logs to be merged according to preset log merging rules, and greatly save storage space of the logs.
The above embodiment describes a method for extracting a log basic data structure, and in this embodiment, a method for merging logs is specifically described.
Referring to fig. 2, the embodiment of the invention discloses a log data management method, which comprises the following steps:
step S21: and acquiring a user instruction for representing the start of log data merging through a preset configuration interface, so as to trigger the step of extracting the basic data structure of each log to be merged based on the log function type based on the user instruction.
In this embodiment, because the requirements of different users on log management are different, the log data amounts generated by different users are different, so that the starting time of the merging operation of the log information is determined by the users according to their own actual conditions, and when a user instruction indicating that the log data merging can be started is obtained through a preset configuration interface, the step of extracting the basic data structure of each log to be merged based on the log function type is triggered to start the log merging operation. Therefore, log merging work can be carried out according to the actual situation of the user, and user experience is improved.
Step S22: extracting the basic data structure of each log to be combined based on the log function type to obtain the basic data structure corresponding to each log to be combined.
Step S23: and determining importance values of the logs to be combined based on the basic data structures corresponding to the logs to be combined.
Step S24: and screening a plurality of entry mark logs meeting a preset importance condition from the logs to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in the logs to be combined.
Step S25: and screening logs with the same log function type and basic data structure from each target log, and carrying out merging operation on the screened logs to obtain corresponding merged logs.
In this embodiment, after screening out the target logs, it is necessary to screen out logs with the same basic data structure in the log function type meter from each target log, and then perform a merging operation on the logs. Therefore, the combined logs based on the same functional type and the basic data structure have definite format, extremely high readability and structured data, and the query efficiency of the logs can be greatly improved. The operation logs aiming at the same object and the same target can be repeatedly combined based on subjects and predicates, objects and data information, for example, two logs: the user A views the article A and the time point a; user a views article a, time point a. The user views, the article a, and the time point a are the same (the time point a may have a certain increment or decrement), and other data are not considered, such as success or failure, result data, and the like, and may be directly combined. The two logs can be combined into one log, and then the original log is deleted.
In this embodiment, after performing a merging operation on each target log based on a preset log merging rule to obtain a corresponding merged log, the method further includes: and determining the importance value of the combined log, and compressing the combined log with the importance value smaller than a preset degree threshold value to obtain a compressed log. The combined log with the importance degree lower than the preset importance degree threshold value can be further compressed, so that the storage space of the log can be further saved. It can be appreciated that the preset importance threshold value can be set by the user according to the actual situation of the user.
It can be seen that, in this embodiment, a user instruction for starting log data merging is obtained through a preset configuration interface, so that the step of extracting the basic data structure of each log to be merged based on the log function type is triggered based on the user instruction; extracting the basic data structure of each log to be combined based on the log function type to obtain a basic data structure corresponding to each log to be combined; determining importance values of the logs to be combined based on basic data structures corresponding to the logs to be combined; screening a plurality of entry mark logs meeting a preset importance condition from each log to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in each log to be combined; and screening logs with the same log function type and basic data structure from each target log, and carrying out merging operation on the screened logs to obtain corresponding merged logs. In this way, the log information with the same function and the same data structure is combined, so that the format of the log is clear, the log information has extremely high readability, and the log information is combined according to the basic data structure, so that the query efficiency of the log after the combination can be greatly improved.
The above embodiment specifically describes a method for merging logs, and this embodiment specifically describes another process for merging logs to be merged based on a basic data structure of the logs to be merged.
Referring to fig. 3, the embodiment of the invention discloses a log data management method, which comprises the following steps:
step S31: and extracting the basic data structure of each log to be combined based on the log function type to obtain the basic data structure corresponding to each log to be combined.
Step S32: and determining importance values of the logs to be combined based on the basic data structures corresponding to the logs to be combined.
Step S33: and screening a plurality of entry mark logs meeting a preset importance condition from the logs to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in the logs to be combined.
Step S34: and screening logs with the same log function type from the target logs to obtain a first screened target log.
In this embodiment, after determining the importance value of each log to be combined, the logs with the same function type are screened from each target log, so as to obtain a first screened target log.
In this embodiment, the screening the logs with the same log function type from the target logs to obtain the first screened target log further includes: judging whether the number of the logs corresponding to the first screened target logs is one, if so, directly marking the first screened target logs as a combined state. If the number of the first screened target logs is one, the logs cannot be combined with other logs, and the first screened target logs are directly marked as combined, so that the logs can be prevented from being selected when the subsequent log combining operation is performed, and errors in log combining work are avoided.
Step S35: and screening the logs with the difference degree of the basic data structure in a preset difference degree range from the first screened target logs to obtain second screened target logs.
In this embodiment, after logs with the same function type of each log to be combined are screened out, because each basic data structure of each log to be combined can be used as a part of the combining policy, after the first screened target log is screened out, the establishment of the combining policy is performed according to the requirement of the user, and the log with the difference degree in the preset difference degree range is selected to obtain the second screened log.
Step S36: and merging the second screened target logs into one log to obtain merged logs.
In this embodiment, the second filtered target log is merged into a log according to a preset merging rule, so as to obtain a merged log. Therefore, the original logs to be combined can be deleted, and the storage space of the logs is greatly saved.
In the present embodiment, step S35 and step S36 are exemplified. In business system processing, time is often required to be extremely accurate, so the generation of the original log will typically record very accurate time, e.g., to the order of seconds, but in post-log processing, such an accuracy is often not required. After screening out log information of record time based on the function types of the logs to be combined, selecting a system to scan a file A and a time point a (time of year, month, day, time and second) based on a principle that a subject combines predicates and the difference degree of time-like words is in a preset difference degree range; the system scans the file B, and the two logs at the time point B (time of year, month, day, time of second) are combined. In these two logs, the system, the scanning and the file are the key points, namely a time point a and a time point b, and only one day of the day may need to be recorded after one or two years, so that the two logs can be combined into the system to scan the file, namely a time point C. Wherein, the time point C may be only the year, month and day information of the time point a and the time point b. Thus, a log of the time point C is generated, and the original two logs can be deleted.
For the specific processes of step S31, step S32 and step S33, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and the details are not repeated here.
In this embodiment, the basic data structure of each log to be combined is extracted based on the log function type to obtain the basic data structure corresponding to each log to be combined; determining importance values of the logs to be combined based on basic data structures corresponding to the logs to be combined; screening a plurality of entry mark logs meeting a preset importance condition from each log to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in each log to be combined; screening logs with the same log function type from the target logs to obtain first screened target logs; and merging the second screened target logs into one log to obtain merged logs. Therefore, the logs are screened according to different function types of the logs, and different merging strategies are formulated according to the different function types of the logs, so that each basic data structure of the logs can be used as a merging basis. Therefore, the storage space of the log can be greatly saved, the merged log has clear format and extremely strong readability, and when the log is queried later, the targeted query can be performed according to different parts of the log, so that the query efficiency can be greatly improved.
Referring to fig. 4, an embodiment of the present invention discloses a log data management device, including:
the basic data structure acquisition module 11 is configured to extract a basic data structure of each log to be combined based on a log function type, so as to obtain a basic data structure corresponding to each log to be combined;
the importance value determining module 12 is configured to determine an importance value of each log to be merged based on a basic data structure corresponding to each log to be merged;
the log to be combined selecting module 13 is configured to screen a plurality of entry target logs that meet a preset importance condition from each log to be combined, where the preset importance condition is that importance values of each target log are equal and smaller than importance values of other logs in each log to be combined;
the log merging module 14 is configured to perform a merging operation on each of the target logs based on a preset log merging rule, so as to obtain a corresponding merged log.
In this embodiment, the basic data structure of each log to be combined is extracted based on the log function type to obtain the basic data structure corresponding to each log to be combined; determining importance values of the logs to be combined based on basic data structures corresponding to the logs to be combined; screening a plurality of entry mark logs meeting a preset importance condition from each log to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in each log to be combined; and carrying out merging operation on each target log based on a preset log merging rule so as to obtain a corresponding log after merging. Therefore, the method and the device for merging the logs to be merged in the log merging mode divide and extract basic data structures of the logs to be merged, then determine importance values of the logs to be merged based on the basic data structures, select the logs with the same importance values and lower than importance values of other logs in the logs to be merged, merge the logs to be merged according to preset log merging rules, and greatly save storage space of the logs.
In a specific embodiment, the log data management device specifically further includes:
the log to be combined judging module is used for judging whether the log to be combined which is not subjected to the combining operation exists currently;
and the step jump module is used for re-jumping to the step of selecting a plurality of item mark logs meeting the preset importance condition from the logs to be combined if the log to be combined exists, until the logs to be combined which are not subjected to the combining operation exist currently.
In a specific embodiment, the log data management device specifically further includes:
the log compression module is used for determining the importance value of the combined log, compressing the combined log with the importance value smaller than a preset degree threshold value, and obtaining the compressed log.
In a specific embodiment, the log data management device specifically further includes:
the user instruction acquisition module is used for acquiring a user instruction representing the beginning of log data merging through a preset configuration interface so as to trigger the step of extracting the basic data structure of each log to be merged based on the log function type based on the user instruction.
In a specific embodiment, the log merging module 14 specifically includes:
the first log merging unit is used for screening logs with the same log function type and basic data structure from each target log, and merging the screened logs to obtain corresponding merged logs.
In a specific embodiment, the log merging module 14 specifically includes:
the first screened target log obtaining unit is used for screening logs with the same log function type from the target logs to obtain first screened target logs;
the second screened target log obtaining unit is used for screening the logs with the difference degree of the basic data structure in the preset difference degree range from the first screened target log to obtain a second screened target log;
and the combined log obtaining unit is used for combining the second filtered target log into one log so as to obtain the combined log.
In a specific embodiment, the log data management device specifically further includes:
and the log marking unit is used for judging whether the number of the logs corresponding to the first screened target log is one, and if so, directly marking the first screened target log as a combined state.
Further, the embodiment of the present application further discloses an electronic device, and fig. 5 is a block diagram of the electronic device 20 according to an exemplary embodiment, where the content of the figure is not to be considered as any limitation on the scope of use of the present application.
Fig. 5 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. Wherein the memory 22 is configured to store a computer program that is loaded and executed by the processor 21 to implement the relevant steps in the log data management method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the present embodiment may be specifically an electronic computer.
In this embodiment, the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.
The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon may include an operating system 221, a computer program 222, and the like, and the storage may be temporary storage or permanent storage.
The operating system 221 is used for managing and controlling various hardware devices on the electronic device 20 and computer programs 222, which may be Windows Server, netware, unix, linux, etc. The computer program 222 may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the log data management method performed by the electronic device 20 disclosed in any of the foregoing embodiments.
Further, the application also discloses a computer readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the previously disclosed log data management method. For specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing has outlined the detailed description of the preferred embodiment of the present application, and the detailed description of the principles and embodiments of the present application has been provided herein by way of example only to facilitate the understanding of the method and core concepts of the present application; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.

Claims (10)

1. A log data management method, comprising:
extracting the basic data structure of each log to be combined based on the log function type to obtain the basic data structure corresponding to each log to be combined;
determining importance values of the logs to be combined based on basic data structures corresponding to the logs to be combined;
screening a plurality of entry mark logs meeting a preset importance condition from each log to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in each log to be combined;
and carrying out merging operation on each target log based on a preset log merging rule so as to obtain a corresponding log after merging.
2. The method of claim 1, wherein after performing a merging operation on each of the target logs based on a preset log merging rule to obtain a corresponding merged log, further comprising:
judging whether the logs to be combined which are not subjected to the combination operation exist currently or not;
if so, the step of selecting a plurality of item mark logs meeting the preset importance condition from the logs to be combined is re-skipped until the logs to be combined which are not subjected to the combining operation exist currently.
3. The method of claim 2, wherein after performing a merging operation on each of the target logs based on a preset log merging rule to obtain a corresponding merged log, further comprising:
and determining the importance value of the combined log, and compressing the combined log with the importance value smaller than a preset degree threshold value to obtain a compressed log.
4. The log data management method of claim 1, further comprising:
and acquiring a user instruction for representing the start of log data merging through a preset configuration interface, so as to trigger the step of extracting the basic data structure of each log to be merged based on the log function type based on the user instruction.
5. The log data management method according to any one of claims 1 to 4, wherein the merging operation of each target log based on a preset log merging rule to obtain a corresponding merged log includes:
and screening logs with the same log function type and basic data structure from each target log, and carrying out merging operation on the screened logs to obtain corresponding merged logs.
6. The log data management method according to any one of claims 1 to 4, wherein the merging operation of each target log based on a preset log merging rule to obtain a corresponding merged log includes:
screening logs with the same log function type from the target logs to obtain first screened target logs;
screening logs with the difference degree of the basic data structure in a preset difference degree range from the first screened target logs to obtain second screened target logs;
and merging the second screened target logs into one log to obtain the merged log.
7. The method for log data management according to claim 6, wherein after the log having the same log function type is selected from the target logs to obtain the first screened target log, further comprising:
judging whether the number of the logs corresponding to the first screened target logs is one, if so, directly marking the first screened target logs as a combined state.
8. A log data management apparatus, comprising:
the basic data structure acquisition module is used for extracting the basic data structure of each log to be combined based on the log function type so as to obtain the basic data structure corresponding to each log to be combined;
the importance value determining module is used for determining importance values of the logs to be combined based on basic data structures corresponding to the logs to be combined;
the log to be combined selecting module is used for screening a plurality of item mark logs meeting a preset importance condition from each log to be combined, wherein the preset importance condition is that the importance value of each target log is equal to or smaller than the importance value of other logs in each log to be combined;
and the log merging module is used for merging the target logs based on a preset log merging rule so as to obtain corresponding merged logs.
9. An electronic device comprising a processor and a memory; wherein the memory is for storing a computer program that is loaded and executed by the processor to implement the log data management method of any of claims 1 to 7.
10. A computer readable storage medium for storing a computer program, wherein the computer program when executed by a processor implements the log data management method of any of claims 1 to 7.
CN202310064209.3A 2023-01-16 2023-01-16 Log data management method, device, equipment and storage medium Pending CN116303315A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310064209.3A CN116303315A (en) 2023-01-16 2023-01-16 Log data management method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310064209.3A CN116303315A (en) 2023-01-16 2023-01-16 Log data management method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116303315A true CN116303315A (en) 2023-06-23

Family

ID=86821270

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310064209.3A Pending CN116303315A (en) 2023-01-16 2023-01-16 Log data management method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116303315A (en)

Similar Documents

Publication Publication Date Title
CN107943951B (en) Method and system for retrieving block chain service information
CN110569214B (en) Index construction method and device for log file and electronic equipment
KR101999409B1 (en) Formatting data by example
US10248674B2 (en) Method and apparatus for data quality management and control
CN103379099A (en) Hostile attack identification method and system
CN111881133A (en) Storage bucket management method and device, computer equipment and readable storage medium
CN111026728A (en) Log data processing method and related device
CN111061681A (en) Method and device for partitioning directory based on case insensitivity and storage medium
CN110941632A (en) Database auditing method, device and equipment
CN110191097B (en) Method, system, equipment and storage medium for detecting security of login page
CN114757639A (en) Data processing method, device, equipment and storage medium
CN112559538A (en) Incidence relation generation method and device, computer equipment and storage medium
US5826104A (en) Batch program status via tape data set information for dynamically determining the real time status of a batch program running in a main frame computer system
CN113901441A (en) User abnormal request detection method, device, equipment and storage medium
CN112084199A (en) Scene-based general parameter maintenance method and device
CN116303315A (en) Log data management method, device, equipment and storage medium
CN112019546A (en) Protection strategy adjusting method, system, equipment and computer storage medium
CN106326400A (en) Multi-dimension data set-based data processing system
CN114531306B (en) Real-time detection method and system based on threat behaviors
CN111092879A (en) Log association method and device, electronic equipment and storage medium
CN116074095A (en) Log analysis method, device, equipment and storage medium
CN111159142A (en) Data processing method and device
CN114297657A (en) File behavior detection baseline determination and file behavior anomaly detection method and device
CN110213055B (en) Information updating method and device, computer equipment and computer readable storage medium
CN111026748B (en) Data compression method, device and system for network access frequency management and control

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination