CN115967615A - Equipment log processing method, device and system and storage medium - Google Patents

Equipment log processing method, device and system and storage medium Download PDF

Info

Publication number
CN115967615A
CN115967615A CN202211630391.6A CN202211630391A CN115967615A CN 115967615 A CN115967615 A CN 115967615A CN 202211630391 A CN202211630391 A CN 202211630391A CN 115967615 A CN115967615 A CN 115967615A
Authority
CN
China
Prior art keywords
log
original
hash value
storage
processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211630391.6A
Other languages
Chinese (zh)
Inventor
田静伟
于星杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Topsec Technology Co Ltd
Beijing Topsec Network Security Technology Co Ltd
Beijing Topsec Software Co Ltd
Original Assignee
Beijing Topsec Technology Co Ltd
Beijing Topsec Network Security Technology Co Ltd
Beijing Topsec Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Topsec Technology Co Ltd, Beijing Topsec Network Security Technology Co Ltd, Beijing Topsec Software Co Ltd filed Critical Beijing Topsec Technology Co Ltd
Priority to CN202211630391.6A priority Critical patent/CN115967615A/en
Publication of CN115967615A publication Critical patent/CN115967615A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Debugging And Monitoring (AREA)

Abstract

The embodiment of the invention provides a method, a device, a system and a storage medium for processing equipment logs, wherein the method comprises the following steps: obtaining an original log sent by a log source; processing the original log, and determining a hash value of the original log; and determining whether the hash value is stored, if so, at least refusing to forward the original log to a log storage service for storage, and if not, at least forwarding the original log and the corresponding hash value to the log storage service for storage. The device log processing method can effectively reduce network transmission flow, save log storage space and improve log storage efficiency.

Description

Equipment log processing method, device and system and storage medium
Technical Field
The embodiment of the invention relates to the technical field of network security and data storage, in particular to a method, a device and a system for processing equipment logs and a storage medium.
Background
As more and more devices are accessed in the network system, the types and the number of device logs to be processed by the management system are more and more. The current management network has the characteristic of wide equipment distribution, a large amount of data needs to be transmitted in the internet or a private network, the internet has the characteristic of unstable transmission quality, and the private transmission is stable but the cost is high. The management system needs to store the received logs locally for subsequent log analysis, statistics and audit, and a large amount of redundant logs cause waste of storage space in reality.
Therefore, how to realize that the management system manages the equipment based on the statistics and analysis of the logs by stably transmitting less log data and occupies less space when storing the log data becomes a problem to be considered.
In order to solve the above problems, the existing solution includes that the weblog storage method based on multi-attribute hash deduplication in the intrusion detection system includes log deduplication and log storage, and the intrusion detection system can deduplicate and store the weblog on a local server, and the intrusion detection system includes a data acquisition server and a plurality of data storage and analysis servers, and the data acquisition server, the data storage and analysis servers are all connected with the switch.
However, this solution has drawbacks including:
1. the problem of local storage is only solved, the problem of log transmission is not considered, and a specific processing scheme for effectively reducing network traffic in a log transmission scene is not provided.
2. The hash processing of the log is centralized in the log service, so the requirement on the performance of the equipment system is high.
3. The scheme based on multi-attribute hash deduplication is not suitable for the situation that a plurality of log types exist in the system and a large number of logs exist. Therefore, in this scenario, the processing efficiency of the system on the log is greatly reduced.
Disclosure of Invention
The invention provides a device log processing method, a device, a system and a storage medium, which can effectively reduce network transmission flow, save log storage space and improve log storage efficiency.
In order to solve the above technical problem, an embodiment of the present invention provides an apparatus log processing method, including:
obtaining an original log sent by a log source;
processing the original log, and determining a hash value of the original log;
and determining whether the hash value is stored, if so, at least refusing to forward the original log to a log storage service for storage, and if not, at least forwarding the original log and the corresponding hash value to the log storage service for storage.
As an optional embodiment, the processing the original log and determining the hash value of the original log includes:
and calculating and determining the hash value of the original log based on the log contents except the log time in the original log.
As an optional embodiment, the method further comprises:
for the original log which is sent for the first time, determining a hash value and log time of the original log;
locally storing and recording at least the hash value;
and forwarding the hash value of the original log, the log time and the original log to the log storage service for matching storage.
As an optional embodiment, the determining whether the hash value is already stored, and if yes, at least refusing to forward the original log to a log storage service for storage includes:
and determining whether the hash value is stored locally, if so, refusing to forward the original log to a log storage service for storage.
As an alternative embodiment, the method further comprises the following steps:
and forwarding the hash value and the log time of the original log to the log storage service for corresponding storage, and updating the locally stored forwarding times corresponding to the original log.
As an optional embodiment, the method further comprises:
when the log storage service receives the original log, determining a compression state of the original log to determine whether to perform recovery processing on the original log based on the compression state.
Another embodiment of the present invention provides an apparatus for processing device logs, including:
the acquisition module is used for acquiring an original log sent by a log source;
the processing module is used for processing the original log and determining a hash value of the original log;
and the determining module is used for determining whether the hash value is stored, if so, at least refusing to forward the original log to a log storage service for storage, and if not, at least forwarding the original log and the corresponding hash value to the log storage service for storage.
As an optional embodiment, the processing the original log and determining the hash value of the original log include:
and calculating and determining the hash value of the original log based on the log contents except the log time in the original log.
Another embodiment of the present invention further provides a device log processing system, including:
one or more processors;
a memory configured to store one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement a device log processing method as in any above.
Another embodiment of the present invention further provides a storage medium, on which a computer program is stored, which when executed by a processor implements the device log processing method as described in any one of the above.
Based on the disclosure of the above embodiments, it can be known that the embodiments of the present invention have the beneficial effects of processing the original log by obtaining the original log sent by the log source, determining the hash value of the original log, and determining whether the hash value is already stored, if yes, at least refusing to forward the original log to the log storage service for storage, and if not, at least forwarding the original log and the corresponding hash value to the log storage service for storage.
Additional features and advantages of the present application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
The technical solution of the present application is further described in detail by the accompanying drawings and embodiments.
Drawings
The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a flowchart of a device log processing method in an embodiment of the present invention.
Fig. 2 is an application flowchart of a device log processing method in the embodiment of the present invention.
Fig. 3 is a flowchart of a device log processing method according to another embodiment of the present invention.
Fig. 4 is a block diagram of a device log processing apparatus according to an embodiment of the present invention.
Detailed Description
The following detailed description of specific embodiments of the present invention is provided in connection with the accompanying drawings, which are not intended to limit the invention.
It will be understood that various modifications may be made to the embodiments disclosed herein. The following description is, therefore, not to be taken in a limiting sense, but is made merely as an exemplification of embodiments. Other modifications will occur to those skilled in the art within the scope and spirit of the disclosure.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the disclosure and, together with a general description of the disclosure given above, and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.
These and other characteristics of the invention will become apparent from the following description of a preferred form of embodiment, given as a non-limiting example, with reference to the attached drawings.
It should also be understood that although the present invention has been described with reference to some specific examples, a person of skill in the art shall certainly be able to achieve many other equivalent forms of the invention, having the characteristics as set forth in the claims and hence all coming within the field of protection defined thereby.
The above and other aspects, features and advantages of the present disclosure will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.
Specific embodiments of the present disclosure are described hereinafter with reference to the accompanying drawings; however, it is to be understood that the disclosed embodiments are merely examples of the disclosure that may be embodied in various forms. Well-known and/or repeated functions and structures have not been described in detail so as not to obscure the present disclosure with unnecessary or unnecessary detail. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailed structure.
The specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," which may each refer to one or more of the same or different embodiments in accordance with the disclosure.
The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
As shown in fig. 1, an embodiment of the present invention provides an apparatus log processing method, including:
obtaining an original log sent by a log source;
processing the original log and determining a hash value of the original log;
and determining whether the hash value is stored, if so, at least refusing to forward the original log to the log storage service for storage, and if not, at least forwarding the original log and the corresponding hash value to the log storage service for storage.
Based on the method described in this embodiment, the verification on the original log is performed through the HASH value (HASH value), and the log reporting processing flow is optimized, so that the traffic consumed by network transmission on log transmission is effectively reduced, and the user bandwidth use cost is reduced; secondly, the existing network can be fully utilized to provide more efficient log collection with larger data volume, the same bit bears higher effective log information, and the method in the embodiment can avoid repeated storage of the same original log, effectively reduce the occupation of the log storage space, and improve the storage space utilization rate and the log storage efficiency.
As shown in FIG. 2, the log sources shown in FIG. 2: the original syslog is generated and forwarded to a log proxy service/module (hereinafter referred to as log proxy), and the method in this embodiment is executed by the log proxy by default.
Log agent: the log agent performs optimization processing on the log and forwards the log after the optimization processing to a log service;
and log service: and receiving the log and processing the log, wherein the processing comprises further performing log forwarding or log recovery and local storage of the baseline log.
Storage service: the log service stores the received log in a storage service, and the log storage service in the embodiment is composed of the log service and the storage service. In this embodiment, the storage of the original log is realized based on the storage service, and the HASH value (HASH value) of the original log of the log is stored. Subsequent logs with the same HASH value will only hold the log time and the corresponding HASH value (HASH value), as will be described in detail below.
From the process analysis, the log processing in the present embodiment is roughly divided into several parts, such as log generation, log forwarding, log receiving, and log storing, and is similar to the general log processing process, but the present embodiment is different from the existing scheme in that the present embodiment mainly calculates the HASH value (HASH value) of the original log at the time of log forwarding and log storing, and performs determination based on the HASH value (HASH value) at the time of receiving the log again subsequently, and the like.
Specifically, in this embodiment, when processing the original log and determining the hash value of the original log, the method includes:
and calculating and determining the hash value of the original log based on the log contents except the log time in the original log. And then, whether the hash value is forwarded to the log storage service for storage can be judged based on the hash value.
Further, in general, syslog (raw log of devices) has two common styles:
log time + semantic descriptions such as:
2022-09-0108
Or log time + KEY = VALUE, as:
time = "2019-10-2508:
by analyzing the logs generated by various systems/devices, no matter which format of log is, a large number of logs with repeated contents exist, and the only difference is that the logs are generated at different times. In extreme cases, such as when the system has similar alarm or security events, the log repetition rate is as high as 90% or more. For this case, as shown in fig. 3, the method of this embodiment further includes:
for an original log which is sent for the first time, determining a hash value and log time of the original log;
locally storing and recording at least the hash value;
and forwarding the hash value of the original log, the log time and the original log to a log storage service for matching storage.
For example, the log agent may determine an original log which has not been processed before and does not store a corresponding hash value as the original log which is sent by the log source for the first time, and after calculating the hash value corresponding to the log, may record the hash value in a locally established hash value table, and may match the log recording time. Then, the hash value, the log time and the original log of the original log can be sent to a log service, and the log service sends the hash value, the log time and the original log to a storage service for matching storage.
Optionally, since the state of the original log when being received may be a compressed state or an uncompressed state, and the original log in the compressed state is not easy to query and process, the method in this embodiment further includes:
when the log storage service receives the original log, a compression state of the original log is determined to determine whether to perform a recovery process on the original log based on the compression state.
Preferably, for example, when the log proxy forwards the log, the state of the original log may be determined first, and may be determined by whether the log proxy has a flag of compression or not, and if yes, the format of the forwarded log is configured as whether the flag is compressed + HASH value + log time + original log, and then the log is forwarded to the log storage service, and the log storage service determines the compression state of the original log according to whether the flag is compressed, and if the log is in the compression state, decompresses the log, and then stores the log.
The compression flag is used for identifying whether the original log is compressed or not, when the original log is large, compression transmission can be performed, and because the compression algorithm is preset at the sending end and the receiving end, the compression algorithm only needs the flag without providing information related to an additional algorithm. Moreover, the HASH value of the original log content in this embodiment is calculated by the HASH algorithm that is preset at both the sending end and the receiving end, so that only the HASH value needs to be carried during forwarding, and information related to an additional algorithm does not need to be provided. In addition, when the original log is forwarded for the first time, the log service can select to send the original log at random time, and can also determine when to send the original log according to the network congestion state, the current or recent occurrence of an event affecting data security, or whether the current time period is a high occurrence period of the event affecting data security, so as to reduce the risk of data loss during log transmission on an unreliable link.
Further, in this embodiment, determining whether the hash value is already stored, and if so, rejecting to forward the original log to the log storage service for storage at least includes:
and determining whether the hash value is stored locally, if so, refusing to forward the original log to a log storage service for storage.
The method further comprises the following steps:
and forwarding the hash value and the log time of the original log to a log storage service for corresponding storage, and updating the forwarding times of the locally stored corresponding original log.
For example, whether a hash value is stored in a locally established hash table is determined, if yes, the original log is refused to be forwarded to the log storage service for storage, meanwhile, the hash value of the original log and the log time are forwarded to the log service, and the log service forwards the log time to the storage service. In this embodiment, the log proxy further records the forwarding times corresponding to each original log and the hash value thereof, and updates the corresponding forwarding times when the log proxy forwards the information, so that when the log proxy forwards the information, it will determine whether the log is forwarded first based on the hash value, and if so, only forward the hash value and the log time, and update the local forwarding times related to the original log. If not, the original log is forwarded for the first time, the original log is forwarded to the log service together with the hash value of the original log, the log time and the compression flag, matching storage is performed locally, and the forwarding times are set to be 1. That is, in addition to storing the original log, the HASH value of the original log is also stored, and in the case where the original log already exists, the original content of the subsequent log having the same HASH value will not be saved, and only the log time and its HASH value will be saved.
In practical application, the method of the embodiment can be applied to a security gateway and a corresponding log analysis processing system, and can also be applied to a large data processing system of the SASE solution in an expanded manner. Moreover, a log agent needs to be deployed in the log source device to realize the optimization processing before log forwarding, and meanwhile, a corresponding log service program needs to be deployed in the log service to realize the log processing analysis after log receiving.
When the method is applied specifically, the log processing flow analysis comprises the following steps:
1. the log source generates a log and forwards the log to a log agent;
2. the log agent performs preliminary analysis on the log and extracts the log time. The log time is identified as LogTime.
3. The log agent performs HASH calculation on other log contents except the log time in the log, and the calculation result is marked as LogHASH.
4. The log agent stores the log related information into a local memory, and the stored content comprises the following contents:
the method comprises the steps of (1) logging HASH value, logging time, log forwarding times and original logs;
the function of some fields is described as follows:
the log HASH value is the result of HASH calculation of other log contents except the log time in the log;
the log time, the time of log generation, and the latest log generation time;
the log forwarding times are accumulated, and the log can be checked subsequently;
the original log is stored and can be sent at a proper time, so that data loss caused by network instability or system disconnection is prevented;
5. the log proxy pair determines log forwarding contents, and comprises the following steps:
if the LogHASH value does not exist in the current memory, the log agent carries the original log to forward;
if the LogHASH value exists in the current memory, the log agent does not carry the original log for forwarding;
after the transfer, the accumulated value of the transfer times of the log is added by 1 to count.
6. After receiving the log, the log service determines whether to recover the log according to whether the log is compressed;
7. the log service forwards the log to the storage service, and the storage service stores the log into a warehouse. When storing the log, the storage service uses the HASH value as a check condition, and only stores the corresponding log time and HASH value for the log with the same HASH value.
Taking the log with an average length of 256 bytes as an example, after the log is processed by the method of the embodiment, the size of the space occupied by storing the original log and storing the log data optimized by the method of the embodiment is compared, and it can be known that:
the Hash algorithm adopts MD5, the calculation result is 128 bits with fixed length, namely 16 bytes, and 32 bytes are occupied by converting the MD5 into characters for storage;
the log time is saved as a shaping number, which takes 4 bytes.
Thus, the optimized log occupies space: s2=32+4=36 bytes;
the original log occupies space: s1=256 bytes
The space occupation ratio of the two is T =36
The space saving is: 1-36, approximately 0.85,
namely, the method of the embodiment can save 85% of storage space when being used for processing and storing the logs.
As shown in fig. 4, another embodiment of the present invention also provides a device log processing apparatus 100, including:
the acquisition module is used for acquiring an original log sent by a log source;
the processing module is used for processing the original log and determining a hash value of the original log;
and the determining module is used for determining whether the hash value is stored, if so, at least refusing to forward the original log to a log storage service for storage, and if not, at least forwarding the original log and the corresponding hash value to the log storage service for storage.
As an optional embodiment, the processing the original log and determining the hash value of the original log includes:
and calculating and determining the hash value of the original log based on the log contents except the log time in the original log.
As an optional embodiment, the processing the original log and determining the hash value of the original log includes:
and calculating and determining the hash value of the original log based on the log contents except the log time in the original log.
As an alternative embodiment, the method further comprises the following steps:
for the original log which is sent for the first time, determining a hash value and log time of the original log;
locally storing and recording at least the hash value;
and forwarding the hash value, the log time and the original log of the original log to the log storage service for matching storage.
As an optional embodiment, the determining whether the hash value is already stored, and if yes, at least refusing to forward the original log to a log storage service for storage includes:
and determining whether the hash value is stored locally, if so, refusing to forward the original log to a log storage service for storage.
As an alternative embodiment, the method further comprises the following steps:
and forwarding the hash value and the log time of the original log to the log storage service for corresponding storage, and updating the locally stored forwarding times corresponding to the original log.
As an alternative embodiment, the method further comprises the following steps:
when the log storage service receives the original log, determining a compression state of the original log to determine whether to perform recovery processing on the original log based on the compression state.
Another embodiment of the present application further provides an apparatus log processing system, including:
one or more processors;
a memory configured to store one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the device log processing method described above.
Further, an embodiment of the present application also provides a storage medium, on which a computer program is stored, and the program, when executed by a processor, implements the device log processing method as described above. It should be understood that each solution in this embodiment has a corresponding technical effect in the foregoing method embodiments, and details are not described here.
Further, embodiments of the present application also provide a computer program product that is tangibly stored on a computer-readable medium and includes computer-executable instructions that, when executed, cause at least one processor to perform a device log processing method, such as the embodiments described above.
It should be noted that the computer storage media of the present application can be computer readable signal media or computer readable storage media or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory medium (RAM), a read-only memory medium (ROM), an erasable programmable read-only memory medium (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, antenna, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
In addition, as will be appreciated by one of skill in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a system for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
The above embodiments are only exemplary embodiments of the present invention, and are not intended to limit the present invention, and the scope of the present invention is defined by the claims. Various modifications and equivalents may be made by those skilled in the art within the spirit and scope of the present invention, and such modifications and equivalents should also be considered as falling within the scope of the present invention.

Claims (10)

1. A device log processing method, comprising:
obtaining an original log sent by a log source;
processing the original log, and determining a hash value of the original log;
and determining whether the hash value is stored, if so, at least refusing to forward the original log to a log storage service for storage, and if not, at least forwarding the original log and the corresponding hash value to the log storage service for storage.
2. The device log processing method according to claim 1, wherein the processing the raw log to determine a hash value of the raw log comprises:
and calculating and determining the hash value of the original log based on the log contents except the log time in the original log.
3. The device log processing method according to claim 1, further comprising:
for the original log which is sent for the first time, determining a hash value and log time of the original log;
locally storing and recording at least the hash value;
and forwarding the hash value, the log time and the original log of the original log to the log storage service for matching storage.
4. The device log processing method of claim 3, wherein the determining whether the hash value has been stored, and if so, at least denying forwarding the raw log to a log storage service for storage comprises:
and determining whether the hash value is stored locally, if so, refusing to forward the original log to a log storage service for storage.
5. The device log processing method according to claim 4, further comprising:
and forwarding the hash value and the log time of the original log to the log storage service for corresponding storage, and updating the locally stored forwarding times corresponding to the original log.
6. The device log processing method according to claim 1, further comprising:
when the log storage service receives the original log, determining a compression state of the original log to determine whether to perform recovery processing on the original log based on the compression state.
7. An apparatus log processing apparatus, comprising:
the acquisition module is used for acquiring an original log sent by a log source;
the processing module is used for processing the original log and determining a hash value of the original log;
and the determining module is used for determining whether the hash value is stored, if so, at least refusing to forward the original log to a log storage service for storage, and if not, at least forwarding the original log and the corresponding hash value to the log storage service for storage.
8. The device log processing apparatus of claim 7, wherein the processing the raw log to determine the hash value of the raw log comprises:
and calculating and determining the hash value of the original log based on the log contents except the log time in the original log.
9. A device log processing system, comprising:
one or more processors;
a memory configured to store one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the device log processing method of any of claims 1-6.
10. A storage medium, characterized in that a computer program is stored thereon, which when executed by a processor implements the device log processing method according to any one of claims 1 to 6.
CN202211630391.6A 2022-12-19 2022-12-19 Equipment log processing method, device and system and storage medium Pending CN115967615A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211630391.6A CN115967615A (en) 2022-12-19 2022-12-19 Equipment log processing method, device and system and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211630391.6A CN115967615A (en) 2022-12-19 2022-12-19 Equipment log processing method, device and system and storage medium

Publications (1)

Publication Number Publication Date
CN115967615A true CN115967615A (en) 2023-04-14

Family

ID=87353722

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211630391.6A Pending CN115967615A (en) 2022-12-19 2022-12-19 Equipment log processing method, device and system and storage medium

Country Status (1)

Country Link
CN (1) CN115967615A (en)

Similar Documents

Publication Publication Date Title
US10984018B2 (en) System, methods, and media for compressing non-relational database objects
CN107332876B (en) Method and device for synchronizing block chain state
US9158806B2 (en) Integrity checking and selective deduplication based on network parameters
US20190222603A1 (en) Method and apparatus for network forensics compression and storage
CN105991412B (en) Information push method and device
CN111740868B (en) Alarm data processing method and device and storage medium
CN108062419B (en) File storage method, electronic equipment, system and medium
CN107454120A (en) The method of network attack defending system and defending against network attacks
CN109905479B (en) File transmission method and device
CN111464630A (en) Transaction broadcasting method, apparatus and storage medium
CN112817602A (en) JSON format data sending and receiving method, device and medium
CN114048201A (en) Distributed stream computing engine Flink-based key field real-time deduplication method
CN115499230A (en) Network attack detection method and device, equipment and storage medium
US10355866B2 (en) File reputation acquiring method, gateway device, and file reputation server
CN110784501B (en) Method, system, device and storage medium for packaging and broadcasting transaction group containing block
CN109274720B (en) Method and system for transmitting data
CN103841144A (en) Cloud storage system and method, user terminal and cloud storage server
CN108809678A (en) A kind of method and server of information push
US10498825B2 (en) Method and system for selecting a transport mechanism and a storage process
CN115967615A (en) Equipment log processing method, device and system and storage medium
CN106304122B (en) Business data analysis method and system
CN107092529B (en) OLAP service method, device and system
US9130827B2 (en) Sampling from distributed streams of data
US20210117096A1 (en) Method, device and computer program product for backuping data
CN114157716A (en) Data processing method and device based on block chain and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination