CN115757304A - Log storage method, device and system, electronic equipment and storage medium - Google Patents
Log storage method, device and system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115757304A CN115757304A CN202211351136.8A CN202211351136A CN115757304A CN 115757304 A CN115757304 A CN 115757304A CN 202211351136 A CN202211351136 A CN 202211351136A CN 115757304 A CN115757304 A CN 115757304A
- Authority
- CN
- China
- Prior art keywords
- log information
- key value
- frequency
- dimension
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a log storage method, a log storage device, a log storage system, electronic equipment and a log storage medium. The method comprises the following steps: obtaining log information, setting dimension labels for the log information, and obtaining labeled log information; dimension splitting is carried out on the marked log information to obtain key value log information of each dimension; determining a hash value of the key value log information, and repeatedly judging the key value log information marked with the log information based on the hash value to obtain a judgment result; and storing each key value log information in the labeling log information and the hash value of each key value log information based on the judgment result. According to the invention, the dimension of the marked log information is split to obtain the key value log information with different dimensions, and the key value log information is stored based on the hash value of the key value log information, so that the key value log information with the same hash value is not stored repeatedly and is used for replacing copy, the problem of redundant storage of the same log information in the prior art is solved, and the storage space occupied by the log information is reduced.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a log storage method, apparatus, system, electronic device, and storage medium.
Background
With the wide application of big data and cloud computing in the internet, the pace of the fusion development of finance and science and technology is accelerated, and more data are generated in the process.
Currently, log information is stored by using data collection tools such as Sqoop and jump to introduce related data into a Hadoop cluster, so as to realize data collection, processing, storage and loading, and files and data are stored by using HDFS.
However, all log information is stored in the hadoop cluster, and some meaningless log information can be stored repeatedly, so that resource waste is caused; in addition, when the log information is imported into the hadoop cluster, the dimension data in each log are the same, so that in the processing process, more noise data exist, and the processing efficiency is low.
Disclosure of Invention
The invention provides a log storage method, a log storage device, a log storage system, electronic equipment and a log storage medium, which aim to solve the problem of redundant log information storage in the prior art.
According to an aspect of the present invention, there is provided a log storage method, including:
obtaining log information, setting dimension labels for the log information, and obtaining labeled log information;
dimension splitting is carried out on the labeled log information to obtain key value log information of each dimension;
determining a hash value of the key value log information, and repeatedly judging each key value log information of the labeled log information based on the hash value to obtain a judgment result;
and storing each piece of key value log information in the labeling log information and the hash value of each piece of key value log information based on the judgment result.
According to another aspect of the present invention, there is provided a log storage apparatus, comprising:
the log information marking module is used for acquiring log information and setting dimension marking on the log information to obtain marked log information;
the log information splitting module is used for carrying out dimension splitting on the labeled log information to obtain key value log information of each dimension;
the hash value judgment module is used for determining the hash value of the key value log information and repeatedly judging each key value log information of the labeled log information based on the hash value to obtain a judgment result;
and the log storage module is used for storing each piece of key value log information in the labeling log information and the hash value of each piece of key value log information based on the judgment result.
According to another aspect of the present invention, there is provided a log storage system, comprising: log collector, key value sorter and storage cluster, wherein:
the log collector is used for acquiring log information, setting dimension marking on the log information to obtain marked log information and sending the marked log information to the key value sorter;
the key value sorter is used for receiving the labeled log information, performing dimension splitting on the labeled log information to obtain key value log information of each dimension, determining a hash value of the key value log information, and sending the key value log information and the hash value of the key value log information to the storage cluster;
the storage cluster is used for receiving the key value log information and the hash value of the key value log information, and repeatedly judging each key value log information of the labeled log information based on the hash value to obtain a judgment result; and storing each piece of key value log information in the labeling log information and the hash value of each piece of key value log information based on the judgment result.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the log storing method according to any of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement a log storage method according to any one of the embodiments of the present invention when the computer instructions are executed.
According to the technical scheme of the embodiment of the invention, the dimension of the marked log information is split to obtain the key value log information, and the key value log information is stored based on the hash value of the key value log information, so that the key value log information with the same hash value is not stored repeatedly and is copied by reference, the problem of redundant storage of the same log information in the prior art is solved, and the storage space occupied by the log information is reduced; in addition, the log information is stored separately through different splitting dimensions, so that the processing of the log information is facilitated, and the processing efficiency of the log information is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram of a data storage structure implementing an embodiment of the invention;
fig. 2 is a flowchart of a log storage method according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a log storage device according to a second embodiment of the present invention;
fig. 4 is a schematic structural diagram of a journal storage system according to a third embodiment of the present invention;
fig. 5 is a schematic structural diagram of a memory architecture according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
FIG. 1 is a schematic diagram of a data storage structure (Merkle DAG structure) implementing an embodiment of the invention. As shown in fig. 1, the Merkle DAG structure includes a root data block, a low frequency key-value data layer, and a high frequency key-value data layer.
The root data block stores a storage log information table, and the information such as the hash value, the reference times and the like of the stored log information is maintained through the storage log information table.
The low-frequency key value data layer is used for storing low-frequency key value log information, including data nodes of the low-frequency key value log information; the low-frequency key value log information is a combination of key dimension log fields in the log information, can fully express each piece of log information and embody the value of the log information, and has no noise data.
The high-frequency key value data layer is used for storing high-frequency key value log information, including data nodes of the high-frequency key value log information, wherein the high-frequency key value log information is part of the key value log information determined by system configuration or a plurality of key combinations of non-key log fields in the log information. Often, the combination of these key values can represent a business event that occurs during the operation of a system, such as: high-frequency key value groups such as a system, an operating environment, an event label, an event word prompt, an error detailed information translation and the like are generated, the information is generally long and basically the same in content, and a large amount of information can be generated in the scenes of a high-flow period, failure of upstream and downstream services, complex operation overtime and the like. The high-frequency key value log information is recorded and compared in a key mode, when a new high-frequency key value data block is met, the data block is added and copied dynamically, and when a repeated data block is met, the data block is quoted through a hash value to replace the copy of redundant data.
Example one
Fig. 2 is a flowchart of a log storage method according to an embodiment of the present invention, where the present embodiment is applicable to a case where log information is stored according to different dimensions, and the method may be executed by a log storage device and/or a log storage system, where the log storage device and/or the log storage system may be implemented in a form of hardware and/or software, and the log storage device and/or the log storage system may be configured in an electronic device according to an embodiment of the present invention. As shown in fig. 2, the method includes:
s210, obtaining log information, setting dimension marking on the log information, and obtaining marked log information.
The log information is information for recording user operation and system operation state. In the embodiment, log information sent by a consumer system is collected, dimension labels are set for log fields in the collected log information according to different dimensions, and labeled log information is obtained; the dimension marking refers to marking set according to the dimension of the log field, and correspondingly, the marked log information is the log information with the dimension marking set.
On the basis of the foregoing embodiment, optionally, the setting a dimension label for the log information to obtain labeled log information includes: analyzing the log information to obtain each log field of the log information, setting dimension labels for each log field, and forming labeled log information based on each labeled log field; wherein the dimensions include a low frequency dimension and a high frequency dimension.
The log field refers to a data field in the log information, and specifically, the log field includes, but is not limited to, an event label, an event text prompt, an error detailed information translation, and the like, which is not limited herein. In the embodiment, the log information is analyzed, log fields in the log information are extracted, the log fields are labeled according to the set dimensions of the log fields, and labeled log information is formed based on the labeled log fields; the frequency dimension comprises a low-frequency dimension and a high-frequency dimension, the low-frequency dimension is used for marking log fields with low occurrence frequency in the log information, and correspondingly, the high-frequency dimension is used for marking log fields with high occurrence frequency in the log information.
It is understood that the log fields that frequently appear in the log information are log fields of non-critical dimensions, such as event labels, event text prompts, error detailed information translations, and the like, and correspondingly, the log fields that frequently appear in the log information are log fields of critical dimensions.
S220, dimension splitting is carried out on the labeled log information, and key value log information of each dimension is obtained.
The dimension splitting refers to splitting the marked log information according to the dimension marking dimension, and specifically, the dimension of the dimension splitting includes, but is not limited to, a low-frequency dimension, a high-frequency dimension and the like, and is not limited herein. In the embodiment, the marked log information is split according to the dimension of the dimension mark to obtain key value log information of each dimension; the key value log information refers to log information of each dimension obtained after the marked log information is split.
On the basis of the foregoing embodiment, optionally, the performing dimension splitting on the labeled log information to obtain key value log information of each dimension includes: determining the service type of the log information, determining a key value sorter based on the service type, and performing dimension splitting on the labeled log information based on the key value sorter to obtain the key value log information of each dimension.
The service type refers to the service type of a service system generating log information, the service type is associated with a key value sorter, and the key value sorter is used for splitting labeled log information after labeling. In this embodiment, the service type of the service system that generates the log information is determined, the key value sorter corresponding to the service type is determined based on the service type, and the dimension splitting is performed on the labeled log information based on the key value sorter, so as to obtain the key value log information of each dimension.
For example, assuming that the labeled log information is X (a, B, C1, D, E, F, G, H, I, J), where a, B, C, D are low-frequency dimensions and E, F, G, H, I, J are high-frequency dimensions, after splitting the labeled log information X, (a, B, C, D) are low-frequency dimension log information, and (E, F, G, H, I, J) are high-frequency dimension log information.
It should be noted that the splitting dimension of the labeled log information in the configuration information of the key value sorter is the same as the labeling dimension of the labeled log information; illustratively, a high frequency dimension and a low frequency dimension.
S230, determining a hash value of the key value log information, and repeatedly judging each key value log information of the labeled log information based on the hash value to obtain a judgment result.
The repeated judgment refers to judging whether the key value log information is stored in the storage cluster, and specifically, judging whether the key value log information is stored by judging whether the hash value of the key value log information and the hash value of the stored key value log information are repeated. In this embodiment, the hash value of each key value log information is calculated, the hash value of each key value log information is compared with the hash value of the stored key value log information, and if the hash value of the stored key value log information is repeated with the hash value of the key value log information, the determination result of the key value log information is stored; and if the hash value of any stored key value log information is not repeated with the hash value of the key value log information, judging that the key value log information is not stored.
On the basis of the above embodiment, optionally, the key value log information includes high-frequency key value log information and low-frequency key value log information; correspondingly, the determining the hash value of the key value log information, and repeatedly determining each key value log information of the labeled log information based on the hash value to obtain a determination result specifically includes: establishing a reference relation between the high-frequency key value log information and the low-frequency key value log information, respectively determining hash values of the high-frequency key value log information and the low-frequency key value log information, and repeatedly judging based on the hash values and a stored log information table to obtain a judgment result; the storage log information table is used for storing the hash value of the stored key value log information.
The key value log information comprises high-frequency key value log information and low-frequency key value log information, wherein the high-frequency key value log information is a combination of high-frequency dimensionality log fields, namely a combination of log fields of non-key dimensionality; correspondingly, the low-frequency key value log information is a combination of low-frequency dimension log fields, that is, a combination of key dimension log fields. In the embodiment, a reference relation between high-frequency key value log information and low-frequency key value log information is established, hash values of the high-frequency key value log information and the low-frequency key value log information are respectively calculated, the hash values of the high-frequency key value log information and the low-frequency key value log information are respectively compared with hash values in a storage log information table, if any hash value in the storage log information table is repeated with the hash values of the high-frequency key value log information or the low-frequency key value log information, a judgment result of the high-frequency key value log information or the low-frequency key value log information is stored, and otherwise, the judgment result of the high-frequency key value log information or the low-frequency key value log information is not stored; the storage log information table is used for storing a hash value of stored key value log information, the storage log information table is stored in a root data block (namely a root node), and the storage log information table is also used for storing the number of times of referencing the high-frequency key value log information to the low-frequency key value log information.
And S240, storing the key value log information in the labeling log information and the hash value of the key value log information based on the judgment result.
In the embodiment, according to the determination result, if the determination result of the key value log information is that the key value log information is not stored, a new data node is generated, the key value log information is stored in the new data node, and the hash value of the key value log information is stored in a stored log information table; and if the judgment result of the key value log information is stored, the key value log information is not stored.
On the basis of the foregoing embodiment, optionally, the storing each piece of key-value log information in the labeled log information and the hash value of each piece of key-value log information based on the determination result specifically includes: if the judgment result of the low-frequency key value log information is stored, the marked log information is not stored; if the judgment result of the low-frequency key value log information is not stored, storing the low-frequency key value log information and the hash value of the low-frequency key value log information, and determining the judgment result of the high-frequency key value log information; if the judgment result of the high-frequency key value log information is stored, updating the reference relationship between the data node used for storing the high-frequency key value log information and the data node used for storing the low-frequency key value log information in the log information storage table; and if the judgment result of the high-frequency key value log information is that the high-frequency key value log information is not stored, storing the high-frequency key value log information and the hash value of the high-frequency key value log information, and updating the reference relationship between the data node used for storing the high-frequency key value log information and the data node used for storing the low-frequency key value log information in the stored log information table.
In this embodiment, the determination result includes a determination result of the high-frequency key value log information and a determination result of the low-frequency key value log information; if the judgment result of the low-frequency key value log information is stored, the marked log information is not stored; if the judgment result of the low-frequency key value log information is not stored, generating a new data node to store the low-frequency key value log information, storing the hash value of the low-frequency key value log information in a log information storage table, and determining the judgment result of the high-frequency key value log information in a linkage manner; if the judgment result of the high-frequency key value log information is stored, the high-frequency key value log information is not stored, and the reference relation between a data node for storing the high-frequency key value log information and a new data node for storing the low-frequency key value log information in a log information storage table is updated; and if the judgment result of the high-frequency key value log information is that the high-frequency key value log information is not stored, generating a new data node to store the high-frequency key value log information, storing the hash value of the high-frequency key value log information in a log information storage table, and updating the reference relationship between the new data node used for storing the high-frequency key value log information and the new data node used for storing the low-frequency key value log information in the log information storage table.
It can be understood that, under the condition that the log information of the low-frequency key value is stored, the log information of the key dimension in the log information is stored, the log information is discarded without information loss, and the marked log information can be directly not stored; and under the condition that the log information of the low-frequency key value is not stored, the log information of the key dimension in the log information is not stored, and the log information needs to be stored.
On the basis of the foregoing embodiment, optionally, after storing each piece of key-value log information and the hash value of each piece of key-value log information, the method further includes: and updating the storage state version based on the stored log information table after the log information storage is finished.
In this embodiment, after the log information is stored, a root hash value is calculated based on the hash value of each key value log information in the stored log information table after the log information is stored, and the root hash value is used as the storage state version of the current storage node. Illustratively, the hash values of the key value log information are added, the hash value of the addition result is calculated as a root hash value, and the root hash value is used as a storage state version.
According to the technical scheme, the dimension of the marked log information is split to obtain the key value log information, and the key value log information is stored based on the hash value of the key value log information, so that the key value log information with the same hash value is not stored repeatedly and is used for reference to replace copy, the problem of redundant storage of the same log information in the prior art is solved, and the storage space occupied by the log information is reduced; in addition, the log information is stored separately through different splitting dimensions, so that the processing of the log information is facilitated, and the processing efficiency of the log information is improved.
Example two
Fig. 3 is a schematic structural diagram of a log storage device according to a second embodiment of the present invention. As shown in fig. 3, the apparatus includes:
the log information labeling module 310 is configured to obtain log information, and set dimension labels for the log information to obtain labeled log information;
the log information splitting module 320 is configured to perform dimension splitting on the labeled log information to obtain key value log information of each dimension;
the judging module 330 is configured to determine a hash value of the key value log information, and repeatedly judge each key value log information of the labeled log information based on the hash value to obtain a judgment result;
the log storage module 340 is configured to store, based on the determination result, each piece of key-value log information in the labeled log information and a hash value of each piece of key-value log information.
On the basis of the foregoing embodiment, optionally, the log information labeling module 310 is specifically configured to analyze the log information to obtain each log field of the log information, set a dimension label for each log field, and form the labeled log information based on each labeled log field; wherein the dimensions include a low frequency dimension and a high frequency dimension.
On the basis of the foregoing embodiment, optionally, the log information splitting module 320 is specifically configured to determine the service type of the log information, determine a key value sorter based on the service type, and perform dimension splitting on the labeled log information based on the key value sorter to obtain the key value log information of each dimension.
On the basis of the above embodiment, optionally, the key value log information includes high-frequency key value log information and low-frequency key value log information; correspondingly, the determining module 330 is specifically configured to establish a reference relationship between the high-frequency key value log information and the low-frequency key value log information, determine hash values of the high-frequency key value log information and the low-frequency key value log information respectively, and perform repeated determination based on the hash values and a stored log information table to obtain a determination result; the storage log information table is used for storing the hash value of the stored key value log information.
On the basis of the foregoing embodiment, optionally, the log storage module 340 is specifically configured to not store the labeled log information if the determination result of the low-frequency key value log information is already stored; if the judgment result of the low-frequency key value log information is not stored, storing the low-frequency key value log information and the hash value of the low-frequency key value log information, and determining the judgment result of the high-frequency key value log information; if the judgment result of the high-frequency key value log information is stored, updating the reference relationship between the data node used for storing the high-frequency key value log information and the data node used for storing the low-frequency key value log information in the log information storage table; if the judgment result of the high-frequency key value log information is not stored, storing the high-frequency key value log information and the hash value of the high-frequency key value log information, and establishing a reference relationship between a data node used for storing the high-frequency key value log information and a data node used for storing the low-frequency key value log information in the stored log information table.
On the basis of the foregoing embodiment, optionally, the apparatus further includes a storage status version updating module, configured to update the storage status version based on the storage log information table after the log information storage is completed.
The log storage device provided by the embodiment of the invention can execute the log storage method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE III
Fig. 4 is a schematic structural diagram of a log storage system according to a third embodiment of the present invention. As shown in fig. 4, the system includes: log collector 410, key-value sorter 420, and storage cluster 430, wherein:
the log collector 410 is configured to obtain log information, set a dimension label for the log information, obtain labeled log information, and send the labeled log information to the key value sorter 420;
the key value sorter 420 is configured to receive the annotation log information, perform dimension splitting on the annotation log information to obtain key value log information of each dimension, determine a hash value of the key value log information, and send the key value log information and the hash value of the key value log information to the storage cluster 430;
the storage cluster 430 is configured to receive the key value log information and the hash value of the key value log information, and perform repeated determination on each key value log information of the labeled log information based on the hash value to obtain a determination result; and storing each piece of key value log information in the labeling log information and the hash value of each piece of key value log information based on the judgment result.
Fig. 5 is a schematic structural diagram of a memory architecture according to a third embodiment of the present invention. As shown in fig. 5, the log collector collects log information of each system from each system, and sends the collected log information to key value sorters corresponding to different clusters. The key value sorter is responsible for preprocessing the log information before the log information is stored in the storage cluster, wherein the log information is mainly divided into high-frequency key value information and low-frequency key value information according to a configuration file of the key value sorter or according to labeling dimensions (key dimensions and non-key dimensions) of the log information, and hash values of the high-frequency key value information and the low-frequency key value information are calculated; the low-frequency key value information corresponds to key dimension data, and the high-frequency key value information corresponds to non-key dimension data. After preprocessing the log information, the key value sorter sends the high-frequency key value information, the low-frequency key value information and the hash value of the high-frequency key value information and the low-frequency key value information to the storage cluster; and the storage cluster repeatedly judges the high-frequency key value information and the low-frequency key value information, stores the high-frequency key value information and the low-frequency key value information which are not stored, and the hash values of the high-frequency key value information and the low-frequency key value information, and updates the reference in the storage log information table.
Example four
Fig. 6 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 6, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to the bus 14.
A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as a log storage method.
In some embodiments, the log storage method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the log storing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log storage method by any other suitable means (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for implementing the log storage method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.
EXAMPLE five
An embodiment of the present invention further provides a computer-readable storage medium, where a computer instruction is stored in the computer-readable storage medium, where the computer instruction is used to enable a processor to execute a log storage method, where the method includes:
obtaining log information, setting dimension labels for the log information, and obtaining labeled log information; dimension splitting is carried out on the marked log information to obtain key value log information of each dimension; determining a hash value of the key value log information, and repeatedly judging the key value log information marked with the log information based on the hash value to obtain a judgment result; and storing each key value log information in the labeling log information and the hash value of each key value log information based on the judgment result.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of log storage, comprising:
obtaining log information, setting dimension labels for the log information, and obtaining labeled log information;
dimension splitting is carried out on the labeled log information to obtain key value log information of each dimension;
determining a hash value of the key value log information, and repeatedly judging each key value log information of the labeled log information based on the hash value to obtain a judgment result;
and storing each piece of key value log information in the labeling log information and the hash value of each piece of key value log information based on the judgment result.
2. The method of claim 1, wherein the setting a dimension label to the log information to obtain a labeled log information comprises:
analyzing the log information to obtain each log field of the log information, setting dimension labels for each log field, and forming labeled log information based on each labeled log field; wherein the dimensions include a low frequency dimension and a high frequency dimension.
3. The method of claim 1, wherein the performing dimension splitting on the labeled log information to obtain key value log information of each dimension comprises:
determining the service type of the log information, determining a key value sorter based on the service type, and performing dimension splitting on the labeled log information based on the key value sorter to obtain the key value log information of each dimension.
4. The method of claim 3, wherein the key-value log information comprises high-frequency key-value log information and low-frequency key-value log information;
correspondingly, the determining the hash value of the key value log information, and repeatedly determining each key value log information of the labeled log information based on the hash value to obtain a determination result includes:
establishing a reference relation between the high-frequency key value log information and the low-frequency key value log information, and respectively determining hash values of the high-frequency key value log information and the low-frequency key value log information; repeatedly judging based on the hash value and a storage log information table to obtain a judgment result; the storage log information table is used for storing the hash value of the stored key value log information.
5. The method of claim 4, wherein storing each of the key-value log information and the hash value of each of the key-value log information in the annotation log information based on the determination result comprises:
if the judgment result of the low-frequency key value log information is stored, the marked log information is not stored;
if the judgment result of the low-frequency key value log information is not stored, storing the low-frequency key value log information and the hash value of the low-frequency key value log information, and determining the judgment result of the high-frequency key value log information;
if the judgment result of the high-frequency key value log information is stored, updating the reference relationship between the data node for storing the high-frequency key value log information and the data node for storing the low-frequency key value log information in the log information storage table;
and if the judgment result of the high-frequency key value log information is that the high-frequency key value log information is not stored, storing the high-frequency key value log information and the hash value of the high-frequency key value log information, and updating the reference relationship between the data node used for storing the high-frequency key value log information and the data node used for storing the low-frequency key value log information in the stored log information table.
6. The method of claim 5, wherein after storing each of the key-value log information and the hash value of each of the key-value log information, the method further comprises:
and updating the storage state version based on the stored log information table after the log information storage is finished.
7. A log storage device, comprising:
the log information marking module is used for acquiring log information and setting dimension marking on the log information to obtain marked log information;
the log information splitting module is used for carrying out dimension splitting on the labeled log information to obtain key value log information of each dimension;
the judging module is used for determining a hash value of the key value log information and repeatedly judging each key value log information of the labeled log information based on the hash value to obtain a judging result;
and the log storage module is used for storing each piece of key value log information in the labeled log information and the hash value of each piece of key value log information based on the judgment result.
8. A log storage system, comprising: log collector, key value sorter and storage cluster, wherein:
the log collector is used for acquiring log information, setting dimension marking on the log information to obtain marked log information and sending the marked log information to the key value sorter;
the key value sorter is used for receiving the annotation log information, performing dimension splitting on the annotation log information to obtain key value log information of each dimension, determining a hash value of the key value log information, and sending the key value log information and the hash value of the key value log information to the storage cluster;
the storage cluster is used for receiving the key value log information and the hash value of the key value log information, and repeatedly judging each key value log information of the labeled log information based on the hash value to obtain a judgment result; and storing each piece of key value log information in the labeling log information and the hash value of each piece of key value log information based on the judgment result.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the log storage method of any one of claims 1-7.
10. A computer-readable storage medium storing computer instructions for causing a processor to implement the log storage method of any one of claims 1 to 7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211351136.8A CN115757304A (en) | 2022-10-31 | 2022-10-31 | Log storage method, device and system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211351136.8A CN115757304A (en) | 2022-10-31 | 2022-10-31 | Log storage method, device and system, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115757304A true CN115757304A (en) | 2023-03-07 |
Family
ID=85354717
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211351136.8A Pending CN115757304A (en) | 2022-10-31 | 2022-10-31 | Log storage method, device and system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115757304A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117539840A (en) * | 2024-01-10 | 2024-02-09 | 杭州新中大科技股份有限公司 | Log acquisition method, device, equipment and medium |
-
2022
- 2022-10-31 CN CN202211351136.8A patent/CN115757304A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117539840A (en) * | 2024-01-10 | 2024-02-09 | 杭州新中大科技股份有限公司 | Log acquisition method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114579104A (en) | Data analysis scene generation method, device, equipment and storage medium | |
| CN115757304A (en) | Log storage method, device and system, electronic equipment and storage medium | |
| CN114064925A (en) | Knowledge graph construction method, data query method, device, equipment and medium | |
| CN117055851A (en) | Software architecture recovery method and device, electronic equipment and storage medium | |
| CN115186738B (en) | Model training method, device and storage medium | |
| CN115048352B (en) | Log field extraction method, device, equipment and storage medium | |
| CN113051479B (en) | File processing and recommendation information generation methods, devices, equipment and storage medium | |
| CN115563310A (en) | Method, device, equipment and medium for determining key service node | |
| CN115328898A (en) | Data processing method and device, electronic equipment and medium | |
| CN114996930A (en) | Modeling method and device, electronic equipment and storage medium | |
| CN114968950A (en) | Task processing method and device, electronic equipment and medium | |
| CN114817223A (en) | Service data extraction method and device, electronic equipment and storage medium | |
| CN114064803A (en) | Data synchronization method and device | |
| CN113656239A (en) | Monitoring method and device for middleware and computer program product | |
| CN113742321A (en) | Data updating method and device | |
| CN116431698B (en) | Data extraction method, device, equipment and storage medium | |
| CN117093638B (en) | Micro-service data initialization method, system, electronic equipment and storage medium | |
| CN116562050B (en) | Visual modeling method of distributed simulation platform and distributed simulation platform | |
| CN113553836B (en) | Method and device for updating dialogue dictionary, electronic equipment and storage medium | |
| CN109871318B (en) | Key class identification method based on software operation network | |
| CN114706578A (en) | Data processing method, device, equipment and medium | |
| CN114492393A (en) | Text theme determination method and device and electronic equipment | |
| US10909079B1 (en) | Data-driven reduction of log message data | |
| CN115567624A (en) | Message processing method and device, electronic equipment and medium | |
| CN117764560A (en) | Vehicle maintenance suggestion acquisition method and vehicle maintenance suggestion generation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |