CN108038018B - Extensible log data storage method and device - Google Patents
Extensible log data storage method and device Download PDFInfo
- Publication number
- CN108038018B CN108038018B CN201711416857.1A CN201711416857A CN108038018B CN 108038018 B CN108038018 B CN 108038018B CN 201711416857 A CN201711416857 A CN 201711416857A CN 108038018 B CN108038018 B CN 108038018B
- Authority
- CN
- China
- Prior art keywords
- field
- change rate
- dictionary
- pattern
- log data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000013500 data storage Methods 0.000 title claims abstract description 35
- 230000008859 change Effects 0.000 claims abstract description 78
- 238000004590 computer program Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 abstract description 19
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 230000006872 improvement Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0787—Storage of error reports, e.g. persistent data storage, storage using memory protection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0775—Content or structure details of the error report, e.g. specific table structure, specific error fields
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
- G06F11/3082—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by aggregating or compressing the monitored data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides an expandable log data storage method and device, and relates to the technical field of computers, wherein the expandable log data storage method comprises the following steps: firstly, comparing the change rate of each field in the log data with the preset standard change rate, then when the change rate is smaller than the standard change rate, putting the field into a field dictionary for storage, storing different key values in each field by a field ID, storing different values in each field by field contents, when the change rate is larger than or equal to the standard change rate, putting the field into a pattern dictionary for storage, storing different key values in each field by the pattern ID, storing different values in each field by the pattern contents, namely dividing the log data into a part with a small change rate and a part with a large change rate, and storing the key values in the field in a dictionary form, thereby avoiding the problem of redundant information in the storage process.
Description
Technical Field
The invention relates to the technical field of computers, in particular to an extensible log data storage method and device.
Background
The log data (English is called log data) is the inherent meaning of a log, i.e. the log data is the information in a log used to indicate why the log is generated. General log data comprises information such as time, client IP, server IP, client hostname, server hostname, user, operation and the like, the log storage equipment divides the data into strips after receiving the log data or stores the data as a disk file as a whole, and the specific storage structure of the log storage equipment is shown in FIG. 1.
However, most of the log data are duplicated, such as IP, port, host name, operation type, etc., and storing the log data item by item during the operation process may cause the part of the log data to be duplicated, thereby occupying a large amount of disk space.
In summary, at present, no effective solution exists for the problem that log data has a large amount of information redundancy in the storage process.
Disclosure of Invention
In view of this, embodiments of the present invention provide an extensible log data storage method and apparatus, which effectively solve the problem of a large amount of redundant information in the storage process of log data by setting a field dictionary and a pattern dictionary.
In a first aspect, an embodiment of the present invention provides an extensible log data storage method, including: comparing the change rate of each field in the log data with the preset standard change rate;
when the change rate is smaller than the standard change rate, the fields are placed into a field dictionary for storage, wherein the storage format of data in the field dictionary comprises field IDs and field contents, the field IDs are used for storing different key values in the fields, and the field contents are used for storing different numerical values in the fields;
and when the change rate is greater than or equal to the standard change rate, the fields are put into a pattern dictionary for storage, wherein the storage format of the data in the pattern dictionary comprises a pattern ID and pattern content, the pattern ID is used for storing different key values in the fields, and the pattern content is used for storing different numerical values in the fields.
With reference to the first aspect, an embodiment of the present invention provides a first possible implementation manner of the first aspect, where the method further includes:
the storage format of the data in the field dictionary further comprises a field counter, and when the change rate is smaller than the standard change rate, the field counter is used for storing the occurrence times of each field;
when the number of times of occurrence of each field is counted, the number of times corresponding to the counted field is extracted in the field counter.
With reference to the first aspect, an embodiment of the present invention provides a second possible implementation manner of the first aspect, where the method further includes:
the storage format of the data in the pattern dictionary further comprises a pattern counter, and when the change rate is greater than or equal to the standard change rate, the pattern counter is used for storing the occurrence times of each field;
when the number of times each field appears is counted, the number of times corresponding to the counted field is extracted in the pattern counter.
With reference to the first aspect, an embodiment of the present invention provides a third possible implementation manner of the first aspect, where the method further includes:
and respectively establishing a field dictionary and a pattern dictionary according to the prior values.
With reference to the first aspect, an embodiment of the present invention provides a fourth possible implementation manner of the first aspect, where the method further includes:
and dynamically establishing a field dictionary and a pattern dictionary according to the fields.
With reference to the first aspect, an embodiment of the present invention provides a fifth possible implementation manner of the first aspect, where the method further includes:
and when the log data come from different application programs or different devices, establishing the field dictionary and the pattern dictionary according to an open architecture and a uniform interface.
With reference to the first aspect, an embodiment of the present invention provides a sixth possible implementation manner of the first aspect, where the field ID and the pattern ID are both represented by binary codes.
In a second aspect, an embodiment of the present invention provides an extensible log data storage apparatus, including:
the change rate comparison module is used for comparing the change rate of each field in the log data with the preset standard change rate;
the field storage module is used for storing the fields in the field dictionary when the change rate is smaller than the standard change rate, wherein the storage format of the data in the field dictionary comprises field IDs (identity) and field contents, the field IDs are used for storing different key values in the fields, and the field contents are used for storing different numerical values in the fields;
and the mode storage module is used for storing the fields in the mode dictionary when the change rate is greater than or equal to the standard change rate, wherein the storage format of the data in the mode dictionary comprises a mode ID and mode contents, the mode ID is used for storing different key values in the fields, and the mode contents are used for storing different numerical values in the fields.
In a third aspect, an embodiment of the present invention further provides a terminal, including a memory and a processor, where the memory is used to store a program that supports the processor to execute the method for extensible log data storage provided in the foregoing aspect, and the processor is configured to execute the program stored in the memory.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to perform the steps of any one of the methods described above.
The embodiment of the invention provides an expandable log data storage method and device, wherein the expandable log data storage method comprises the following steps: firstly, comparing the change rate of each field in the log data with the preset standard change rate, wherein the preset standard change rate is a reference value, the value of the change rate can be flexibly set according to the actual use scene, after the comparison process, when the change rate is smaller than the standard change rate, the field is placed in a field dictionary for storage, wherein the storage format of the data in the field dictionary comprises a field ID and field contents, the field ID is used for storing different key values in each field, correspondingly, the same key value is represented by the same field ID in the field dictionary, so that the space waste caused by storing the key values in the field dictionary one by one can be saved, the field contents are used for storing different values in each field, and then, when the change rate is larger than or equal to the standard change rate, the field is placed in a pattern dictionary for storage, in this application, the storage format of the data in the pattern dictionary includes a pattern ID and pattern contents, and the pattern ID is used to store different key values in respective fields, and accordingly, the same key values are represented by the same pattern ID in the pattern dictionary, thereby saving the space waste caused by storing key values one by one in a pattern dictionary, storing different key values in each field by a pattern ID, storing different numerical values in each field by pattern content, storing log data by adopting the key value-numerical value storage mode in the application, compared with the mode that each log data is stored one by one in the prior art, the method reduces the process of recording a large amount of log data repeatedly, effectively saves the size of a disk space occupied by log data storage, and further avoids the phenomenon of large amount of redundancy of information in the storage process of the log data.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 illustrates a prior art structure of a log data store;
FIG. 2 illustrates a first flowchart of a method for extensible log data storage provided by an embodiment of the present invention;
FIG. 3 illustrates a second flowchart of a method for scalable log data storage provided by embodiments of the present invention;
FIG. 4 is a connection diagram illustrating the structure of a scalable log data storage device according to an embodiment of the present invention.
Icon: 11-a rate of change comparison module; 12-field storage module; 13-pattern storage module.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
Currently, in the log data storage process, information of log data time, client IP, server IP, client hostname, server hostname, user, operation, and the like needs to be put into a storage space one by one for storage, that is, after log data is received by log storage equipment, the data is striped, or stored as a disk file as a whole. Therefore, most repeated data in the log data, such as IP, ports, host names, operation types and the like, can be repeatedly recorded in the storage process, so that a large amount of disk space is occupied, and serious information redundancy is caused while a large amount of storage space is wasted.
Based on this, embodiments of the present invention provide an extensible log data storage method and apparatus, which are described below by way of embodiments.
Example 1
Referring to fig. 2 and fig. 3, the method for storing extensible log data provided by this embodiment specifically includes the following steps:
step S101: and comparing the change rate of each field in the log data with the preset standard change rate. Here, the predetermined standard change rate is a priori value, and the size of the priori value is related to information such as the size of a storage space and the requirement of information redundancy. The size of the change rate of the fields is compared in order to distinguish the fields in the log data.
Step S102: when the change rate is smaller than the standard change rate, the field is put into the field dictionary for storage, namely, the field with smaller change is put into the field dictionary for storage, it should be noted that, the storage format of the data in the field dictionary includes field ID and field content, and in the implementation process, the field ID is used to store different key values in each field, i.e., each different key value occupies one field ID, and the same key value occupies the same field ID, so that, in practice, the name of a key can be represented by the same field ID when it is repeated multiple times, thereby effectively saving the space of the disk, and the specific numerical value corresponding to each key value is respectively stored, that is, the field contents are used to store different values in each field, and when the log data is searched, the combination can be performed through the field ID and the corresponding field contents.
Step S103: when the change rate is greater than or equal to the standard change rate, the field is put into a pattern dictionary for storage, that is, for the storage of the field with larger change, the field is put into the field dictionary, it should be noted here that the storage format of the data in the pattern dictionary includes the pattern ID and the pattern content, the pattern ID may also be generally referred to as a fixed part, the fixed part may be represented by a string pattern, and the data of the fixed part is less changed or limited for a specific system, therefore, a pattern dictionary may be established, the pattern ID is used to store different key values in each field, that is, each different key value occupies a pattern ID, the same key value occupies the same pattern ID, so that in the implementation process, when the key values are repeated many times, the name of the key value can be represented by the same pattern ID, thereby effectively saving the space of the disk, and the specific value corresponding to each key value is stored separately, the schema contents are used to store different values in the various fields. I.e. a combination of a series of pattern IDs and more varied data.
In addition, the expandable log data storage method further comprises the following steps:
step S001: the storage format of the data in the field dictionary further comprises a field counter, namely in the implementation process, each field in the log data is represented in the form of a key value-reference count, and for the storage modes of the two fields, when the change rate is smaller than the standard change rate, the field counter is used for storing the occurrence number of each field, namely the total number of each identical field ID stored in the field counter, in the implementation process, the field IDs corresponding to the fields are respectively counted, and when repeated field IDs occur again, the count is increased by one again.
Step S002: when the number of times of occurrence of each field is counted, the number of times corresponding to the counted field is extracted in the field counter. When other application terminals want to call the occurrence times of the field ID, the times of the field ID corresponding to the counted field are directly extracted from the field counter, and the method is convenient and fast.
Note that, the field ID and the pattern ID are both expressed by binary codes. In the digital circuit, a binary number is a number represented by a binary notation system, i.e., a notation system with a base 2, and is generally represented by two different symbols 0 and 1. The binary code is used for representing the code and is convenient for the computer to store and call.
In addition, in the prior art, in order to check the problem through log data, the occurrence frequency of a certain data in the log needs to be counted, so that when the occurrence frequency of a certain data in the log is counted, all data needs to be scanned and counted, and the efficiency is very low. The expandable log data storage method further comprises the following steps:
(1) the storage format of the data in the pattern dictionary further comprises a pattern counter, namely in the implementation process, each field in the log data is represented in the form of key value-reference counting, when the change rate is greater than or equal to the standard change rate, the pattern counter is used for storing the occurrence number of each field, namely the total number of each identical pattern ID stored in the pattern counter, in the implementation process, the pattern IDs corresponding to the fields are respectively counted, when repeated pattern IDs occur again, the counting is increased once again, and the counting is very convenient.
(2) When the number of times each field appears is counted, the number of times corresponding to the counted field is extracted in the pattern counter. Similarly, when other application terminals want to call the occurrence times of the mode ID, the mode ID times corresponding to the counted fields are directly extracted from the mode counter, and the method is convenient and fast.
In addition, the expandable log data storage method further comprises the following steps: and respectively establishing a field dictionary and a pattern dictionary according to the prior values. A method for establishing a field dictionary and a pattern dictionary is described, namely, the field dictionary and the pattern dictionary are established in advance according to a plurality of fields in previous log data, namely, prior values. The field dictionary and the pattern dictionary are respectively established through the prior value and the occurrence frequency thereof, so that the consumption can be saved for processing the log data occurring in the application operation, and the log data occurring in the operation can be conveniently stored after being directly matched with the fields from the field dictionary and the pattern dictionary.
In addition, the expandable log data storage method further comprises the following steps: and dynamically establishing a field dictionary and a pattern dictionary according to the fields. Another method of field dictionary and schema dictionary establishment is described herein, namely, a field dictionary and a schema dictionary are established by field dynamics, fields of log data occurring in application running are dynamically added to the field dictionary and the schema dictionary, and the field dictionary and the schema dictionary become huge as the log data is generated. Since the field dictionary and the pattern dictionary are established and the corresponding log data fields are required to be stored during the running of the application, the implementation method has a slow processing speed.
In addition, the expandable log data storage method further comprises the following steps: and when the log data come from different application programs or different devices, establishing the field dictionary and the pattern dictionary according to an open architecture and a uniform interface. It should be noted here that the open architecture has portability and tailorability of application systems, strong interoperability between nodes on the network, and is easily available. The field dictionary and the pattern dictionary are established through an open architecture and a uniform interface, so that the migration and the operation among different application programs are facilitated.
In summary, the extensible log data storage method provided in this embodiment includes: firstly, comparing the change rate of each field in the log data with the preset standard change rate, then when the change rate is smaller than the standard change rate, putting the fields into a field dictionary for storage, in the embodiment, the storage format of the data in the field dictionary comprises field IDs and field contents, the field IDs are used for storing different key values in each field, the field contents are used for storing different numerical values in each field, and when the change rate is larger than or equal to the standard change rate, putting the fields into a pattern dictionary for storage, correspondingly, the storage format of the data in the pattern dictionary comprises pattern IDs and pattern contents, the pattern IDs are used for storing different key values in each field, the pattern contents are used for storing different numerical values in each field, and each field in the log data is stored separately through the form of the dictionary, therefore, the phenomenon that repeated fields are stored for multiple times is effectively avoided, a large amount of disk space is saved, information redundancy of log data in the storage process is reduced to a great extent, and the method is convenient and fast.
Example 2
Referring to fig. 4, the present embodiment provides an expandable log data storage apparatus including: and the change rate comparison module 11 is used for comparing the change rate of each field in the log data with the preset standard change rate.
And the field storage module 12 is configured to, when the change rate is smaller than the standard change rate, place the field into a field dictionary for storage, where a storage format of data in the field dictionary includes a field ID and field contents, the field ID is used to store different key values in each field, and the field contents are used to store different values in each field.
And the pattern storage module 13 is configured to, when the change rate is greater than or equal to the standard change rate, put the fields into a pattern dictionary for storage, where a storage format of data in the pattern dictionary includes a pattern ID and pattern content, the pattern ID is used to store different key values in the fields, and the pattern content is used to store different values in the fields.
The expandable log data storage device provided by the embodiment of the invention has the same technical characteristics as the expandable log data storage method provided by the embodiment, so that the same technical problems can be solved, and the same technical effects can be achieved.
An embodiment of the present invention further provides a terminal, including a memory and a processor, where the memory is used to store a program that supports the processor to execute the method of the above embodiment, and the processor is configured to execute the program stored in the memory.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to perform the steps of any one of the above methods.
It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. The implementation principle and the generated technical effect of the expandable log data storage method and the device provided by the embodiment of the invention are the same as those of the method embodiment, and for brief description, the corresponding content in the method embodiment can be referred to where the device embodiment is not mentioned.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, each functional module or unit in each embodiment of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions or without necessarily implying any relative importance. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. An expandable log data storage method, comprising:
comparing the change rate of each field in the log data with the preset standard change rate;
when the change rate is smaller than the standard change rate, the fields are placed into a field dictionary for storage, wherein the storage format of data in the field dictionary comprises field IDs and field contents, the field IDs are used for storing different key values in the fields, and the field contents are used for storing different numerical values in the fields;
and when the change rate is greater than or equal to the standard change rate, the fields are placed into a pattern dictionary for storage, wherein the storage format of data in the pattern dictionary comprises a pattern ID and pattern content, the pattern ID is used for storing different key values in the fields, and the pattern content is used for storing different numerical values in the fields.
2. The scalable log data storage method of claim 1, further comprising:
the storage format of the data in the field dictionary further comprises a field counter, and when the change rate is smaller than the standard change rate, the field counter is used for storing the occurrence times of each field;
and when the occurrence times of each field are counted, extracting the times corresponding to the counted fields in the field counter.
3. The scalable log data storage method of claim 1, further comprising:
the storage format of the data in the pattern dictionary further comprises a pattern counter, and when the change rate is greater than or equal to the standard change rate, the pattern counter is used for storing the occurrence times of each field;
when the number of times of occurrence of each field is counted, the number of times corresponding to the counted field is extracted in the pattern counter.
4. The scalable log data storage method of claim 1, further comprising:
and respectively establishing the field dictionary and the pattern dictionary according to the prior value.
5. The scalable log data storage method of claim 1, further comprising:
and dynamically establishing the field dictionary and the pattern dictionary according to the field.
6. The scalable log data storage method of claim 1, further comprising:
and when the log data come from different application programs or different devices, establishing the field dictionary and the mode dictionary according to an open architecture and a uniform interface.
7. The scalable log data storage method of claim 1, wherein the field ID and the pattern ID are each represented in binary code.
8. An expandable log data storage device, comprising:
the change rate comparison module is used for comparing the change rate of each field in the log data with the preset standard change rate;
a field storage module, configured to, when the change rate is smaller than the standard change rate, put the field into a field dictionary for storage, where a storage format of data in the field dictionary includes a field ID and field contents, the field ID is used to store different key values in each field, and the field contents are used to store different values in each field;
and the mode storage module is used for storing the fields in a mode dictionary when the change rate is greater than or equal to the standard change rate, wherein the storage format of data in the mode dictionary comprises a mode ID and mode contents, the mode ID is used for storing different key values in the fields, and the mode contents are used for storing different numerical values in the fields.
9. A terminal, comprising a memory for storing a program that enables the processor to perform the method of any of claims 1 to 7 and a processor configured to execute the program stored in the memory.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of the claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711416857.1A CN108038018B (en) | 2017-12-22 | 2017-12-22 | Extensible log data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711416857.1A CN108038018B (en) | 2017-12-22 | 2017-12-22 | Extensible log data storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108038018A CN108038018A (en) | 2018-05-15 |
| CN108038018B true CN108038018B (en) | 2020-09-29 |
Family
ID=62100835
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711416857.1A Active CN108038018B (en) | 2017-12-22 | 2017-12-22 | Extensible log data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108038018B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111107420B (en) * | 2018-10-25 | 2022-08-16 | 武汉斗鱼网络科技有限公司 | Medal loading method, storage medium, electronic device and system of live APP |
| CN111913922B (en) * | 2019-05-10 | 2024-02-20 | 北京百度网讯科技有限公司 | Binary structured log generation method, device, equipment and storage medium |
| CN110474807B (en) * | 2019-08-16 | 2022-06-21 | 北京云中融信网络科技有限公司 | Log processing method and device |
| CN110516122B (en) * | 2019-09-03 | 2022-05-13 | 网易（杭州）网络有限公司 | Embedded data merging method and device and electronic equipment |
| CN110797082A (en) * | 2019-10-24 | 2020-02-14 | 福建和瑞基因科技有限公司 | Method and system for storing and reading gene sequencing data |
| CN112181765A (en) * | 2020-09-25 | 2021-01-05 | 武汉光庭信息技术股份有限公司 | Navigation map capacity monitoring method and system |
| CN115100757B (en) * | 2022-06-20 | 2023-05-09 | 重庆长安汽车股份有限公司 | Method and device for storing automobile data, vehicle and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101539921A (en) * | 2008-03-19 | 2009-09-23 | 国际商业机器公司 | Optimal writing method, optimal writing equipment and data generating and writing equipment |
| CN104283723A (en) * | 2014-10-31 | 2015-01-14 | 北京蓝汛通信技术有限责任公司 | Network access log processing method and device |
| US9542430B1 (en) * | 2015-10-30 | 2017-01-10 | International Business Machines Corporation | Ensuring data maintenance within a LDAP environment |
| CN106354617A (en) * | 2016-08-29 | 2017-01-25 | 广州华多网络科技有限公司 | Program compression log file output method and device |
| CN107133151A (en) * | 2017-05-24 | 2017-09-05 | 努比亚技术有限公司 | A kind of daily record data processing method, equipment and computer-readable recording medium |
-
2017
- 2017-12-22 CN CN201711416857.1A patent/CN108038018B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101539921A (en) * | 2008-03-19 | 2009-09-23 | 国际商业机器公司 | Optimal writing method, optimal writing equipment and data generating and writing equipment |
| CN104283723A (en) * | 2014-10-31 | 2015-01-14 | 北京蓝汛通信技术有限责任公司 | Network access log processing method and device |
| US9542430B1 (en) * | 2015-10-30 | 2017-01-10 | International Business Machines Corporation | Ensuring data maintenance within a LDAP environment |
| CN106354617A (en) * | 2016-08-29 | 2017-01-25 | 广州华多网络科技有限公司 | Program compression log file output method and device |
| CN107133151A (en) * | 2017-05-24 | 2017-09-05 | 努比亚技术有限公司 | A kind of daily record data processing method, equipment and computer-readable recording medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108038018A (en) | 2018-05-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108038018B (en) | Extensible log data storage method and device | |
| US9836346B2 (en) | Error troubleshooting using a correlated knowledge base | |
| CN106547644B (en) | Incremental backup method and equipment | |
| WO2014037767A1 (en) | Multi-level inline data deduplication | |
| US8301650B1 (en) | Bloom filter compaction | |
| CN106649708A (en) | Method and device for storing data | |
| CN108932268B (en) | Data processing method and device | |
| CN107480205B (en) | Method and device for partitioning data | |
| US20140019586A1 (en) | Methods for format preserving and data masking and devices thereof | |
| CN105900093A (en) | Keyvalue database data table updating method and data table updating device | |
| CN109753573B (en) | Processing method and device for constructing preset model based on graph database | |
| CN111277461A (en) | Method, system and equipment for identifying content distribution network node | |
| CN106933916B (en) | JSON character string processing method and device | |
| US11126520B2 (en) | Skew detector for data storage system | |
| CN111753019B (en) | Data partitioning method and device applied to data warehouse | |
| CN108363727B (en) | Data storage method and device based on ZFS file system | |
| CN105204782B (en) | A kind of method and device for realizing data storage | |
| CN111090623B (en) | Data auditing method and device, electronic equipment and storage medium | |
| CN105589683B (en) | Sample extraction method and device | |
| CN113132241B (en) | ACL template dynamic configuration method and device | |
| CN110990799A (en) | Data processing method, device and system for anti-crawler and storage medium | |
| CN113420000A (en) | Storage management method, device, equipment and machine-readable storage medium | |
| CN112463741A (en) | Cleaning method for aggregated large files and related equipment | |
| CN106326425B (en) | Data classification treating method and apparatus | |
| US20240078222A1 (en) | Selective Addition of Datum to a Tree Data Structure |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 310000 Room 608, building 5, No. 998, Wenyi West Road, Wuchang Street, Yuhang District, Hangzhou City, Zhejiang Province Applicant after: Flash it Co.,Ltd. Address before: 310000 401A 5, 998 West Wen Yi Road, Wuchang Street, Yuhang District, Hangzhou, Zhejiang. Applicant before: HANGZHOU SECSMART INFORMATION TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20180515 Assignee: Hangzhou Jintou Finance Leasing Co.,Ltd. Assignor: Flash it Co.,Ltd. Contract record no.: X2022980028282 Denomination of invention: Scalable log data storage method and device Granted publication date: 20200929 License type: Exclusive License Record date: 20230112 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Scalable log data storage method and device Effective date of registration: 20230115 Granted publication date: 20200929 Pledgee: Hangzhou Jintou Finance Leasing Co.,Ltd. Pledgor: Flash it Co.,Ltd. Registration number: Y2023980031389 |
|
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: 311121 Room 101, Building 9, No. 998, Wenyi West Road, Wuchang Subdistrict, Yuhang District, Hangzhou City, Zhejiang Province Patentee after: Flash it Co.,Ltd. Address before: 310000 Room 608, Building No. 998 Wenyi West Road, Wuchang Street, Yuhang District, Hangzhou City, Zhejiang Province Patentee before: Flash it Co.,Ltd. |
|
| EC01 | Cancellation of recordation of patent licensing contract | ||
| EC01 | Cancellation of recordation of patent licensing contract |
Assignee: Hangzhou Jintou Finance Leasing Co.,Ltd. Assignor: Flash it Co.,Ltd. Contract record no.: X2022980028282 Date of cancellation: 20240327 |
|
| PC01 | Cancellation of the registration of the contract for pledge of patent right | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Granted publication date: 20200929 Pledgee: Hangzhou Jintou Finance Leasing Co.,Ltd. Pledgor: Flash it Co.,Ltd. Registration number: Y2023980031389 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Scalable method and device for storing log data Granted publication date: 20200929 Pledgee: The Bank of Hangzhou branch of Limited by Share Ltd. sea park Pledgor: Flash it Co.,Ltd. Registration number: Y2024980014339 |