CN102929936A - Log recording method, log inquiring method and system - Google Patents
Log recording method, log inquiring method and system Download PDFInfo
- Publication number
- CN102929936A CN102929936A CN2012103661952A CN201210366195A CN102929936A CN 102929936 A CN102929936 A CN 102929936A CN 2012103661952 A CN2012103661952 A CN 2012103661952A CN 201210366195 A CN201210366195 A CN 201210366195A CN 102929936 A CN102929936 A CN 102929936A
- Authority
- CN
- China
- Prior art keywords
- daily record
- play amount
- index file
- log
- field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a log system. The system comprises a log recording device and a log inquiring device. According to the log recording device, a recording unit records a shift quantity of a writing position of a log to a starting position of a log data file; and an index file writing unit writes a field value of every field, the log shift quantity and/or log length into corresponding index files in a correlating mode. According to the log inquiring device, a log shift quantity acquiring unit reads all log shift quantities which meet separate inquiring conditions from the corresponding index files according to a plurality of inquiring conditions; an intersection calculating unit calculates intersections of the read log shift quantities; and a log reading unit reads logs which meet the inquiring conditions from a log data file according to the log shift quantities in the intersections. By using the log system, the inquiring performance of the logs can be obviously improved, and storage space occupied by the index files can be reduced.
Description
Technical field
The present invention relates to the log processing field, more specifically, the log system that relates to a kind of log recording method and device, a kind of log inquiring method and device and have said apparatus.
Background technology
Log system is responsible for receiving the daily record data that gateway sends, and stores the daily record data that receives according to certain format, and the query function to daily record data is provided.The daily record data that gateway sends generally includes three types: admin log, running log and access log.Admin log record be the keeper of gateway to the bookkeeping of gateway, the running log record be event in the operational process of gateway own, and the access log record is Access Events by gateway, a common session meeting produces an access log.Generally, the admin log of gateway and running log can be not a lot, and the access log per second may reach thousands of.
Fig. 1 shows an example of the daily record data of gateway transmission.As shown in Figure 1, the field that comprises in this daily record data is the time, module, grade, source IP, purpose IP, destination interface, agreement and action.
Present log system mainly operates on Linux/Unix or the Windows server, and the employing relational database is realized storage and the query function to daily record data.In addition, in order to improve the query performance of daily record, also can set up index for relational database.
Yet, the poor performance of the log system of realizing based on relational database.On the one hand, in the log system of realizing based on relational database, the insertion poor performance of daily record data.If set up index for the query performance that improves daily record to relational database, then the insertion performance of daily record data can further descend.In addition, the insertion performance of relational database also can descend along with increasing of record.On the other hand, even be the Database index, along with the daily record of recording increases, query performance still can't meet the demands.
Storage space (for example, the disk space) utilization factor of the log system of realizing based on relational database in addition, is low.In the log system of realizing based on relational database, in order to improve the query performance of daily record, be necessary for relational database foundation search and draw, this moment is the shared storage space of increase relational database files obviously, thereby so that the storage space utilization factor of log system is low.
Summary of the invention
In view of above-mentioned, the object of the present invention is to provide a kind of log recording method and device, the method and device can reduce the shared storage space of index file.
Another object of the present invention is to provide a kind of log inquiring method and device, the method and device can improve the query performance of daily record.
Another object of the present invention is to provide the log system that comprises said apparatus.
According to an aspect of the present invention, provide a kind of log recording method, having comprised: the field value of from the daily record of obtaining, obtaining each field in this daily record; The daily record of obtaining is inserted in the daily record data file, and described daily record data file and a plurality of index file that is pre-created are stored in the memory storage explicitly; Record the side-play amount of the reference position of the described daily record data file of writing position distance that aims at described day in the described daily record data file; And the side-play amount of the field value of each field and daily record is written in the corresponding index file explicitly, index file of each Related fields, wherein, at least one index file in described a plurality of index file is the master index file, in described master index file, also write explicitly the length of described daily record with the side-play amount of the field value of corresponding field and daily record.
In one or more examples aspect above-mentioned, in non-master index file, also write explicitly the length of described daily record with the side-play amount of the field value of corresponding field and daily record.
In one or more examples aspect above-mentioned, described field comprises time type field, enumeration type field and types of variables field, for the time type field, the field value of obtaining is zero-time and the concluding time of daily record, for the enumeration type field, the field value of obtaining is the enumeration type of field, and for the types of variables field, the field value of obtaining is variate-value.
In one or more examples aspect above-mentioned, the described index file corresponding with time index, index file described and the enumeration type Related fields adopt the array mode, and the index file of described and types of variables Related fields adopts the mode of HASH table.
According to a further aspect in the invention, provide a kind of log recording apparatus, having comprised: field value acquiring unit, for the field value of obtaining each field this daily record from the daily record of obtaining; The daily record plug-in unit is used for the daily record of obtaining is inserted into the daily record data file, and described daily record data file and a plurality of index file that is pre-created are stored in the memory storage explicitly; Record cell is for the side-play amount of the reference position that records the described daily record data file of writing position distance that aims at described daily record data file described day; And index file writing unit, be used for the field value of each field and the side-play amount of daily record are written to corresponding index file explicitly, index file of each Related fields, wherein, at least one index file in described a plurality of index file is the master index file, when the index file of correspondence was the master index file, described index file writing unit also was used for the length of described daily record and the field value of corresponding field and the side-play amount of daily record are written to described master index file explicitly.
In one or more examples aspect above-mentioned, in non-master index file, also write explicitly the length of described daily record with the side-play amount of the field value of corresponding field and daily record.
According to a further aspect in the invention, a kind of log inquiring method is provided, record is carried out in described daily record as described above method, described log inquiring method comprises: according to the multiple queries condition of user's input, in the index file of correspondence, read all daily record side-play amounts that satisfy the single query condition, to obtain the daily record side-play amount set corresponding with this single query condition; Calculate all daily record side-play amount intersection of sets collection that read; Daily record side-play amount intersection of sets collection according to calculating obtains the master index file corresponding with the daily record side-play amount in this common factor; And according to the daily record side-play amount in the master index file that obtains and daily record length, from described daily record data file, read the daily record of satisfying the multiple queries condition.
In one or more examples aspect above-mentioned, calculate all the daily record side-play amount intersection of sets collection read and comprise: to the daily record side-play amount set of obtaining, sort from small to large according to the number of the data element that comprises; And for the set of the daily record side-play amount after the ordering, the first two daily record side-play amount intersection of sets collection in the daily record side-play amount set after the iterative computation ordering, until finish iterative computation for the daily record side-play amount set of being obtained to some extent, wherein, the result of each iterative computation replaces two daily record side-play amount set carrying out this iterative computation, first daily record side-play amount set during as next iterative computation.
In one or more examples aspect above-mentioned, common factor between two daily record side-play amounts are gathered adopts dichotomy to calculate, and when carrying out dichotomy, for first element of formerly daily record side-play amount set, the beginning of dichotomy and end position are respectively first element and last element in rear daily record side-play amount set; For the element except first element of described formerly daily record side-play amount set, the starting position of dichotomy is the as a result position of a front binary search, and end position is last element of formerly daily record side-play amount set.
According to a further aspect in the invention, a kind of log query device is provided, record is carried out in described daily record as described above method, described log query device comprises: daily record side-play amount acquiring unit, be used for the multiple queries condition according to user's input, in the index file of correspondence, read all daily record side-play amounts that satisfy the single query condition, to obtain the daily record side-play amount set corresponding with this single query condition; The common factor computing unit is used for calculating all daily record side-play amount intersection of sets collection that read; Master index file acquisition unit is used for according to the daily record side-play amount intersection of sets collection that calculates, and obtains the master index file corresponding with the daily record side-play amount in this common factor; And the log read unit, be used for daily record side-play amount and daily record length according to the master index file that obtains, from described daily record data file, read the daily record of satisfying the multiple queries condition.
In one or more examples aspect above-mentioned, described common factor computing unit comprises: order module, be used for the daily record side-play amount set to obtaining, and sort from small to large according to the number of the data element that comprises; And iterative computation module, be used for for the set of the daily record side-play amount after the ordering, the first two daily record side-play amount intersection of sets collection in the daily record side-play amount set after the iterative computation ordering, until finish iterative computation for the daily record side-play amount set of being obtained to some extent, wherein, the result of each iterative computation replaces two daily record side-play amount set carrying out this iterative computation, first daily record side-play amount set during as next iterative computation.
According to a further aspect in the invention, provide a kind of log system, having comprised: aforesaid log recording apparatus; And aforesaid log query device.
Utilize above-mentioned log recording method and device, according to the mode that the daily record data file is combined with the daily record index file record is carried out in daily record, thereby need not to store based on relational database, reduce thus the shared storage space of index file.
Utilize above-mentioned log inquiring method and device, read the daily record of satisfying the multiple queries condition by the common factor that obtains between the side-play amount that satisfies the multiple queries condition, can significantly improve the query performance of daily record.
In order to realize above-mentioned and relevant purpose, one or more aspects of the present invention comprise the feature that the back will describe in detail and particularly point out in the claims.Following explanation and accompanying drawing describe some illustrative aspects of the present invention in detail.Yet, the indication of these aspects only be some modes that can use in the variety of way of principle of the present invention.In addition, the present invention is intended to comprise all these aspects and their equivalent.
Description of drawings
According to following detailed description of carrying out with reference to accompanying drawing, above and other purpose of the present invention, feature and advantage will become more apparent.In the accompanying drawings:
Fig. 1 shows an example of daily record;
Fig. 2 shows the according to an embodiment of the invention process flow diagram of log recording method;
Fig. 3 shows the according to an embodiment of the invention block diagram of log recording apparatus;
Fig. 4 shows the process flow diagram according to the log inquiring method of the embodiment of the invention;
Fig. 5 shows the process flow diagram of an example of the computation process that the side-play amount among Fig. 4 occurs simultaneously;
Fig. 6 shows the according to an embodiment of the invention block diagram of log query device;
Fig. 7 shows the block diagram of a realization example of the common factor computing unit among Fig. 6; With
Fig. 8 shows the according to an embodiment of the invention block diagram of log system.
Identical label is indicated similar or corresponding feature or function in institute's drawings attached.
Embodiment
Various aspects of the present disclosure are described below.Should be understood that the instruction of this paper can be with varied form imbody, and disclosed any concrete structure, function or both only are representational in this article.Based on the instruction of this paper, those skilled in the art should be understood that an aspect disclosed herein can be independent of any other side and realize, and the two or more aspects in these aspects can make up according to variety of way.For example, can use the aspect of any number described in this paper, implement device or hands-on approach.In addition, can use other structure, function or except one or more aspects described in this paper or be not the 26S Proteasome Structure and Function of one or more aspects described in this paper, realize this device or put into practice this method.In addition, any aspect described herein can comprise at least one element of claim.
The below describes each embodiment of the present invention with reference to Fig. 2 to Fig. 7.
Fig. 2 shows the according to an embodiment of the invention process flow diagram of log recording method.
As shown in Figure 2, after obtaining daily record, for example, receive daily record data from gateway after, in step S210, from the daily record of obtaining, obtain the field value of each field in this daily record.Usually, daily record is made of a plurality of fields.Field in the described daily record can comprise time type field, enumeration type field and types of variables field.Described time type field is used for the temporal information of log, such as the information about the time of origin of daily record.Described enumeration type field can be the field of daily record rank field, module name field etc., is used for log grade point, module title etc.The types of variables field can be the field such as IP address field, username field etc., is used for record IP address, user name etc.For the time type field, the field value of obtaining is the time of origin of daily record.For the enumeration type field, the field value of obtaining is the enumeration type value of field, such as daily record grade point, module title etc.For the types of variables field, the field value of obtaining is variate-value, such as IP address, user name etc.
Then, at step S220, the daily record of obtaining is inserted in the daily record data file.Here, described daily record data file is used for log data, and with being stored in the memory storage that the index file that is pre-created is associated.Usually, in the daily record data file, every delegation writes a daily record.In other example, in delegation, also can write many daily records.The described index file that is pre-created can comprise the index file corresponding with the time type field (hereinafter being called the time index file), with the index file of enumeration type Related fields and with the index file of types of variables Related fields.Described index file can comprise master index file and non-master index file.Record field value, side-play amount and daily record length in the master index file.Record field value and side-play amount in non-master index file.In addition, alternatively, in non-master index file, log length not also can log length.In the described index file that is pre-created, comprise at least a master index file, be the master index file such as the time index file.In other example of the present invention, the described index file that is pre-created can all be the master index file.Preferably, only comprise a master index file in the described index file that is pre-created.
Then, at step S230, record the side-play amount of the reference position of the described daily record data file of writing position distance that aims at described day in the described daily record data file.
Then, at step S240, first field from daily record begins, and for each field, judges whether master index file of the corresponding index file of this field.For example, according to the type of field, determine whether master index file of the corresponding index file of this field.
In step S240, judge when the corresponding index file of this field is the master index file, at step S250, the side-play amount of the field value of this field of obtaining, daily record and daily record length are written in the master index file with this Related fields explicitly.In step S240, judge when the corresponding index file of this field is non-master index file, in step S260, the side-play amount of the field value of this field of obtaining and daily record is written in the index file with this Related fields explicitly.
For dissimilar index files, adopt different forms.In an example of the present invention, time index file and enumerate index file and adopt the array mode.When time index file and enumeration type file were the master index file, each array element was three-number set, such as, in the time index file, side-play amount and the daily record length of the zero-time of a daily record of record and concluding time, daily record.In enumerating index file, enumeration type (such as daily record grade, module title), side-play amount and the daily record length of a daily record of record.
Such as, for the time index file, if one the time of daily record generation is 0x5059bd05, side-play amount in the daily record data file is 256, length is 100, then need to add 12 bytes (binary mode): 0x5059bd050x000001000x00000064 in the time index file, the time index file is the array of following data structure:
When time index file and enumeration type file were non-master index file, each array element was the binary array, such as, in the time index file, the zero-time of a daily record of record and the side-play amount of concluding time and daily record.In enumerating index file, enumeration type (such as daily record grade, module title) and the side-play amount of a daily record of record.The variable index file adopts HASH table mode.When the variable index file is the master index file, each nodes records of HASH table side-play amount and the daily record length of the key assignments of daily record field all daily records corresponding with this key assignments.For example each nodes records in the user name index file HASH table user name, this user aim at side-play amount and daily record length in the daily record data file all corresponding days.When the variable index file is non-master index file, each nodes records of HASH table the key assignments of daily record field and the side-play amount of all daily records corresponding with this key assignments.
Then, in step S270, judge whether all to finish above-mentioned index file ablation process for all fields in the daily record.If finish, then flow process finishes.Otherwise, in step S280, forward next field to, then turn back to step S240.
Fig. 2 only is a realization example of log record process of the present invention.For above-mentioned example, can also carry out various modification.
Such as, in a modification of the present invention, can think that all index files all are the master index files, thereby omit step S240 and step S260 among Fig. 2.In another modification of the present invention, before step S240, can also comprise deterministic process, be used for for each field, judge in the index file that is pre-created, whether there is corresponding index file.When being judged as the index file that does not have correspondence, creating corresponding index file, and the field value of this field and the side-play amount of daily record are written in the index file that creates explicitly.When having corresponding index file, flow process proceeds to step S240.
Fig. 3 shows the according to an embodiment of the invention block diagram of log recording apparatus 300.As shown in Figure 3, log recording apparatus 300 comprises field value acquiring unit 310, daily record plug-in unit 320, record cell 330 and index file writing unit 340.
Field value acquiring unit 310 is used for obtaining from the daily record of obtaining the field value of each field this daily record.Daily record plug-in unit 320 is used for the daily record of obtaining is inserted into the daily record data file, and described daily record data file and a plurality of index file that is pre-created are stored in the memory storage explicitly.At least one index file in described a plurality of index file that is pre-created is the master index file.
Record cell 330 is used for recording the side-play amount of the reference position of the described daily record data file of writing position distance that aims at described daily record data file described day.
Index file writing unit 340 is used for the side-play amount of the field value of each field and daily record is written to corresponding index file, index file of each Related fields explicitly.In addition, when the index file of correspondence was the master index file, described index file writing unit 340 also was used for the length of described daily record and the field value of corresponding field and the side-play amount of daily record are written to described master index file explicitly.In addition, in another embodiment of the present invention, for non-master index file, also the length of described daily record and the field value of corresponding field and the side-play amount of daily record can be written in the described non-master index file explicitly.
In another example of the present invention, described log recording apparatus can also comprise: the judging unit (not shown), be used for for each field, before the side-play amount with the field value of this field and daily record is written in the corresponding index file explicitly, judge in the index file that is pre-created, whether there is corresponding index file; And the creating unit (not shown), be used for when judgment unit judges goes out not exist corresponding index file, creating corresponding index file.When not having corresponding index file, described index file writing unit 340 is configured to the side-play amount of the field value of this field and daily record is written in the index file that creates explicitly, and when having corresponding index file, described index file writing unit 340 is configured to the side-play amount of the field value of this field and daily record is written in this corresponding index file explicitly.
Fig. 4 shows the process flow diagram according to the log inquiring method of the embodiment of the invention.
As shown in Figure 4, utilizing the multiple queries condition when as above described daily record of recording is inquired about with reference to Fig. 2, at first, at step S410, multiple queries condition according to user's input, in the index file of correspondence, read all daily record side-play amounts that satisfy the single query condition, to obtain the daily record side-play amount set corresponding with this single query condition.Described multiple queries condition can comprise such as with the querying condition of time correlation, the querying condition relevant with given level, the querying condition relevant with user name etc.For the querying condition of given zero-time and concluding time daily record, in the time index file, read all side-play amounts that satisfy this querying condition, and form daily record side-play amount set offsets_1[].For the querying condition of given level, in daily record hierarchy index file, read the side-play amount of all daily records of satisfying this given level, and form daily record side-play amount set offsets_2[].For the querying condition that provides user name, in the user name index file, read the side-play amount of all daily records of satisfying designated user, and form daily record side-play amount offsets_n[].The side-play amount offsets_1[here], offsets_2[] and side-play amount offsets_n[] normally side-play amount set.
Then, at step S420, calculate all daily record side-play amount intersection of sets collection that read, namely ask side-play amount set offsets_1[], offsets_2[] ... offsets_n[] common factor.
After as above obtaining daily record side-play amount intersection of sets collection, at step S430, according to the daily record side-play amount intersection of sets collection that calculates, obtain the master index condition corresponding with the daily record side-play amount that comprises in this common factor.Here, may there be a plurality of daily record side-play amounts in the common factor that calculates, thereby can obtain a plurality of master index conditions.
Then, at step S440, according to the daily record side-play amount in the master index condition of obtaining and daily record length, from described daily record data file, read the daily record of satisfying the multiple queries condition.
Fig. 5 shows the process flow diagram of an example of the computation process that the side-play amount set among Fig. 4 occurs simultaneously.
As shown in Figure 5, at first, in step S421, for the set of the side-play amount obtained, sort from small to large according to the number of the array element that comprises.Suppose side-play amount set offsets_1[]＜offsets_2[]＜...＜offsets_n[].Then, in step S423, the set of first side-play amount is set to current side-play amount set, that is, with offsets_1[] be set to current side-play amount set.Then, at step S425, calculating the set of current side-play amount and next side-play amount intersection of sets collection, that is, is offsets_1[in current side-play amount set], calculate offsets_1[] and offsets_2[] common factor.Offsets_1[in the set of current side-play amount] and offsets_2[] common factor the time, calculate this common factor and offsets_3[] common factor.So analogize.
After calculating common factor, in step S427, judge whether to calculate all side-play amount intersection of sets collection.If calculate, then flow process finishes.Otherwise flow process proceeds to step S429, the common factor that calculates is gathered as current side-play amount, and flow process turns back to step S425 subsequently, proceeds to occur simultaneously and calculates.
In another example of the present invention, calculating two side-play amounts set offsets_1[] and offsets_2[] during the common factor of (supposing that both have respectively M and N element), if first side-play amount Element of a set number will adopt dichotomy to calculate common factor much smaller than second side-play amount Element of a set number (being that M is much smaller than N).Namely for each element in first side-play amount set, use dichotomy in the set of the second side-play amount, to search whether there is identical element.
Because the set of two side-play amounts all is monotonically increasing, when using dichotomy, for offsets_1[] first element, beginning and the end position of dichotomy are respectively offsets_2[] first element and last element.For offsets_1[] second element, the starting position of dichotomy is the as a result position of a front binary search, end position or offsets_2[] last element.Go down successively, can reduce thus by two minutes number of times.
Because these array elements all are monotonically increasings, calculate in theory two and increase progressively array offsets_m[], offsets_n[] (having respectively a M and N element) the time complexity best result of occuring simultaneously is o (M+N), thereby reduced to calculate the complexity of occuring simultaneously.And then improved query performance.
Fig. 6 shows the according to an embodiment of the invention block diagram of log query device 600.As shown in Figure 6, log query device 600 comprises daily record side-play amount acquiring unit 610, common factor computing unit 620, master index file acquisition unit 630 and log read unit 640.
The multiple queries condition that daily record side-play amount acquiring unit 610 is used for according to user's input reads all daily record side-play amounts that satisfy the single query condition in the index file of correspondence, to obtain the daily record side-play amount set corresponding with this single query condition.Common factor computing unit 620 is used for calculating all daily record side-play amount intersection of sets collection that read.
Master index file acquisition unit 630 is used for according to the daily record side-play amount intersection of sets collection that calculates, and obtains the master index condition corresponding with the daily record side-play amount that comprises in this common factor.
Daily record side-play amount and daily record length that log read unit 630 is used for according to the master index condition of obtaining read the daily record of satisfying the multiple queries condition from described daily record data file.
Common factor computing unit among Fig. 6 can adopt various ways to realize.Fig. 7 shows the block diagram of a realization example of the common factor computing unit among Fig. 6.As shown in Figure 7, described common factor computing unit 620 comprises order module 621 and iterative computation module 623.Order module 621 is used for the daily record side-play amount set of obtaining, and sorts from small to large according to the number of the data element that comprises.Iterative computation module 623 is used for for the set of the daily record side-play amount after the ordering, the first two daily record side-play amount intersection of sets collection in the daily record side-play amount set after the iterative computation ordering, until finish iterative computation for the daily record side-play amount set of being obtained to some extent, wherein, the result of each iterative computation replaces two daily record side-play amount set carrying out this iterative computation, first daily record side-play amount set during as next iterative computation.
Fig. 8 shows the according to an embodiment of the invention block diagram of log system 10.As shown in Figure 8, log system 10 comprises log recording apparatus 300 and log query device 600.The various modification that also can comprise log recording apparatus 300 and log query device here, in the log system 10.
Utilize above-mentioned log recording method and device, according to the mode that the daily record data file is combined with the daily record index file daily record is stored, thereby need not to store based on relational database, reduce thus the shared storage space of index file.
Utilize above-mentioned log inquiring method and device, read the daily record of satisfying the multiple queries condition by the common factor that obtains between the side-play amount that satisfies the multiple queries condition, can significantly improve the query performance of daily record.
Through overtesting, adopt the present invention, can significantly improve the performance of log system, and the disk space that index file takies is less than 20% of daily record data file.
Utilization is according to technical scheme of the present invention, and on DELL R71 server, for having 5 fields, length is the daily record of 128-256 byte, and inserting 1,000,000 daily records only needs 12 seconds.And according to the difference of querying condition, query time is between 0.3 millisecond to 5 milliseconds.And under identical hardware environment, use the technical scheme based on relational database (such as the sqlite database), in the situation of opening affairs, insert 1,000,000 37 minutes and 53 seconds times spent of record, and query time is between 3 milliseconds to 15 milliseconds.In addition, the insertion performance of sqlite database obviously descends along with the increase of inserting daily record.Adopt the solution of the present invention then almost not descend.
In addition, ask a plurality of algorithms that array is occured simultaneously that increase progressively after the above-mentioned optimization by adopting, than common algorithm performance boost 3-10 doubly, use the algorithm after optimizing, the performance boost of inquiry 40%.
Although the disclosed content in front shows exemplary embodiment of the present invention, should be noted that under the prerequisite of the scope of the present invention that does not deviate from the claim restriction, can carry out multiple change and modification.Function, step and/or action according to the claim to a method of inventive embodiments described herein do not need to carry out with any particular order.In addition, although element of the present invention can be with individual formal description or requirement, also it is contemplated that a plurality of, unless clearly be restricted to odd number.
Be described although as above described each embodiment according to the present invention with reference to figure, it will be appreciated by those skilled in the art that each embodiment that the invention described above is proposed, can also make various improvement on the basis that does not break away from content of the present invention.Therefore, protection scope of the present invention should be determined by the content of appending claims.
Claims (12)
1. log recording method comprises:
From the daily record of obtaining, obtain the field value of each field in this daily record;
The daily record of obtaining is inserted in the daily record data file, and described daily record data file and a plurality of index file that is pre-created are stored in the memory storage explicitly;
Record the side-play amount of the reference position of the described daily record data file of writing position distance that aims at described day in the described daily record data file; And
The field value of each field and the side-play amount of daily record are written in the corresponding index file explicitly, index file of each Related fields,
Wherein, at least one index file in described a plurality of index files is the master index file, in described master index file, also writes explicitly the length of described daily record with the side-play amount of the field value of corresponding field and daily record.
2. log recording method as claimed in claim 1 wherein, in non-master index file, also writes the length of described daily record explicitly with the side-play amount of the field value of corresponding field and daily record.
3. log recording method as claimed in claim 1, wherein, described field comprises time type field, enumeration type field and types of variables field, for the time type field, the field value of obtaining is zero-time and the concluding time of daily record, and for the enumeration type field, the field value of obtaining is the enumeration type of field, and for the types of variables field, the field value of obtaining is variate-value.
4. log recording method as claimed in claim 3, wherein, the described index file corresponding with time index, index file described and the enumeration type Related fields all adopt the array mode, and the index file of described and types of variables Related fields adopts the mode of HASH table.
5. log recording apparatus comprises:
Field value acquiring unit is for the field value of obtaining each field this daily record from the daily record of obtaining;
The daily record plug-in unit is used for the daily record of obtaining is inserted into the daily record data file, and described daily record data file and a plurality of index file that is pre-created are stored in the memory storage explicitly;
Record cell is for the side-play amount of the reference position that records the described daily record data file of writing position distance that aims at described daily record data file described day; And
The index file writing unit is used for the field value of each field and the side-play amount of daily record are written to corresponding index file explicitly, index file of each Related fields,
Wherein, at least one index file in described a plurality of index file is the master index file, when the index file of correspondence was the master index file, described index file writing unit also was used for the length of described daily record and the field value of corresponding field and the side-play amount of daily record are written to described master index file explicitly.
6. log recording apparatus as claimed in claim 5 in non-master index file, also writes the length of described daily record explicitly with the side-play amount of the field value of corresponding field and daily record.
7. log inquiring method, described daily record is carried out record according to the method for claim 1, and described log inquiring method comprises:
Multiple queries condition according to user's input reads all daily record side-play amounts that satisfy the single query condition in the index file of correspondence, to obtain the daily record side-play amount set corresponding with this single query condition;
Calculate all daily record side-play amount intersection of sets collection that read;
Daily record side-play amount intersection of sets collection according to calculating obtains the master index file corresponding with the daily record side-play amount in this common factor; And
According to the daily record side-play amount in the master index file that obtains and daily record length, from described daily record data file, read the daily record of satisfying described multiple queries condition.
8. log inquiring method as claimed in claim 7, wherein, calculate all daily record side-play amount intersection of sets collection that read and comprise:
To the set of the daily record side-play amount obtained, sort from small to large according to the number of the data element that comprises; And
For the set of the daily record side-play amount after the ordering, the first two daily record side-play amount intersection of sets collection in the daily record side-play amount set after the iterative computation ordering, until finish iterative computation for the daily record side-play amount set of being obtained to some extent, wherein, the result of each iterative computation replaces two daily record side-play amount set carrying out this iterative computation, first daily record side-play amount set during as next iterative computation.
9. log inquiring method as claimed in claim 8, wherein, common factor between two daily record side-play amounts are gathered adopts dichotomy to calculate, and when carrying out dichotomy calculating, first element for the set of formerly daily record side-play amount, the beginning of dichotomy and end position are respectively first element and last element in rear daily record side-play amount set, and for the element except first element of described formerly daily record side-play amount set, the starting position of dichotomy is the as a result position of a front binary search, and end position is last element of formerly daily record side-play amount set.
10. log query device, described daily record is carried out record according to the method for claim 1, and described log query device comprises:
Daily record side-play amount acquiring unit is used for the multiple queries condition according to user's input, reads all daily record side-play amounts that satisfy the single query condition in the index file of correspondence, to obtain the daily record side-play amount set corresponding with this single query condition;
The common factor computing unit is used for calculating all daily record side-play amount intersection of sets collection that read;
Master index file acquisition unit is used for according to the daily record side-play amount intersection of sets collection that calculates, and obtains the master index file corresponding with the daily record side-play amount in this common factor; And
The log read unit is used for daily record side-play amount and daily record length according to the master index file that obtains, reads the daily record of satisfying querying condition from described daily record data file.
11. log query device as claimed in claim 10, wherein, described common factor computing unit comprises:
Order module is used for the daily record side-play amount set to obtaining, and sorts from small to large according to the number of the data element that comprises; And
The iterative computation module, be used for for the set of the daily record side-play amount after the ordering, the first two daily record side-play amount intersection of sets collection in the daily record side-play amount set after the iterative computation ordering, until finish iterative computation for the daily record side-play amount set of being obtained to some extent, wherein, the result of each iterative computation replaces two daily record side-play amount set carrying out this iterative computation, first daily record side-play amount set during as next iterative computation.
12. a log system comprises:
Such as claim 5 or 6 described log recording apparatus; And
Such as claim 10 or 11 described log query devices.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210366195.2A CN102929936B (en) | 2012-09-26 | 2012-09-26 | Log recording method, log inquiring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210366195.2A CN102929936B (en) | 2012-09-26 | 2012-09-26 | Log recording method, log inquiring method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102929936A true CN102929936A (en) | 2013-02-13 |
| CN102929936B CN102929936B (en) | 2016-08-24 |
Family
ID=47644734
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210366195.2A Active CN102929936B (en) | 2012-09-26 | 2012-09-26 | Log recording method, log inquiring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102929936B (en) |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103177116A (en) * | 2013-04-08 | 2013-06-26 | 国电南瑞科技股份有限公司 | Distributed log handling and inquiring method based on two-stage index |
| CN104035839A (en) * | 2014-06-12 | 2014-09-10 | 上海交通大学 | Method for implementation of recovery of Android system private data |
| CN104794123A (en) * | 2014-01-20 | 2015-07-22 | 阿里巴巴集团控股有限公司 | Method and device for establishing NoSQL database index for semi-structured data |
| CN104834648A (en) * | 2014-02-12 | 2015-08-12 | 华为技术有限公司 | Log query method and device |
| CN106021058A (en) * | 2015-03-24 | 2016-10-12 | 想象技术有限公司 | Logging events with timestamps |
| CN106649627A (en) * | 2016-12-06 | 2017-05-10 | 杭州迪普科技股份有限公司 | Log searching method and device |
| CN106951557A (en) * | 2017-03-31 | 2017-07-14 | 北京希嘉创智教育科技有限公司 | Daily record correlating method, device and apply its computer system |
| CN107220287A (en) * | 2017-04-24 | 2017-09-29 | 东软集团股份有限公司 | For the index managing method of log query, device, storage medium and equipment |
| CN107688624A (en) * | 2017-08-18 | 2018-02-13 | 杭州迪普科技股份有限公司 | A kind of daily record index structuring method and device |
| CN107861956A (en) * | 2016-09-22 | 2018-03-30 | 杭州海康威视系统技术有限公司 | A kind of bayonet socket crosses the querying method and device of car data record |
| CN107943807A (en) * | 2016-10-13 | 2018-04-20 | 华为技术有限公司 | A kind of data processing method and storage device |
| CN108427705A (en) * | 2018-01-17 | 2018-08-21 | 平安科技（深圳）有限公司 | Electronic device, distributed system journal querying method and storage medium |
| CN108572914A (en) * | 2018-03-14 | 2018-09-25 | 口碑(上海)信息技术有限公司 | A kind of method and device generating the daily record for pressure test |
| CN110515803A (en) * | 2019-08-27 | 2019-11-29 | 联想(北京)有限公司 | For the processing method of log information, device and electronic equipment |
| CN112800006A (en) * | 2021-01-27 | 2021-05-14 | 杭州迪普科技股份有限公司 | Log storage method and device for network equipment |
| CN113342768A (en) * | 2021-06-01 | 2021-09-03 | 北京奇艺世纪科技有限公司 | Log reading method and device, electronic equipment and readable storage medium |
| CN114567503A (en) * | 2022-03-04 | 2022-05-31 | 南京联成科技发展股份有限公司 | Encryption method for centrally controlled and trusted data acquisition |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107644070B (en) * | 2017-09-13 | 2020-09-15 | 北京柠檬微趣科技股份有限公司 | Data indexing method, data query method and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101620617A (en) * | 2009-07-28 | 2010-01-06 | 中兴通讯股份有限公司 | Method and device for searching and processing log file |
| US20100023545A1 (en) * | 2008-07-25 | 2010-01-28 | Tibbo Technology, Inc. | Data logging system and method thereof for heterogeneous data |
| CN102239472A (en) * | 2008-09-05 | 2011-11-09 | Arc景象有限责任公司 | Storing log data efficiently while supporting querying |
-
2012
- 2012-09-26 CN CN201210366195.2A patent/CN102929936B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100023545A1 (en) * | 2008-07-25 | 2010-01-28 | Tibbo Technology, Inc. | Data logging system and method thereof for heterogeneous data |
| CN102239472A (en) * | 2008-09-05 | 2011-11-09 | Arc景象有限责任公司 | Storing log data efficiently while supporting querying |
| CN101620617A (en) * | 2009-07-28 | 2010-01-06 | 中兴通讯股份有限公司 | Method and device for searching and processing log file |
Cited By (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103177116B (en) * | 2013-04-08 | 2016-05-04 | 国电南瑞科技股份有限公司 | A kind of distributed information log based on two-stage index is processed and querying method |
| CN103177116A (en) * | 2013-04-08 | 2013-06-26 | 国电南瑞科技股份有限公司 | Distributed log handling and inquiring method based on two-stage index |
| CN104794123B (en) * | 2014-01-20 | 2018-07-27 | 阿里巴巴集团控股有限公司 | A kind of method and device building NoSQL database indexes for semi-structured data |
| CN104794123A (en) * | 2014-01-20 | 2015-07-22 | 阿里巴巴集团控股有限公司 | Method and device for establishing NoSQL database index for semi-structured data |
| WO2015106711A1 (en) * | 2014-01-20 | 2015-07-23 | 阿里巴巴集团控股有限公司 | Method and device for constructing nosql database index for semi-structured data |
| CN104834648A (en) * | 2014-02-12 | 2015-08-12 | 华为技术有限公司 | Log query method and device |
| CN104834648B (en) * | 2014-02-12 | 2018-07-31 | 华为技术有限公司 | Log inquiring method and device |
| CN104035839A (en) * | 2014-06-12 | 2014-09-10 | 上海交通大学 | Method for implementation of recovery of Android system private data |
| CN106021058A (en) * | 2015-03-24 | 2016-10-12 | 想象技术有限公司 | Logging events with timestamps |
| CN106021058B (en) * | 2015-03-24 | 2021-07-27 | 北欧半导体公司 | Logging events with timestamps |
| CN107861956B (en) * | 2016-09-22 | 2020-10-02 | 杭州海康威视系统技术有限公司 | Method and device for inquiring data record of bayonet passing vehicle |
| CN107861956A (en) * | 2016-09-22 | 2018-03-30 | 杭州海康威视系统技术有限公司 | A kind of bayonet socket crosses the querying method and device of car data record |
| CN107943807A (en) * | 2016-10-13 | 2018-04-20 | 华为技术有限公司 | A kind of data processing method and storage device |
| CN107943807B (en) * | 2016-10-13 | 2020-06-16 | 华为技术有限公司 | Data processing method and storage device |
| CN106649627A (en) * | 2016-12-06 | 2017-05-10 | 杭州迪普科技股份有限公司 | Log searching method and device |
| CN106649627B (en) * | 2016-12-06 | 2019-09-17 | 杭州迪普科技股份有限公司 | A kind of method and apparatus that log is searched |
| CN106951557B (en) * | 2017-03-31 | 2020-02-07 | 北京希嘉创智教育科技有限公司 | Log association method and device and computer system applying log association method and device |
| CN106951557A (en) * | 2017-03-31 | 2017-07-14 | 北京希嘉创智教育科技有限公司 | Daily record correlating method, device and apply its computer system |
| CN107220287A (en) * | 2017-04-24 | 2017-09-29 | 东软集团股份有限公司 | For the index managing method of log query, device, storage medium and equipment |
| CN107688624A (en) * | 2017-08-18 | 2018-02-13 | 杭州迪普科技股份有限公司 | A kind of daily record index structuring method and device |
| CN107688624B (en) * | 2017-08-18 | 2020-12-29 | 杭州迪普科技股份有限公司 | Log index construction method and device |
| CN108427705B (en) * | 2018-01-17 | 2022-04-12 | 平安科技（深圳）有限公司 | Electronic device, distributed system log query method and storage medium |
| CN108427705A (en) * | 2018-01-17 | 2018-08-21 | 平安科技（深圳）有限公司 | Electronic device, distributed system journal querying method and storage medium |
| CN108572914A (en) * | 2018-03-14 | 2018-09-25 | 口碑(上海)信息技术有限公司 | A kind of method and device generating the daily record for pressure test |
| CN108572914B (en) * | 2018-03-14 | 2020-12-18 | 口碑(上海)信息技术有限公司 | Method and device for generating log for pressure test |
| CN110515803B (en) * | 2019-08-27 | 2021-04-13 | 联想(北京)有限公司 | Processing method and device for log message and electronic equipment |
| CN110515803A (en) * | 2019-08-27 | 2019-11-29 | 联想(北京)有限公司 | For the processing method of log information, device and electronic equipment |
| CN112800006A (en) * | 2021-01-27 | 2021-05-14 | 杭州迪普科技股份有限公司 | Log storage method and device for network equipment |
| CN112800006B (en) * | 2021-01-27 | 2023-05-26 | 杭州迪普科技股份有限公司 | Log storage method and device for network equipment |
| CN113342768A (en) * | 2021-06-01 | 2021-09-03 | 北京奇艺世纪科技有限公司 | Log reading method and device, electronic equipment and readable storage medium |
| CN113342768B (en) * | 2021-06-01 | 2023-10-10 | 北京奇艺世纪科技有限公司 | Log reading method and device, electronic equipment and readable storage medium |
| CN114567503A (en) * | 2022-03-04 | 2022-05-31 | 南京联成科技发展股份有限公司 | Encryption method for centrally controlled and trusted data acquisition |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102929936B (en) | 2016-08-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102929936A (en) | Log recording method, log inquiring method and system | |
| CN110321344B (en) | Information query method and device for associated data, computer equipment and storage medium | |
| US20100161565A1 (en) | Cluster data management system and method for data restoration using shared redo log in cluster data management system | |
| CN111177277B (en) | Data storage method, transaction storage method and device | |
| CN102122285B (en) | Data cache system and data inquiry method | |
| CN103150397B (en) | A kind of data directory creation method, data retrieval method and system | |
| CN100440221C (en) | Apparatus and method for a managing file system | |
| CN103020204A (en) | Method and system for carrying out multi-dimensional regional inquiry on distribution type sequence table | |
| CN111190904B (en) | Method and device for hybrid storage of graph-relational database | |
| CN103703467A (en) | Method and apparatus for storing data | |
| CN102867071A (en) | Management method for massive network management historical data | |
| CN102016789A (en) | Data processing apparatus and method of processing data | |
| CN107807932B (en) | Hierarchical data management method and system based on path enumeration | |
| CN107918620A (en) | Wiring method and device, the electronic equipment of a kind of database | |
| CN103488710B (en) | The non-fixed-length data method of efficient storage in big data page | |
| CN102880541A (en) | Log information acquisition system and log information acquisition method | |
| CN112347076B (en) | Data storage method and device of distributed database and computer equipment | |
| Ramamohanarao et al. | Recursive linear hashing | |
| CN108363791A (en) | A kind of method of data synchronization and device of database | |
| CN103856462A (en) | Method and system for managing sessions | |
| CN102521304A (en) | Hash based clustered table storage method | |
| CN103310008A (en) | Cloud control server and file index method | |
| CN110008030A (en) | A kind of method of metadata access, system and equipment | |
| CN102137086B (en) | Method, device and system for processing data transmission | |
| CN116756253A (en) | Data storage and query methods, devices, equipment and media of relational database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |