CN108121645A - A kind of daily record method for evaluating quality, device, server and storage medium - Google Patents
A kind of daily record method for evaluating quality, device, server and storage medium Download PDFInfo
- Publication number
- CN108121645A CN108121645A CN201711420458.2A CN201711420458A CN108121645A CN 108121645 A CN108121645 A CN 108121645A CN 201711420458 A CN201711420458 A CN 201711420458A CN 108121645 A CN108121645 A CN 108121645A
- Authority
- CN
- China
- Prior art keywords
- acquisition
- file destination
- daily record
- information
- record data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a kind of daily record method for evaluating quality, device, server and storage mediums.This method is included in when listening to that event occurs in file destination, acquisition in real time and the corresponding daily record data of file destination, and according to the acquisition attributes information of daily record data that is gathered within a preset time interval of record in prefixed time interval cycle；The primitive attribute information of event has occurred in record file destination；Acquisition Error rate is determined according to each acquisition attributes information of primitive attribute information and file destination, to carry out quality evaluation to the daily record data of file destination.It is realized using above-mentioned technical proposal and quality evaluation is carried out to the daily record data of acquisition, the reliability of daily record data is gathered with effective detection, it avoids since the error of data acquisition influences the final result of follow-up data processing, and then daily record data is avoided to bring misguidance to system maintenance, system development and user data excavation etc..
Description
Technical field
The present embodiments relate to technical field of data processing more particularly to a kind of daily record method for evaluating quality, device, clothes
Business device and storage medium.
Background technology
In a complete information system, log system is a very important functional component.Log system
According to all behaviors that certain canonical form record system generates, form substantial amounts of daily record data and understand as exploitation maintenance personnel
The important evidence of information system.
And Flume is as a High Availabitity, highly reliable, distributed massive logs acquisition, polymerization and the system of transmission, it can
To be performed from console Console, RPC (Thrift-RPC), file (Text), Tail (UNIX Tail), Syslog and order
(EXEC) etc. gather daily record data in data sources in real time and be widely used.
If attached drawing 1A is the schematic diagram of the agency of Flume and its accessory structure in the prior art.Wherein, 100A is acted on behalf of
(Agent) it is made of information source 110 (Source), channel 120 (Channel) and the stay of two nights 130 (Sink).Wherein, information source 110 is fixed
The justice source of daily record data, for when collecting enough daily record datas and can generate a Flume event, new establishment
Event be sent to channel 120；The stay of two nights 130 defines the outlet of gathered data, for being connect as consumer unit from channel 120
Receipts event, and in the data distribution received to target memory 140, for subsequently handling the daily record data of acquisition；
And channel 120 is as information source 110 and the intermediate channel of the stay of two nights 130, as the communication mechanism and retention mechanism of event, in information source
110 and the stay of two nights 130 reading and writing data ability there are during larger difference, be used for temporal data as buffer cell.
However, it is carried out using Flume in data acquisition, since its own does not provide the quality evaluation of daily record data
Function causes server is restarted, log content covering or deletion are when abnormal event occurs, and people is safeguarded in system development
Member can not have found the exception of daily record data in time, cause subsequent log analysis result reliability low, be information where server
System maintenance, system development and user data excavation of system etc. bring misguidance.
The content of the invention
An embodiment of the present invention provides a kind of daily record method for evaluating quality, device, server and storage medium, to realize
Quality evaluation is carried out to the daily record data of acquisition, the reliability of daily record is gathered with effective monitoring, is avoided due to data acquisition
Error influence follow-up data processing final result.
In a first aspect, an embodiment of the present invention provides a kind of daily record method for evaluating quality, this method includes：
When event occurs in listening to file destination, acquisition in real time and the corresponding daily record number of the file destination
According to, and according to the acquisition attributes for being recorded in the daily record data gathered in the prefixed time interval in prefixed time interval cycle
Information；
Record the primitive attribute information that event has occurred in the file destination；
Acquisition Error rate is determined according to each acquisition attributes information of the primitive attribute information and the file destination, with
Quality evaluation is carried out to the daily record data of the file destination.
Second aspect, the embodiment of the present invention additionally provide a kind of daily record quality assessment device, which includes：
First logging modle, for when event occurs in listening to file destination, acquisition in real time and the target to be literary
The corresponding daily record data of part, and according to the day that is gathered in the prefixed time interval of being recorded in prefixed time interval cycle
The acquisition attributes information of will data；
Second logging modle, for recording the primitive attribute information that event has occurred in the file destination；
Quality assessment modules, for each acquisition attributes information according to the primitive attribute information and the file destination
Acquisition Error rate is determined, to carry out quality evaluation to the daily record data of the file destination.
The third aspect, the embodiment of the present invention additionally provide a kind of server, and including input unit, which further includes：
One or more processors；
Storage device, for storing one or more programs；
One or more of programs are performed by one or more of processors, use one or more of processors
Realize the daily record method for evaluating quality that any embodiment of the present invention provides.
Fourth aspect, the embodiment of the present invention additionally provide a kind of computer readable storage medium, are stored thereon with computer
Program, the daily record method for evaluating quality which realizes when being executed by processor.
The embodiment of the present invention is by the way that when event occurs in listening to file destination, acquisition is opposite with file destination in real time
The daily record data answered, and adopted according to the record daily record data that is gathered within a preset time interval in prefixed time interval cycle
Set attribute information；The primitive attribute information of event has occurred in record file destination；According to primitive attribute information and target text
Each acquisition attributes information of part determines Acquisition Error rate, to carry out quality evaluation to the daily record data of file destination.Using above-mentioned
Technical solution solves the exception that system development maintenance personnel can not have found daily record data in time, causes subsequent log analysis knot
The technical issues of fruit reliability is low realizes and carries out quality evaluation to the daily record data of acquisition, and daily record number is gathered with effective detection
According to reliability, avoid since the error of data acquisition influences the final result of follow-up data processing, and then avoid day
Will data bring misguidance to system maintenance, system development and user data excavation etc..
Description of the drawings
Figure 1A is the schematic diagram of the agency of Flume and its accessory structure in the prior art；
Figure 1B is the schematic diagram of the agency of Flume and its accessory structure in various embodiments of the present invention；
Fig. 2 is a kind of flow diagram for daily record method for evaluating quality that the embodiment of the present invention one provides；
Fig. 3 A are a kind of flow diagrams of daily record method for evaluating quality provided by Embodiment 2 of the present invention；
Fig. 3 B are the flow diagrams of another daily record method for evaluating quality provided by Embodiment 2 of the present invention；
Fig. 4 is a kind of structure diagram for daily record quality assessment device that the embodiment of the present invention three provides；
Fig. 5 is a kind of hardware architecture diagram for server that the embodiment of the present invention four provides.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining the present invention rather than limitation of the invention.It also should be noted that in order to just
Part related to the present invention rather than entire infrastructure are illustrated only in description, attached drawing.
Embodiment one
Figure 1B is the schematic diagram of the agency of Flume and its accessory structure in various embodiments of the present invention.Wherein, 100B bags are acted on behalf of
Include information source 110, channel 120, the stay of two nights 130, information collector 150 and database 160.Wherein, information source 110, channel 120 and
The function of the stay of two nights 130 is identical with the function of acting on behalf of 100A in Figure 1A.It is further included in addition, acting on behalf of 100B：Information collector 150,
It is connected with information source 110, for recording the attribute information that event has occurred in server；Database 160, with information collector 150
It is connected, for the target text for storing acquisition attributes information collected by information collector 150 and information recorder 180 is recorded
The primitive attribute information of event has occurred in part.Further, 100B is acted on behalf of to further include：Speed limiting device 170, is connected to information source
Between 110 and channel 120, for controlling information source 110 to the rate of the transmission daily record data of channel 120, and then information source 110 is limited
Log collection rate.Report form generator 190 is used to believe based on the primitive attribute information stored in database 160 and acquisition attributes
Breath generates corresponding data sheet.
Fig. 2 is a kind of flow diagram for daily record method for evaluating quality that the embodiment of the present invention one provides, and the present embodiment can
Situation suitable for carrying out quality evaluation to daily record data during log collection is carried out to the file server, this method
It can be performed by daily record quality assessment device, which is realized by software and/or hardware, and is configured on the server.It should
Method specifically comprises the following steps：
S210, when event occurs in listening to file destination, in real time acquisition with the file destination corresponding day
Will data, and according to the acquisition for being recorded in the daily record data gathered in the prefixed time interval in prefixed time interval cycle
Attribute information.
In this step, when event occurs in listening to file destination, show there is user to carry out file destination
Certain operation processing so that corresponding variation has occurred in the data in file destination.At this point, the acquisition in real time of information source 110 and prison
The corresponding daily record data of file destination listened, and information collector 150, according to prefixed time interval, the cycle is recorded in this
The acquisition attributes information of the daily record of acquisition in prefixed time interval.It should be noted that the daily record of the file destination gathered in real time
Data are identical to data content caused by file destination execution operation with user.When according to the prefixed time interval cycle
It records the acquisition attributes information that the daily record data in the file destination gathered within a preset time interval is formed and includes at least text
Part identification information, acquisition total amount identification information and acquisition time information.Wherein, file identification information is for right in spatial domain
The daily record data of acquisition distinguishes, the file destination currently gathered with unique mark；Acquisition total amount identification information is used to record
Total collection capacity of this log collection；Acquisition time information is used to distinguish the daily record data of acquisition in time-domain.
S220, the primitive attribute information that event has occurred in the file destination is recorded.
In this step, information recorder 180 records the primitive attribute information that event has occurred in file destination.Wherein,
Primitive attribute information includes at least target identification information, target record total amount information and recording time information.Wherein, target mark
Information is known for identifying recorded file destination in spatial domain, with the file destination of unique mark current record；Target is remembered
Record total amount information is used to record the event total amount occurred in the current file that event occurs；Recording time information was used in the time
The fileinfo of record is distinguished on domain.
S230, Acquisition Error is determined according to each acquisition attributes information of the primitive attribute information and the file destination
Rate, to carry out quality evaluation to the daily record data of the file destination.
In this step, file identification information is compared with target identification information, when file identification information and target
When the file destination that identification information is identified is same file destination, according to each acquisition total amount identification information and the mesh of file destination
Mark record total amount information is compared, and determines Acquisition Error rate.Subsequently, supervisory user by according to the size of Acquisition Error rate to mesh
Marking file, once the corresponding daily record data gathered of complete processing operation carries out quality evaluation.
The embodiment of the present invention is by the way that when event occurs in listening to file destination, acquisition is opposite with file destination in real time
The daily record data answered, and adopted according to the record daily record data that is gathered within a preset time interval in prefixed time interval cycle
Set attribute information；The primitive attribute information of event has occurred in record file destination；According to primitive attribute information and target text
Each acquisition attributes information of part determines Acquisition Error rate, to carry out quality evaluation to the daily record data of file destination.Using above-mentioned
Technical solution solves the exception that system development maintenance personnel can not have found daily record data in time, causes subsequent log analysis knot
The technical issues of fruit reliability is low realizes and carries out quality evaluation to the daily record data of acquisition, and daily record number is gathered with effective detection
According to reliability, avoid since the error of data acquisition influences the final result of follow-up data processing, and then avoid day
Will data bring misguidance to system maintenance, system development and user data excavation etc..
Embodiment two
Fig. 3 A and Fig. 3 B be two kinds of daily record method for evaluating quality provided by Embodiment 2 of the present invention flow diagram, this reality
Example is applied on the basis of the technical solution of the various embodiments described above, step is further refined and added.
Further, before " acquisition in real time and the corresponding daily record data of the file destination ", addition feature " according to
Default file path information, determines file destination to be monitored ", specifically to limit the method for determination of file destination.
Further, step " in real time acquisition with the corresponding daily record data of the file destination " is refined as " according to by
The acquisition in real time of the mode of row acquisition and the corresponding daily record data of the file destination ", specifically to determine the acquisition of daily record data
Mode.
Correspondingly, the step primitive attribute information of event " record occurred in the file destination " is refined as " record
The line number information of event has occurred in the preset time period in the file destination as target line number ", it is former to improve record
The content of beginning attribute information；
It is correspondingly, step is " true according to each acquisition attributes information of the primitive attribute information and the file destination
Determine Acquisition Error rate " be refined as " file identification information that each acquisition attributes information in the preset time period is included with
When target identification information is corresponding, the total line number of log collection is determined according to each acquisition attributes information in the preset time period；
The Acquisition Error rate of the daily record data of the file destination is determined according to formula η=(1-m/n) × 100%；Wherein, m is described
The total line number of log collection, n are the target line number, and η is the Acquisition Error rate ", to improve the specific of definite Acquisition Error rate
Step.
Fig. 3 A are participated in, which specifically includes following steps：
S310, according to default file path information, determine file destination to be monitored.
S320, monitor in file destination whether event occurs；If it is, continue to execute S330；If it is not, then continue
Perform S320.
By S310 and S320, information source 110 can monitor definite target in real time according to default file path information
File, when event occurs in listening to file destination, it will carry out subsequent log collection step；Target is not being heard
When event occurs in file, monitoring step will be continued to execute.
S330, acquisition in real time and the corresponding daily record data of the file destination in the way of gathering line by line.
In this step, information source 110 acquisition in real time and corresponding daily record number of file destination in the way of gathering line by line
According to.
S340, the daily record data that is gathered in the prefixed time interval of being recorded according to the prefixed time interval cycle
Acquisition attributes information.
In this step, being recorded in the prefixed time interval according to the prefixed time interval cycle of information collector 150
The acquisition attributes information of the daily record data gathered.Wherein, acquisition attributes information includes at least the file identification letter of file destination
Breath, log collection line number and acquisition time information.Wherein, file identification information is used in spatial domain to the daily record number of acquisition
According to distinguishing, the file destination currently gathered with unique mark；Log collection line number is used to record within a preset time interval
Total line number of the daily record data of acquisition；Acquisition time information is used to distinguish the daily record data of acquisition in time-domain.
Illustratively, file identification information includes the IP address and file inode of server.Wherein, server
IP address is to distinguish different servers, and file inode is distinguishing file destination different in same server.It needs
It is noted that by daily record data that different server is gathered most summarize at last it is into target memory 140 namely multiple
The stay of two nights 130 on server may be coupled in a destination server 140, therefore the different target text on to same server
While part is distinguished, it is still desirable to which server is distinguish between.
Illustratively, the traffic type information of daily record data can also be included in acquisition attributes information, with to currently adopting
The business datum of collection is described, and facilitates processing of the business development personnel to daily record data.
S350, the line number information that event has occurred in the file destination is recorded as target line number.
In this step, information recorder 180 records the head office that event has occurred in preset time period in each file destination
Number as target line number, to as the daily record data for judging to be gathered in corresponding file destination whether the reference of exception according to
According to.
S360, the file identification information that is included of each acquisition attributes information and target identification information in the file destination
When corresponding, the total line number of log collection is determined according to each acquisition attributes information of the file destination.
In this step, believe in the primitive attribute of file identification information and record included in each acquisition attributes information
In the case that target identification information is consistent included in breath, it is determined that file destination to be assessed.Obtain target text to be assessed
The log collection line number of all acquisition attributes information in part, and each log collection line number is determined that daily record is adopted by way of summation
Lump line number, to the evaluation points whether daily record data as file destination to be assessed is abnormal.
S370, according to formula η=(1-m/n) × 100% determine the file destination daily record data Acquisition Error rate；
Wherein, m is the total line number of the log collection, and n is the target line number, and η is the Acquisition Error rate.
In this step, by the ratio of the total line number of log collection and target line number, file destination to be assessed is directly determined
In daily record data acquisition rate, and the Acquisition Error rate of file destination to be assessed is determined according to the acquisition rate indirectly, by each
The Acquisition Error rate of file destination characterizes the quality of each daily record data in the file destination.
The embodiment of the present invention specifically defines file destination really by the definite step of additional file destination to be monitored
Determine mode；By further refine the acquisition mode of daily record data for the follow-up quality evaluation for carrying out daily record data provide evaluation because
Son；By refining determining for primitive attribute information and Acquisition Error rate, the content of the perfect primitive attribute information recorded
And the specific determination process of Acquisition Error rate.It solves system development maintenance personnel using above-mentioned technical proposal not sending out in time
The technical issues of showing the exception of daily record data, causing subsequent log analysis result reliability low, realize the daily record to acquisition
Data carry out quality evaluation, and the reliability of daily record data is gathered with effective detection, avoids the error due to data acquisition
The final result of follow-up data processing is influenced, and then daily record data is avoided to dig system maintenance, system development and user data
Pick etc. brings misguidance.
On the basis of the technical solution of the various embodiments described above, further, " target text can also be monitored performing step
Whether event occurs for part " after, additional feature " go by acquisition time information and acquisition according to daily record data has currently been gathered
Number information determines current log acquisition rate；If the current log acquisition rate is more than limitation acquisition rate, according to described
Limit acquisition rate acquisition and the corresponding daily record data of the file destination ", prevent data volume from increasing sharply, capture program acquisition
Quantitative commitments cause occupancy server resource excessively high, are provided with balancing the log collection real-time during log collection with server
Source occupancy.
Referring to Fig. 3 B, which specifically includes following steps：
S310, according to default file path information, determine file destination to be monitored.
S320, monitor in file destination whether event occurs；If it is, continue to execute S321；If it is not, then continue
Perform S320.
In this step, information source 110 monitors definite file destination in real time according to default file path information, is monitoring
When event occurs into file destination, determine that current log acquisition rate has with the acquisition rate to log collection process
Effect limitation；When not listening to that event occurs in file destination, monitoring step will be continued to execute.
S321, current log is determined according to the acquisition time information and acquisition line number information that have currently gathered daily record data
Acquisition rate.
In this step, the daily record data gathered can similarly record acquisition time information and be adopted in gatherer process
Collect line number information, rate conduct when can determine currently to carry out log collection according to acquisition time information and acquisition line number information
Current log acquisition rate.Illustratively, the current log gathered data finally determined can also be prestored to acquisition attributes
It in information, obtains directly by way of inquiry, is operated with the comparison performed subsequently with limiting acquisition rate.
S322, judge whether the current log acquisition rate is more than limitation acquisition rate；If it is, perform S330B；
If it is not, then perform S330A.
In this step, fixed acquisition rate has been prestored in server as limitation acquisition rate.Working as the day before yesterday
When will acquisition rate is more than limitation acquisition rate, represents that current log acquisition rate is too fast, server resource can be caused to occupy
Phenomena such as high, it is therefore desirable to rate limit is carried out to current log collection process using limitation acquisition rate, it subsequently will be according to
Limit acquisition rate acquisition in real time and the corresponding daily record data of file destination；It is adopted in current log acquisition rate no more than limitation
When collecting rate, represent that current log acquisition rate will not cause server resource materially affect, need not subsequently carry out rate limit
System can be gathered directly and the corresponding daily record data of file destination in real time.It should be noted that the technical solution of the present embodiment
The mode synthesis speed limiter 170 of token bucket algorithm may be employed to carry out rate limit to log collection process.It needs to illustrate
, limiting acquisition rate can be manually set by developing maintenance personnel according to previous experiences, can also need people according to acquisition
For adjustment.Illustratively, it is 100 batches/second to limit acquisition rate, wherein, 1 batch is 100 row data, at this time in acquisition daily record
Maximum acquisition rate is 10000 rows/second in the process.
It has been sent out it should be noted that can also gather by comparing in the byte number of daily record data and record file destination
It makes trouble the byte number of part.At this point, limitation acquisition rate will also be set by way of bytes-per-second.
S330A, in real time acquisition and the corresponding daily record data of the file destination；Continue to execute S340.
S330B, gathered and the corresponding daily record data of the file destination according to the limitation acquisition rate；It continues to execute
S340。
S340, the daily record data that is gathered in the prefixed time interval of being recorded according to the prefixed time interval cycle
Acquisition attributes information.
S350, the line number information that event has occurred in the file destination is recorded as target line number.
S360, the file identification information that is included of each acquisition attributes information and target identification information in the file destination
When corresponding, the total line number of log collection is determined according to each acquisition attributes information of the file destination.
S370, according to formula η=(1-m/n) × 100% determine the file destination daily record data Acquisition Error rate；
Wherein, m is the total line number of the log collection, and n is the target line number, and η is the Acquisition Error rate.
The embodiment of the present invention is judged by additional rate and rate-limiting step is adjusted the rate of log collection, prevents
Only data volume is increased sharply, and the quantitative commitments of capture program acquisition cause occupancy server resource excessively high, to balance log collection process
In log collection real-time and server resource occupancy.Using above-mentioned technical proposal solve system development maintenance personnel without
Method finds the exception of daily record data in time, the technical issues of causing subsequent log analysis result reliability low, realizes to adopting
The daily record data of collection carries out quality evaluation, and the reliability of daily record data is gathered with effective detection, is avoided due to data acquisition
The error of journey influences the final result of follow-up data processing, and then avoids daily record data to system maintenance, system development and use
User data excavation etc. brings misguidance.
Embodiment three
Fig. 4 is a kind of structure diagram for daily record quality assessment device that the embodiment of the present invention three provides, and the present embodiment can
The situation of quality evaluation, the daily record matter are carried out to daily record data during log collection is carried out to the file server
Measuring apparatus for evaluating includes the first logging modle 410, the second logging modle 420 and quality assessment modules 430.
Wherein, the first logging modle 410, for when event occurs in listening to file destination, acquisition and institute in real time
The corresponding daily record data of file destination is stated, and institute in the prefixed time interval is recorded according to the prefixed time interval cycle
The acquisition attributes information of the daily record data of acquisition；
Second logging modle 420, for recording the primitive attribute information that event has occurred in the file destination；
Quality assessment modules 430, for each acquisition attributes according to the primitive attribute information and the file destination
Information determines Acquisition Error rate, to carry out quality evaluation to the daily record data of file destination.
The embodiment of the present invention is by the way that when event occurs in listening to file destination, the first logging modle 410 is adopted in real time
Collection and the corresponding daily record data of file destination, and it is recorded in institute in the prefixed time interval according to the prefixed time interval cycle
The acquisition attributes information of the daily record data of acquisition；Second logging modle 420 records the original that event has occurred in the file destination
Beginning attribute information；Quality assessment modules 430 determine to adopt according to each acquisition attributes information of primitive attribute information and file destination
Collect error rate, to carry out quality evaluation to the daily record data gathered in preset time period.It is solved using above-mentioned technical proposal and is
System exploitation maintenance personnel can not have found the exception of daily record data, the technology for causing subsequent log analysis result reliability low in time
Problem realizes and carries out quality evaluation to the daily record data of acquisition, and the reliability of daily record data is gathered with effective detection, is avoided
Due to data acquisition error influence follow-up data processing final result, and then avoid daily record data to system maintenance,
System development and user data excavation etc. bring misguidance.
The schematic diagram of agency and its accessory structure for the Flume as described in Figure 1B, in daily record quality evaluation device
First logging modle 410 corresponds to information source 110, information collector 150 and database 160；Second logging modle, 420 corresponding informance
Logger 180 and database 160；Quality assessment modules 430 correspond to report form generator 190.
Further, first logging modle 410, including：
Collecting unit, for acquisition in real time and the corresponding daily record number of the file destination in the way of gathering line by line
According to.
Further, second logging modle 420, including：
Recording unit, for recording the line number information that event has occurred in the file destination as target line number；
Correspondingly, quality assessment modules 430, are specifically used for：
It is opposite with target identification information in the file identification information that each acquisition attributes information of the file destination is included
At once, the total line number of log collection is determined according to each acquisition attributes information of the file destination；
The Acquisition Error rate of the daily record data of the file destination is determined according to formula η=(1-m/n) × 100%；Wherein,
M is the total line number of the log collection, and n is the target line number, and η is the Acquisition Error rate.
Further, which further includes：
Rate acquisition module, in the acquisition in real time of the first logging modle and the corresponding daily record data of the file destination
While, current log acquisition speed is determined according to the acquisition time information and acquisition line number information that have currently gathered daily record data
Rate；
Limit acquisition module, for the current log acquisition rate be more than limitation acquisition rate when, according to the limit
Acquisition rate acquisition processed and the corresponding daily record data of the file destination.
Further, which further includes：
Monitor module, for it is described it is real-time acquisition with the corresponding daily record data of the file destination before, according to pre-
If file path information, determine file destination to be monitored.
The schematic diagram of agency and its accessory structure for the Flume as described in Figure 1B, in daily record quality evaluation device
Rate acquisition module and limitation acquisition module corresponding speed limiter 170；It monitors module and corresponds to information source 110.
Above-mentioned daily record quality assessment device can perform the daily record method for evaluating quality that any embodiment of the present invention is provided, tool
For the corresponding function module of execution journal quality method and advantageous effect.
Example IV
Fig. 5 is a kind of hardware architecture diagram for server that the embodiment of the present invention four provides.It is as shown in figure 5, of the invention
The server that example IV provides, including input unit 510, processor 520 and storage device 530.
Wherein, input unit 510, for implement acquisition with the corresponding daily record data of file destination, and according to it is default when
Between gap periods the acquisition attributes information for being recorded in the daily record data gathered in the prefixed time interval；It is additionally operable to record
The primitive attribute information of a generation event in the file destination；
One or more processors 520；
Storage device 530, for storing one or more programs.
In Fig. 5 by taking a processor 520 as an example, the input unit 510 in the server can pass through bus or its other party
Formula is connected with processor 520 and storage device 530, and processor 520 and storage device 530 are also by bus or other modes
It connects, in Fig. 5 exemplified by being connected by bus.
In the present embodiment, the processor 520 in server can be according to each of primitive attribute information and file destination
Acquisition attributes information determines Acquisition Error rate；File destination to be monitored can also be determined according to default file path information；
May also listen in file destination whether event occurs；It can also judge whether current log acquisition rate is more than limitation acquisition
Rate.
Storage device 530 in the server is used as a kind of computer readable storage medium, available for storage one or more
A program, described program can be software program, computer executable program and module, such as daily record matter in the embodiment of the present invention
Corresponding program instruction/the module of appraisal procedure is measured (for example, attached first logging modle 410 shown in Fig. 4, the second logging modle 420
And quality assessment modules 430).Processor 520 by operation be stored in software program in storage device 530, instruction and
Module, various function application and data processing so as to execute server, that is, realize daily record quality in above method embodiment
Appraisal procedure.
Storage device 530 may include storing program area and storage data field, wherein, storing program area can store operation system
Application program needed for system, at least one function；Storage data field can store (the daily record number in such as above-described embodiment such as data
According to, prefixed time interval, acquisition attributes information, primitive attribute information and Acquisition Error rate etc.).In addition, storage device 530 can
To include high-speed random access memory, nonvolatile memory can also be included, a for example, at least disk memory is dodged
Memory device or other non-volatile solid state memory parts.In some instances, storage device 530 can further comprise compared with
The remotely located memory of processor 520, these remote memories can pass through network connection to equipment.The example of above-mentioned network
Including but not limited to internet, intranet, LAN, mobile radio communication and combinations thereof.
In addition, the embodiment of the present invention also provides a kind of computer readable storage medium, computer program is stored thereon with, it should
Program be used to realize that the present invention implements the daily record method for evaluating quality provided, this method bag when daily record quality assessment device performs
It includes：When event occurs in listening to file destination, acquisition in real time and the corresponding daily record data of the file destination, and press
According to the acquisition attributes information for being recorded in the daily record data gathered in the prefixed time interval in prefixed time interval cycle；Note
Record the primitive attribute information that event has occurred in the file destination；According to the primitive attribute information and the file destination
Each acquisition attributes information determine Acquisition Error rate, with to the daily record data of the file destination carry out quality evaluation.
By the description above with respect to embodiment, it is apparent to those skilled in the art that, the present invention
It can be realized by software and required common hardware, naturally it is also possible to which by hardware realization, but the former is more in many cases
Good embodiment.Based on such understanding, what technical scheme substantially in other words contributed to the prior art
Part can be embodied in the form of software product, which can be stored in computer readable storage medium
In, floppy disk, read-only memory (Read-Only Memory, ROM), random access memory (Random such as computer
Access Memory, RAM), flash memory (FLASH), hard disk or CD etc., including some instructions with so that a computer is set
Standby (can be personal computer, server or the network equipment etc.) performs the method described in each embodiment of the present invention.
Note that it above are only presently preferred embodiments of the present invention and institute's application technology principle.It will be appreciated by those skilled in the art that
The invention is not restricted to specific embodiment described here, can carry out for a person skilled in the art various apparent variations,
It readjusts and substitutes without departing from protection scope of the present invention.Therefore, although being carried out by above example to the present invention
It is described in further detail, but the present invention is not limited only to above example, without departing from the inventive concept, also
It can include other more equivalent embodiments, and the scope of the present invention is determined by scope of the appended claims.
Claims (10)
1. a kind of daily record method for evaluating quality, which is characterized in that including：
When event occurs in listening to file destination, acquisition in real time and the corresponding daily record data of the file destination, and
According to the acquisition attributes information for being recorded in the daily record data gathered in the prefixed time interval in prefixed time interval cycle；
Record the primitive attribute information that event has occurred in the file destination；
Acquisition Error rate is determined according to each acquisition attributes information of the primitive attribute information and the file destination, with to institute
The daily record data for stating file destination carries out quality evaluation.
2. according to the method described in claim 1, it is characterized in that, the real-time acquisition and the file destination corresponding day
Will data, including：
Acquisition in real time and the corresponding daily record data of the file destination in the way of gathering line by line.
3. according to the method described in claim 2, it is characterized in that, described record the original that event has occurred in the file destination
Beginning attribute information, including：
The line number information that event has occurred in the file destination is recorded as target line number；
Correspondingly, Acquisition Error is determined according to each acquisition attributes information of the primitive attribute information and the file destination
Rate, including：
When the file identification information that each acquisition attributes information of the file destination is included is corresponding with target identification information,
The total line number of log collection is determined according to each acquisition attributes information of the file destination；
The Acquisition Error rate of the daily record data of the file destination is determined according to formula η=(1-m/n) × 100%；Wherein, m is
The total line number of log collection, n are the target line number, and η is the Acquisition Error rate.
4. according to claim 1 any one of them method, which is characterized in that corresponding with the file destination in real-time acquisition
Daily record data when, further include：
Acquisition time information and acquisition line number information according to daily record data has currently been gathered determine current log acquisition rate；
If the current log acquisition rate is more than limitation acquisition rate, according to the limitation acquisition rate acquisition and the mesh
Mark the corresponding daily record data of file.
5. according to claim 1-4 any one of them methods, which is characterized in that in the real-time acquisition and the file destination
Before corresponding daily record data, further include：
According to default file path information, file destination to be monitored is determined.
6. a kind of daily record quality assessment device, which is characterized in that including：
First logging modle, for when event occurs in listening to file destination, acquisition and the file destination phase in real time
Corresponding daily record data, and according to the daily record number that is gathered in the prefixed time interval of being recorded in prefixed time interval cycle
According to acquisition attributes information；
Second logging modle, for recording the primitive attribute information that event has occurred in the file destination；
Quality assessment modules, for being determined according to each acquisition attributes information of the primitive attribute information and the file destination
Acquisition Error rate, to carry out quality evaluation to the daily record data of the file destination.
7. device according to claim 6, which is characterized in that first logging modle, including：
Collecting unit, for acquisition in real time and the corresponding daily record data of the file destination in the way of gathering line by line.
8. device according to claim 7, which is characterized in that second logging modle, including：
Recording unit, for recording the line number information that event has occurred in the file destination as target line number；
Correspondingly, quality assessment modules, are specifically used for：
When the file identification information that each acquisition attributes information of the file destination is included is corresponding with target identification information,
The total line number of log collection is determined according to each acquisition attributes information of the file destination；
The Acquisition Error rate of the daily record data of the file destination is determined according to formula η=(1-m/n) × 100%；Wherein, m is
The total line number of log collection, n are the target line number, and η is the Acquisition Error rate.
9. a kind of server, including input unit, which is characterized in that further include：
One or more processors；
Storage device, for storing one or more programs；
One or more of programs are performed by one or more of processors, are realized using one or more of processors
Such as claim 1-5 any one of them daily record method for evaluating quality.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor
Such as claim 1-5 any one of them daily record method for evaluating quality is realized during execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711420458.2A CN108121645A (en) | 2017-12-25 | 2017-12-25 | A kind of daily record method for evaluating quality, device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711420458.2A CN108121645A (en) | 2017-12-25 | 2017-12-25 | A kind of daily record method for evaluating quality, device, server and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108121645A true CN108121645A (en) | 2018-06-05 |
Family
ID=62231764
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711420458.2A Pending CN108121645A (en) | 2017-12-25 | 2017-12-25 | A kind of daily record method for evaluating quality, device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108121645A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109885547A (en) * | 2019-02-22 | 2019-06-14 | 安徽继远软件有限公司 | The behavior cognitive method of multimode log in multi-source operation data based on IT infrastructure |
| CN110543452A (en) * | 2019-08-07 | 2019-12-06 | 浙江大华技术股份有限公司 | data acquisition method and equipment |
| CN111177076A (en) * | 2019-12-30 | 2020-05-19 | 腾讯科技（深圳）有限公司 | File information management method, device, equipment and storage medium |
| CN112035456A (en) * | 2020-08-31 | 2020-12-04 | 重庆长安汽车股份有限公司 | Real-time detection method for user behavior data quality and storage medium |
| CN112380105A (en) * | 2020-11-23 | 2021-02-19 | 华人运通(上海)云计算科技有限公司 | Log collection method, device, system, equipment, storage medium and plug-in |
| CN112612673A (en) * | 2020-12-24 | 2021-04-06 | 青岛海尔科技有限公司 | Analysis method and device for dial test log, storage medium and electronic device |
| CN113138967A (en) * | 2021-05-08 | 2021-07-20 | 贵州全安密灵科技有限公司 | Data information acquisition method, black box, detonator and storage medium |
| CN115168030A (en) * | 2022-06-24 | 2022-10-11 | 天翼爱音乐文化科技有限公司 | Dynamic regulation and control log acquisition and processing method and device and storage medium |
| CN115296986A (en) * | 2022-06-27 | 2022-11-04 | 青岛海尔科技有限公司 | Event recording method and device, storage medium and electronic device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140244581A1 (en) * | 2012-01-17 | 2014-08-28 | Amazon Technologies, Inc. | System and method for log conflict detection and resolution in a data store |
| CN106708648A (en) * | 2015-07-14 | 2017-05-24 | 阿里巴巴集团控股有限公司 | Text data storage checking method and system |
| CN106991177A (en) * | 2017-04-06 | 2017-07-28 | 广东亿迅科技有限公司 | The nondestructive collection method and its system of real-time logs under a kind of distributed environment |
| CN107273266A (en) * | 2017-06-09 | 2017-10-20 | 上海艾融软件股份有限公司 | A kind of processing method and processing device of application daily record |
| CN107301094A (en) * | 2017-05-10 | 2017-10-27 | 南开大学 | The dynamic self-adapting data model inquired about towards extensive dynamic transaction |
-
2017
- 2017-12-25 CN CN201711420458.2A patent/CN108121645A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140244581A1 (en) * | 2012-01-17 | 2014-08-28 | Amazon Technologies, Inc. | System and method for log conflict detection and resolution in a data store |
| CN106708648A (en) * | 2015-07-14 | 2017-05-24 | 阿里巴巴集团控股有限公司 | Text data storage checking method and system |
| CN106991177A (en) * | 2017-04-06 | 2017-07-28 | 广东亿迅科技有限公司 | The nondestructive collection method and its system of real-time logs under a kind of distributed environment |
| CN107301094A (en) * | 2017-05-10 | 2017-10-27 | 南开大学 | The dynamic self-adapting data model inquired about towards extensive dynamic transaction |
| CN107273266A (en) * | 2017-06-09 | 2017-10-20 | 上海艾融软件股份有限公司 | A kind of processing method and processing device of application daily record |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109885547A (en) * | 2019-02-22 | 2019-06-14 | 安徽继远软件有限公司 | The behavior cognitive method of multimode log in multi-source operation data based on IT infrastructure |
| CN110543452A (en) * | 2019-08-07 | 2019-12-06 | 浙江大华技术股份有限公司 | data acquisition method and equipment |
| CN111177076A (en) * | 2019-12-30 | 2020-05-19 | 腾讯科技（深圳）有限公司 | File information management method, device, equipment and storage medium |
| CN112035456A (en) * | 2020-08-31 | 2020-12-04 | 重庆长安汽车股份有限公司 | Real-time detection method for user behavior data quality and storage medium |
| CN112035456B (en) * | 2020-08-31 | 2024-05-03 | 重庆长安汽车股份有限公司 | Real-time detection method for user behavior data quality and storage medium |
| CN112380105A (en) * | 2020-11-23 | 2021-02-19 | 华人运通(上海)云计算科技有限公司 | Log collection method, device, system, equipment, storage medium and plug-in |
| CN112612673B (en) * | 2020-12-24 | 2022-12-06 | 青岛海尔科技有限公司 | Analysis method and device for dial testing log, storage medium and electronic device |
| CN112612673A (en) * | 2020-12-24 | 2021-04-06 | 青岛海尔科技有限公司 | Analysis method and device for dial test log, storage medium and electronic device |
| CN113138967A (en) * | 2021-05-08 | 2021-07-20 | 贵州全安密灵科技有限公司 | Data information acquisition method, black box, detonator and storage medium |
| CN113138967B (en) * | 2021-05-08 | 2023-03-21 | 贵州全安密灵科技有限公司 | Data information acquisition method, black box, detonator and storage medium |
| CN115168030A (en) * | 2022-06-24 | 2022-10-11 | 天翼爱音乐文化科技有限公司 | Dynamic regulation and control log acquisition and processing method and device and storage medium |
| CN115168030B (en) * | 2022-06-24 | 2023-10-20 | 天翼爱音乐文化科技有限公司 | Dynamic regulation log acquisition and processing method, device and storage medium |
| CN115296986A (en) * | 2022-06-27 | 2022-11-04 | 青岛海尔科技有限公司 | Event recording method and device, storage medium and electronic device |
| CN115296986B (en) * | 2022-06-27 | 2024-03-22 | 青岛海尔科技有限公司 | Event recording method and device, storage medium and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108121645A (en) | A kind of daily record method for evaluating quality, device, server and storage medium | |
| CN111212038B (en) | Open data API gateway system based on big data artificial intelligence | |
| US10482242B2 (en) | System and method for performing event inquiries in a network | |
| US7788198B2 (en) | Method for detecting anomalies in server behavior using operational performance and failure mode monitoring counters | |
| US8531984B2 (en) | Recording medium storing analysis program, analyzing method, and analyzing apparatus | |
| US20050262237A1 (en) | Dynamic incident tracking and investigation in service monitors | |
| JP2010511359A (en) | Method and apparatus for network anomaly detection | |
| US20050097517A1 (en) | Method and system for adjusting the relative value of system configuration recommendations | |
| CN112132687B (en) | Resource exchange wind control method and device, computer equipment and storage medium | |
| CN108632111A (en) | Service link monitoring method based on log | |
| US20200328978A1 (en) | Dynamic policy implementation for application-aware routing based on granular business insights | |
| CN111835681B (en) | Large-scale flow abnormal host detection method and device | |
| CN113660115B (en) | Alarm-based network security data processing method, device and system | |
| CN111444072A (en) | Client abnormality identification method and device, computer equipment and storage medium | |
| CN114039900A (en) | Efficient network data packet protocol analysis method and system | |
| CN115622867A (en) | Industrial control system safety event early warning classification method and system | |
| US11962666B2 (en) | User-configurable end user monitoring (EUM) | |
| CN108092849A (en) | Business datum monitoring method, apparatus and system | |
| CN104461847B (en) | Data processor detection method and device | |
| US11675647B2 (en) | Determining root-cause of failures based on machine-generated textual data | |
| CN110990223A (en) | Monitoring alarm method and device based on system log | |
| US11480955B2 (en) | Smart building sensor network fault diagnostics platform | |
| EP2993827B1 (en) | System and method for monitoring computer servers | |
| CN112835780B (en) | Service detection method and device | |
| CN113392079A (en) | Distributed storage cluster log storage optimization method, system and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180605 |
|
| RJ01 | Rejection of invention patent application after publication |