CN109861843B - Method, device and equipment for completely collecting and confirming log files - Google Patents
Method, device and equipment for completely collecting and confirming log files Download PDFInfo
- Publication number
- CN109861843B CN109861843B CN201811448703.5A CN201811448703A CN109861843B CN 109861843 B CN109861843 B CN 109861843B CN 201811448703 A CN201811448703 A CN 201811448703A CN 109861843 B CN109861843 B CN 109861843B
- Authority
- CN
- China
- Prior art keywords
- log
- time
- newly added
- server
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
A method, a device and equipment for confirming complete collection of log files are disclosed. The client device can judge whether the following conditions are met according to the relevant state value of the local log file: and no new log record exists, or the new log record is acquired by the server. And when the condition is met, sending integrity information containing the time parameter to the server. The server side can confirm the respective latest update time of the log files corresponding to the client side equipment at the current time according to the time parameters in the integrity information, wherein the log files corresponding to all the equipment are updated at the time point, and further can confirm the time when the application log files are completely acquired on all the client side equipment sides where the application is deployed, namely the complete acquisition time.
Description
Technical Field
The embodiment of the specification relates to the technical field of information, in particular to a complete collection confirmation method, a complete collection confirmation device and complete collection confirmation equipment for log files.
Background
Currently, applications are often deployed in a distributed manner on multiple client devices. Thus, each client device may generate a log file independently of the other.
When the server side counts the log files, the log files generated by each device need to be summarized so as to perform further monitoring or analysis. In this process, it is necessary to know whether the resulting summary log file is complete. For example, if an application is deployed on one hundred devices and a third party needs to aggregate log files from 10:00 to 20:00 today for analysis, then at least one of the 10: after 00, 20: before 00, all log records generated by the one hundred devices are received by the service end, otherwise, the analysis result is inaccurate.
Based on this, a more accurate validation scheme for the complete collection of log files is needed.
Disclosure of Invention
To solve the problem that the log file cannot be acquired in the prior art, and to achieve more accurate confirmation of complete collection of the log file, embodiments of the present specification provide a method for confirming complete collection of the log file in terms of client devices, which is applied to a system including a plurality of client devices, where the plurality of client devices each generate a log file of the same application program, and the method includes:
the client equipment selects any log file stored locally;
judging whether the selected log file meets the following conditions: no newly added log record exists, or the newly added log record is acquired by the server;
if so, confirming and sending the integrity information and the client equipment identification corresponding to the selected log file so that the server side can confirm the complete acquisition time of the log files generated by the plurality of client equipment;
the integrity information corresponding to the log file comprises time parameters related to newly added log records of the log file, and the complete acquisition time is used for indicating that the log files generated by the plurality of client devices before the time point are completely acquired by the server.
Meanwhile, an embodiment of the present specification further provides a method for completely acquiring and confirming a log file in a server, which is applied to a system including a plurality of client devices, where the plurality of client devices all generate a log file of a same application program, and the method includes:
the method comprises the steps that a server side receives integrity information and client side equipment identifications sent by each client side equipment, wherein the integrity information comprises time parameters related to newly added log records of log files;
according to the time parameters contained in the client equipment identification and the integrity information, confirming the latest updating time of the log file of each client equipment at the server;
and determining the latest updating time with the largest interval with the current time as the complete acquisition time of the log files generated by the server side to the plurality of client devices.
Correspondingly, an embodiment of the present specification further provides a complete collection and confirmation apparatus for a log file in a client device, which is applied to a system including a plurality of client devices, where the plurality of client devices all generate a log file of a same application program, and the complete collection and confirmation apparatus includes:
the device is applied to a system comprising a plurality of client devices, wherein the plurality of client devices respectively generate log files of the same application program, and the device comprises:
the client equipment selects any log file stored locally;
the judging module is used for judging whether the selected log file meets the following conditions: no newly added log record exists, or the newly added log record is acquired by the server;
if the log files are satisfied, the first confirmation module confirms the integrity information and the client equipment identification corresponding to the selected log files so that the server side can confirm the complete acquisition time of the log files generated by the plurality of client equipment;
the sending module is used for sending the integrity information and the client equipment identification corresponding to the selected log file;
the integrity information corresponding to the log file comprises time parameters related to newly added log records of the log file, and the complete acquisition time is used for indicating that the log files generated by the plurality of client devices before the time point are completely acquired by the server.
Correspondingly, an embodiment of the present specification further provides a complete collection and confirmation apparatus for a log file in a server, which is applied to a system including a plurality of client devices, where the plurality of client devices all generate a log file of a same application program, and the complete collection and confirmation apparatus includes:
the receiving module is used for receiving integrity information and client equipment identifications sent by each client equipment by the server, wherein the integrity information comprises time parameters related to newly added log records of the log file;
the latest update time confirmation module confirms the latest update time of the log file of each client device at the server side according to the time parameters contained in the client device identification and the integrity information;
and the complete acquisition time confirmation module is used for determining the latest update time with the largest interval with the current time as the complete acquisition time of the log files generated by the server side to the plurality of client devices.
In the solution provided in the embodiment of the present specification, the client may determine according to the relevant state value of the local log file whether the following condition is satisfied: and no new log record exists, or the new log record is acquired by the server. And when the condition is met, sending integrity information containing the time parameter to the server. The server side can confirm the respective latest update time of the log files corresponding to the client side equipment at the current time according to the time parameters in the integrity information, wherein the log files corresponding to all the equipment are updated at the time point, and further can confirm the time when the application log files are completely acquired on all the client side equipment sides where the application is deployed, namely the complete acquisition time. And the corresponding relation between the current time and the complete acquisition time can be established, and the complete acquisition time is inserted into the log files summarizing all the devices according to the corresponding relation, so that the function of marking is played, and the log files of all the devices before the time point are completely acquired at the corresponding current time. Therefore, a consumer can know the log collection state when consuming the log file, the application state can be analyzed more accurately, and support is provided for monitoring, alarming or other service index calculation.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of embodiments of the invention.
In addition, any one of the embodiments in the present specification is not required to achieve all of the effects described above.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present specification, and other drawings can be obtained by those skilled in the art according to the drawings.
Fig. 1 is a schematic flowchart of a method for confirming complete collection of a log file in a client device according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of a complete acquisition time provided by an embodiment of the present disclosure;
fig. 3 is a flowchart illustrating a method for completely collecting and confirming a log file in a server according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of a complete collection confirmation apparatus for a log file in the aspect of a client device according to an embodiment of the present specification;
fig. 5 is a schematic structural diagram of a complete collection confirmation apparatus for log files in the aspect of a server according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an apparatus for configuring a method according to an embodiment of the present disclosure.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the embodiments of the present specification, the technical solutions in the embodiments of the present specification will be described in detail below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only a part of the embodiments of the present specification, and not all the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of protection.
The technical solutions provided by the embodiments of the present description are described in detail below with reference to the accompanying drawings. As shown in fig. 1, fig. 1 is a schematic flowchart of a method for completely collecting and confirming a log file in a client device aspect provided in an embodiment of the present specification, where the flowchart specifically includes the following steps:
s101, the client device selects any log file stored locally.
One application program can be deployed on a plurality of client devices in a mode of installing the client. Thus, each client device generates a log file for the application and saves it in the client device. The log files generated by the client devices are generally in the same format and can be summarized by the server. The log file records the system operation events, typically in chronological order, on a case-by-case basis. Each log record will contain the corresponding time of generation.
Also, there may be multiple log files local to the client device. Over time, new log records are continuously written into each log file.
In the case that there are possibly a plurality of log files saved locally, the manner in which the client device selects the log file may be a manner of timed traversal, so as to avoid omission. For example, the log files are sorted according to file names, one log file is selected every 3 seconds in sequence, and corresponding judgment is carried out.
S103, judging whether the selected log file meets the following conditions: and no new log record exists, or the new log record is acquired by the server.
For example, a certain status flag may be set in advance for each log file, and the status flag may have a plurality of values, which respectively represent different statuses, for example, the values: A. b and C, wherein A is used for representing that no newly added log record exists in the log file, B is used for representing that the newly added log record is obtained by the server side, and C represents other states. The value mode of the state flag of the selected log file is usually determined before the log file is selected, and the state value of any log file can be updated by the client device at any time according to actual situations.
And S105, if the log file integrity information is met, the integrity information and the client equipment identification corresponding to the selected log file are confirmed and sent, so that the server side can confirm the complete collection time of the log files generated by the plurality of client equipment.
The integrity information is information that includes a time parameter, and the integrity information may also include a status value. The time parameter is related to the newly added log record and is used for reflecting the related updating time of the newly added log record in the selected log file in the local client device.
For example, when it is determined that no newly added log record exists in the log file, the generation time of the last log record in the log file is obtained and determined as the time parameter; or if the time of generating the newly added log record of the log file fails to be analyzed, determining that the time parameter in the integrity information is a default value, wherein the default value is used for indicating that the newly added log record exists, but the format is wrong, for example, the log time generally and uniformly indicates the number of seconds passed since 1 month and 1 day in 1970 by unix time, so that the current log time cannot be a negative number, and the default value can be set to be-1 for indicating the format error of the newly added log record; or analyzing and acquiring the generation time of the newly added log record of the log file, and determining the generation time of the newly added log record as a time parameter.
At the server, the update progress of the log file of each client device can be counted according to the client device identifier and the time parameter, and then the integral complete acquisition time of the log files generated by the plurality of client devices is determined.
For example, an application is deployed on 100 client devices, and when the server side counts the current time, all update log records generated before 15 points on the 100 client devices are collected by the server side, the complete collection time of the log file is 15 points; if 99 client devices have collected new log records 15 points ago and 1 has collected new log records 14 points before 55 points, the complete collection time of the log file is 14 points and 55 points. In other words, the full collection time is used to indicate that the log files generated by the plurality of client devices before the time point have been completely collected by the server.
Furthermore, the complete acquisition instant is a dynamically changing value. The server can continuously update the log update time of each client device according to the integrity information reported by each device, so that the complete acquisition time is further continuously updated. If the server only keeps one complete acquisition time, the newly generated complete acquisition time can cover the previous complete acquisition time. In an embodiment, the server side records the current system time while generating the complete acquisition time, and then the complete acquisition time and the current system time can be established and stored to obtain a data sequence of the complete acquisition time and the current system time.
For example, at point 17, 24 points, the full acquisition time is 17, 10 points, and possibly 18, 24 points after one hour, and the full acquisition time is 18, 15 points. As shown in fig. 2, fig. 2 is a schematic diagram of a complete acquisition time provided in the embodiment of the present disclosure. The solid line part in fig. 2 represents the log files of the devices that have been received at a certain time, and the dotted line part represents that at another time, the log files of the devices are continuously updated by the server based on the newly added log files, thereby generating another complete collection time.
Due to various reasons such as networks and devices, not every time a new log record of a log file sent by a client device is received by a server. Therefore, if the client fails to send a new log record, the new log record can be sent again until a certain specified sending frequency is reached, and the sending is stopped. For example, after a client fails to send a certain newly added log record, if the client does not receive feedback information of the server, the client considers that the sending fails, and thus the newly added log record information is sent again, and after the sending fails for ten times, the newly added log record information is not sent again. The newly added log record is not received by the server and can be recorded in the integrity information.
In one embodiment, a send flag may be set for each log file. The value of the transmission flag includes a first flag value or a second flag value. For example, the first flag value is "T" and the second flag value is "F". The specific value of the sending flag bit may be various, and the specific form is not limited here.
When the log file in the client device is updated and a system event is generated, the client can read the newly added log record and analyze the generation time of the newly added log record. And after the generation moment of the newly added log record is analyzed successfully, modifying the value of the sending flag bit to be F for indicating that the newly added log record exists, wherein the newly added log record is not obtained by the server at the moment. And then, sending the newly added log record to a server, and modifying the value of the sending flag bit to be T after receiving confirmation information which is returned by the server and aims at the newly added log record, so as to indicate that the server has obtained the newly added log record. The server may use a callback function to feed back to the client which new log records it has received.
In this embodiment, when the client device determines the selected log file, the value of the corresponding sending flag bit may be directly checked, and if the value of the sending flag bit is "T", it may be known that a newly added log record in the selected log file has been acquired by the server.
Further, the client device may also perform state assignment on the integrity information of the log file. For example, the state values of the integrity information may include a first state value for indicating a resolution failure; the second state value is used for indicating that the analysis is successful; and a third state value to indicate that no new log record has been added.
For example, if the client device fails to analyze the generation time of the newly added log record, it is determined that the state value of the integrity information is a first state value, for example, the value is "ParseFailed" and is used to indicate that the analysis fails; if the analysis of the generation time of the newly added log record is successful, determining that the state value of the integrity information is a second state value, for example, the value is 'OK', which is used for indicating the success of the analysis.
Then, if there is a new log record, even if it is not received by the service end, the value of its sending flag bit should be "F"; or, when there is a new added log record and the analysis is not successful, the state value of the integrity information should be "ParseFailed". Therefore, if the client device detects that the value of the sending flag bit is "T" (representing that the previously generated new journal has been sent and received by the server), and the state of the integrity information is a second state value (representing that the previously generated new journal is indeed successfully parsed), it may be determined that no other new journal record exists in the journal file after the new journal record has been received.
When the client device determines that no other newly added log record exists after the last feedback message of the server is received in the log file, the state value of the integrity information may be modified to a third state value, for example, "nonetwolog," indicating that no newly added log record exists. The integrity information may include a state value of the integrity information, so that the server can more conveniently determine a new log state of the log file. For example, if the server finds the third status value in the received integrity information, it may directly determine that the last update time of the log file of the client apparatus remains unchanged from the previous time.
In addition, the integrity information may further include the number of records of the newly added log record received by the server. The specific mode is that the server side feeds back the number of the received newly-added diary records to the client side equipment, and the client side equipment writes the number information into the integrity information. The server side can further carry out corresponding statistics according to the integrity information.
The above section describes the client side aspect of the embodiment of the present specification, and as for the server side aspect, the embodiment of the present specification further provides a complete collection and confirmation method of a log file, which is applied to a system including a plurality of client devices, where the plurality of client devices all generate a log file of the same application program, as shown in fig. 3, and fig. 3 is a flowchart of the complete collection and confirmation method of a log file of the server side aspect provided by the embodiment of the present specification, and includes:
s301, the server receives integrity information and client device identifications sent by each client device, wherein the integrity information comprises time parameters related to newly added log records of log files.
And S303, according to the time parameters contained in the client equipment identification and the integrity information, confirming the latest update time of the log file of each client equipment at the server.
Specifically, the server may find the client device log file corresponding to the received integrity information according to the client device identifier. And then, the time parameter in the integrity information is taken out and compared with the time of the last record in the log file of the client equipment stored by the server.
For example, if the time parameter in the integrity information is a default value (e.g., -1), then the last update time of the client device log file is not updated; if the time parameter is the same as the time of the last record in the client device log file stored by the server, the latest update time of the client device log file is not updated (at this time, it may be considered that no log record is added, and the state value of the integrity information may be the aforementioned "nonetwolog"); if the time parameter is later than the latest update time of the log file of the client device, the time parameter can be confirmed as the latest update time of the log file of the client device at the server.
After the server receives the integrity information, the server may also send feedback information to the client to notify the client that the integrity information has been successfully received.
S305, determining the latest updating time with the largest interval with the current time as the complete acquisition time of the log files generated by the server to the plurality of client devices.
There is a corresponding most recent update time in the log file for each client device. Therefore, all the clients finish reporting the newly added log files at the latest updating time with the maximum interval with the current time.
Because each client device continuously reports the integrity information of the log files (for example, all local log files are traversed every 3 seconds), the server can continuously correct the latest update time of each client device, and thus the complete acquisition time of all log files is dynamically obtained.
In the solution provided in the embodiment of the present specification, the client may determine according to the relevant state value of the local log file whether the following condition is satisfied: and no new log record exists, or the new log record is acquired by the server. And when the condition is met, sending integrity information containing the time parameter to the server. The server side can confirm the respective latest update time of the log files corresponding to the client side equipment at the current time according to the time parameters in the integrity information, wherein the log files corresponding to all the equipment are updated at the time point, and further can confirm the time when the application log files are completely acquired on all the client side equipment sides where the application is deployed, namely the complete acquisition time. Therefore, a consumer can know the log collection state when consuming the log file, the application state can be analyzed more accurately, and support is provided for monitoring, alarming or other service index calculation.
In a specific embodiment, for the step S303, the latest update time of the log file of each client device at the server is confirmed, and an abnormal client device in the plurality of client devices may also be eliminated; and confirming the latest updating time of the log files of the rest client devices at the server.
For example, if a certain client device does not report new integrity information for more than 30 minutes, or the time parameter in the reported integrity information is not changed for 30 minutes all the time, the client device is considered to have failed. Alternatively, the failed client device is identified from some other channel (e.g., system notification). Then, the latest update time of the abnormal devices can be counted, and the abnormal client devices can be not included in the calculation of the integrity time.
In one embodiment, the full acquisition condition may be predetermined, for example, at a specified latest update time, when 90% of the client devices have completed acquisition, the latest update time may be confirmed as the full acquisition time. In this case, there may be a plurality of latest update times satisfying the condition, the latest update time with the largest current time interval is determined as the complete collection time of the log file, and the complete collection condition may also be set to a plurality of levels, for example, 90%, 95%, 99%, and so on.
After the complete acquisition time is determined, the server can collect the log files generated by the client device to generate a service log collection file, and the complete acquisition time is written into the service log collection file, so that the complete acquisition time of the log files generated by all devices of the application program can be known at any time.
In one implementation, the server can also establish a corresponding relationship between the current time and the complete acquisition time, and store the corresponding relationship; summarizing the log files generated by the client devices to generate a service log summarizing file; and marking the complete acquisition time corresponding to any time point in the service log summary file according to the corresponding relation between the current time and the complete time. Therefore, when the log file of the specified time period needs to be taken out later, the complete acquisition time corresponding to the log file related to the application program in the time period can be known.
In addition, when the integrity information further includes the number of records of the newly added log record received by the server, the server may also count the total number of the newly added log record at regular time, for example, count the number of the newly added log record of each client device received once per minute, and count the total number received, so that when a third party uses the service log summary file, the third party may check according to the above information. It will be readily appreciated that the statistical time interval may be set on an as-needed basis, such as 30 seconds, 1 minute, 3 minutes, and so forth.
For example, when the consumer needs to count how many logs are from 1 point to 3 points altogether at 3 points 08 minutes, then, first, it can be confirmed at the time of 3 points 08 minutes according to the complete acquisition time whether all logs generated by the devices from 1 point to 3 points are sent, and further, the total number of records is calculated according to the number records in the integrity information to check whether the number of newly added logs from 1 point to 3 points in the log summary file generated by the server is correct, so as to ensure that the statistics made on the basis of the log file is more accurate.
Correspondingly, in the aspect of a client, an embodiment of the present specification further provides a complete collection confirmation apparatus for a log file, which is used in a system including a plurality of client devices, where the plurality of client devices each generate a log file of a same application program, as shown in fig. 4, and fig. 4 is a schematic structural diagram of the complete collection confirmation apparatus for a log file in the aspect of a client device provided in the embodiment of the present specification, and includes:
a selecting module 401, in which the client device selects any locally stored log file;
the determining module 403 determines whether the selected log file satisfies the following conditions: no newly added log record exists, or the newly added log record is acquired by the server;
if the log file integrity information and the client device identifier are met, the first confirmation module 405 confirms the integrity information and the client device identifier corresponding to the selected log file so that the server side can confirm the complete acquisition time of the log files generated by the plurality of client devices;
a sending module 407, configured to send integrity information and a client device identifier corresponding to the selected log file;
the integrity information corresponding to the log file comprises time parameters related to newly added log records of the log file, and the complete acquisition time is used for indicating that the log files generated by the plurality of client devices before the time point are completely acquired by the server.
Further, the apparatus further includes a second determining module 409, configured to, when it is determined that no newly added log record exists in the log file, obtain a generation time of a last log record in the log file, and determine the generation time as the time parameter; or if the generation moment of the newly added log record of the log file fails to be analyzed, determining the time parameter in the integrity information as a default value for indicating that the newly added log record exists but the format is wrong; or analyzing and acquiring the generation time of the newly added log record of the log file, and determining the generation time of the newly added log record as a time parameter.
Further, the determining module 403 obtains a value of the sending flag bit of the log file; if the value of the sending zone bit is a first zone bit value, judging that a newly added log record in the log file is acquired by a server;
further, the apparatus further includes a dereferencing module 411, which analyzes the generation time of the newly added log record, and after the analysis of the generation time of the newly added log record is successful, modifies the dereferencing of the sending flag bit to be a second flag bit value, which is used to indicate that the newly added log record exists and the server side has not obtained the newly added log record; and sending the newly added log record to a server, and modifying the value of the sending flag bit into a first flag bit value after receiving confirmation information which is returned by the server and aims at the newly added log record, so as to indicate that the server has obtained the newly added log record.
Further, the determining module 403 determines that no newly added log record exists in the log file if the value of the sending flag bit is the first flag bit value and the state value of the integrity information is the second state value; the value taking module 411 is further configured to determine that the state value of the integrity information is a first state value if the analysis of the generation time of the newly added log record fails, and is used to indicate that the analysis fails; and if the analysis of the generation moment of the newly added log record is successful, determining the state value of the integrity information as a second state value for indicating the success of the analysis.
Furthermore, the integrity information corresponding to the log file also includes the number of records of the newly added log record received by the server.
Correspondingly, in the aspect of the server, an embodiment of the present specification further provides a complete collection confirmation apparatus for a log file, which is used in a system including a plurality of client devices, where the plurality of client devices each generate a log file of a same application program, as shown in fig. 5, and fig. 5 is a schematic structural diagram of the complete collection confirmation apparatus for a log file in the aspect of the server, which is provided in the embodiment of the present specification, and includes:
a receiving module 501, configured to receive integrity information and a client device identifier sent by each client device by a server, where the integrity information includes a time parameter related to a newly added log record of a log file;
a latest update time confirmation module 503, configured to confirm the latest update time of the log file of each client device at the server according to the time parameter included in the client device identifier and the integrity information;
the complete collection time confirmation module 505 determines the latest update time with the largest interval with the current time as the complete collection time of the log files generated by the server to the plurality of client devices.
Further, the latest update time confirmation module 503 eliminates an abnormal client device in the plurality of client devices; and confirming the latest updating time of the log files of the rest client devices at the server.
Further, the latest update time confirmation module 503 determines a plurality of latest update times satisfying the complete acquisition condition; determining the latest update time with the largest time interval with the current time among a plurality of latest update times meeting the complete acquisition condition as the complete acquisition time of the log file; the complete acquisition conditions include: at the selected most recent update time, the number or proportion of devices that have completed log file updates at the server exceeds a threshold.
Further, the device further comprises a write-in module 507 for summarizing the log files generated by the client devices to generate a service log summarizing file; and writing the complete acquisition time into the service log summary file.
Further, the apparatus further includes a marking module 509, which establishes a corresponding relationship between the current time and the complete acquisition time, and stores the relationship; summarizing the log files generated by the client devices to generate a service log summarizing file; and marking the complete acquisition time corresponding to any time point in the service log summary file according to the corresponding relation between the current time and the complete time.
Further, the apparatus further includes a counting module 511, which counts the total number of newly added log records in a specified time interval, so that a third party can check when using the service log summary file.
The embodiment of the present specification further provides a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the complete collection confirmation method of the log file shown in fig. 1 or fig. 3 when executing the program.
Fig. 6 is a schematic diagram illustrating a more specific hardware structure of a computing device according to an embodiment of the present disclosure, where the computing device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein the processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 are communicatively coupled to each other within the device via bus 1050.
The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present disclosure.
The Memory 1020 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 1020 may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present specification is implemented by software or firmware, the relevant program codes are stored in the memory 1020 and called to be executed by the processor 1010.
The input/output interface 1030 is used for connecting an input/output module to input and output information. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The communication interface 1040 is used for connecting a communication module (not shown in the drawings) to implement communication interaction between the present apparatus and other apparatuses. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
It should be noted that although the above-mentioned device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present description, and not necessarily all of the components shown in the figures.
The present specification further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the complete collection confirmation method for log files shown in fig. 1 or fig. 3.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
From the above description of the embodiments, it is clear to those skilled in the art that the embodiments of the present disclosure can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the embodiments of the present specification may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments of the present specification.
The systems, methods, modules or units described in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the method embodiment, since it is substantially similar to the method embodiment, it is relatively simple to describe, and reference may be made to the partial description of the method embodiment for relevant points. The above-described method embodiments are merely illustrative, wherein the modules described as separate components may or may not be physically separate, and the functions of the modules may be implemented in one or more software and/or hardware when implementing the embodiments of the present specification. And part or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The foregoing is only a specific embodiment of the embodiments of the present disclosure, and it should be noted that, for those skilled in the art, a plurality of modifications and decorations can be made without departing from the principle of the embodiments of the present disclosure, and these modifications and decorations should also be regarded as the protection scope of the embodiments of the present disclosure.
Claims (22)
1. A complete collection confirmation method of log files is applied to a system comprising a plurality of client devices, wherein the plurality of client devices respectively generate the log files of the same application program, and the method comprises the following steps:
the client equipment selects any log file stored locally;
judging whether the selected log file meets the following conditions: no newly added log record exists, or the newly added log record is acquired by the server;
if so, confirming and sending the integrity information and the client equipment identification corresponding to the selected log file so that the server side can confirm the complete acquisition time of the log files generated by the plurality of client equipment;
the integrity information corresponding to the log file comprises time parameters related to newly added log records of the log file, and the complete acquisition time is used for indicating that the log files generated by the plurality of client devices before the time are completely acquired by the server; the integrity information corresponding to the log file also comprises the number of newly added log records received by the server, and the number of the newly added log records is used for counting the total number of the newly added log records in a specified time interval by the server so as to be checked by a third party.
2. The method as claimed in claim 1, wherein the time parameter associated with the newly added log record of the log file is determined by a method comprising:
when determining that no newly added log record exists in the log file, acquiring the generation time of the last log record in the log file, and determining the generation time as the time parameter; alternatively, the first and second electrodes may be,
if the generation moment of the newly added log record of the log file fails to be analyzed, determining that the time parameter in the integrity information is a default value and is used for indicating that the newly added log record exists but the format is wrong; alternatively, the first and second electrodes may be,
analyzing and acquiring the generation time of a newly added log record of the log file, and determining the generation time of the newly added log record as a time parameter.
3. The method of claim 1, wherein determining whether the following conditions are satisfied in the selected log file comprises:
obtaining the value of the sending zone bit of the log file;
if the value of the sending zone bit is a first zone bit value, judging that a newly added log record in the log file is acquired by a server;
the value generation mode of the sending zone bit comprises the following steps:
analyzing the generation time of the newly added log record, and after the generation time of the newly added log record is analyzed successfully, modifying the value of the sending flag bit into a second flag bit value for indicating that the newly added log record exists and the server side does not acquire the newly added log record;
and sending the newly added log record to a server, and modifying the value of the sending flag bit into a first flag bit value after receiving confirmation information which is returned by the server and aims at the newly added log record, so as to indicate that the server has obtained the newly added log record.
4. The method of claim 3, wherein determining whether the following conditions are satisfied in the selected log file comprises:
if the value of the sending zone bit is a first zone bit value and the state value of the integrity information is a second state value, determining that no newly added log record exists in the log file;
wherein, the state value generation mode of the integrity information comprises:
if the analysis of the generation moment of the newly added log record fails, determining that the state value of the integrity information is a first state value and is used for indicating the analysis failure;
and if the analysis of the generation moment of the newly added log record is successful, determining the state value of the integrity information as a second state value for indicating the success of the analysis.
5. A complete collection confirmation method of log files is applied to a system comprising a plurality of client devices, wherein the plurality of client devices generate the log files of the same application program, and the method comprises the following steps:
the method comprises the steps that a server receives integrity information and client equipment identifications sent by client equipment, wherein the integrity information comprises time parameters related to newly added log records of log files and the number of the newly added log records received by the server;
according to the time parameters contained in the client equipment identification and the integrity information, confirming the latest updating time of the log file of each client equipment at the server;
determining the latest updating time with the largest interval with the current time as the complete acquisition time of the log files generated by the server side to the plurality of client devices;
and counting the total number of newly added log records in a specified time interval so as to facilitate the verification by a third party.
6. The method of claim 5, confirming a most recent update time of the log file of each client device at the server, comprising:
rejecting abnormal client devices in the plurality of client devices;
and confirming the latest updating time of the log files of the rest client devices at the server.
7. The method of claim 5, determining a most recent update time having a greatest separation from a current time as a full collection time of log files generated by the server for the plurality of client devices, comprising:
determining a plurality of latest update moments meeting a complete acquisition condition;
determining the latest update time with the largest time interval with the current time among a plurality of latest update times meeting the complete acquisition condition as the complete acquisition time of the log file;
the complete acquisition conditions include: at the selected most recent update time, the number or proportion of devices that have completed log file updates at the server exceeds a threshold.
8. The method of claim 5, further comprising:
summarizing the log files generated by the client devices to generate a service log summarizing file;
and writing the complete acquisition time into the service log summary file.
9. The method of claim 5, further comprising:
establishing a corresponding relation between the current time and the complete acquisition time, and storing;
summarizing the log files generated by the client devices to generate a service log summarizing file;
and marking the complete acquisition time corresponding to any time point in the service log summary file according to the corresponding relation between the current time and the complete time.
10. The method of claim 8 or 9, wherein counting the total number of newly added log records in a specified time interval for checking by a third party comprises:
and counting the total number of newly added log records in a specified time interval so as to facilitate the third party to check when using the service log summary file.
11. A complete collection confirmation device of log files is applied to a system comprising a plurality of client devices, wherein the plurality of client devices respectively generate the log files of the same application program, and the device comprises:
the client equipment selects any log file stored locally;
the judging module is used for judging whether the selected log file meets the following conditions: no newly added log record exists, or the newly added log record is acquired by the server;
if the log files are satisfied, the first confirmation module confirms the integrity information and the client equipment identification corresponding to the selected log files so that the server side can confirm the complete acquisition time of the log files generated by the plurality of client equipment;
the sending module is used for sending the integrity information and the client equipment identification corresponding to the selected log file;
the integrity information corresponding to the log file comprises time parameters related to newly added log records of the log file, and the complete acquisition time is used for indicating that the log files generated by the plurality of client devices before the time are completely acquired by the server; the integrity information corresponding to the log file also comprises the number of newly added log records received by the server, and the number of the newly added log records is used for counting the total number of the newly added log records in a specified time interval by the server so as to be checked by a third party.
12. The apparatus as claimed in claim 11, further comprising a second confirming module for acquiring a generation time of a last log record in the log file as the time parameter when it is determined that there is no newly added log record in the log file; or if the generation moment of the newly added log record of the log file fails to be analyzed, determining the time parameter in the integrity information as a default value for indicating that the newly added log record exists but the format is wrong; or analyzing and acquiring the generation time of the newly added log record of the log file, and determining the generation time of the newly added log record as a time parameter.
13. The apparatus according to claim 11, wherein the determining module obtains a value of a sending flag bit of the log file; if the value of the sending zone bit is a first zone bit value, judging that a newly added log record in the log file is acquired by a server;
the device also comprises a dereferencing module which is used for analyzing the generation moment of the newly added log record, and after the generation moment of the newly added log record is analyzed successfully, the dereferencing of the sending zone bit is modified into a second zone bit value which is used for indicating that the newly added log record exists and the newly added log record is not obtained by the server; and sending the newly added log record to a server, and modifying the value of the sending flag bit into a first flag bit value after receiving confirmation information which is returned by the server and aims at the newly added log record, so as to indicate that the server has obtained the newly added log record.
14. The apparatus according to claim 13, wherein the determining module determines that no new log record exists in the log file if the value of the sending flag is a first flag value and the state value of the integrity information is a second state value;
the value taking module is further used for determining that the state value of the integrity information is a first state value if the analysis of the generation moment of the newly added log record fails, and the first state value is used for indicating that the analysis fails; and if the analysis of the generation moment of the newly added log record is successful, determining the state value of the integrity information as a second state value for indicating the success of the analysis.
15. A complete collection confirmation device of log files is applied to a system comprising a plurality of client devices, wherein the plurality of client devices generate the log files of the same application program, and the complete collection confirmation device comprises:
the system comprises a receiving module, a server side and a plurality of client side equipment, wherein the server side receives integrity information and client side equipment identifications sent by each client side equipment, and the integrity information comprises time parameters related to newly added log records of log files and the number of records of the newly added log records received by the server side;
the latest update time confirmation module confirms the latest update time of the log file of each client device at the server side according to the time parameters contained in the client device identification and the integrity information;
the complete acquisition time confirmation module is used for determining the latest update time with the largest interval with the current time as the complete acquisition time of the log files generated by the server side to the plurality of client side devices;
and the counting module is used for counting the total number of newly added log records in a specified time interval so as to facilitate the checking by a third party.
16. The apparatus of claim 15, the most recent update time validation module to cull anomalous client devices of the plurality of client devices; and confirming the latest updating time of the log files of the rest client devices at the server.
17. The apparatus of claim 15, the most recent update time validation module to determine a plurality of most recent update times that satisfy a complete acquisition condition; determining the latest update time with the largest time interval with the current time among a plurality of latest update times meeting the complete acquisition condition as the complete acquisition time of the log file; the complete acquisition conditions include: at the selected most recent update time, the number or proportion of devices that have completed log file updates at the server exceeds a threshold.
18. The apparatus of claim 15, further comprising a write module that aggregates log files generated by the client devices to generate a service log aggregated file; and writing the complete acquisition time into the service log summary file.
19. The device of claim 15, further comprising a marking module for establishing and storing a corresponding relationship between the current time and the complete acquisition time; summarizing the log files generated by the client devices to generate a service log summarizing file; and marking the complete acquisition time corresponding to any time point in the service log summary file according to the corresponding relation between the current time and the complete time.
20. The apparatus of claim 18 or 19, wherein the statistical module is configured to count a total number of newly added log records in a specified time interval for checking by a third party when the service log summary file is used.
21. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1 to 4 when executing the program.
22. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 5 to 10 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811448703.5A CN109861843B (en) | 2018-11-28 | 2018-11-28 | Method, device and equipment for completely collecting and confirming log files |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811448703.5A CN109861843B (en) | 2018-11-28 | 2018-11-28 | Method, device and equipment for completely collecting and confirming log files |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109861843A CN109861843A (en) | 2019-06-07 |
| CN109861843B true CN109861843B (en) | 2021-11-23 |
Family
ID=66890410
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811448703.5A Active CN109861843B (en) | 2018-11-28 | 2018-11-28 | Method, device and equipment for completely collecting and confirming log files |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109861843B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112181915B (en) * | 2019-07-03 | 2023-12-08 | 成都华为技术有限公司 | Method, device, terminal and storage medium for executing service |
| CN110995531A (en) * | 2019-11-15 | 2020-04-10 | 苏州浪潮智能科技有限公司 | Method, system, terminal and storage medium for testing rsyslog accuracy |
| CN114238018B (en) * | 2021-12-17 | 2023-03-24 | 天翼爱音乐文化科技有限公司 | Method, system and device for detecting integrity of log collection file and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103200046A (en) * | 2013-03-28 | 2013-07-10 | 青岛海信传媒网络技术有限公司 | Method and system for monitoring network cell device performance |
| CN103927252A (en) * | 2014-04-18 | 2014-07-16 | 安徽科大讯飞信息科技股份有限公司 | Cross-component log recording method, device and system |
| CN107451034A (en) * | 2017-08-17 | 2017-12-08 | 浪潮软件股份有限公司 | A kind of big data cluster log management apparatus, method and system |
| CN108062323A (en) * | 2016-11-08 | 2018-05-22 | 北京国双科技有限公司 | A kind of log read method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10180951B2 (en) * | 2013-03-15 | 2019-01-15 | Amazon Technologies, Inc. | Place snapshots |
| CN104778188B (en) * | 2014-02-24 | 2018-10-19 | 贵州电网公司信息通信分公司 | A kind of distributed apparatus log collection method |
-
2018
- 2018-11-28 CN CN201811448703.5A patent/CN109861843B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103200046A (en) * | 2013-03-28 | 2013-07-10 | 青岛海信传媒网络技术有限公司 | Method and system for monitoring network cell device performance |
| CN103927252A (en) * | 2014-04-18 | 2014-07-16 | 安徽科大讯飞信息科技股份有限公司 | Cross-component log recording method, device and system |
| CN108062323A (en) * | 2016-11-08 | 2018-05-22 | 北京国双科技有限公司 | A kind of log read method and device |
| CN107451034A (en) * | 2017-08-17 | 2017-12-08 | 浪潮软件股份有限公司 | A kind of big data cluster log management apparatus, method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109861843A (en) | 2019-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109739727B (en) | Service monitoring method and device in micro-service architecture | |
| CN108156006B (en) | Buried point data reporting method and device and electronic equipment | |
| CN109861843B (en) | Method, device and equipment for completely collecting and confirming log files | |
| CN110213068B (en) | Message middleware monitoring method and related equipment | |
| CN108737132B (en) | Alarm information processing method and device | |
| CN109039819B (en) | Time delay statistical method, device, system and storage medium | |
| CN112311617A (en) | Configured data monitoring and alarming method and system | |
| CN110224885B (en) | Equipment monitoring alarm method and device, storage medium and electronic equipment | |
| CN104202201A (en) | Log processing method and device and terminal | |
| CN110727560A (en) | Cloud service alarm method and device | |
| CN111078513A (en) | Log processing method, device, equipment, storage medium and log alarm system | |
| CN108363098B (en) | Controllable seismic source force signal data quality control method, device and system | |
| CN116204385A (en) | Computer log monitoring method and system | |
| CN113032412A (en) | Data synchronization method and device, electronic equipment and computer readable medium | |
| WO2017007981A1 (en) | Action correlation framework | |
| CN113495820A (en) | Method and device for collecting and processing abnormal information and abnormal monitoring system | |
| CN111124859A (en) | Log processing method, device, equipment and storage medium | |
| CN110855484B (en) | Method, system, electronic device and storage medium for automatically detecting traffic change | |
| CN108959657B (en) | Data change processing method and device | |
| CN111427749A (en) | Monitoring tool and method for ironic service in openstack environment | |
| WO2020044090A1 (en) | Method and apparatus for determining change event publishing success | |
| CN115580647A (en) | Service call link tracking method and system and electronic equipment | |
| CN108712284B (en) | Fault service positioning method and device and service server | |
| CN112395155A (en) | Service monitoring method and device, storage medium and electronic device | |
| CN110020348B (en) | Early warning method and device for circled events |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20221118 Address after: Room 554, floor 5, building 3, No. 969, Wenyi West Road, Wuchang Street, Yuhang District, Hangzhou City, Zhejiang Province Patentee after: TAOBAO (CHINA) SOFTWARE CO.,LTD. Address before: Box 847, four, Grand Cayman capital, Cayman Islands, UK Patentee before: ALIBABA GROUP HOLDING Ltd. |
|
| TR01 | Transfer of patent right |