CN111858529B - Log playback method, device, equipment and computer readable storage medium - Google Patents
Log playback method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111858529B CN111858529B CN202010605964.4A CN202010605964A CN111858529B CN 111858529 B CN111858529 B CN 111858529B CN 202010605964 A CN202010605964 A CN 202010605964A CN 111858529 B CN111858529 B CN 111858529B
- Authority
- CN
- China
- Prior art keywords
- log
- serial number
- sequence number
- previous
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000004590 computer program Methods 0.000 claims description 30
- 230000000875 corresponding effect Effects 0.000 description 112
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000005192 partition Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1435—Saving, restoring, recovering or retrying at system level using file system or storage system metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1438—Restarting or rejuvenating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The application discloses a log playback method, a log playback device, log playback equipment and a computer readable storage medium, which are applied to a distributed file system and are used for obtaining a first serial number of a first effective log in the log system; acquiring a second serial number written into a log in an OSD local file system; judging whether the difference value between the first serial number and the second serial number is larger than 1; if yes, taking the log serial number which is one larger than the second serial number as a third serial number; judging whether a log corresponding to the third serial number exists in the log system or not; if the log corresponding to the third serial number exists in the log system, starting from the log corresponding to the third serial number, and performing log playback on the OSD local file system. In the application, when the difference between the first serial number and the second serial number is greater than 1, the log serial number which is one larger than the second serial number is used as the third serial number, and the log corresponding to the third serial number is searched in the log system through the third serial number, so that the success rate of log playback can be ensured.
Description
Technical Field
The present disclosure relates to the field of distributed storage technologies, and in particular, to a log playback method, system, device, and computer readable storage medium.
Background
In the distributed file system, the main functions of OSD (Object-based Storage Device, object storage) are storing data, copying data, balancing data, restoring data, and the like. In general, a hard disk corresponds to an OSD, and the OSD manages the hard disk storage. When data is written into the OSD, the data is written into the log system, and then the data is written into the OSD local file system by the log system, namely, one data can be written into the OSD through twice writing operation.
However, during the log playback process, it may occur that the log sequence number in the OSD file system is at least one number smaller than the log sequence number in the log system, and thus, when the log playback is performed on the distributed storage system, a case where the log to be played back cannot be found in the log system, resulting in a log playback failure.
In summary, how to improve the log playback success rate of the distributed file system is a problem to be solved by those skilled in the art.
Disclosure of Invention
The purpose of the application is to provide a log playback method, which can solve the technical problem of how to improve the log playback success rate of a distributed file system to a certain extent. The application also provides a log playback device, equipment and a computer readable storage medium.
In order to achieve the above object, the present application provides the following technical solutions:
a log playback method is applied to a distributed file system, and comprises the following steps:
acquiring a first serial number of a first effective log in a log system;
acquiring a second serial number written into a log in an OSD local file system;
judging whether the difference value between the first serial number and the second serial number is larger than 1;
if the difference value between the first serial number and the second serial number is greater than 1, taking the log serial number which is one larger than the second serial number as a third serial number;
judging whether a log corresponding to the third serial number exists in the log system or not;
and if the log corresponding to the third serial number exists in the log system, starting from the log corresponding to the third serial number, and performing log playback on the OSD local file system.
Preferably, the determining whether the log corresponding to the third serial number exists in the log system includes:
taking the log corresponding to the first serial number as a current log;
judging whether the previous log before the current log in the log system is the log corresponding to the third serial number or not;
if the previous log is not the log corresponding to the third sequence number, judging whether the sequence number of the previous log is smaller than the first sequence number, if the sequence number of the previous log is smaller than the first sequence number, taking the previous log as the current log, returning to the step of executing the judgment of whether the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the sequence number of the previous log is larger than the first sequence number, judging that the log corresponding to the third sequence number does not exist in the log system;
if the previous log of the current log is the log corresponding to the third serial number, judging that the log corresponding to the third serial number exists in the log system.
Preferably, the determining whether the previous log before the current log in the log system is the log corresponding to the third serial number includes:
in the log system, the log head information of the last log tail is read;
Judging whether the log sequence number in the log header information is the third sequence number or not;
if the log sequence number in the log header information is the third sequence number, acquiring log data of a previous log based on the log header information, calculating a real-time check value of the log data, judging whether the real-time check value is equal to the check value in the log header information, if the real-time check value is equal to the check value in the log header information, judging that the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the real-time check value is not equal to the check value in the log header information, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number;
if the log sequence number in the log header information is not the third sequence number, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number.
Preferably, the acquiring log data of the previous log based on the log header information includes:
acquiring the length of the log data from the log header information and then filling the length of zero;
And reading the log data based on the offset address of the current log, the length of the log header information, the length of the log data and the zero padding length after the log data.
Preferably, the reading the log data based on the offset address of the current log, the length of the log header information, the length of the log data, and the zero-padding length after the log data includes:
calculating a first offset address obtained by subtracting the length of the log header information from the offset address of the current log;
calculating a second offset address obtained by subtracting the log data from the first offset address and then adding zero length;
judging whether the second offset address is smaller than the log data length;
if the second offset address is smaller than the length of the log data, calculating a length difference value between the length of the log data and the length of the second offset address, reading first data which is equal to the length difference value from the tail part of the log system forwards, reading second data from the head part of the log system to the second offset address, and combining the first data and the second data into the log data;
and if the second offset address is greater than or equal to the log data length, reading the log data with the log data length from the second offset address.
A log playback device for use in a distributed file system, comprising:
the first acquisition module is used for acquiring a first serial number of a first effective log in the log system;
the second acquisition module is used for acquiring a second serial number written into the log in the OSD local file system;
the first setting module is used for taking a log serial number which is one larger than the second serial number as a third serial number;
the first judging module is used for judging whether the log corresponding to the third serial number exists in the log system or not; and if the log corresponding to the third serial number exists in the log system, starting from the log corresponding to the third serial number, and performing log playback on the OSD local file system.
Preferably, the first judging module includes:
the first setting submodule is used for taking the log corresponding to the first serial number as a current log;
the first judging submodule is used for judging whether the previous log before the current log in the log system is the log corresponding to the third serial number or not; if the previous log is not the log corresponding to the third sequence number, judging whether the sequence number of the previous log is smaller than the first sequence number, if the sequence number of the previous log is smaller than the first sequence number, taking the previous log as the current log, returning to the step of executing the judgment of whether the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the sequence number of the previous log is larger than the first sequence number, judging that the log corresponding to the third sequence number does not exist in the log system; if the previous log of the current log is the log corresponding to the third serial number, judging that the log corresponding to the third serial number exists in the log system.
Preferably, the first judging submodule includes:
the first reading unit is used for reading the log head information of the last log tail in the log system;
a first judging unit, configured to judge whether a log sequence number in the log header information is the third sequence number; if the log sequence number in the log header information is the third sequence number, acquiring log data of a previous log based on the log header information, calculating a real-time check value of the log data, judging whether the real-time check value is equal to the check value in the log header information, if the real-time check value is equal to the check value in the log header information, judging that the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the real-time check value is not equal to the check value in the log header information, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number; if the log sequence number in the log header information is not the third sequence number, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number.
A log playback device for use in a distributed file system, comprising:
a memory for storing a computer program;
a processor for implementing the steps of any one of the log playback methods described above when executing the computer program.
A computer readable storage medium for use in a distributed file system, the computer readable storage medium having stored therein a computer program which when executed by a processor performs the steps of the log playback method as described in any one of the above.
The log playback method is applied to a distributed file system and used for obtaining a first serial number of a first effective log in the log system; acquiring a second serial number written into a log in an OSD local file system; judging whether the difference value between the first serial number and the second serial number is larger than 1; if the difference value between the first serial number and the second serial number is greater than 1, taking the log serial number which is one larger than the second serial number as a third serial number; judging whether a log corresponding to the third serial number exists in the log system or not; if the log corresponding to the third serial number exists in the log system, starting from the log corresponding to the third serial number, and performing log playback on the OSD local file system. In the application, when the difference between the first serial number and the second serial number is greater than 1, the log serial number which is one larger than the second serial number is used as the third serial number, and the log corresponding to the third serial number is searched in the log system through the third serial number, so that the success rate of log playback can be ensured. The log playback device, the log playback equipment and the computer readable storage medium also solve the corresponding technical problems.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings may be obtained according to the provided drawings without inventive effort to a person skilled in the art.
Fig. 1 is a flowchart of a log playback method provided in an embodiment of the present application;
FIG. 2 is a flowchart for determining whether a log corresponding to a third serial number exists in the log system in the present application;
fig. 3 is a schematic structural diagram of a log playback device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a log playback device according to an embodiment of the present application;
fig. 5 is another schematic structural diagram of a log playback device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
In the distributed file system, the main functions of OSD (Object-based Storage Device, object storage) are storing data, copying data, balancing data, restoring data, and the like. In general, a hard disk corresponds to an OSD, and the OSD manages the hard disk storage. When data is written into the OSD, the data is written into the log system, and then the data is written into the OSD local file system by the log system, namely, one data can be written into the OSD through twice writing operation. However, in the log playback process, there may be a case where the log serial number in the OSD file system is at least one number smaller than the log serial number in the log system, and if the log playback is performed at this time, the log to be played back cannot be found in the log after the first valid log in the log system, and thus the log to be played back cannot be found in the log system when the log playback is performed on the distributed storage system, resulting in log playback failure. The log playback method provided by the application can improve the log playback success rate of the distributed file system.
Referring to fig. 1, fig. 1 is a flowchart of a log playback method according to an embodiment of the present application.
The log playback method provided by the embodiment of the application is applied to a distributed file system, and can comprise the following steps:
step S101: a first sequence number of a first valid log in the log system is obtained.
In practical application, the first serial number of the first effective day in the log system can be obtained first, and the first effective log in the log system is the earliest effective log in the log system; since the oldest sequence number of the valid log written to the log system is called start_seq, and is stored in the header (the first 4K space of the partition) of the Journal (log system) partition of the disk, the start_seq can be read in the header of the Journal partition to obtain the first sequence number.
Step S102: and acquiring a second serial number written into the log in the OSD local file system.
In practical application, after the first sequence number of the first valid log in the log system is obtained, the second sequence number of the log written in the OSD local file system may be obtained, that is, the latest sequence number of the log written in the OSD local file system, and because the log sequence number of the OSD local file system is written in the commit_op_seq file, the sequence number is stored in the commit_op_seq file and is called commit_seq, the commit_seq may be read in the commit_op_seq file to obtain the second sequence number.
Step S103: judging whether the difference value between the first serial number and the second serial number is larger than 1; if the difference between the first sequence number and the second sequence number is greater than 1, step S104 is performed.
Step S104: step S105 is executed with the log number one larger than the second number as the third number.
Step S105: judging whether a log corresponding to the third serial number exists in the log system or not; if the log corresponding to the third serial number exists in the log system, executing step S106; if the log corresponding to the third serial number does not exist in the log system, step S107 is executed.
Step S106: and starting from the log corresponding to the third serial number, performing log playback on the OSD local file system.
Step S107: the playback is ended.
In practical application, because the logs written into the OSD local file system do not need to be replayed again under the conditions of abnormal power failure and the like, the logs after the second serial number are only replayed, and under the normal condition, the second serial number is smaller than the first serial number by 1, so after the second serial number of the logs written into the OSD local file system is obtained, whether the difference between the first serial number and the second serial number is larger than 1 needs to be judged, if the difference between the first serial number and the second serial number is equal to 1, the first serial number is indicated to be connected with the second serial number, at the moment, if the difference between the first serial number and the second serial number is larger than 1, the logs in the log system starts to be replayed, if the corresponding logs in the third log system start to be replayed, the logs in the corresponding system can be ensured to be replayed, and if the corresponding logs in the third log system start to be replayed.
The log playback method is applied to a distributed file system and used for obtaining a first serial number of a first effective log in the log system; acquiring a second serial number written into a log in an OSD local file system; judging whether the difference value between the first serial number and the second serial number is larger than 1; if the difference value between the first serial number and the second serial number is greater than 1, taking the log serial number which is one larger than the second serial number as a third serial number; judging whether a log corresponding to the third serial number exists in the log system or not; if the log corresponding to the third serial number exists in the log system, starting from the log corresponding to the third serial number, and performing log playback on the OSD local file system. In the application, when the difference between the first serial number and the second serial number is greater than 1, the log serial number which is one larger than the second serial number is used as the third serial number, and the log corresponding to the third serial number is searched in the log system through the third serial number, so that the success rate of log playback can be ensured.
Referring to fig. 2, fig. 2 is a flowchart of determining whether a log corresponding to a third serial number exists in the log system in the present application.
The log playback method provided in the embodiment of the present application is applied to a distributed file system, and the step of determining whether a log corresponding to a third serial number exists in the log system may specifically be:
Step S201: and taking the log corresponding to the first serial number as a current log.
Step S202: judging whether the previous log before the current log in the log system is the log corresponding to the third serial number; if the previous log is not the log corresponding to the third sequence number, executing step S203; if the previous log of the current log is the log corresponding to the third serial number, step S206 is executed.
In practical application, because the third sequence number is smaller than the first sequence number, the log corresponding to the third sequence number should be stored before the log corresponding to the first sequence number in the log system, so the log corresponding to the first sequence number needs to be used as the current log, whether the last log before the current log in the log system is the log corresponding to the third sequence number is judged, and if the last log of the current log is the log corresponding to the third sequence number, the log corresponding to the third sequence number is judged to exist in the log system.
Step S203: judging whether the sequence number of the last log is smaller than the first sequence number, if so, executing step S204; if the sequence number of the previous log is greater than the first sequence number, step S205 is performed.
Step S204: the previous log is returned to step S202 as the current log.
In practical application, since the log system stores the log by overwriting, the sequence number of the last log may be greater than the first sequence number, at this time, the log corresponding to the third sequence number may not exist in the log system, and if the sequence number of the last log is less than the first sequence number, the log corresponding to the third sequence number may exist in the log system, so if the last log is not the log corresponding to the third sequence number, it is necessary to determine whether the sequence number of the last log is less than the first sequence number, if the sequence number of the last log is less than the first sequence number, the last log is taken as the current log, and step S202 is executed, and if the sequence number of the last log is greater than the first sequence number, it is determined that the log corresponding to the third sequence number does not exist in the log system.
Step S205: and judging that the log corresponding to the third serial number does not exist in the log system.
Step S206: and judging that a log corresponding to the third serial number exists in the log system.
The log playback method provided by the embodiment of the application is applied to a distributed file system, because each log in the log system comprises a log head of a head, a pre_pad, a post_pad and a log head of a tail, and the log head comprises a serial number of the log, a crc check value of a log data part, a length of the log data part, the pre_pad, the post_pad and the like, wherein the pre_pad represents a zero padding length before log data, and the post_pad represents a zero padding length after log data, in the process of judging whether a previous log before a current log in the log system is a log corresponding to a third serial number, log head information of the last log tail can be read in the log system; judging whether the log sequence number in the log header information is a third sequence number; if the log sequence number in the log head information is the third sequence number, acquiring log data of the previous log based on the log head information, calculating a real-time check value of the log data, judging whether the real-time check value is equal to the check value in the log head information, if the real-time check value is equal to the check value in the log head information, judging that the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the real-time check value is not equal to the check value in the log head information, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number; if the log sequence number in the log header information is not the third sequence number, judging that the last log before the current log in the log system is not the log corresponding to the third sequence number.
That is, when the log sequence number of the previous log is the same as the third sequence number and the log data of the previous log is the same as the log data corresponding to the third sequence number, the previous log can be judged to be the log corresponding to the third sequence number, so that the situation that the previous log is wrongly judged to be the log corresponding to the third sequence number when the previous log is partially covered or completely covered due to the overwriting is avoided, and the accuracy of log playback in the application is ensured.
The method for replaying the log provided in the embodiment of the present application is applied to a distributed file system, because the log data is searched in the log system by an offset address, and the length of the log header information is known after the log data is recorded in the log header, the offset address of the previous log can be determined based on the offset address of the current log, the length of the log header information, the length of the log data, and the length of the log data after the log data is zero-added, and then the log data of the previous log is determined according to the offset address and the length of the log data, that is, the process of obtaining the log data of the previous log based on the log header information can be specifically: acquiring the length of the log data from the log header information and then filling the length of zero; and reading the log data based on the offset address of the current log, the length of the log header information, the length of the log data and the zero padding length after the log data.
In practical application, because the log system stores the log in an overwriting manner, the situation that the log is stored at the head and the tail of the log system may occur, so that the first offset address obtained by subtracting the length of the log head information from the offset address of the current log is needed to be calculated in the process of reading the log data based on the offset address of the current log, the length of the log head information, the length of the log data and the zero-padding length after the log data; calculating a second offset address obtained by subtracting the zero length of log data from the first offset address; judging whether the second offset address is smaller than the length of the log data; if the second offset address is smaller than the length of the log data, calculating the length difference between the length of the log data and the length of the second offset address, reading first data which are equal to the length difference from the tail part of the log system forwards, reading second data from the head part of the log system to the second offset address, and combining the first data and the second data into the log data; and if the second offset address is greater than or equal to the length of the log data, reading the log data with the length equal to the length of the log data from the second offset address.
Referring to fig. 3, fig. 3 is a schematic structural diagram of a log playback device according to an embodiment of the present application.
The log playback device provided in the embodiment of the application is applied to a distributed file system, and may include:
a first obtaining module 101, configured to obtain a first sequence number of a first valid log in the log system;
a second obtaining module 102, configured to obtain a second serial number of the log written in the OSD local file system;
a first setting module 103, configured to use a log sequence number that is one greater than the second sequence number as a third sequence number;
a first judging module 104, configured to judge whether a log corresponding to the third serial number exists in the log system; if the log corresponding to the third serial number exists in the log system, starting from the log corresponding to the third serial number, and performing log playback on the OSD local file system.
The log playback device provided in the embodiment of the present application is applied to a distributed file system, and the first judging module may include:
the first setting submodule is used for taking the log corresponding to the first serial number as a current log;
the first judging submodule is used for judging whether the previous log before the current log in the log system is the log corresponding to the third serial number or not; if the previous log is not the log corresponding to the third sequence number, judging whether the sequence number of the previous log is smaller than the first sequence number, if the sequence number of the previous log is smaller than the first sequence number, taking the previous log as the current log, returning to the step of executing the step of judging whether the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the sequence number of the previous log is larger than the first sequence number, judging that the log corresponding to the third sequence number does not exist in the log system; if the previous log of the current log is the log corresponding to the third serial number, judging that the log corresponding to the third serial number exists in the log system.
The log playback device provided in the embodiment of the present application is applied to a distributed file system, and the first judging submodule may include:
the first reading unit is used for reading the log head information of the last log tail in the log system;
the first judging unit is used for judging whether the log sequence number in the log header information is a third sequence number or not; if the log sequence number in the log head information is the third sequence number, acquiring log data of the previous log based on the log head information, calculating a real-time check value of the log data, judging whether the real-time check value is equal to the check value in the log head information, if the real-time check value is equal to the check value in the log head information, judging that the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the real-time check value is not equal to the check value in the log head information, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number; if the log sequence number in the log header information is not the third sequence number, judging that the last log before the current log in the log system is not the log corresponding to the third sequence number.
The log playback device provided in the embodiment of the present application is applied to a distributed file system, and the first determining unit may include:
The first acquisition unit is used for acquiring the length of the log data and the length of zero padding after the log data in the log header information;
the first reading unit is used for reading the log data based on the offset address of the current log, the length of the log header information, the length of the log data and the zero padding length after the log data.
The log playback device provided in the embodiment of the present application is applied to a distributed file system, and the first reading unit may include:
the first calculation unit is used for calculating a first offset address obtained by subtracting the length of the log head information from the offset address of the current log;
the second calculating unit is used for calculating a second offset address obtained by subtracting the log data from the first offset address and then adding zero length;
the second judging unit is used for judging whether the second offset address is smaller than the length of the log data; if the second offset address is smaller than the length of the log data, calculating the length difference between the length of the log data and the length of the second offset address, reading first data which are equal to the length difference from the tail part of the log system forwards, reading second data from the head part of the log system to the second offset address, and combining the first data and the second data into the log data; and if the second offset address is greater than or equal to the length of the log data, reading the log data with the length equal to the length of the log data from the second offset address.
The application also provides log playback equipment and a computer readable storage medium, which have the corresponding effects of the log playback method. Referring to fig. 4, fig. 4 is a schematic structural diagram of a log playback device according to an embodiment of the present application.
The log playback device provided in the embodiment of the present application includes a memory 201 and a processor 202, where the memory 201 stores a computer program, and the processor 202 implements the following steps when executing the computer program:
acquiring a first serial number of a first effective log in a log system;
acquiring a second serial number written into a log in an OSD local file system;
judging whether the difference value between the first serial number and the second serial number is larger than 1;
if the difference value between the first serial number and the second serial number is greater than 1, taking the log serial number which is one larger than the second serial number as a third serial number;
judging whether a log corresponding to the third serial number exists in the log system or not;
if the log corresponding to the third serial number exists in the log system, starting from the log corresponding to the third serial number, and performing log playback on the OSD local file system.
The log playback device provided in the embodiment of the present application includes a memory 201 and a processor 202, where the memory 201 stores a computer program, and the processor 202 implements the following steps when executing the computer program: taking the log corresponding to the first serial number as a current log; judging whether the previous log before the current log in the log system is the log corresponding to the third serial number; if the previous log is not the log corresponding to the third sequence number, judging whether the sequence number of the previous log is smaller than the first sequence number, if the sequence number of the previous log is smaller than the first sequence number, taking the previous log as the current log, returning to the step of executing the step of judging whether the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the sequence number of the previous log is larger than the first sequence number, judging that the log corresponding to the third sequence number does not exist in the log system; if the previous log of the current log is the log corresponding to the third serial number, judging that the log corresponding to the third serial number exists in the log system.
The log playback device provided in the embodiment of the present application includes a memory 201 and a processor 202, where the memory 201 stores a computer program, and the processor 202 implements the following steps when executing the computer program: in a log system, reading log head information of the last log tail; judging whether the log sequence number in the log header information is a third sequence number; if the log sequence number in the log head information is the third sequence number, acquiring log data of the previous log based on the log head information, calculating a real-time check value of the log data, judging whether the real-time check value is equal to the check value in the log head information, if the real-time check value is equal to the check value in the log head information, judging that the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the real-time check value is not equal to the check value in the log head information, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number; if the log sequence number in the log header information is not the third sequence number, judging that the last log before the current log in the log system is not the log corresponding to the third sequence number.
The log playback device provided in the embodiment of the present application includes a memory 201 and a processor 202, where the memory 201 stores a computer program, and the processor 202 implements the following steps when executing the computer program: acquiring the length of the log data from the log header information and then filling the length of zero; and reading the log data based on the offset address of the current log, the length of the log header information, the length of the log data and the zero padding length after the log data.
The log playback device provided in the embodiment of the present application includes a memory 201 and a processor 202, where the memory 201 stores a computer program, and the processor 202 implements the following steps when executing the computer program: calculating a first offset address obtained by subtracting the length of the log header information from the offset address of the current log; calculating a second offset address obtained by subtracting the zero length of log data from the first offset address; judging whether the second offset address is smaller than the length of the log data; if the second offset address is smaller than the length of the log data, calculating the length difference between the length of the log data and the length of the second offset address, reading first data which are equal to the length difference from the tail part of the log system forwards, reading second data from the head part of the log system to the second offset address, and combining the first data and the second data into the log data; and if the second offset address is greater than or equal to the length of the log data, reading the log data with the length equal to the length of the log data from the second offset address.
Referring to fig. 5, another log playback device provided in an embodiment of the present application includes a memory 201 and a processor 202, where the memory 201 stores a computer program, and the processor 202 implements the following steps when executing the computer program: the device may further include: an input port 203 connected to the processor 202 for transmitting an externally input command to the processor 202; a display unit 204 connected to the processor 202, for displaying the processing result of the processor 202 to the outside; the communication module 205 connected to the processor 202 is configured to implement a log playback device provided in the embodiment of the present application, including a memory 201 and the processor 202, where the memory 201 stores a computer program, and the processor 202 implements the following steps when executing the computer program: the device communicates with the outside world. The display unit 204 may be a display panel, a laser scanning display, or the like; communication means employed by the communication module 205 include, but are not limited to, mobile high definition link technology (HML), universal Serial Bus (USB), high Definition Multimedia Interface (HDMI), wireless connection: wireless fidelity (WiFi), bluetooth communication, bluetooth low energy communication, ieee802.11s based communication.
The embodiment of the application provides a computer readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
acquiring a first serial number of a first effective log in a log system;
acquiring a second serial number written into a log in an OSD local file system;
judging whether the difference value between the first serial number and the second serial number is larger than 1;
if the difference value between the first serial number and the second serial number is greater than 1, taking the log serial number which is one larger than the second serial number as a third serial number;
judging whether a log corresponding to the third serial number exists in the log system or not;
if the log corresponding to the third serial number exists in the log system, starting from the log corresponding to the third serial number, and performing log playback on the OSD local file system.
The embodiment of the application provides a computer readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: taking the log corresponding to the first serial number as a current log; judging whether the previous log before the current log in the log system is the log corresponding to the third serial number; if the previous log is not the log corresponding to the third sequence number, judging whether the sequence number of the previous log is smaller than the first sequence number, if the sequence number of the previous log is smaller than the first sequence number, taking the previous log as the current log, returning to the step of executing the step of judging whether the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the sequence number of the previous log is larger than the first sequence number, judging that the log corresponding to the third sequence number does not exist in the log system; if the previous log of the current log is the log corresponding to the third serial number, judging that the log corresponding to the third serial number exists in the log system.
The embodiment of the application provides a computer readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: in a log system, reading log head information of the last log tail; judging whether the log sequence number in the log header information is a third sequence number; if the log sequence number in the log head information is the third sequence number, acquiring log data of the previous log based on the log head information, calculating a real-time check value of the log data, judging whether the real-time check value is equal to the check value in the log head information, if the real-time check value is equal to the check value in the log head information, judging that the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the real-time check value is not equal to the check value in the log head information, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number; if the log sequence number in the log header information is not the third sequence number, judging that the last log before the current log in the log system is not the log corresponding to the third sequence number.
The embodiment of the application provides a computer readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: acquiring the length of the log data from the log header information and then filling the length of zero; and reading the log data based on the offset address of the current log, the length of the log header information, the length of the log data and the zero padding length after the log data.
The embodiment of the application provides a computer readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: calculating a first offset address obtained by subtracting the length of the log header information from the offset address of the current log; calculating a second offset address obtained by subtracting the zero length of log data from the first offset address; judging whether the second offset address is smaller than the length of the log data; if the second offset address is smaller than the length of the log data, calculating the length difference between the length of the log data and the length of the second offset address, reading first data which are equal to the length difference from the tail part of the log system forwards, reading second data from the head part of the log system to the second offset address, and combining the first data and the second data into the log data; and if the second offset address is greater than or equal to the length of the log data, reading the log data with the length equal to the length of the log data from the second offset address.
The computer readable storage medium referred to in this application includes Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The description of the relevant parts in the log playback device, the device and the computer readable storage medium provided in the embodiments of the present application refers to the detailed description of the corresponding parts in the log playback method provided in the embodiments of the present application, and will not be repeated here. In addition, the parts of the above technical solutions provided in the embodiments of the present application, which are consistent with the implementation principles of the corresponding technical solutions in the prior art, are not described in detail, so that redundant descriptions are avoided.
It is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (6)
1. A log playback method, applied to a distributed file system, comprising:
acquiring a first serial number of a first effective log in a log system;
acquiring a second serial number written into a log in an OSD local file system;
judging whether the difference value between the first serial number and the second serial number is larger than 1;
if the difference value between the first serial number and the second serial number is greater than 1, taking the log serial number which is one larger than the second serial number as a third serial number;
judging whether a log corresponding to the third serial number exists in the log system or not;
if the log corresponding to the third serial number exists in the log system, starting from the log corresponding to the third serial number, performing log playback on the OSD local file system;
Wherein the determining whether the log corresponding to the third serial number exists in the log system includes:
taking the log corresponding to the first serial number as a current log;
judging whether the previous log before the current log in the log system is the log corresponding to the third serial number or not;
if the previous log is not the log corresponding to the third sequence number, judging whether the sequence number of the previous log is smaller than the first sequence number, if the sequence number of the previous log is smaller than the first sequence number, taking the previous log as the current log, returning to the step of executing the judgment of whether the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the sequence number of the previous log is larger than the first sequence number, judging that the log corresponding to the third sequence number does not exist in the log system;
if the previous log of the current log is the log corresponding to the third serial number, judging that the log corresponding to the third serial number exists in the log system;
the determining whether the previous log before the current log in the log system is the log corresponding to the third serial number includes:
in the log system, the log head information of the last log tail is read;
Judging whether the log sequence number in the log header information is the third sequence number or not;
if the log sequence number in the log header information is the third sequence number, acquiring log data of a previous log based on the log header information, calculating a real-time check value of the log data, judging whether the real-time check value is equal to the check value in the log header information, if the real-time check value is equal to the check value in the log header information, judging that the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the real-time check value is not equal to the check value in the log header information, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number;
if the log sequence number in the log header information is not the third sequence number, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number.
2. The method according to claim 1, wherein the acquiring log data of a previous log based on the log header information includes:
acquiring the length of the log data from the log header information and then filling the length of zero;
And reading the log data based on the offset address of the current log, the length of the log header information, the length of the log data and the zero padding length after the log data.
3. The method of claim 2, wherein the reading the log data based on the offset address of the current log, the length of the log header information, the log data length, the log data zero-post-padding length, comprises:
calculating a first offset address obtained by subtracting the length of the log header information from the offset address of the current log;
calculating a second offset address obtained by subtracting the log data from the first offset address and then adding zero length;
judging whether the second offset address is smaller than the log data length;
if the second offset address is smaller than the length of the log data, calculating a length difference value between the length of the log data and the length of the second offset address, reading first data which is equal to the length difference value from the tail part of the log system forwards, reading second data from the head part of the log system to the second offset address, and combining the first data and the second data into the log data;
And if the second offset address is greater than or equal to the log data length, reading the log data with the log data length from the second offset address.
4. A log playback device for use in a distributed file system, comprising:
the first acquisition module is used for acquiring a first serial number of a first effective log in the log system;
the second acquisition module is used for acquiring a second serial number written into the log in the OSD local file system;
the first setting module is used for taking a log serial number which is one larger than the second serial number as a third serial number;
the first judging module is used for judging whether the log corresponding to the third serial number exists in the log system or not; if the log corresponding to the third serial number exists in the log system, starting from the log corresponding to the third serial number, performing log playback on the OSD local file system;
wherein, the first judging module includes:
the first setting submodule is used for taking the log corresponding to the first serial number as a current log;
the first judging submodule is used for judging whether the previous log before the current log in the log system is the log corresponding to the third serial number or not; if the previous log is not the log corresponding to the third sequence number, judging whether the sequence number of the previous log is smaller than the first sequence number, if the sequence number of the previous log is smaller than the first sequence number, taking the previous log as the current log, returning to the step of executing the judgment of whether the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the sequence number of the previous log is larger than the first sequence number, judging that the log corresponding to the third sequence number does not exist in the log system; if the previous log of the current log is the log corresponding to the third serial number, judging that the log corresponding to the third serial number exists in the log system;
Wherein, the first judging submodule comprises:
the first reading unit is used for reading the log head information of the last log tail in the log system;
a first judging unit, configured to judge whether a log sequence number in the log header information is the third sequence number; if the log sequence number in the log header information is the third sequence number, acquiring log data of a previous log based on the log header information, calculating a real-time check value of the log data, judging whether the real-time check value is equal to the check value in the log header information, if the real-time check value is equal to the check value in the log header information, judging that the previous log before the current log in the log system is the log corresponding to the third sequence number, and if the real-time check value is not equal to the check value in the log header information, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number; if the log sequence number in the log header information is not the third sequence number, judging that the previous log before the current log in the log system is not the log corresponding to the third sequence number.
5. A log playback device for use in a distributed file system, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the log playback method of any one of claims 1 to 3 when executing the computer program.
6. A computer readable storage medium, for application to a distributed file system, having stored therein a computer program which when executed by a processor performs the steps of the log playback method of any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010605964.4A CN111858529B (en) | 2020-06-29 | 2020-06-29 | Log playback method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010605964.4A CN111858529B (en) | 2020-06-29 | 2020-06-29 | Log playback method, device, equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111858529A CN111858529A (en) | 2020-10-30 |
| CN111858529B true CN111858529B (en) | 2024-04-09 |
Family
ID=72988175
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010605964.4A Active CN111858529B (en) | 2020-06-29 | 2020-06-29 | Log playback method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858529B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107733716A (en) * | 2017-10-26 | 2018-02-23 | 郑州云海信息技术有限公司 | Distributed file system log analysis method, system, equipment and storage medium |
| WO2018094961A1 (en) * | 2016-11-28 | 2018-05-31 | 华为技术有限公司 | Write request processing method, device, and data center |
| CN109213617A (en) * | 2018-09-25 | 2019-01-15 | 郑州云海信息技术有限公司 | A kind of determination method, system and the associated component of osd failure cause |
| CN109408482A (en) * | 2018-12-17 | 2019-03-01 | 浪潮电子信息产业股份有限公司 | A kind of date storage method, system, equipment and computer readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9405643B2 (en) * | 2013-11-26 | 2016-08-02 | Dropbox, Inc. | Multi-level lookup architecture to facilitate failure recovery |
-
2020
- 2020-06-29 CN CN202010605964.4A patent/CN111858529B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018094961A1 (en) * | 2016-11-28 | 2018-05-31 | 华为技术有限公司 | Write request processing method, device, and data center |
| CN107733716A (en) * | 2017-10-26 | 2018-02-23 | 郑州云海信息技术有限公司 | Distributed file system log analysis method, system, equipment and storage medium |
| CN109213617A (en) * | 2018-09-25 | 2019-01-15 | 郑州云海信息技术有限公司 | A kind of determination method, system and the associated component of osd failure cause |
| CN109408482A (en) * | 2018-12-17 | 2019-03-01 | 浪潮电子信息产业股份有限公司 | A kind of date storage method, system, equipment and computer readable storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 分布式I/O日志回放系统的设计与实现;詹玲;马骏;陈伯江;陈维梁;吕睿;;计算机工程与应用(36);全文 * |
| 面向移动通信大数据的云存储系统优化;杨洪章;罗圣美;施景超;王志坤;季一木;;计算机应用(S1);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111858529A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9015695B2 (en) | Information processing apparatus and information processing method | |
| CN104811645B (en) | A kind of embedded Video data storage method | |
| CN107315659B (en) | Metadata redundancy backup method and device | |
| CN109445685B (en) | User mode file system processing method | |
| CN106446097B (en) | File reading method and system | |
| KR20170010810A (en) | Method, device and user equipment for reading/writing data in nand flash | |
| CN110888851B (en) | Method and device for creating and decompressing compressed file, and electronic and storage device | |
| CN108874441B (en) | Board card configuration method, device, server and storage medium | |
| CN111143290A (en) | Memory management method, system, equipment and computer readable storage medium | |
| CN110727597B (en) | Method for checking invalid code completion case based on log | |
| CN111752482A (en) | Method and system for writing data in full-flash distributed storage system | |
| US20240045763A1 (en) | A data reconstruction method based on erasure coding, an apparatus, a device and a storage medium | |
| CN111858529B (en) | Log playback method, device, equipment and computer readable storage medium | |
| US20160292035A1 (en) | Indicating rebuild state of storage devices | |
| CN109408482A (en) | A kind of date storage method, system, equipment and computer readable storage medium | |
| US9965491B2 (en) | Method and device for recording system log | |
| CN110134340B (en) | Method, device, equipment and storage medium for updating metadata | |
| CN110187837B (en) | File access method, device and file system | |
| CN111913913A (en) | Access request processing method and device | |
| CN111399857A (en) | FRU data burning method, system, equipment and computer storage medium | |
| CN108052294B (en) | Modification writing method and modification writing system of distributed storage system | |
| CN114780489B (en) | Method and device for realizing distributed block storage bottom layer GC | |
| CN110750496A (en) | File copying method, system, equipment and computer readable storage medium | |
| CN115576484A (en) | Data reading and writing method and device, electronic equipment and storage medium | |
| CN107908500A (en) | A kind of bad block identifying method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |