CN115904885A

CN115904885A - Log management method and device, electronic equipment and computer readable storage medium

Info

Publication number: CN115904885A
Application number: CN202310226530.7A
Authority: CN
Inventors: 夏令洲; 张柯
Original assignee: Inspur Electronic Information Industry Co Ltd
Current assignee: Inspur Electronic Information Industry Co Ltd
Priority date: 2023-03-10
Filing date: 2023-03-10
Publication date: 2023-04-04
Anticipated expiration: 2043-03-10
Also published as: CN115904885B

Abstract

The invention discloses a log management method, a log management device, electronic equipment and a computer readable storage medium, and relates to the technical field of computer networks, wherein the method comprises the following steps: carrying out data monitoring on the target file and determining modified data in the target file; determining a target Bit corresponding to the modified data and a target bitmap where the target Bit is located in all bitmaps corresponding to the target file; wherein, each Bit in each bitmap is used for recording the modification state of corresponding data in the target file; determining a bitmap identifier of a target bitmap, and generating a modification identifier corresponding to modified data according to the target Bit and the modification state recorded in the target Bit; and correspondingly storing the bitmap identification and the modification identification to the bitmap log corresponding to the target file. By applying the technical scheme provided by the invention, the data volume of the difference log can be reduced while the precision of the difference log is effectively improved.

Description

Log management method and device, electronic equipment and computer readable storage medium

Technical Field

The present invention relates to the field of computer network technologies, and in particular, to a log management method and apparatus, an electronic device, and a computer-readable storage medium.

Background

With the digitization of information, data gradually becomes the service core of users, the performance requirements of users on various functions of a storage system bearing data are gradually improved, and a difference log technology is used as a basic technology of a distributed file system and is widely applied to various service functions of the distributed storage system, for example, one of the key functions of a disaster-tolerant file remote copy technology is the difference log technology.

File remote replication essentially synchronizes the data of the master cluster to the slave cluster. In order to save bandwidth and improve synchronization speed, the remote copy only synchronously changes the difference data, and the more the difference data is simplified, the more the bandwidth can be saved, thereby improving the synchronization efficiency.

At present, a common implementation manner of a difference log in the industry usually uses a file as a basic unit of log recording, and one log is recorded for each modification, but when the size of a single file is very large, even if the modification of the file is very small, the whole file needs to be transferred, which causes a higher bandwidth pressure. However, since remote copy has relatively high requirements on bandwidth and transmission data volume, the smaller the difference data volume is, the more the transmission efficiency can be improved, and therefore, the accuracy of the difference log needs to be improved as much as possible in such a scenario.

Therefore, how to reduce the data volume of the difference log while effectively improving the precision of the difference log is an urgent problem to be solved by those skilled in the art.

Disclosure of Invention

The invention aims to provide a log management method, which can effectively improve the precision of difference logs and reduce the data volume of the difference logs; another object of the present invention is to provide a log management apparatus, an electronic device and a computer-readable storage medium, all of which have the above advantages.

In a first aspect, the present invention provides a log management method, including:

carrying out data monitoring on a target file and determining modified data in the target file;

determining a target Bit corresponding to the modified data and a target bitmap where the target Bit is located in all bitmaps corresponding to the target file; wherein, each Bit in each bitmap is used for recording the modification state of the corresponding data in the target file;

determining a bitmap identifier of the target bitmap, and generating a modification identifier corresponding to the modified data according to the target Bit and the modification state recorded in the target Bit;

and correspondingly storing the bitmap identification and the modification identification to a bitmap log corresponding to the target file.

Optionally, the monitoring the target file to determine modified data in the target file includes:

performing data monitoring on the target file, and determining a data change type when monitoring that the target file has data change;

and if the data change type is data modification operation, determining modified data in the target file.

Optionally, the data change type further includes a data add operation, and the method further includes:

when the data change type is the data adding operation, determining added data in the target file;

determining an offset address and an incremental data length of the incremental data;

and correspondingly storing the offset address and the increased data length to a data newly-increased log corresponding to the target file.

Optionally, the correspondingly storing the offset address and the increased data length to a data new log corresponding to the target file includes:

generating a key value pair corresponding to the added data according to the offset address and the length of the added data; wherein a key value in the key value pair is used for recording the offset address, and a value in the key value pair is used for recording the increased data length;

and recording the key value pairs to a data newly-added log corresponding to the target file.

Optionally, the log management method further includes:

when a data synchronization request is received, determining difference data according to the bitmap log and the data newly added log;

performing a data synchronization operation on the difference data.

Optionally, the determining, in all bitmaps corresponding to the target file, a target Bit corresponding to the modified data and a target bitmap where the target Bit is located includes:

determining an offset address and a modified data length of the modified data;

and inquiring to obtain a target Bit corresponding to the modified data and a target bitmap where the target Bit is located in all bitmaps corresponding to the target file according to the offset address and the length of the modified data.

Optionally, when the record value in the target Bit is 1, the modified state is modified, and when the record value in the target Bit is 0, the modified state is unmodified;

correspondingly, the generating a modification identifier corresponding to the modified data according to the target Bit and the modification state recorded in the target Bit includes:

combining the recording values in each target Bit into a binary modification identifier;

determining a data type of the bitmap;

and converting the binary modification identifier according to the data type to obtain the modification identifier.

Optionally, the combining the recording values in each target Bit into a binary modification identifier includes:

and combining the record values in each target Bit into the binary modification identifier according to a small-end sorting method.

Optionally, the correspondingly storing the bitmap identifier and the modification identifier to the bitmap log corresponding to the target file includes:

generating a key value pair corresponding to the modified data according to the bitmap identification and the modified identification; the key value in the key value pair is used for recording the bitmap identification, and the value in the key value pair is used for recording the modification identification;

and recording the key value pair to a bitmap log corresponding to the target file.

Optionally, the bitmap log comprises an exact bitmap log and an imprecise bitmap log; after determining the target Bit corresponding to the modified data and the target bitmap where the target Bit is located in all bitmaps corresponding to the target file, the method further includes:

determining a fully modified Bit and an incompletely modified Bit in all the target Bit bits;

correspondingly, the correspondingly storing the bitmap identification and the modification identification to the bitmap log corresponding to the target file includes:

correspondingly storing the bitmap identification and the modification identification corresponding to the completely modified Bit to the accurate bitmap log;

and correspondingly storing the bitmap identification and the modification identification corresponding to the incomplete modification Bit to the inaccurate bitmap log.

Optionally, the correspondingly storing the bitmap identifier and the modification identifier corresponding to the completely modified Bit into the precision bitmap log includes:

judging whether the bitmap identification exists in the accurate bitmap log and the inaccurate bitmap log or not;

if the bitmap identification does not exist in the accurate bitmap log or the inaccurate bitmap log, the bitmap identification and the modification identification corresponding to the completely modified Bit are correspondingly stored in the accurate bitmap log;

if the bitmap identification exists in the accurate bitmap log, updating the existing identification corresponding to the bitmap identification in the accurate bitmap log by using the modification identification to obtain a first update identification, and replacing the existing identification with the first update identification;

and if the bitmap identification exists in the inaccurate bitmap log, updating the existing identification corresponding to the bitmap identification in the inaccurate bitmap log by using the modification identification to obtain a second update identification, and correspondingly storing the bitmap identification and the second update identification to the accurate bitmap log.

Optionally, the updating, by using the modification identifier, an existing identifier corresponding to the bitmap identifier in the precision bitmap log to obtain a first update identifier includes:

and carrying out binary OR operation on the modified identifier and the existing identifier to obtain the first updating identifier.

Optionally, after the storing the bitmap identifier and the second update identifier to the precision bitmap log correspondingly, the method further includes:

and deleting the bitmap identification and the existing identification which correspond to each other in the inaccurate bitmap log.

Optionally, the correspondingly storing the bitmap identifier and the modified identifier corresponding to the incomplete modified Bit into the inexact bitmap log includes:

if the bitmap identification does not exist in the accurate bitmap log or the inaccurate bitmap log, correspondingly storing the bitmap identification and the modification identification corresponding to the incompletely modified Bit to the inaccurate bitmap log;

if the bitmap identification exists in the accurate bitmap log, discarding the bitmap identification and the modification identification corresponding to the incompletely modified Bit;

and if the bitmap identification exists in the inaccurate bitmap log, updating the existing identification corresponding to the bitmap identification in the inaccurate bitmap log by using the modification identification to obtain a third update identification, and correspondingly storing the bitmap identification and the third update identification to the inaccurate bitmap log.

Optionally, after the storing the bitmap identifier corresponding to the incomplete modification Bit and the modification identifier corresponding to the incomplete modification Bit in the inexact bitmap log, the method further includes:

determining target modification data corresponding to the incomplete modification Bit;

when the bitmap level of the bitmap corresponding to the target file is not 1, determining a target lower-level Bit corresponding to the target modification data and a target lower-level bitmap where the target lower-level Bit is located; wherein the target lower level bitmap is the lower level bitmap of the incomplete modified Bit;

determining a lower bitmap identification of the target lower bitmap, and generating a lower modification identification corresponding to the target modification data according to the target lower Bit and the modification state recorded in the target lower Bit;

and correspondingly storing the lower bitmap identification and the lower modification identification to the lower bitmap log corresponding to the inaccurate bitmap log until the tail bitmap log is completely stored.

Optionally, the log management method further includes:

deleting a lower bitmap of the incomplete modification Bit when the incomplete modification Bit is updated to the complete modification Bit.

Optionally, each bitmap corresponding to the target file is an int64 integer bitmap.

In a second aspect, the present invention also discloses a log management apparatus, including:

the monitoring module is used for carrying out data monitoring on a target file and determining modified data in the target file;

a determining module, configured to determine, in all bitmaps corresponding to the target file, a target Bit corresponding to the modified data and a target bitmap where the target Bit is located; wherein, each Bit in each bitmap is used for recording the modification state of corresponding data in the target file;

a generating module, configured to determine a bitmap identifier of the target bitmap, and generate a modification identifier corresponding to the modified data according to the target Bit and a modification status recorded in the target Bit;

and the storage module is used for correspondingly storing the bitmap identification and the modification identification to the bitmap log corresponding to the target file.

In a third aspect, the present invention also discloses an electronic device, including:

a memory for storing a computer program;

a processor for implementing the steps of any of the log management methods as described above when executing the computer program.

In a fourth aspect, the present invention also discloses a computer readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of any of the log management methods described above.

By applying the technical scheme provided by the invention, for each target file in the storage system, a bitmap corresponding to the target file is created, and the modification state of each data in the target file is recorded through each Bit in the bitmap, so that the modified data in the target file can be determined by monitoring the data of the target file, the target Bit corresponding to the modified data and the target bitmap where the target Bit is located are further determined, further, the modified identification corresponding to the modified data is generated by using the target Bit and the modification state recorded inside the target Bit, and is corresponding to the bitmap identification of the target bitmap and is stored in the bitmap log, so that the modified data in the target file is recorded through the bitmap log, namely, the difference log is realized based on the bitmap log, on the basis, the modified data in the target file can be determined through the bitmap log regardless of the size of the target file, and further, the data synchronization based on the modified data is realized, but not based on the data synchronization of the whole target file, obviously, the realization mode can effectively improve the precision of the difference log, and simultaneously reduce the data volume of the difference log.

Drawings

In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present invention, the drawings that are needed to be used in the description of the prior art and the embodiments of the present invention will be briefly described below. Of course, the following description of the drawings related to the embodiments of the present invention is only a part of the embodiments of the present invention, and it will be obvious to those skilled in the art that other drawings can be obtained from the provided drawings without any creative effort, and the obtained other drawings also belong to the protection scope of the present invention.

Fig. 1 is a schematic flowchart of a log management method according to the present invention;

FIG. 2 is a schematic diagram of a method for adding a log record to data according to the present invention;

FIG. 3 is a schematic diagram of a bitmap logging method provided by the present invention;

FIG. 4 is a schematic diagram of a multi-level bitmap logging method provided by the present invention;

FIG. 5 is a schematic diagram of a log reduction method provided by the present invention;

FIG. 6 is a schematic structural diagram of a log management device according to the present invention;

fig. 7 is a schematic structural diagram of an electronic device provided in the present invention.

Detailed Description

The core of the invention is to provide a log management method, which can effectively improve the precision of the difference log and reduce the data volume of the difference log; another core of the present invention is to provide a log management apparatus, an electronic device and a computer-readable storage medium, all of which have the above advantages.

In order to more clearly and completely describe the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The embodiment of the invention provides a log management method.

Referring to fig. 1, fig. 1 is a flowchart illustrating a log management method according to the present invention, where the log management method includes the following steps S101 to S104.

S101: and monitoring data of the target file, and determining modified data in the target file.

This step is intended to implement data monitoring of the target file, so as to determine the modified data in the target file, i.e. the modified data. The target file is a data file that needs to be log-managed in the storage system, and may be all data files in the storage system or a part of data files specified in the storage system.

In the implementation process, the target file in the storage system can be subjected to real-time data monitoring, and when the data modification operation occurs in the target file, the modified data is determined. The determining of the modified data may specifically be determining an offset address and a data modification length of target data in which a data modification operation occurs.

S102: determining a target Bit corresponding to the modified data and a target bitmap where the target Bit is located in all bitmaps corresponding to the target file; and each Bit in each bitmap is used for recording the modification state of corresponding data in the target file.

The step aims to realize the determination of the target Bit and the target bitmap where the target Bit is located. Specifically, for each target file in the storage system, a bitmap corresponding to the target file may be created in advance, and the modification state of each data in the target file is recorded through each Bit in the bitmap, where one target file may correspond to one or more bitmaps, and one data (here, a unit of data, which may be set according to actual needs) in the target file corresponds to one Bit, that is, one Bit records the modification state of one unit of data. It should be noted that the number of bitmaps and the type of bitmaps corresponding to the target file are determined by the file size of the target file, and since the size of each file in the storage system may be different, the number of bitmaps corresponding to different target files may also be different, which is not limited in the present invention.

Therefore, after determining the modified data in the target file, the target Bit corresponding to the modified data may be determined in all the bitmaps corresponding to the target file, so as to obtain the modified state of the modified data recorded in the target Bit.

In one possible implementation, each bitmap corresponding to the target file may be an int64 integer bitmap.

Specifically, data of an int64 type includes 8 bytes and 8 Bit bits of each byte, so that an int64 integer bitmap includes 64 Bit bits, and when the difference log precision is 1M (that is, the data unit is 1M), an int64 integer bitmap can identify the modification state of the data of 64M. Assuming that the file size of the target file is 640M, the number of bitmaps corresponding to the target file is 10.

S103: and determining a bitmap identifier of the target bitmap, and generating a modification identifier corresponding to the modified data according to the target Bit and the modification state recorded in the target Bit.

This step is intended to enable determination of the bitmap identification and the modification identification in order to enable logging of the modified data. The bitmap identifier refers to a bitmap identifier of a target bitmap, and when the number of bitmaps corresponding to one target file is multiple, different bitmaps may have different bitmap identifiers for bitmap differentiation, for example, a sequence number (0,1,2, … …, n) may be set for each bitmap corresponding to one target file, and the bitmap sequence number is used as the bitmap identifier. The modification flag is flag information of modified data, and is generated automatically according to a target Bit (here, the Bit serial number of the target Bit, and one bitmap includes multiple bits) and a modification status recorded in the target Bit, where the modification status is divided into modified status and unmodified status.

S104: and correspondingly storing the bitmap identification and the modification identification to the bitmap log corresponding to the target file.

The step aims to realize the generation and storage of the difference log corresponding to the modified data. After determining the bitmap identification and the modification identification corresponding to the modified data, a corresponding relationship can be generated between the bitmap identification and the modification identification, and the bitmap identification and the modification identification are correspondingly stored in a bitmap log. Therefore, the bitmap log is used for realizing the log record of the modified data in the target file, namely realizing the difference log record. On the basis, when data synchronization is needed subsequently, the modified data in the target file can be directly determined based on the bitmap log, so that the data synchronization of the modified data is realized, and the data synchronization does not need to be carried out based on the whole target file.

It can be seen that, in the log management method provided in the embodiment of the present invention, for each target file in the storage system, a bitmap corresponding to the target file is created, and the modification state of each data in the target file is recorded by each Bit in the bitmap, so that the modified data in the target file can be determined by performing data monitoring on the target file, and then the target Bit corresponding to the modified data and the target bitmap where the target Bit is located are determined, further, a modification identifier corresponding to the modified data is generated by using the target Bit and the modification state recorded inside the target Bit, and is corresponding to the bitmap identifier of the target bitmap and stored in the bitmap log, so that the modified data in the target file is recorded by the bitmap log, that is, a difference log is realized based on the bitmap log, based on this, the modified data in the target file can be determined by the bitmap log regardless of the size of the target file, and further, data synchronization based on the modified data is realized, rather than data synchronization of the whole target file, obviously, the realization method can reduce the data amount of the difference log while effectively improving the precision of the difference log.

In an embodiment of the present invention, the monitoring the data of the target file and determining the modified data in the target file may include:

carrying out data monitoring on the target file, and determining the data change type when monitoring that the target file has data change;

and if the data change type is the data modification operation, determining modified data in the target file.

It can be understood that, regarding the data change in the target file, it is not limited to the data modification operation, for example, there may also be a data addition operation, so that, in the process of monitoring the data of the target file, when the data change in the target file is monitored, the data change type may be determined first, and only when the data change type is the data modification operation (the data amount of the data in the target file is not increased or decreased), the step of determining the modified data in the target file may be executed continuously.

In an embodiment of the present invention, the data change type further includes a data addition operation, and the log management method further includes:

when the data change type is data adding operation, determining added data in the target file;

determining an offset address of the added data and an added data length;

and correspondingly storing the offset address and the increased data length into a data newly-added log corresponding to the target file.

The log management method provided by the embodiment of the invention provides another log type, namely a data newly-added log, which is used for recording the added data in the target file. As described above, the data change in the target file is not limited to the data modification operation, and there may also be a data addition operation, and when the data change type is determined to be the data addition operation, the added data in the target file may be determined first, which mainly refers to determining the offset address and the length of the added data, so that the offset address and the length of the added data may form a corresponding relationship, and the corresponding relationship may be stored in the data addition log.

Obviously, in the embodiment of the present invention, the log record of the modified data in the target file is realized based on the bitmap log, and the log record of the added data in the target file is realized based on the data addition log, on this basis, when data synchronization is subsequently required, the difference data (i.e., modified data + added data) in the target file can be directly determined based on the data addition log and the bitmap log, so as to realize the data synchronization of the difference data, without performing data synchronization based on the whole target file.

In an embodiment of the present invention, the storing the offset address and the increased data length into the data new log corresponding to the target file correspondingly may include:

generating a key value pair corresponding to the added data according to the offset address and the length of the added data; the key value in the key value pair is used for recording the offset address, and the value in the key value pair is used for recording the increased data length;

and recording the key value pair to a data new log corresponding to the target file.

The embodiment of the invention provides a data storage method of a data newly added log, namely, an offset address and an added data length which correspond to each other can be stored in a key value pair mode, wherein the offset address is used as a key value in the key value pair, and the added data length is used as a value in the key value pair, so that the key value pair which is generated based on the offset address and the added data length and corresponds to the added data can be recorded in the data newly added log, and the log recording of the added data in a target file is realized.

In an embodiment of the present invention, the log management method may further include:

a data synchronization operation is performed on the difference data.

It can be understood that the log management method provided by the present invention can be used for data synchronization of differential data in different storage systems, therefore, on the basis of implementing log record of modified data in a target file based on a bitmap log and implementing log record of added data in the target file based on a data addition log, when a data synchronization request is received and data synchronization operation needs to be performed, differential data between two storage systems, that is, modified data and added data, can be directly determined according to the bitmap log and the data addition log, and further data synchronization between the two storage systems is implemented based on the differential data.

In an embodiment of the present invention, the determining, in all the bitmaps corresponding to the target file, the target Bit corresponding to the modified data and the target bitmap where the target Bit is located may include:

determining an offset address of the modified data and a length of the modified data;

and in all bitmaps corresponding to the target file, inquiring according to the offset address and the length of the modified data to obtain a target Bit corresponding to the modified data and a target bitmap where the target Bit is located.

The embodiment of the invention provides a method for realizing the determination of a target Bit corresponding to modified data and a target bitmap in which the target Bit corresponds. As described above, each Bit in the bitmap records the modification state of the corresponding data in the target file, and one Bit corresponds to one unit of data, so that the actual position of the modified data in the target file needs to be determined first when determining the target Bit corresponding to the modified data, and thus after determining the modified data in the target file, the offset address and the length of the modified data in the target file can be further determined, and thus the determination of the target Bit and the target bitmap where the target Bit is located can be realized according to the offset address and the length of the modified data in the target file.

In one embodiment of the present invention, when the record value in the target Bit is 1, the corresponding modification status is modified, and when the record value in the target Bit is 0, the corresponding modification status is unmodified;

correspondingly, the generating of the modification identifier corresponding to the modification data according to the target Bit and the modification state recorded in the target Bit may include:

combining the recording values in the Bit bits of each target into a binary modification identifier;

determining the data type of the bitmap;

The embodiment of the invention provides a method for generating a modification identifier corresponding to modified data. Firstly, each Bit in the bitmap can realize the record of the modification state of the modification data by recording binary values, wherein, binary 1 corresponds to the modified state as modified, and binary 0 corresponds to the modified state as unmodified; further, after determining the target Bit corresponding to the modified data, combining the record value (1 or 0) in each target Bit into a binary modified flag, for example, if the four target Bit bits are 1100, determining the data type of the bitmap corresponding to the target file at the same time, and then converting the binary modified flag into the data type to obtain the modified flag, for example, if the bitmap mentioned above may be an int64 integer bitmap, the binary modified flag may be converted into int64 integer data, and when the 64 Bit bits in the target bitmap are all binary 1, the binary modified flag is converted into int64 18446744073709551615.

In an embodiment of the present invention, the combining the record values in each target Bit into the binary modification identifier may include: and combining the recording values in the Bit positions of the targets into a binary modification identifier according to a small-end sorting method.

The embodiment of the invention provides a method for realizing a combined binary modification identifier, namely a small-end sequencing method, wherein the small-end sequencing method is a Little-endian, specifically, a low-order byte is stored into a low address, and a high-order byte is stored into a high address.

In an embodiment of the present invention, the storing the bitmap identification and the modification identification in the bitmap log corresponding to the target file correspondingly may include:

and recording the key value pairs to a bitmap log corresponding to the target file.

The embodiment of the invention provides a data storage method of a bitmap log, which can store bitmap marks and modification marks corresponding to each other in a key value pair mode, wherein the bitmap marks are used as key values in the key value pair, and the modification marks are used as value values in the key value pair, so that the key value pairs which are generated based on the bitmap marks and the modification marks and correspond to the modification data can be recorded in the bitmap log, and the log recording of the modification data in a target file is realized.

In one embodiment of the invention, the bitmap log may comprise an exact bitmap log and an inexact bitmap log; after determining the target Bit corresponding to the modified data and the target bitmap in which the target Bit is located in all the bitmaps corresponding to the target file, the method may further include: determining a complete modification Bit and an incomplete modification Bit in all target Bit bits;

correspondingly, the correspondingly storing the bitmap identification and the modification identification to the bitmap log corresponding to the target file may include:

correspondingly storing the bitmap identification corresponding to the completely modified Bit and the modified identification into an accurate bitmap log;

and correspondingly storing the bitmap identification corresponding to the incompletely modified Bit and the modified identification to an inaccurate bitmap log.

As described above, one Bit in the bitmap is used to record the modification state of one unit of data, where the unit of data is customized by the user according to the actual situation, and may be, for example, 1M (1m = 1024kb) data, but in the data modification process, the 1M data may not be completely modified, but only a part of the data therein, such as 250kb, is modified, and then the corresponding target Bit may be referred to as an incompletely modified Bit, whereas, if the entire 1M data is completely modified, the corresponding target Bit may be referred to as a completely modified Bit.

On this basis, after determining each target Bit corresponding to the modified data, all target bits can be divided into a completely modified Bit and an incompletely modified Bit according to the actual modification condition of the target data corresponding to each target Bit, and then, when storing the bitmap log, the identifier pairs (composed of the modified identifier and the bitmap identifier) corresponding to the completely modified Bit and the incompletely modified Bit can be separately stored, wherein the identifier pair corresponding to the completely modified Bit is stored into the accurate bitmap log, and the identifier pair corresponding to the incompletely modified Bit is stored into the inaccurate bitmap log.

In an embodiment of the present invention, the storing the bitmap identifier corresponding to the completely modified Bit and the modified identifier corresponding to the completely modified Bit into the precise bitmap log may include:

judging whether the bitmap identification exists in the accurate bitmap log and the inaccurate bitmap log;

if no bitmap identification exists in the accurate bitmap log or the inaccurate bitmap log, the bitmap identification and the modification identification corresponding to the completely modified Bit are correspondingly stored in the accurate bitmap log;

if the accurate bitmap log stores the bitmap identifier, updating the corresponding existing identifier of the bitmap identifier in the accurate bitmap log by using the modification identifier to obtain a first updating identifier, and replacing the existing identifier with the first updating identifier;

and if the bitmap identification exists in the inaccurate bitmap log, updating the existing identification corresponding to the bitmap identification in the inaccurate bitmap log by using the modification identification to obtain a second update identification, and correspondingly storing the bitmap identification and the second update identification into the accurate bitmap log.

The embodiment of the invention provides a method for storing an identifier pair corresponding to a completely modified Bit into an accurate bitmap log. It can be understood that the log record of the modified data of the target file is performed in real time, when a user performs a data modification operation on the target file, the target data corresponding to the data modification operation may be the same for a plurality of times, that is, one data may be modified for a plurality of times, and then, for one unit data, in the case that the target data has been previously modified (may be the whole unit data modified, or may be a part of the unit data modified), the log corresponding to the unit data must have been recorded in the bitmap log (which may be an inaccurate bitmap log, or may be an accurate bitmap log), and at this time, the identification pair corresponding to the unit data in the bitmap log needs to be updated.

In the process of correspondingly storing the bitmap identification and the modification identification corresponding to the completely modified Bit into the accurate bitmap log, whether the bitmap identification exists in the accurate bitmap log and the inaccurate bitmap log or not can be judged first to determine whether the data corresponding to the completely modified Bit is modified or not, if the bitmap identification does not exist in the accurate bitmap log or the inaccurate bitmap log, the data corresponding to the completely modified Bit is not modified, and the bitmap identification and the modification identification corresponding to the completely modified Bit can be directly and correspondingly stored into the accurate bitmap log; if the bitmap identification exists in the accurate bitmap log, the data corresponding to the completely modified Bit is completely modified, and at this time, the modification identification corresponding to the bitmap identification can be used to update the existing identification (the identification recorded in the last modification) corresponding to the bitmap identification in the accurate bitmap log, so as to obtain a first update identification, and replace the existing identification with the first update identification; if the bitmap identification exists in the inaccurate bitmap log, the data corresponding to the completely modified Bit is partially modified, at this time, the modification identification corresponding to the bitmap identification can be used to update the existing identification (the identification recorded in the last modification) corresponding to the bitmap identification in the inaccurate bitmap log, so as to obtain a second update identification.

In an embodiment of the present invention, the updating the existing identifier corresponding to the bitmap identifier in the precision bitmap log by using the modification identifier to obtain the first update identifier may include: and carrying out binary OR operation on the modified identifier and the existing identifier to obtain a first updated identifier.

The embodiment of the invention provides an identifier updating method, which can directly carry out binary system or operation on a modification identifier and an existing identifier in an accurate bitmap log after the modification identifier corresponding to a Bit is completely modified in the data modification process is determined, so that the first updating identifier is obtained. Of course, the above-mentioned calculation manner of the second update identifier may be implemented in the same manner, and is not described herein again.

In an embodiment of the present invention, after the correspondingly storing the bitmap identifier and the second update identifier into the precision bitmap log, the method may further include: and deleting the bitmap identification and the existing identification which correspond to each other in the inaccurate bitmap log.

As described above, when the bitmap identifier corresponding to the Bit that is completely modified exists in the inaccurate bitmap log, the second update identifier may be obtained by updating the identifier, but since the bitmap identifier and the second update identifier are completely modified, the bitmap identifier and the second update identifier may be stored in the accurate bitmap log, and at this time, in order to avoid that the same modified unit data is recorded for many times and is repeated, the bitmap identifier and the existing identifier corresponding to each other in the inaccurate bitmap log may be directly deleted. That is, for a unit of data that is modified, only the most recent modification log is recorded.

In an embodiment of the present invention, the storing the bitmap identifier corresponding to the incompletely modified Bit and the modified identifier corresponding to the incompletely modified Bit in the inexact bitmap log may include:

judging whether bitmap identification exists in the accurate bitmap log and the inaccurate bitmap log or not;

if no bitmap identification exists in the accurate bitmap log and the inaccurate bitmap log, the bitmap identification corresponding to the incompletely modified Bit and the modified identification are correspondingly stored in the inaccurate bitmap log;

if the bitmap identification exists in the inaccurate bitmap log, the current identification corresponding to the bitmap identification in the inaccurate bitmap log is updated by using the modification identification to obtain a third update identification, and the bitmap identification and the third update identification are correspondingly stored in the inaccurate bitmap log.

The embodiment of the invention provides a method for storing an identifier pair corresponding to an incompletely modified Bit into an inaccurate bitmap log, which is similar to the method for storing the identifier pair corresponding to the completely modified Bit into the accurate bitmap log.

In the process of correspondingly storing the bitmap identification and the modification identification corresponding to the incompletely modified Bit into the inaccurate bitmap log, whether the bitmap identification exists in the accurate bitmap log and the inaccurate bitmap log or not can be judged first to determine whether the data corresponding to the incompletely modified Bit is modified or not, if the bitmap identification does not exist in the accurate bitmap log or the inaccurate bitmap log, the data corresponding to the incompletely modified Bit is not modified, and the bitmap identification and the modification identification corresponding to the incompletely modified Bit can be directly and correspondingly stored into the inaccurate bitmap log; if the bitmap identification exists in the accurate bitmap log, the data corresponding to the incompletely modified Bit is completely modified, and for the unit data which is completely modified before, log recording is not required for the incomplete modification, so that the bitmap identification and the modified identification corresponding to the incompletely modified Bit can be directly discarded; if the bitmap identification exists in the inaccurate bitmap log, it indicates that the data corresponding to the incompletely modified Bit is partially modified, at this time, the modification identification corresponding to the bitmap identification may be used to update the existing identification (the identification recorded at the last modification) corresponding to the bitmap identification in the inaccurate bitmap log, so as to obtain a third update identification, and the bitmap identification and the third update identification are correspondingly stored in the inaccurate bitmap log. Of course, the calculation manner of the third update flag may refer to the first update flag and the second update flag, which is not described herein again.

In an embodiment of the present invention, after storing the bitmap identifier corresponding to the incomplete modified Bit and the modified identifier corresponding to the incomplete modified Bit in the inexact bitmap log, the method may further include:

when the bitmap level of the bitmap corresponding to the target file is not 1, determining a target subordinate Bit corresponding to the target modification data and a target subordinate bitmap where the target subordinate Bit is located; the target lower bitmap is a lower bitmap with an incompletely modified Bit position;

determining a lower bitmap identification of the target lower bitmap, and generating a lower modification identification corresponding to target modification data according to the target lower Bit and the modification state recorded in the target lower Bit;

It can be understood that the inexact bitmap log is used for realizing log recording of unit data which is not completely modified, in order to further improve the precision of the difference log, the unit data which is not completely modified can be continuously divided into next-level precision, and a lower-level bitmap can be continuously created for Bit bits which are not completely modified. The specific implementation flow of the method can be exemplified by referring to fig. 4 described later.

In an embodiment of the present invention, the log management method may further include: when the incomplete modification Bit is updated to the complete modification Bit, the lower bitmap of the incomplete modification Bit is deleted.

It can be understood that when the incomplete modification Bit is updated to the complete modification Bit, it indicates that the unit data corresponding to the target Bit has been completely modified, and it is not necessary to continue to use the lower bitmap to perform multi-level log recording on part of the modified data in the unit data, so that the lower bitmap of the incomplete modification Bit can be directly deleted to avoid unnecessary waste of resources.

The embodiment of the invention provides another log management method.

The log management method provided by the embodiment of the invention mainly comprises four functional modules: data newly added logs, bitmap logs, precision grading indexes and log simplification.

1. Newly adding a log to the data:

referring to fig. 2, fig. 2 is a schematic diagram of a method for recording new logs of data according to the present invention. When the file1 (target file) is newly added for the first time or additional writing occurs, recording the data change of each time to a data newly added log: key is the offset of the file change (increasing the offset of the data in the file), and value is the length of the file change (increasing the data length). The data addition log in fig. 2 includes two logs, the first one: adding a new file, wherein a new write with the length of 350 occurs at the offset 0; a second bar: the file has been appended with a 150-length write at offset 350.

2. Bitmap log:

referring to fig. 3, fig. 3 is a schematic diagram of a bitmap logging method according to the present invention. When the modification occurs to the file1 (target file), the file length is not changed, and the bitmap log is used to record the difference. The principle of the bitmap log is mainly to use an int64 type bitmap to identify data changes, one Bit in the bitmap is used to identify 1M data, one int64 bitmap can identify 64M data, wherein a key value is a bitmap sequence number (bitmap identification), starting from 0 and being up to 63. When the file is modified, which bits of the file are changed can be calculated according to the offset and the change range, so that the key and the value which need to be recorded are calculated, and the value is sorted by using a small end. It should be noted that the bitmap log records only the changed Bit, and when there is a change in the repeated Bit, a new value may be recalculated and updated according to the newly changed Bit and the changed Bit that has been recorded.

Further, the bitmap log is divided into an exact bitmap log and an inexact bitmap log. The precision bitmap log records a completely changed data block (unit data), and when the data block size is 1M as shown in fig. 3, if the data range modified once completely covers the Bit, the Bit is recorded into the precision bitmap log. And recording the incompletely changed data block by the imprecise bitmap, and recording the Bit into the imprecise bitmap log if the data range modified once does not completely cover the Bit and the incompletely modified Bit does not exist in the exactness bitmap when the size of the data block is 1M. If a Bit that has not been completely modified is present in the precision bitmap, indicating that this Bit has been completely modified, then the change is ignored.

The implementation flow shown in fig. 3 has been modified 3 times, and the recording process of the bitmap log is as follows:

(1) First modification and rewriting: the offset of data modification is 1024 × 1024 × 2+512, and the change length is 1024 × 1024 × 5, and since the size of each data block is 1M, it can be calculated that the corresponding 3-8 blocks have changed. Because one int64 bit can identify the change of the 64-bit data block, the bitmap log is stored by using map, so that the key value of the bitmap log needing to be recorded is calculated to be 0. Of the data blocks 3 to 8, 4,5,6,7 are completely covered, the small-end ordering binary is 1111000, and the value is 120 after converting into int64, so the key value of the exact bitmap log needs to be recorded as 0, and the value is 120. Before the log is written into the accurate bitmap log, whether the accurate bitmap log and the inaccurate bitmap log have records corresponding to key values or not needs to be judged, and if yes, recalculation is needed; if not, the record is inserted directly. At this time, for the first time of modification, it is determined that there is no record with the key value of 0 in both the accurate bitmap log and the inaccurate bitmap log, and then "key:0, value:120 ". And 3, 8 data blocks in the 3 rd to 8 th data blocks only modify parts and are not completely covered, the binary system of the small end sequencing is 10000100, and the binary system is converted into int64 of 132, before the binary system is written into the inaccurate bitmap log, whether the record corresponding to the key value exists in the inaccurate bitmap log needs to be judged, if so, recalculation is needed, and if not, the record is directly inserted. At this time, the first time of modification and rewriting is performed, it is determined that no record with the key value of 0 exists in the inaccurate bitmap log, and then "key:0, value:132".

(2) And (3) second modification: the offset of data modification is 1024 × 1024 × 66, the change length is 1024 × 1024 × 2, and since the size of each data block is 1M, the corresponding 66 and 67 blocks are calculated to be changed, because one int64 bit can identify the change of a 64-bit data block, the bitmap log is stored by using map, and the key value of the bitmap log to be recorded can be calculated to be 1. The 66 th and 67 th data blocks are completely covered, the value is 12 after the data blocks are converted into int64 by using a small-end sorting binary system of 1100, so that the key value of the accurate bitmap log is required to be recorded to be 1, and the value is 12. Before writing into the accurate bitmap log, whether the record corresponding to the key value exists in the accurate bitmap log and the inaccurate bitmap log needs to be judged, if yes, recalculation is needed, and if not, the record is directly inserted. At this time, the second modification is performed, it is determined that there is no record with the key value of 1 in the accurate bitmap log and the inaccurate bitmap log, and then "key:1, value:12 ".

(3) And (3) modifying and rewriting for the third time: the offset of data modification is 1024 × 1024 × 65, the change length is 1024 × 1024 × 3+200, because the size of each data block is 1M, the corresponding 65-68 block is calculated to be changed, because one int64 Bit can identify the change of the 64-Bit data block, the bitmap log is stored by using map, the change of the bitmap log whose key value is 1 and value corresponds to Bit which needs to be recorded can be calculated by using small-end sorting, the data blocks 65, 66 and 67 in the data blocks 65-68 are completely covered, the binary value of the small-end sorting is 1110, the value is 14 after the data block is converted into int64, and therefore the key value of the accurate bitmap log needs to be recorded is 1, and the value is 14. Before writing into the accurate bitmap log, whether the accurate bitmap log and the inaccurate bitmap log have records corresponding to key values or not needs to be judged, if yes, recalculation is needed, and if not, direct insertion is carried out. At this time, for the third modification, it is determined that there is already a record with a key value of 1 in the precision bitmap, the value is 12, the new value record is 14, and binary or operation is performed on the new value 14 and the old value 12, and the result is 14, so that the value record with the key value of 1 in the precision bitmap log is updated to 14. In the data blocks 65 to 68, the data block 68 only modifies part of the data block, is not completely covered, uses a small-end ordering binary system as 10000, converts the data block into int64 and then into value as 16, and before writing into an inaccurate bitmap log, needs to judge whether the inaccurate bitmap log has records of corresponding key values, and if so, needs to recalculate, and if not, directly inserts the data block. At this time, the third time of modification is performed, it is determined that there is no record with the key value of 1 in the inaccurate bitmap log, and then "key:1, value:16".

When new modification writing is carried out subsequently, the accurate bitmap log and the inaccurate bitmap log can be modified or inserted according to the rules, the file size is 640M, and at most 10 records can be respectively inserted into the accurate bitmap log and the inaccurate bitmap log.

Before the accurate bitmap log is inserted, whether corresponding keys exist in the accurate bitmap log and the inaccurate bitmap log or not needs to be judged, if records exist in the accurate bitmap log, the records of corresponding key values in the accurate bitmap log are updated, and if not, the accurate bitmap log is directly inserted. If there is a record in the inaccurate bitmap log, which indicates that the originally locally modified Bit is completely modified, the corresponding value in the inaccurate bitmap log also needs to be recalculated, so as to clear the Bit converted into the accurate bitmap log record.

Before the inaccurate bitmap log is inserted, whether corresponding keys exist in the accurate bitmap log and the inaccurate bitmap log needs to be judged, and if the calculated change Bit exists in the accurate bitmap log, the record is ignored. Recalculation if the same Bit already exists in the inexact bitmap log, and immediate insertion if it does not exist.

3. Precision grading index:

referring to fig. 4, fig. 4 is a schematic diagram of a multi-level bitmap log recording method according to the present invention. The precision hierarchical index can record multi-level bitmap logs from coarse granularity to fine granularity according to precision, a lower-level bitmap log can be recorded only when the bitmap logs are inaccurate, and blocks of a higher-level inaccurate bitmap are divided into the lower-level accurate bitmap log and/or the inaccurate bitmap log by using lower-level precision. The block sequence number of the upper-level inaccurate bitmap log can be used as an index, each level of lower-level bitmap is obtained by re-dividing the upper-level inaccurate block, the inaccurate block of the lower-level bitmap may also have the lower-level bitmap, and each level of bitmap log has an independent bitmap log.

As shown in fig. 4, a 3-level index is adopted, the precision of the first-level bitmap is 1M, and the whole file is divided according to 1M. When a modified write occurs with an offset of 1024 × 1024 × 1 and a change length of 1024 × 1024 × 3+1024 × 5+5, the corresponding 2-5 blocks are changed. Wherein 2, 3 and 4 are complete modification and record to the accurate bitmap log, and 5 is incomplete modification and record to the inaccurate bitmap log.

Each imprecise bit of the first level corresponds to a secondary bitmap, the 5 th block of the first level in fig. 4 is an imprecise block, and the block number 5 serves as an index to point to the corresponding secondary bitmap. Because the precision of the first level bitmap is 1M, and the precision of the second level bitmap is 1kb, the second level bitmap is divided into 1M/1kb =1024 bits, the offset of the data mapped to the second level bitmap of the generated modified write action is 0, the change length is 1024 × 5+5, and the change is generated corresponding to 1-6 blocks. Wherein 1-5 blocks are completely modified and recorded into a secondary accurate bitmap log; and 6, recording to a secondary inaccurate bitmap log for incomplete modification.

Each inexact bit of the second level corresponds to a three-level bitmap, the 6 th block of the second level in fig. 4 is an inexact block, and the block number 6 points to the corresponding third-level bitmap as an index. Because the precision of the second level bitmap is 1kb and the precision of the third level bitmap is 1b, the third level bitmap is sliced to 1kb/1b =1024 bits. The offset of the data mapped to the third-level bitmap by the generated modified write behavior is 0, the change length is 5, corresponding 1-5 blocks are changed, all the blocks are completely modified, and the modified write behavior is recorded into a three-level accurate bitmap log.

The hierarchy and precision of the bitmap can be configured according to actual conditions, and the precision of the bitmap at the lowest level (tail level bitmap) is the basic recording unit of the change of the difference data, so that incomplete modified data blocks cannot occur at the tail level.

4. The log is simplified:

referring to fig. 5, fig. 5 is a schematic diagram of a log reduction method provided by the present invention. Each time a modification occurs, if the inaccurate bit is completely modified, it is converted into an accurate bit, and the lower bitmap of the corresponding bit is deleted to simplify the log. As shown in FIG. 5, after the first modified write, three levels of bitmaps are generated, each level having an independent bitmap log; and at the second time of modification, the 6 th block of the second-level bitmap is converted from the incomplete modification state into a complete modification state, at this time, the inexact bitmap log of the second-level bitmap can be converted into an exact bitmap log, and meanwhile, the correspondingly indexed third-level bitmap log is deleted.

The embodiment of the invention provides a log management device.

Referring to fig. 6, fig. 6 is a schematic structural diagram of a log management device provided in the present invention, where the log management device may include:

the monitoring module 1 is used for carrying out data monitoring on the target file and determining modified data in the target file;

the determining module 2 is used for determining a target Bit corresponding to the modified data and a target bitmap where the target Bit is located in all bitmaps corresponding to the target file; wherein, each Bit in each bitmap is used for recording the modification state of corresponding data in the target file;

the generating module 3 is used for determining a bitmap identifier of the target bitmap and generating a modification identifier corresponding to the modified data according to the target Bit and the modification state recorded in the target Bit;

and the storage module 4 is used for correspondingly storing the bitmap identification and the modification identification to the bitmap log corresponding to the target file.

It can be seen that, the log management apparatus provided in the embodiment of the present invention creates a bitmap corresponding to each target file in a storage system, and records the modification state of each data in the target file through each Bit in the bitmap, so that the modified data in the target file can be determined by performing data monitoring on the target file, and further determines a target Bit corresponding to the modified data and a target bitmap where the target Bit is located, and further, a modification identifier corresponding to the modified data is generated by using the target Bit and the modification state recorded inside the target Bit, and is corresponding to the bitmap identifier of the target bitmap and stored in a bitmap log, thereby realizing that the modified data in the target file is recorded through the bitmap log, that is, a difference log is realized based on the bitmap log.

In an embodiment of the present invention, the monitoring module 1 may be specifically configured to perform data monitoring on the target file, and determine a data change type when it is monitored that data change occurs in the target file; and if the data change type is data modification operation, determining modified data in the target file.

In an embodiment of the present invention, the data change type may further include a data add operation, and the log management apparatus may further include a data add logging module, configured to determine, when the data change type is the data add operation, to add data in the target file; determining an offset address and an incremental data length of the incremental data; and correspondingly storing the offset address and the increased data length to a data newly-increased log corresponding to the target file.

In an embodiment of the present invention, the incremental data logging module may be specifically configured to generate a key-value pair corresponding to the incremental data according to the offset address and the incremental data length; wherein a key value in the key value pair is used for recording the offset address, and a value in the key value pair is used for recording the increased data length; and recording the key value pairs to a data newly-added log corresponding to the target file.

In an embodiment of the present invention, the log management apparatus may further include a data synchronization module, configured to determine, when a data synchronization request is received, difference data according to the bitmap log and the data addition log; performing a data synchronization operation on the difference data.

In an embodiment of the present invention, the determining module 2 may be specifically configured to determine an offset address and a modified data length of the modified data; and inquiring to obtain a target Bit corresponding to the modified data and a target bitmap where the target Bit is located in all bitmaps corresponding to the target file according to the offset address and the length of the modified data.

In one embodiment of the present invention, when the record value in the target Bit is 1, the modified state is modified, and when the record value in the target Bit is 0, the modified state is unmodified;

accordingly, the generating module 3 may include:

the combination unit is used for combining the recording values in the target Bit bits into binary modification identifiers;

a determining unit configured to determine a data type of the bitmap;

and the conversion unit is used for converting the binary modification identifier according to the data type to obtain the modification identifier.

In an embodiment of the present invention, the combining unit may be specifically configured to combine the record values in each target Bit into the binary modification flag according to a small-end sorting method.

In an embodiment of the present invention, the storage module 4 may be specifically configured to generate a key-value pair corresponding to the modified data according to the bitmap identifier and the modification identifier; the key value in the key value pair is used for recording the bitmap identification, and the value in the key value pair is used for recording the modification identification; and recording the key value pairs to a bitmap log corresponding to the target file.

In an embodiment of the present invention, the bitmap log may include an exact bitmap log and an inaccurate bitmap log, and the log management apparatus may further include a Bit determination module, configured to determine, in all bitmaps corresponding to the target file, a target Bit corresponding to the modified data and a target bitmap where the target Bit is located, and then determine, in all target bits, a fully modified Bit and an incompletely modified Bit;

accordingly, the storage module 4 may include:

the first storage unit is used for correspondingly storing the bitmap identification and the modification identification corresponding to the completely modified Bit to the accurate bitmap log;

and the second storage unit is used for correspondingly storing the bitmap identification and the modification identification corresponding to the incomplete modification Bit to the inaccurate bitmap log.

In an embodiment of the present invention, the first storage unit may be specifically configured to determine whether the bitmap identifier exists in the precise bitmap log and the imprecise bitmap log; if the bitmap identification does not exist in the accurate bitmap log or the inaccurate bitmap log, the bitmap identification and the modification identification corresponding to the completely modified Bit are correspondingly stored in the accurate bitmap log; if the bitmap identification exists in the accurate bitmap log, updating the existing identification corresponding to the bitmap identification in the accurate bitmap log by using the modification identification to obtain a first update identification, and replacing the existing identification with the first update identification; and if the bitmap identification exists in the inaccurate bitmap log, updating the existing identification corresponding to the bitmap identification in the inaccurate bitmap log by using the modification identification to obtain a second update identification, and correspondingly storing the bitmap identification and the second update identification to the accurate bitmap log.

In an embodiment of the present invention, the first storage unit may be specifically configured to perform a binary or operation on the modification identifier and the existing identifier to obtain the first update identifier.

In an embodiment of the present invention, the first storage unit may be further configured to delete the bitmap identifier and the existing identifier, which correspond to each other in the inexact bitmap log, after the bitmap identifier and the second update identifier are correspondingly stored in the exact bitmap log.

In an embodiment of the present invention, the second storage unit may be specifically configured to determine whether the bitmap identifier exists in the precise bitmap log and the imprecise bitmap log; if the bitmap identification does not exist in the precise bitmap log or the imprecise bitmap log, the bitmap identification and the modification identification corresponding to the incompletely modified Bit are correspondingly stored in the imprecise bitmap log; if the bitmap identification exists in the accurate bitmap log, discarding the bitmap identification and the modification identification corresponding to the incompletely modified Bit; and if the bitmap identification exists in the inaccurate bitmap log, updating the existing identification corresponding to the bitmap identification in the inaccurate bitmap log by using the modification identification to obtain a third update identification, and correspondingly storing the bitmap identification and the third update identification to the inaccurate bitmap log.

In an embodiment of the present invention, the log management apparatus may further include a lower level storage module, configured to determine target modification data corresponding to the incomplete modification Bit after the bitmap identifier and the modification identifier corresponding to the incomplete modification Bit are correspondingly stored in the inexact bitmap log; when the bitmap level of the bitmap corresponding to the target file is not 1, determining a target next-level Bit corresponding to the target modification data and a target next-level bitmap where the target next-level Bit is located; wherein the target lower level bitmap is the lower level bitmap of the incomplete modified Bit; determining a lower bitmap identification of the target lower bitmap, and generating a lower modification identification corresponding to the target modification data according to the target lower Bit and the modification state recorded in the target lower Bit; and correspondingly storing the lower bitmap identification and the lower modification identification to the lower bitmap log corresponding to the inaccurate bitmap log until the tail bitmap log is completely stored.

In an embodiment of the present invention, the log management apparatus may further include a lower deletion module configured to delete a lower bitmap of the incomplete modification Bit when the incomplete modification Bit is updated to the complete modification Bit.

In an embodiment of the present invention, each bitmap corresponding to the target file may be an int64 integer bitmap.

For the introduction of the apparatus provided in the embodiment of the present invention, please refer to the above method embodiment, which is not described herein again.

The embodiment of the invention provides electronic equipment.

Referring to fig. 7, fig. 7 is a schematic structural diagram of an electronic device according to the present invention, where the electronic device includes:

a memory for storing a computer program;

a processor, configured to execute a computer program, may implement the steps of any of the log management methods described above.

As shown in fig. 7, which is a schematic diagram of a structure of an electronic device, the electronic device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through a communication bus 13.

In the embodiment of the present invention, the processor 10 may be a Central Processing Unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, etc.

The processor 10 may call a program stored in the memory 11, and in particular, the processor 10 may perform operations in an embodiment of the log management method.

The memory 11 is used for storing one or more programs, the program may include program codes, the program codes include computer operation instructions, and in the embodiment of the present invention, at least the program for implementing the following functions is stored in the memory 11:

In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created during use.

Further, the memory 11 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device or other volatile solid state storage device.

The communication interface 12 may be an interface of a communication module for connecting with other devices or systems.

Of course, it should be noted that the structure shown in fig. 7 does not constitute a limitation to the electronic device in the embodiment of the present invention, and in practical applications, the electronic device may include more or less components than those shown in fig. 7, or some components in combination.

An embodiment of the present invention provides a computer-readable storage medium.

The computer-readable storage medium provided by the embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, the steps of any one of the log management methods can be implemented.

The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

For the introduction of the computer-readable storage medium provided by the embodiment of the present invention, please refer to the above method embodiment, which is not described herein again.

The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed in the embodiment corresponds to the method disclosed in the embodiment, so that the description is simple, and the relevant points can be referred to the description of the method part.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The technical solution provided by the present invention is described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, without departing from the principle of the present invention, it can make several improvements and modifications to the present invention, and those improvements and modifications also fall into the protection scope of the present invention.

Claims

1. A method of log management, comprising:

2. The log management method of claim 1, wherein the monitoring data of the target file and determining the modified data in the target file comprises:

3. The log management method of claim 2, wherein the data change type further comprises a data add operation, the method further comprising:

4. The log management method of claim 3, wherein the storing the offset address and the increased data length correspondence to a data new log corresponding to the target file comprises:

5. The log management method of claim 3, further comprising:

performing a data synchronization operation on the difference data.

6. The log management method according to claim 1, wherein the determining, in all bitmaps corresponding to the target file, a target Bit corresponding to the modified data and a target bitmap in which the target Bit is located includes:

determining an offset address and a modified data length of the modified data;

7. The log management method as claimed in claim 1, wherein when the record value in the target Bit is 1, the modified state is modified, and when the record value in the target Bit is 0, the modified state is unmodified;

combining the recording values in the target Bit bits into binary modification identifiers;

determining a data type of the bitmap;

8. The log management method of claim 7, wherein said combining the recorded values in each of the target Bit bits into a binary modification identifier comprises:

9. The log management method of claim 1, wherein the storing the bitmap identification and the modification identification into the bitmap log corresponding to the target file comprises:

10. The log management method according to any one of claims 1 to 9, wherein the bitmap log includes an exact bitmap log and an inexact bitmap log; after determining the target Bit corresponding to the modified data and the target bitmap in which the target Bit is located in all the bitmaps corresponding to the target file, the method further includes:

11. The log management method according to claim 10, wherein the storing the bitmap identification and the modified identification corresponding to the fully modified Bit into the precise bitmap log comprises:

12. The log management method according to claim 11, wherein the updating the existing identifier corresponding to the bitmap identifier in the precision bitmap log by using the modification identifier to obtain a first updated identifier comprises:

13. The log management method according to claim 11, wherein after storing the bitmap identification and the second update identification in correspondence to the precision bitmap log, the method further comprises:

14. The log management method according to claim 10, wherein the storing the bitmap identification and the modified identification corresponding to the incomplete modified Bit into the inexact bitmap log comprises:

if the bitmap identification does not exist in the precise bitmap log or the imprecise bitmap log, the bitmap identification and the modification identification corresponding to the incompletely modified Bit are correspondingly stored in the imprecise bitmap log;

15. The log management method according to claim 10, wherein after storing the bitmap identification and the modified identification corresponding to the incomplete modified Bit into the inexact bitmap log, the method further comprises:

16. The log management method of claim 15, further comprising:

17. The log management method of claim 1, wherein each of the bitmaps corresponding to the target file is an int64 integer bitmap.

18. A log management apparatus, comprising:

the generating module is used for determining the bitmap identification of the target bitmap and generating a modification identification corresponding to the modified data according to the target Bit and the modification state recorded in the target Bit;

19. An electronic device, comprising:

a memory for storing a computer program;

a processor for implementing the steps of the log management method as claimed in any one of claims 1 to 17 when executing said computer program.

20. A computer-readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, carries out the steps of the log management method according to any one of claims 1 to 17.