CN112860649A

CN112860649A - Method, device and system for generating index in increment manner

Info

Publication number: CN112860649A
Application number: CN202110150195.8A
Authority: CN
Inventors: 陈元强; 林柯
Original assignee: Shenzhen Mulang Cloud Data Co ltd
Current assignee: Shenzhen mulangyun Technology Co.,Ltd.
Priority date: 2021-02-03
Filing date: 2021-02-03
Publication date: 2021-05-28

Abstract

The disclosure discloses a method, a device and a system for generating indexes in an increment mode. Wherein, the method comprises the following steps: reading a log of a file system; analyzing the information related to the change of the file system in a period of time from the log; generating an index of the file system in an incremental manner based on the parsed information about the changes. The present disclosure solves the technical problem of [ very large consumption of resources and time due to complete scanning of a file system ].

Description

Method, device and system for generating index in increment manner

Technical Field

The present disclosure relates to the field of data storage, and in particular, to a method, an apparatus, and a system for incrementally generating an index.

Background

In the big data era, the amount of data stored on computers has increased geometrically, and the number of files in file systems is often counted in billions, even billions and billions. These files cannot be searched in various usage scenarios of the data. Searching in a massive file system is a very time and resource consuming matter. To search quickly and also to reduce resource usage, stored files need to be indexed.

The entire file system is scanned and all file metadata therein is recorded in an index database. When searching, firstly searching in the index, and then positioning to a specific file in the file system according to the file path in the record after finding. In order to update changes made in the file system into the index, the file system needs to be periodically scanned.

However, the above method has the following problems:

(1) the consumption of time and resources is still very large per complete scan of the file system.

(2) Changes made to the file system over time may only account for a portion of the entire file system, with too many repetitive tasks per complete scan.

In view of the above problems, no effective solution has been proposed.

Disclosure of Invention

The embodiment of the disclosure provides a method, a device and a system for generating an index in an incremental manner, so as to at least solve the technical problem that the consumption of resources and time is very large due to the fact that a file system is completely scanned.

According to an aspect of the disclosed embodiment, there is provided a method for incrementally generating an index, including: reading a log of a file system; analyzing the information related to the change of the file system in a period of time from the log; generating an index of the file system in an incremental manner based on the parsed information about the changes.

According to another aspect of the disclosed embodiments, there is also provided a reading module configured to read a log of a file system; the analysis module is configured to analyze information related to changes of the file system in a period of time from the log; a delta generation module configured to generate an index of the file system in a delta manner based on the parsed information about the changes.

According to another aspect of the embodiments of the present disclosure, there is also provided an index generating system, including a file system and the apparatus for incrementally generating an index according to any one of the above technical solutions.

According to another aspect of the embodiments of the present disclosure, there is also provided a computer device including a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the incremental generation indexing method as described in any one of the above.

According to another aspect of the embodiments of the present disclosure, there is also provided a readable storage medium storing a computer program, wherein the computer program is configured to implement the incremental generation indexing method according to any one of the above items when the computer program is executed by a processor.

In the embodiment of the present disclosure, an index is generated in an incremental manner, and an index of a file system is generated in an incremental manner by analyzing relevant information of changes of the file system in a period of time from a log of the file system, so that a technical problem that resources and time are consumed very much due to complete scanning of the file system is solved, and a technical effect of saving resources and time is achieved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the disclosure and together with the description serve to explain the disclosure and not to limit the disclosure. In the drawings:

FIG. 1 is a schematic flow diagram of an alternative method of generating an index according to an embodiment of the present disclosure;

FIG. 2 is a schematic flow diagram of another alternative method of generating an index according to an embodiment of the present disclosure;

FIG. 3 is a block diagram of an alternative means for generating an index in accordance with embodiments of the present disclosure;

FIG. 4 is a block diagram of an alternative generate index system according to embodiments of the present disclosure.

Detailed Description

In order to make the technical solutions of the present disclosure better understood by those skilled in the art, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present disclosure, and it is apparent that the described embodiments are only partial embodiments of the present disclosure, not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present disclosure without any creative effort shall fall within the protection scope of the present disclosure.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Definition of key terms

A file system: the system organizes and distributes the space of the file storage device, is responsible for storing the files and protects and retrieves the stored files. Data in a file system is divided into data and metadata.

Data block (data block or block): the storage device performs the minimum unit of data reading and writing files, and the data of one file can be written on a plurality of data blocks.

Data: actual data in a common file.

Metadata (meta data): system data used to describe characteristics of a file, such as the creator of the file, the file size, access rights, distribution information of file data blocks, and the like, is stored in records of the inode or MFT.

inode: an area for storing file metadata in a file system of the unix-like operating system. Each inode has a unique number in a file system.

MFT: the Master File Table (Master File Table), a contiguous disk area of the NTFS File system, is composed of a series of metadata records of files and directories, including all the File and directory records in the NTFS File system. The corresponding record can be found by the MFT number.

Indexing: the file records in the file system, including the file path, the inode number (or MFT number), the file size, the file type (the directory can also be considered as a special file), etc., are grouped together in a structured format, such as stored in a database of MySQL, sqlite, etc., so that the file in the whole file system can be quickly searched. The carriers of these sets of file records are called indices.

SUMMARY

The currently popular file systems, such as NTFS, EXT4, and XFS, are basically journaling file systems (journaling file systems). The journaling file system will draw a continuous area within the file system to record the journal of the file system, which is called a journal area. Before writing data into a specific data block, the file system firstly writes the metadata of the file to be changed into a journal area, and then updates the metadata and the data block of the corresponding file. Replaying the log area records in the event of a system crash can allow for faster recovery of the file system, protecting the file system from damage, without the need to scan the entire file system.

The information in the log area records information related to the changed file, such as a flag of the file (an inode number of the file in the XFS file system and the EXT4 file system, and an MFT number in the NTFS file system), an operation type (e.g., create file, modify file, delete file, etc.), a change of a corresponding data block, and the like. When updating the index, incremental updates may be implemented by records in the log area.

The access modes of the log area information of different file systems are different. The logs of the XFS file system may export log information to a common file via an "XFS _ logprint" command. The log of the EXT4 file system is typically stored in a file with inode number 8 (this file is hidden from view). The logs of the NTFS file system are stored in the "$ LogFile" file under each packing directory.

Each piece of information in the log area corresponds to one file change operation, such as new creation, modification and deletion. Traversing all the information in the log area can reproduce the change of the file system in a period of time.

Embodiment mode 1

According to an embodiment of the present disclosure, there is provided a method for incrementally generating an index, as shown in fig. 1, the method including:

step S101, judging whether the log is updated for the first time.

In the case where the log is first updated, step S109 is executed, otherwise, step S102 is executed.

Step S102, judging whether the file system is a log file system.

In the case where the file system is a journaling file system, step S103 is executed, otherwise, step S109 is executed.

Step S103, judging whether the log is stored in the file system.

In case the log is stored inside the file system, step S104 is performed, otherwise, step S111 is performed.

Step S104, reading the file system log.

Step S105, judging whether the log is sufficient.

And analyzing whether the read log meets the condition of incremental updating. The log area size of the file system is fixed, and writing from the beginning after writing to the end can overwrite the existing content. When the log required by the incremental update is covered and the log is not sufficient, step S109 is executed to perform the full update, otherwise, step S106 is executed.

And step S106, traversing the log.

Traversing the records in the log, and analyzing the file information from the records, wherein the file information comprises the following steps: a file flag (inode number of a file in the XFS file system and the EXT4 file system, MFT number in the NTFS file system), an operation type, and the like.

In step S107, the operation type is determined.

The operation types recorded in the log for operating the file comprise: creation, modification, and deletion. If the operation type is delete, step S110 is executed to delete the corresponding record in the index. In the case where the operation type is creation or modification, step S108 is performed.

And step S108, acquiring file attributes according to the log records.

According to the file marks (the inode number of the file in the XFS file system and the EXT4 file system, and the MFT number in the NTFS file system) in the log record, more complete file attributes such as file path, file size, file type and the like are obtained.

It jumps to step S110.

Step S109, scans the file system.

And performing total updating and scanning the whole file system. After that, step S110 is performed.

Step S110, the index is updated.

And under the condition of generating the index in an increment mode, updating the index based on the file attribute acquired from the log, wherein the updated index is the full index of the whole file system.

In the case of a full generation index, the index is updated based on the file attributes obtained by scanning the file system.

And ending the flow.

And step S111, installing the plug-in and backing up the log.

For a file system using an external device to store a log area, a plug-in may be installed in the target operating system, and after the backup is successful, the log may be obtained and transmitted to the backup server through the plug-in, and then step S104 is performed.

Embodiment mode 2

After the virtual machine is backed up, a file index needs to be made to the file system in the disk mirror image. The mirror image is mounted in a loop device mode, if the mirror image is not backed up for the first time and is a log file system, a log is obtained, the log is analyzed, the log in the period is determined not to be covered if the earliest recording time in the log is less than the last backup time, and the log is traversed from the last backup time to update the incremental index.

Specifically, as shown in fig. 2, the method comprises the following steps:

s201, judging whether a preset condition is met.

And judging whether the file system meets a preset condition or not. Wherein the preset conditions are as follows: 1) the index exists in the file system, i.e., the index is not first updated or backed up; 2) the file system is a log file system; 3) the log is stored inside the file system or, although the log is stored in an external device, the log has been copied by installing a plug-in.

If the preset condition is satisfied, step S202 is executed, otherwise, step S205 is executed.

Step S202, reading the log of the file system.

And reading the log of the file system. Whether the log meets the condition of incremental updating is analyzed. The condition for the incremental update is that the earliest record time in the log is earlier than the time of the last update of the index.

Executing step S203 on the condition that the log satisfies the incremental update; under the condition that the log does not satisfy the incremental update, the whole file system is scanned, and the index is generated in a full-scale mode.

Step S203, analyzing information related to changes of the file system occurring within a period of time from the log.

And traversing the records in the log from the record in the log corresponding to the last updating time of the index, and analyzing the relevant information of the change of the file system from the last updating time to the current time period from the records. Wherein the related information comprises at least one of: file flag and operation type.

Wherein the file mark is an inode number of a file when the file system is an XFS file system or an EXT4 file system, and is an MFT number when the file system is an NTFS file system. The operation type includes at least one of: delete, modify, and create.

Step S204, based on the analyzed relevant information of the change, generating an index of the file system in an incremental mode.

And deleting the corresponding record in the index under the condition that the operation type is deletion.

Under the condition that the operation type is creation or modification, acquiring file attributes based on the file flag, and updating the file attributes into the index, wherein the file attributes comprise at least one of the following: file path, file size and file type.

Step S205 scans the entire file system and generates an index in a full-scale manner.

It is noted that while for simplicity of explanation, the foregoing method embodiments are described as a series of acts or combination of acts, it will be appreciated by those skilled in the art that the present disclosure is not limited by the order of acts, as some steps may, in accordance with the present disclosure, occur in other orders and concurrently. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required for the disclosure.

Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present disclosure.

Embodiment 3

According to an embodiment of the present disclosure, there is also provided a method for implementing the above incremental generation index, as shown in fig. 3, the method includes: a reading module 32, a parsing module 34, and an increment generation module 36.

The reading module 32 is configured to read a log of the file system;

the parsing module 34 is configured to parse the log to obtain information about changes that have occurred to the file system over a period of time;

the delta generation module 36 is configured to generate an index of the file system in an incremental manner based on the parsed information about the changes.

Embodiment 4

According to the embodiment of the present disclosure, there is also provided an index generating system, which includes a file system 42 and an apparatus 44 for incrementally generating an index, where the apparatus 44 for incrementally generating an index is configured to implement any one of the above methods for incrementally generating an index.

Embodiment 5

The embodiment of the disclosure also provides a computer device. Comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, the processor executing the computer program for implementing any of the above methods of incrementally generating an index.

Embodiments of the present disclosure also provide a computer storage medium. Optionally, in this embodiment, the storage medium stores a computer program, and the computer program is used to implement any one of the above methods for incrementally generating an index when executed by a processor.

Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.

The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, network devices, etc.) to execute all or part of the steps of the method according to the embodiments of the present disclosure.

In the above embodiments of the present disclosure, the description of each embodiment is focused on, and for parts that are not described in detail in a certain embodiment, reference may be made to the description of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, and for example, the division of the units is merely a logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units of ` A ` can be selected according to actual needs to achieve the purpose of the embodiment.

In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

Embodiments of the present disclosure may also be configured to:

1. a method of incrementally generating an index, comprising:

reading a log of a file system;

analyzing the information related to the change of the file system in a period of time from the log;

generating an index of the file system in an incremental manner based on the parsed information about the changes.

2. The method according to item 1, wherein,

reading a log of a file system is performed in case the file system satisfies a preset condition.

3. The method of item 2, wherein the preset conditions include:

the index exists in the file system; and

the file system is a log file system.

4. The method of item 3, wherein the preset conditions further comprise: the journal is stored within the file system.

5. The method of item 3, wherein the preset conditions further comprise: in case that the log is stored in the external device, the log is copied by installing a plug-in.

6. The method of any of items 2 to 5, wherein the method further comprises: and under the condition that the preset condition is not met, scanning the whole file system, and generating the index in a full-scale mode.

7. The method of item 1, wherein after reading the log of the file system, the method further comprises:

analyzing whether the log meets the condition of incremental update, wherein the analysis of the information related to the change of the file system in a period of time from the log is performed under the condition that the log meets the condition of incremental update.

8. The method of item 7, wherein the method further comprises: and under the condition that the log does not meet the incremental update, scanning the whole file system and generating the index in a full-scale mode.

9. The method of item 7, wherein the condition of the incremental update is that an earliest record time in the log is earlier than a time of a last update of the index.

10. The method of item 1, wherein parsing out information about changes to the file system over a period of time from the log comprises: traversing the records in the log from the record in the log corresponding to the last update time of the index, and analyzing the relevant information of the change of the file system from the last update time to the current time period from the record, wherein the relevant information comprises at least one of the following information: file flag and operation type.

11. The method of item 10, wherein,

the file mark is an inode number of a file when the file system is an XFS file system or an EXT4 file system, and is an MFT number when the file system is an NTFS file system; and/or

The operation type includes at least one of: delete, modify, and create.

12. The method of item 11, wherein incrementally generating an index of the file system based on the parsed information about the changes comprises:

deleting the corresponding record in the index under the condition that the operation type is deletion;

13. The method according to any one of items 1 to 12, wherein the method is applied to a scene in which a file index is generated for a file system in a disk image after the virtual machine is backed up.

14. The method of item 13, wherein prior to reading the log of the file system, the method further comprises mounting the image in a loopback device.

15. An apparatus for incrementally generating an index, comprising:

a reading module configured to read a log of a file system;

the analysis module is configured to analyze information related to changes of the file system in a period of time from the log;

a delta generation module configured to generate an index of the file system in a delta manner based on the parsed information about the changes.

16. The apparatus of item 15, wherein,

17. The apparatus of item 16, wherein the preset conditions include:

the index exists in the file system; and

the file system is a log file system.

18. The apparatus of item 17, wherein the preset conditions further comprise: the journal is stored within the file system.

19. The apparatus of item 17, wherein the preset conditions further comprise: in case that the log is stored in the external device, the log is copied by installing a plug-in.

20. The apparatus of any of claims 16 to 19, further comprising a full-scale generation module configured to: and under the condition that the preset condition is not met, scanning the whole file system, and generating the index in a full-scale mode.

21. The apparatus of claim 15, further comprising a determining module configured to:

22. The apparatus of item 21, further comprising a full-scale generation module configured to: and under the condition that the log does not meet the incremental update, scanning the whole file system and generating the index in a full-scale mode.

23. The apparatus of item 21, wherein the condition of the incremental update is that an earliest record time in the log is earlier than a time of a last update of the index.

24. The apparatus of item 15, wherein parsing out information about changes to the file system over a period of time from the log comprises: traversing the records in the log from the record in the log corresponding to the last update time of the index, and analyzing the relevant information of the change of the file system from the last update time to the current time period from the record, wherein the relevant information comprises at least one of the following information: file flag and operation type.

25. The apparatus of item 24, wherein,

The operation type includes at least one of: delete, modify, and create.

26. The apparatus of item 25, wherein the delta generation module is further configured to: incrementally generating an index of the file system based on the parsed information about the changes comprises:

27. The apparatus according to any one of items 15 to 16, wherein the apparatus is applied to a scenario in which a file index is generated for a file system in a disk image after a virtual machine backup.

28. The apparatus of item 27, wherein the reading module is further configured to: before reading the log of the file system, the mirror is mounted in a loop device mode.

29. An index generation system comprising a file system and means for incrementally generating an index as described in any of items 15-28.

30. A computer device comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, the processor executing the computer program for implementing the method of any of items 1-14.

31. A readable storage medium, storing a computer program, wherein the computer program is adapted to be executed by a processor to implement the method according to any of the claims 1-14.

The foregoing is merely a preferred embodiment of the present disclosure, and it should be noted that modifications and embellishments could be made by those skilled in the art without departing from the principle of the present disclosure, and these should also be considered as the protection scope of the present disclosure.

Claims

1. A method of incrementally generating an index, comprising:

reading a log of a file system;

2. The method of claim 1,

3. The method according to claim 2, wherein the preset conditions include:

the index exists in the file system; and

the file system is a log file system.

4. The method of claim 3, wherein the preset condition further comprises: the journal is stored within the file system.

5. The method of claim 3, wherein the preset condition further comprises: in case that the log is stored in the external device, the log is copied by installing a plug-in.

6. The method according to any one of claims 2 to 5, further comprising: and under the condition that the preset condition is not met, scanning the whole file system, and generating the index in a full-scale mode.

7. The method of claim 1, wherein after reading the log of the file system, the method further comprises:

8. The method of claim 7, further comprising: and under the condition that the log does not meet the incremental update, scanning the whole file system and generating the index in a full-scale mode.

9. An apparatus for incrementally generating an index, comprising:

a reading module configured to read a log of a file system;

10. An index generation system comprising a file system and the means for incrementally generating an index as recited in claim 9.