CN111367861A

CN111367861A - File caching method, system, device and medium

Info

Publication number: CN111367861A
Application number: CN202010132734.0A
Authority: CN
Inventors: 侯斌
Original assignee: Suzhou Inspur Intelligent Technology Co Ltd
Current assignee: Suzhou Inspur Intelligent Technology Co Ltd
Priority date: 2020-02-29
Filing date: 2020-02-29
Publication date: 2020-07-03

Abstract

The invention discloses a file caching method, which comprises the following steps: creating a dual-layer directory in a storage device; acquiring an absolute path of a file to be cached; computing a hash value based on the absolute path; determining the position of the file to be cached in the double-layer directory according to the hash value, and determining the file name of the file to be cached according to the number of the cached files; and caching the file to be cached named by the file name to a corresponding position under the double-layer directory. The invention also discloses a system, a computer device and a readable storage medium. When the file is stored on the storage device of the client, the file is hashed into a two-layer directory structure, so that the picture files stored in each directory are not too many, and the performance problem of excessive single-directory file number is avoided.

Description

File caching method, system, device and medium

Technical Field

The invention relates to the field of caching, in particular to a file caching method, a file caching system, file caching equipment and a storage medium.

Background

With the vigorous development of the internet, various application scenes generate massive small file images, and the massive small file images are stored in a distributed storage mode in a file mode. Because the cost of the existing full-flash distributed system is still high or low, the storage media of the small pictures still mainly use a mechanical disk, which brings a problem, when AI training is performed on the massive picture disks, the small file pictures need to be read for many times, and the reading of the massive small files becomes a performance bottleneck because the random reading performance of the mechanical disk is poor.

Disclosure of Invention

In view of the above, in order to overcome at least one aspect of the above problems, an embodiment of the present invention provides a file caching method, including the following steps:

creating a dual-layer directory in a storage device;

acquiring an absolute path of a file to be cached;

computing a hash value based on the absolute path;

determining the position of the file to be cached in the double-layer directory according to the hash value, and determining the file name of the file to be cached according to the number of the cached files;

and caching the file to be cached named by the file name to a corresponding position under the double-layer directory.

In some embodiments, creating a dual-layer directory in the storage device further comprises:

creating a first-layer directory containing a preset number of first subdirectories;

and creating a second-layer directory containing a preset number of second sub-directories under each first sub-directory.

In some embodiments, determining, according to the hash value, a location of the file to be cached in the dual-layer directory further includes:

determining a first subdirectory according to the numerical value of a first preset position in the hash value;

and determining a second subdirectory under the first subdirectory according to the numerical value of a second preset position in the hash value.

In some embodiments, determining the file name of the file to be cached according to the number of cached files further includes:

counting the number of shapes of the cached files by a preset digit, and adding one to the number of shapes when caching one file;

and taking the integer number as the file name of the file to be cached.

In some embodiments, further comprising:

taking the absolute path of the file to be cached as a key value, and taking the file name as a value to establish mapping between the absolute path of the file to be cached and the file name;

and acquiring the files cached in the storage equipment according to the mapping.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a file caching system, including:

a creation module configured to create a dual-layer directory in a storage device;

the system comprises an acquisition module, a cache module and a cache module, wherein the acquisition module is configured to acquire an absolute path of a file to be cached;

a computing module configured to compute a hash value based on the absolute path;

the determining module is configured to determine the position of the file to be cached in the double-layer directory according to the hash value and determine the file name of the file to be cached according to the number of cached files;

and the caching module is configured to cache the file to be cached named by the file name to a corresponding position under the double-layer directory.

In some embodiments, the caching module is further configured to:

In some embodiments, the determination module is further configured to:

determining a second subdirectory under the first subdirectory according to a numerical value of a second preset position in the hash value;

and taking the integer number as the file name of the file to be cached.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer apparatus, including:

at least one processor; and

a memory storing a computer program operable on the processor, wherein the processor executes the program to perform any of the steps of the file caching method described above.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which when executed by a processor performs the steps of any of the file caching methods described above.

The invention has one of the following beneficial technical effects: when the file is stored on the storage device of the client, the file is hashed into a two-layer directory structure, so that the picture files stored in each directory are not too many, and the performance problem of excessive single-directory file number is avoided.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.

Fig. 1 is a schematic flowchart of a file caching method according to an embodiment of the present invention;

fig. 2 is a schematic structural diagram of a file caching system according to an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a computer device provided in an embodiment of the present invention;

fig. 4 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.

It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.

According to an aspect of the present invention, an embodiment of the present invention provides a file caching method, as shown in fig. 1, which may include the steps of: s1, creating a double-layer directory in the storage device; s2, acquiring an absolute path of the file to be cached; s3, calculating a hash value based on the absolute path; s4, determining the position of the file to be cached in the double-layer directory according to the hash value, and determining the file name of the file to be cached according to the number of cached files; and S5, caching the file to be cached named by the file name to a corresponding position under the double-layer directory.

When the file is stored on the storage device of the client, the file is hashed into a two-layer directory structure, so that the picture files stored in each directory are not too many, and the performance problem of excessive single-directory file number is avoided.

In some embodiments, creating the dual-layer directory in the storage device in step S1 further includes:

Specifically, when the storage device is initialized, the storage device is formatted into an XFS file system and mounted on a specific directory, and a two-layer directory structure is created in advance, wherein the breadth of each layer of directory structure can be 256 (the number of first subdirectories and second subdirectories); if the SSD has created two-level directories, all the directories need to be emptied.

It should be noted that, in the two-layer directory structure, it is simply said that to create the B directory again under the a directory, for example, the extent of the a directory is 256, that is, 256 subdirectories of the a directory, and then create the B directory under each first subdirectory, since the extent of the B directory is also 256, 256 second subdirectories are under each first subdirectory.

In some embodiments, in the step S2, in the absolute path of the file to be cached, the absolute path may be a current filename and a current directory of the file to be cached, and then the hash value is calculated after splicing the character strings corresponding to the current filename and the current directory.

In some embodiments, determining, in step S4, a location of the file to be cached in the dual-layer directory according to the hash value, further includes:

Specifically, the size of the read file may be determined first, and if the size of the read file is smaller than a threshold (128 KB by default), the computed file is saved in a two-layer directory in the file system of the storage device. And calculating a 64-bit hash value according to an absolute path of the file, wherein the last 8 bits of the hash value can be used as a first preset position, the 8 bits which are the second last of the hash value are used as the first preset position, and each 8 bits correspond to 0-255, so that the position of the first subdirectory can be determined by the corresponding numerical value of the last 8 bits, and the position of the second subdirectory can be determined by the corresponding numerical value of the second last 8 bits.

For example, if the last 8 bits correspond to a value of 20 and the last 8 bits correspond to a value of 30, determining, according to the hash value, that the file to be cached is located in the second subdirectory 30 below the first subdirectory 20.

In some embodiments, the determining, in step S4, the file name of the file to be cached according to the number of cached files further includes:

and taking the integer number as the file name of the file to be cached.

Specifically, the file name of the file to be cached may be named by using a monotonically increasing 64-bit shaping number, and the shaping number is incremented by 1 every time a file is cached.

In some embodiments, the method further comprises:

Specifically, a mapping table of < key, value > indicating the storage location of the small file may be stored in the memory, where key is an absolute path of the file, and value is a file name of the file in the file system.

In this way, when the file is read again in the storage device, the file name stored in the storage device is found by the < key, value > mapping table using the absolute path of the file as a key, and the two-layer directory of the storage device is calculated by the key, thereby reading the file data stored in the storage device. When a file is deleted, an absolute path of a small file picture is used as a key, the file name stored in the storage equipment is found through a mapping table of < key, value >, the two layers of directories of the storage equipment are calculated through the key, the corresponding file is deleted, and then the corresponding record of < key, value > is deleted; and when other clients modify the small picture files, the cache of the storage device of the client needs to be enabled, and the processing flows are the same. When the client modifies the file, the absolute path of the file is used as a key, the file name stored in the storage device is found through a < key, value > mapping table, the two-layer directory of the storage device is calculated through the key, and the stored file data is modified according to the offset and length of the modified data.

In order to solve the problem of repeated reading performance of massive small picture files, the scheme provided by the invention adopts a mode of using a high-capacity SSD as a reading cache at a client, and utilizes the characteristic of high random reading performance of the SSD on small IO (input/output), when the small picture files are read for the first time, file data are cached on the local SSD of the client, and the small picture files can be read by utilizing the SSD cache at the later time. When the small picture files are stored on the client SSD, the small picture files are hashed into a two-layer directory structure, so that the picture files stored in each directory are not too many, and the performance problem of excessive single-directory file number is avoided.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a file caching system, 400, as shown in fig. 2, including:

a creation module 401, the creation module 401 configured to create a dual-layer directory in a storage device;

an obtaining module 402, where the obtaining module 402 is configured to obtain an absolute path of a file to be cached;

a calculation module 403, the calculation module 403 configured to calculate a hash value based on the absolute path;

a determining module 404, where the determining module 404 is configured to determine, according to the hash value, a position of the file to be cached in the double-layer directory, and determine, according to the number of cached files, a file name of the file to be cached;

a caching module 405, where the caching module 405 is configured to cache the file to be cached named by the file name to a corresponding position under the double-layer directory.

In some embodiments, the caching module 405 is further configured to:

In some embodiments, the determination module 404 is further configured to:

and taking the integer number as the file name of the file to be cached.

Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 3, an embodiment of the present invention further provides a computer apparatus 501, comprising:

at least one processor 520; and

the memory 510, the memory 510 storing a computer program 511 executable on the processor, the processor 520 executing the program to perform the steps of any of the above file caching methods.

Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 4, an embodiment of the present invention further provides a computer-readable storage medium 601, where the computer-readable storage medium 601 stores computer program instructions 610, and the computer program instructions 610, when executed by a processor, perform the steps of any of the above file caching methods.

Finally, it should be noted that, as will be understood by those skilled in the art, all or part of the processes of the methods of the above embodiments may be implemented by a computer program to instruct related hardware to implement the methods. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.

In addition, the apparatuses, devices, and the like disclosed in the embodiments of the present invention may be various electronic terminal devices, such as a mobile phone, a Personal Digital Assistant (PDA), a tablet computer (PAD), a smart television, and the like, or may be a large terminal device, such as a server, and the like, and therefore the scope of protection disclosed in the embodiments of the present invention should not be limited to a specific type of apparatus, device. The client disclosed by the embodiment of the invention can be applied to any one of the electronic terminal devices in the form of electronic hardware, computer software or a combination of the electronic hardware and the computer software.

Furthermore, the method disclosed according to an embodiment of the present invention may also be implemented as a computer program executed by a CPU, and the computer program may be stored in a computer-readable storage medium. The computer program, when executed by the CPU, performs the above-described functions defined in the method disclosed in the embodiments of the present invention.

Further, the above method steps and system elements may also be implemented using a controller and a computer readable storage medium for storing a computer program for causing the controller to implement the functions of the above steps or elements.

Further, it should be appreciated that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can include Read Only Memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which can act as external cache memory. By way of example and not limitation, RAM is available in a variety of forms such as synchronous RAM (DRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to comprise, without being limited to, these and other suitable types of memory.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.

The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and/or any other such configuration.

The steps of a method or algorithm described in connection with the disclosure herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.

It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.

The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.

It will be understood by those skilled in the art that all or part of the steps of implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.

Claims

1. A file caching method is characterized by comprising the following steps:

creating a dual-layer directory in a storage device;

acquiring an absolute path of a file to be cached;

computing a hash value based on the absolute path;

2. The method of claim 1, wherein creating a dual-level directory in a storage device further comprises:

3. The method of claim 2, wherein determining the location of the file to be cached in the dual-layer directory according to the hash value, further comprises:

4. The method of claim 1, wherein determining the file name of the file to be cached according to the number of cached files, further comprises:

and taking the integer number as the file name of the file to be cached.

5. The method of claim 1, further comprising:

6. A file caching system, comprising:

7. The system of claim 6, wherein the caching module is further configured to:

8. The system of claim 7, wherein the determination module is further configured to:

and taking the integer number as the file name of the file to be cached.

9. A computer device, comprising:

at least one processor; and

memory storing a computer program operable on the processor, characterized in that the processor executes the program to perform the steps of the method according to any of claims 1-5.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, is adapted to carry out the steps of the method according to any one of claims 1-5.