CN114168540A - File index information processing method and device, electronic equipment and storage medium - Google Patents
File index information processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114168540A CN114168540A CN202111520993.1A CN202111520993A CN114168540A CN 114168540 A CN114168540 A CN 114168540A CN 202111520993 A CN202111520993 A CN 202111520993A CN 114168540 A CN114168540 A CN 114168540A
- Authority
- CN
- China
- Prior art keywords
- sub
- file
- file index
- index
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
- G06F16/152—File search processing using file content signatures, e.g. hash values
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a file index information processing method and device, electronic equipment and a storage medium. The method comprises the following steps: writing target file data to be stored into target storage resources under the storage management node; determining a target file index sub-table from a target file index resource associated with the target storage resource and writing file index information of target file data to be stored into the target file index sub-table; when reading file data, searching a preset file index sub-table matched with target file data to be inquired from preset file index resources associated with preset storage resources under a storage management node; and according to the file index information recorded by searching the adaptive preset file index sub-table, reading the target file data to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs. By adopting the scheme, the problem of low retrieval and reading speed in the process of storing the security service mass image data is solved.
Description
Technical Field
The embodiment of the invention relates to the technical field of storage, in particular to a file index information processing method and device, electronic equipment and a storage medium.
Background
With the continuous development of economic society, the demand of people on safety is higher and higher, and the wide application of the monitoring system is promoted. In the monitoring service, the picture storage service is an important component; various picture services, such as car-passing pictures, illegal pictures, face pictures, structured data, etc., create a large number of picture storage and reading requirements. However, in a common medium city, the image data generated every day is in the tens of millions level or even higher, and in the related scheme, a file name index area is created at one or more positions on the resource according to the resource capacity and the fixed format, and the file information is recorded, which causes the problem that the writing and retrieval of the image are slow when the large number of images are stored in the security service.
Disclosure of Invention
The embodiment of the invention provides a file index information processing method and device, electronic equipment and a storage medium, and aims to solve the problems of slow retrieval and reading speed during storage of massive image data in security and protection services.
In a first aspect, an embodiment of the present invention provides a file index information processing method, where the method includes:
writing target file data to be stored into target storage resources under the storage management node;
determining a target file index sub-table from a target file index resource associated with the target storage resource and writing file index information of target file data to be stored into the target file index sub-table;
when reading file data, searching a preset file index sub-table matched with target file data to be inquired from preset file index resources associated with preset storage resources under a storage management node;
and according to the file index information recorded by searching the adaptive preset file index sub-table, reading the target file data to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs.
In a second aspect, an embodiment of the present invention further provides a file index information processing apparatus, where the apparatus includes:
the data writing module of the target file to be stored is used for writing the data of the target file to be stored into the target storage resource under the storage management node;
the target file index sub-table determining module is used for determining a target file index sub-table from the target file index resources associated with the target storage resources and writing the file index information of the target file data to be stored into the target file index sub-table;
the preset file index sub-table acquisition module is used for searching a preset file index sub-table matched with target file data to be inquired from preset file index resources associated with preset storage resources under a storage management node when file data are read;
and the target file data reading module to be inquired is used for reading the target file data to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs according to the file index information recorded by searching the adaptive preset file index sub-table.
In a third aspect, an embodiment of the present invention further provides an electronic device, including:
one or more processing devices;
storage means for storing one or more programs;
when the one or more programs are executed by the one or more processing apparatuses, the one or more processing apparatuses implement the file index information processing method according to any one of the embodiments of the present invention.
In a fourth aspect, an embodiment of the present invention further provides a readable medium, on which a computer program is stored, where the computer program, when executed by a processing apparatus, implements the file index information processing method described in any one of the embodiments of the present invention.
The embodiment of the invention provides a file index information processing method and device, electronic equipment and a storage medium. The method comprises the following steps: writing target file data to be stored into target storage resources under the storage management node; determining a target file index sub-table from a target file index resource associated with the target storage resource and writing file index information of target file data to be stored into the target file index sub-table; when reading file data, searching a preset file index sub-table matched with target file data to be inquired from preset file index resources associated with preset storage resources under a storage management node; and according to the file index information recorded by searching the adaptive preset file index sub-table, reading the target file data to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs.
By adopting the scheme, the file name index information is separated from the data resource, so that the flexibility of file index storage resource configuration is improved, and the pressure of traversing a magnetic disk with the magnetic disk when the file information is retrieved is avoided; the hash value calculated by adopting the file name is written into the corresponding sub-table, so that the traversal of all the sub-tables during the search of the file index is avoided; by adopting the method for predicting the number of the sub-tables in the directory, the problem of excessive sub-table number or excessive single-table storage is avoided, and the problem of slow retrieval and reading speed during storage of the massive image data in the security service is solved.
The above summary of the present invention is merely an overview of the technical solutions of the present invention, and the present invention can be implemented in accordance with the content of the description in order to make the technical means of the present invention more clearly understood, and the above and other objects, features, and advantages of the present invention will be more clearly understood.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1A is a flowchart of a file index information processing method according to an embodiment of the present invention;
fig. 1B is a schematic diagram of a file name index area management scheme according to an embodiment of the present invention;
FIG. 1C is a schematic diagram of an organization structure of a database according to an embodiment of the present invention;
FIG. 2A is a flowchart of a file index information processing method according to a second embodiment of the present invention;
fig. 2B is a flowchart of a file data writing method according to a second embodiment of the present invention;
fig. 3A is a flowchart of a file index information processing method according to a third embodiment of the present invention;
fig. 3B is a schematic diagram of cloud storage management according to a third embodiment of the present invention;
fig. 3C is a flowchart of a file data reading method according to a third embodiment of the present invention;
FIG. 4 is a block diagram of a file index information processing apparatus according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations (or steps) can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example one
Fig. 1A is a flowchart of a file index information processing method according to an embodiment of the present invention, where this embodiment is applicable to a case of processing file data, and the method of this embodiment may be executed by a file index information processing apparatus, and the apparatus may be implemented in a hardware and/or software manner. The device can be configured in a server for processing file index information. The method specifically comprises the following steps:
and S110, writing the target file data to be stored into the target storage resource under the storage management node.
And creating a file name index area at one or more positions on the resource according to the resource capacity and the fixed format, and recording file information. However, creating a file name index area in the resource itself has certain defects, for example, when reading and writing file information and file data, it is necessary to search in multiple locations of the disk, which increases disk delay; the file name index area retrieval efficiency is strongly bound with the disk performance, and the flexibility is poor; when mass data is searched, the performance is low by comparing the searched data one by one. Therefore, the application provides a file index information processing method.
The target file data to be stored may refer to picture data to be stored in security and protection services, for example, a lot of picture storage and reading requirements are generated by various picture services such as a car passing picture, an illegal picture, a face picture, structured data and the like in a monitoring system.
The storage management node is a storage management node server (CDV), and the CDV is responsible for selecting specific resource record file data and file index data information. Fig. 1B is a schematic diagram of a file name index area management scheme according to an embodiment of the present invention, and as shown in fig. 1B, data resources and file index resources are separated, and the data resources and file index database data adopt a one-to-one correspondence policy.
And writing the picture data into storage management node server management software, wherein the storage resources of the storage management node server CDV comprise a data resource pool RES and database resources. Each CDV manages several resources RES1-RESn and 1 database resource, the size of the database resource is 1/128 the size of the CDV data resource pool. Each data resource corresponds to one sqlite database named as uuid.
Fig. 1C is a schematic diagram of an organization structure of a database according to an embodiment of the present invention, and as shown in fig. 1C, each database uuid.db includes two types of data tables, which are a directory relation data table and a file index information data table. The directory relationship data table tbl _ dir _ tblname _ relationship records the directory information and directory and file index information table relationships, tbl _ dir _ tblname _ relationship _ bak is a backup of tbl _ dir _ tblname _ relationship. The file index information table records detailed file index information, each directory corresponds to a plurality of file index information data tables, and the table name is tbl _ diridx-m _ info, wherein diridx is the index value of the corresponding directory, and m is the mth sub-table. The file name index information and the data resources are separated, so that the flexibility of file index storage resource configuration is improved, and the pressure of traversing a magnetic disk of a magnetic disk when the file information is retrieved is avoided.
S120, determining a target file index sub-table from the target file index resources associated with the target storage resources and writing the file index information of the target file data to be stored into the target file index sub-table.
The target file index sub-table can be an index sub-table established according to file index information of target file data to be stored, when the data volume of the target file data under a single directory is large, writing and reading speed is seriously affected, the target file index sub-table is established for the target file data under each directory, the target file index sub-table to be written can be quickly inquired according to the file index information of the target file data to be stored, and the target file data to be read can be inquired according to the target file index sub-table. The file index information includes file index data for recording file information of the file data, such as creation time, modification time, file index, directory index, and the like.
S130, when reading the file data, searching a preset file index sub-table matched with the target file data to be inquired from preset file index resources associated with preset storage resources under the storage management node.
The preset file index resource may refer to a directory index to be queried including, but not limited to, target file data to be queried, for example, a directory relation table of the preset file index resource is queried according to the directory index to be queried of the target file data to be queried, and the directory relation table of the preset file index resource is determined to include the directory index to be queried of the target file data to be queried; if the directory index to be queried of the target file data to be queried is contained, acquiring the sub-table number of the file index sub-table under the directory index to be queried of the target file data to be queried; and carrying out Hash (hash) on the quantity of the sub-tables and file identifications corresponding to the target file data to be inquired, and searching a preset file index sub-table matched with the target file data to be inquired.
The hash may refer to compressing a message with any length to a message with a fixed length, for example, hashing the number of the sub-tables and a file identifier corresponding to target file data to be stored, and using a processed hash value as a sub-table sequence number to determine a target file index score; that is, the quantity of the sub-tables and the file identification information corresponding to the target file data to be stored are compressed and converted into the sub-table serial numbers for output, so that traversing all the sub-tables when searching the file index is avoided.
The quantity of the sub-tables can be that all the file index sub-tables are sub-tables according to the directory index to be inquired of the target file data to be inquired, so that the situation that all the sub-tables are traversed when the file index is searched is avoided, and the file retrieval efficiency is improved.
S140, according to the file index information recorded by searching the adaptive preset file index sub-table, reading the data of the target file to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs.
The preset file index sub-table can be used for performing hash calculation on file identifications corresponding to target file data to be inquired and the number of the sub-tables, taking a hash processing result as a table number of the file index sub-table, determining the preset file index sub-table according to the table number, recording file index information in the file index sub-table, and performing reading operation on the target file data to be inquired through preset storage resources associated with preset file index resources to which the adaptive preset file index sub-table belongs by the file index information.
For example, the number of the sub-tables of the file index sub-table under the directory index to be queried is calculated, the number of the sub-tables is recorded into a tbl _ dir _ tblname _ relationship table, and according to the file identifier and the number of the sub-tables corresponding to the target file data to be queried, a consistent hash algorithm is used to calculate the hash value corresponding to the target file data to be queried, and the hash value is used as the sub-table sequence number read by the file index information. And when the data of the target file to be inquired is read, searching a preset file index sub-table matched with the data of the target file to be inquired according to the sub-table serial number.
According to the file index information processing method provided by the embodiment of the invention, the file name index information is separated from the data resource, so that the flexibility of file index storage resource configuration is improved, and the pressure of traversing a disk with the disk when the file information is retrieved is avoided; the hash value calculated by adopting the file name is written into the corresponding sub-table, so that the traversal of all sub-tables during the search of the file index is avoided; by adopting the method for predicting the number of the files in the directory, the problems of excessive sub-table number or excessive single-table storage are avoided, and the problem of slow retrieval and reading speed during storage of the massive image data in security service is solved.
Example two
Fig. 2A is a flowchart of a file index information processing method according to a second embodiment of the present invention. Embodiments of the present invention are further optimized on the basis of the above-mentioned embodiments, and the embodiments of the present invention may be combined with various alternatives in one or more of the above-mentioned embodiments. As shown in fig. 2A, the file index information processing method provided in the embodiment of the present invention may include the following steps:
and S210, writing the target file data to be stored into the target storage resource under the storage management node.
S220, determining a target file index sub-table from the target file index resources associated with the target storage resources.
According to the security picture service characteristics, the number of the bayonets and the number of the collected pictures have obvious wave crests and wave troughs and have a certain rule. The number of the stored pictures in a single directory is 10-1000 ten thousand according to time and business scale. When the data size under a single directory is large, the writing and retrieving speed is seriously influenced, so that the corresponding file index table under each directory needs to be sorted.
Optionally, determining a directory index to be stored for storing file index information in the target file index resource and a sub-table number of file index sub-tables under the directory index to be stored;
hashing the file identification corresponding to the sub-table quantity and the target file data to be stored;
and taking the hash processing result as a sub-table sequence number of a file index sub-table used for storing the file index information of the target file data to be stored under the directory index to be stored so as to determine the target file index sub-table.
The sub-table quantity calculation formula is as follows:
the catalog is in hours, with a default value of 3, tblmin of 1, tblmax of 20;
the table of contents is in days, with default values of 20, tblmin of 1, and tblmax of 50.
Wherein:
m may refer to the mth directory;
lastdirtblnum may refer to the number of index data sub-tables for the last directory file;
num [ m-2] may refer to the number of files stored in the first two directories;
num [ m-1] may refer to the number of files stored in the previous directory;
the defapicnum may refer to the default recording picture file index information data amount of each file name index table, and default 20000.
Optionally, determining the sub-table number of the file index sub-table under the last directory index before the directory index to be stored;
determining the data volume of file index information stored under the last directory index and the data volume of file index information under the last two directory indexes before the directory index to be stored;
and determining the sub-table number of the file index sub-table under the directory index to be stored according to the sub-table number of the file index sub-table under the last directory index and the data quantity of the file index information stored under the last directory index and the last two directory indexes so as to create the file index sub-table.
In an alternative to the embodiments of the present invention, the hour list is used as an example to illustrate:
the first directory is 05, the number of default sub-tables is 3, and the number of recorded file indexes is 5000;
the second directory is 06, the default sub-table number is 3, and the number of the recorded file indexes is 15000;
the third directory is 07, the number of corresponding tables max {1,15000/max {5000,20000 }. times.3 or min {20,15000/max {5000,20000 }. times.3, assuming that the number of index records is 50000;
the fourth table is 08, and the number of corresponding tables is max {1,50000/max {15000,20000 }. times.3 } or min {20,50000/max {15000,20000 }. times.3 }. times.9.
After the number of tables for each directory is calculated, the number of tables is recorded in the tbl _ dir _ tblname _ relationship table. And calculating a hash value corresponding to the file identifier by using a consistent hash algorithm according to the file identifier and the sub-table number, wherein the hash value is used as a sub-table serial number for writing the file index information, and then writing the file information into a corresponding file index information data table. And the hash value calculated by adopting the file name is written into the corresponding sub-table, so that the traversing of all sub-tables during the searching of the file index is avoided, and the file data retrieval efficiency is improved.
S230, searching a preset file index sub-table matched with the target file data to be inquired from preset file index resources associated with the preset storage resources under the storage management node.
S240, according to the file index information recorded by searching the adaptive preset file index sub-table, reading the data of the target file to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs.
In an alternative solution of the embodiment of the present invention, fig. 2B is a flowchart of a file data writing method provided in the second embodiment of the present invention, and as shown in fig. 2B, the method includes:
CDM selects CDV according to load balancing principle;
the CDV selects resources according to a load balancing principle;
judging whether the directory is a first picture or not;
if the directory is the first picture, calculating the quantity of the file index sub-tables;
the number of branch tables and the directory information are put into the tbl _ dir _ tblname _ relationship table;
calculating a hash value by adopting a consistent hash algorithm according to the file name and the sub-table number, and determining a table to be written;
file index information is stored;
if the directory is not the first picture, acquiring the sub-table number of the corresponding directory, calculating a hash value by adopting a consistent hash algorithm according to the file name and the sub-table number, and determining a table to be written; and (5) warehousing file index information.
According to the file index information processing method provided by the embodiment of the invention, a directory index to be stored for storing file index information in the target file index resource and the sub-table number of the sub-tables of the file index under the directory index to be stored are determined; hashing the file identification corresponding to the sub-table quantity and the target file data to be stored; and taking the hash processing result as a sub-table sequence number of a file index sub-table used for storing the file index information of the target file data to be stored under the directory index to be stored so as to determine the target file index sub-table. The hash value calculated by the file identification is written into the corresponding sub-table, so that traversing of all sub-tables during searching of the file index is avoided, the retrieval efficiency of file data is improved, and the problems of slow retrieval and reading speed during storage of the large-scale image data in security and protection services are solved.
EXAMPLE III
Fig. 3A is a flowchart of a file index information processing method according to a third embodiment of the present invention. Embodiments of the present invention are further optimized on the basis of the above-mentioned embodiments, and the embodiments of the present invention may be combined with various alternatives in one or more of the above-mentioned embodiments. As shown in fig. 3A, the file index information processing method provided in the embodiment of the present invention may include the following steps:
s310, responding to the storage request of the metadata management server, and determining a target storage resource from preset data resources associated under the storage management node server.
Fig. 3B is a schematic view of cloud storage management according to a third embodiment of the present invention, and as shown in fig. 3B, a CDS cloud storage management system (CDS) is a storage system proposed for solving the defects of small storage capacity, difficult expansion, no disaster tolerance, and the like of a single machine system according to the characteristics of monitoring services. The system is composed of a metadata management server (CDM) and a storage management node server (CDV). The CDM is a lightweight metadata management server, and does not store specific metadata, but selects an appropriate CDV for data to be written or read according to a certain algorithm policy for storage. The CDV is responsible for selecting specific resource record file data and file index data information. When data is written, CDM selects a proper resource of CDV according to the load balancing principle, firstly writes the data into a resource data area, and then writes file index information into a proper position in a file name index area; when reading data, the CDM analyzes the CDV of the recording file according to the file path, then traverses the resource file name index area in the CDV for searching, and finally reads the file data from the data area according to the information recorded in the file name index area. In summary, the good file name index area management scheme has important significance for writing and reading files.
The metadata management server selects a proper storage management node server according to the data writing requirements of the target file to be stored, and determines a target storage resource from preset data resources associated with the storage management node server.
And S320, writing the target file data to be stored into the target storage resource.
S330, determining a target file index sub-table from the target file index resources associated with the target storage resources and writing the file index information of the target file data to be stored into the target file index sub-table.
S340, when reading the file data, searching a preset file index sub-table matched with the target file data to be inquired from preset file index resources associated with preset storage resources under the storage management node.
When the file data is read, determining the sub-table number of the file index sub-tables under the directory index to be inquired, performing hash calculation, and searching the preset file index sub-tables adaptive to the target file data to be inquired according to the sub-table sequence number corresponding to the hash value.
Optionally, if it is determined that the directory relation data table of the preset file index resource associated with the preset storage resource under the storage management node includes the directory index to be queried of the target file data to be queried, determining the number of sub-tables of the file index sub-tables under the directory index to be queried of the target file data to be queried;
hashing the sub-table number and a file identifier corresponding to target file data to be inquired;
and searching a preset file index sub-table matched with the target file data to be inquired under the directory index to be inquired according to the sub-table serial number corresponding to the hash processing result.
When reading file data, searching a preset file index sub-table matched with target file data to be inquired from preset file index resources associated with preset storage resources under a storage management node. The hash value is calculated by adopting the file identifier to search the corresponding sub-tables, so that the traversing of all sub-tables during the searching of the file index is avoided; by adopting the method for predicting the number of the contexts in the directory, the problems of excessive sub-table number or excessive single-table storage are avoided.
And S350, according to the file index information recorded by searching the adaptive preset file index sub-table, reading the data of the target file to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs.
In an alternative of the embodiment of the present invention, optionally, fig. 3C is a flowchart of a file data reading method provided in a third embodiment of the present invention, and as shown in fig. 3C, the method includes:
CDM searches corresponding CDV according to the directory information;
all resources under the CDV are searched concurrently;
judging whether the tbl _ dir _ tblname _ relation contains directory information;
if the directory information is contained, acquiring the quantity of the sub-tables, calculating a hash value according to the file name, searching the files in the corresponding sub-tables, and combining the search results;
if the directory information is not contained, the search results are merged.
According to the file index information processing method provided by the embodiment of the invention, when file data is read, a preset file index sub-table matched with target file data to be inquired is searched from preset file index resources associated with preset storage resources under a storage management node; the hash value is calculated by adopting the file identifier to search the corresponding sub-tables, so that the traversing of all sub-tables during the searching of the file index is avoided; by adopting the method for predicting the number of the files in the directory, the problems of excessive sub-table number or excessive single-table storage are avoided; the problem of retrieval and reading speed are slow when the massive image data is stored in the security service is solved.
Example four
Fig. 4 is a block diagram of a file index information processing apparatus according to a fourth embodiment of the present invention. The scheme of the embodiment can be applied to the situation of processing a large amount of file data, especially the situation of reading a large amount of metadata in monitoring business. The device can be implemented in software and/or hardware and integrated on any electronic equipment with network communication function. The electronic device may be a cloud storage node device in a cloud storage system.
As shown in fig. 4, the file index information processing apparatus in the embodiment of the present application may include the following: the file searching method comprises a to-be-stored target file data writing module 410, a target file index sub-table determining module 420, a preset file index sub-table obtaining module 430 and a to-be-queried target file data reading module 440. Wherein:
a to-be-stored target file data writing module 410, configured to write to-be-stored target file data into a target storage resource under a storage management node;
a target file index sub-table determining module 420, configured to determine a target file index sub-table from a target file index resource associated with the target storage resource and write file index information of target file data to be stored into the target file index sub-table;
a preset file index sub-table obtaining module 430, configured to search a preset file index sub-table adapted to target file data to be queried from a preset file index resource associated with a preset storage resource in a storage management node when reading file data;
and the target file data to be queried reading module 440 is configured to, according to the file index information recorded by searching the adaptive preset file index sub-table, perform a target file data to be queried reading operation in a preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs.
On the basis of the foregoing embodiment, optionally, the to-be-stored target file data writing module 410 includes:
responding to a storage request of a metadata management server, and determining a target storage resource from preset data resources associated under a storage management node server;
and writing the target file data to be stored into the target storage resource.
On the basis of the foregoing embodiment, optionally, determining a target file index sub-table from a target file index resource associated with the target storage resource includes:
determining a directory index to be stored for storing file index information in the target file index resource and the sub-table number of file index sub-tables under the directory index to be stored;
hashing the file identification corresponding to the sub-table quantity and the target file data to be stored;
and taking the hash processing result as a sub-table sequence number of a file index sub-table used for storing the file index information of the target file data to be stored under the directory index to be stored so as to determine the target file index sub-table.
On the basis of the foregoing embodiment, optionally determining the number of sub-tables of the file index sub-table under the directory index to be stored includes:
determining the sub-table number of the file index sub-tables under the last directory index before the directory index to be stored;
determining the data volume of file index information stored under the last directory index and the data volume of file index information under the last two directory indexes before the directory index to be stored;
and determining the sub-table number of the file index sub-table under the directory index to be stored according to the sub-table number of the file index sub-table under the last directory index and the data quantity of the file index information stored under the last directory index and the last two directory indexes so as to create the file index sub-table.
On the basis of the foregoing embodiment, optionally, searching a preset file index sub-table adapted to target file data to be queried from a preset file index resource associated with a preset storage resource under the storage management node includes:
if the directory relation data table of the preset file index resources associated with the preset storage resources under the storage management node comprises the directory index to be inquired of the target file data to be inquired, determining the sub-table number of the file index sub-tables under the directory index to be inquired of the target file data to be inquired;
hashing the sub-table number and a file identifier corresponding to target file data to be inquired;
and searching a preset file index sub-table matched with the target file data to be inquired under the directory index to be inquired according to the sub-table serial number corresponding to the hash processing result.
The file index information processing apparatus provided in the embodiment of the present invention may execute the file index information processing method provided in any embodiment of the present invention, and has a function and an advantageous effect corresponding to the execution of the file index information processing method.
EXAMPLE five
Fig. 5 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention. As shown in fig. 5, the electronic device 500 provided in the embodiment of the present invention includes: one or more storage devices 510 and a processor 520; the storage 510 in the electronic device may be one or more, and fig. 5 illustrates one storage 510 as an example; processor 520 is configured to store one or more programs; the one or more programs are executed by the one or more storage devices 510, so that the one or more storage devices 510 implement the file index information processing method according to any one of the embodiments of the present invention.
The electronic device may further include: an input device 530 and an output device 540.
The storage 510, processor 520, input device 530, and output device 540 in the electronic apparatus may be connected by a bus or other means, and are exemplified by a bus 550 in fig. 5.
The processor 520 in the electronic device, which is a computer-readable storage medium, may be used to store one or more programs, which may be software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the file index information processing method provided in the embodiments of the present invention. The storage 510 executes various functional applications and data processing of the electronic device by running software programs, instructions and modules stored in the processor 520, that is, implements the file index information processing method in the above method embodiments.
The processor 520 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the electronic device, and the like. Further, processor 520 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, processor 520 may further include memory located remotely from storage 510, which may be connected to the devices over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 530 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic apparatus. The output device 540 may include a display device such as a display screen.
And, when one or more programs included in the above-mentioned electronic device are executed by the one or more storage apparatuses 510, the programs perform the following operations:
writing target file data to be stored into target storage resources under the storage management node;
determining a target file index sub-table from a target file index resource associated with the target storage resource and writing file index information of target file data to be stored into the target file index sub-table;
when reading file data, searching a preset file index sub-table matched with target file data to be inquired from preset file index resources associated with preset storage resources under a storage management node;
and according to the file index information recorded by searching the adaptive preset file index sub-table, reading the target file data to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs.
Of course, it can be understood by those skilled in the art that when one or more programs included in the electronic device are executed by the one or more storage devices 510, the programs may also perform related operations in the file index information processing method provided in any embodiment of the present invention.
EXAMPLE six
An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, is configured to perform a file index information processing method, where the method includes:
writing target file data to be stored into target storage resources under the storage management node;
determining a target file index sub-table from a target file index resource associated with the target storage resource and writing file index information of target file data to be stored into the target file index sub-table;
when reading file data, searching a preset file index sub-table matched with target file data to be inquired from preset file index resources associated with preset storage resources under a storage management node;
and according to the file index information recorded by searching the adaptive preset file index sub-table, reading the target file data to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs.
Optionally, the program may be further configured to execute the file index information processing method provided in any embodiment of the present invention when executed by the processor.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), a flash Memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. A computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take a variety of forms, including, but not limited to: an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, Radio Frequency (RF), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A file index information processing method is characterized by comprising the following steps:
writing target file data to be stored into target storage resources under the storage management node;
determining a target file index sub-table from a target file index resource associated with the target storage resource and writing file index information of target file data to be stored into the target file index sub-table;
when reading file data, searching a preset file index sub-table matched with target file data to be inquired from preset file index resources associated with preset storage resources under a storage management node;
and according to the file index information recorded by searching the adaptive preset file index sub-table, reading the target file data to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs.
2. The method of claim 1, wherein writing target file data to be stored to a target storage resource under a storage management node comprises:
responding to a storage request of a metadata management server, and determining a target storage resource from preset data resources associated under a storage management node server;
and writing the target file data to be stored into the target storage resource.
3. The method of claim 1, wherein determining a target file index sublist from a target file index resource associated with the target storage resource comprises:
determining a directory index to be stored for storing file index information in the target file index resource and the sub-table number of file index sub-tables under the directory index to be stored;
hashing the file identification corresponding to the sub-table quantity and the target file data to be stored;
and taking the hash processing result as a sub-table sequence number of a file index sub-table used for storing the file index information of the target file data to be stored under the directory index to be stored so as to determine the target file index sub-table.
4. The method of claim 3, wherein determining the number of sub-tables of the sub-table of the file index under the directory index to be stored comprises:
determining the sub-table number of the file index sub-tables under the last directory index before the directory index to be stored;
determining the data volume of file index information stored under the last directory index and the data volume of file index information under the last two directory indexes before the directory index to be stored;
and determining the sub-table number of the file index sub-table under the directory index to be stored according to the sub-table number of the file index sub-table under the last directory index and the data quantity of the file index information stored under the last directory index and the last two directory indexes so as to create the file index sub-table.
5. The method according to claim 1, wherein searching a preset file index sub-table adapted to target file data to be queried from preset file index resources associated with preset storage resources under a storage management node comprises:
if the directory relation data table of the preset file index resources associated with the preset storage resources under the storage management node comprises the directory index to be inquired of the target file data to be inquired, determining the sub-table number of the file index sub-tables under the directory index to be inquired of the target file data to be inquired;
hashing the sub-table number and a file identifier corresponding to target file data to be inquired;
and searching a preset file index sub-table matched with the target file data to be inquired under the directory index to be inquired according to the sub-table serial number corresponding to the hash processing result.
6. A file index information processing apparatus, characterized in that the apparatus comprises:
the data writing module of the target file to be stored is used for writing the data of the target file to be stored into the target storage resource under the storage management node;
the target file index sub-table determining module is used for determining a target file index sub-table from the target file index resources associated with the target storage resources and writing the file index information of the target file data to be stored into the target file index sub-table;
the preset file index sub-table acquisition module is used for searching a preset file index sub-table matched with target file data to be inquired from preset file index resources associated with preset storage resources under a storage management node when file data are read;
and the target file data reading module to be inquired is used for reading the target file data to be inquired in the preset storage resource associated with the preset file index resource to which the adaptive preset file index sub-table belongs according to the file index information recorded by searching the adaptive preset file index sub-table.
7. The apparatus of claim 6, wherein the target file data to be stored writing module comprises:
responding to a storage request of a metadata management server, and determining a target storage resource from preset data resources associated under a storage management node server;
and writing the target file data to be stored into the target storage resource.
8. The apparatus of claim 6, wherein determining a target file index sub-table from a target file index resource associated with the target storage resource comprises:
determining a directory index to be stored for storing file index information in the target file index resource and the sub-table number of file index sub-tables under the directory index to be stored;
hashing the file identification corresponding to the sub-table quantity and the target file data to be stored;
and taking the hash processing result as a sub-table sequence number of a file index sub-table used for storing the file index information of the target file data to be stored under the directory index to be stored so as to determine the target file index sub-table.
9. An electronic device, comprising:
one or more processing devices;
storage means for storing one or more programs;
when the one or more programs are executed by the one or more processing devices, cause the one or more processing devices to implement the file index information processing method of any one of claims 1 to 5.
10. A readable medium on which a computer program is stored, characterized in that the program, when executed by a processing apparatus, implements a file index information processing method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111520993.1A CN114168540A (en) | 2021-12-13 | 2021-12-13 | File index information processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111520993.1A CN114168540A (en) | 2021-12-13 | 2021-12-13 | File index information processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114168540A true CN114168540A (en) | 2022-03-11 |
Family
ID=80486104
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111520993.1A Pending CN114168540A (en) | 2021-12-13 | 2021-12-13 | File index information processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114168540A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817147A (en) * | 2022-07-01 | 2022-07-29 | 北京网藤科技有限公司 | Method and system for rapidly retrieving file characteristic values through secondary indexes |
| CN115576899A (en) * | 2022-12-09 | 2023-01-06 | 深圳市木浪云科技有限公司 | Index construction method and device and file searching method and device |
| CN116954518A (en) * | 2023-08-09 | 2023-10-27 | 翼存(上海)智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
-
2021
- 2021-12-13 CN CN202111520993.1A patent/CN114168540A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817147A (en) * | 2022-07-01 | 2022-07-29 | 北京网藤科技有限公司 | Method and system for rapidly retrieving file characteristic values through secondary indexes |
| CN115576899A (en) * | 2022-12-09 | 2023-01-06 | 深圳市木浪云科技有限公司 | Index construction method and device and file searching method and device |
| CN116954518A (en) * | 2023-08-09 | 2023-10-27 | 翼存(上海)智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN116954518B (en) * | 2023-08-09 | 2024-06-04 | 翼存(上海)智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109254733B (en) | Method, device and system for storing data | |
| CN114168540A (en) | File index information processing method and device, electronic equipment and storage medium | |
| US7574435B2 (en) | Hierarchical storage management of metadata | |
| CN102725755B (en) | Method and system of file access | |
| CN106970958B (en) | A kind of inquiry of stream file and storage method and device | |
| US11221921B2 (en) | Method, electronic device and computer readable storage medium for data backup and recovery | |
| CN108614837B (en) | File storage and retrieval method and device | |
| JPH086831A (en) | Method and system for tracking of attribute of file | |
| KR20130049111A (en) | Forensic index method and apparatus by distributed processing | |
| CN110825694A (en) | Data processing method, device, equipment and storage medium | |
| WO2014110940A1 (en) | A method, apparatus and system for storing, reading the directory index | |
| CN111177143B (en) | Key value data storage method and device, storage medium and electronic equipment | |
| CN110597762A (en) | File processing method, device, equipment and storage medium | |
| CN105677904A (en) | Distributed file system based small file storage method and device | |
| US9734171B2 (en) | Intelligent redistribution of data in a database | |
| CN114610708A (en) | Vector data processing method and device, electronic equipment and storage medium | |
| CN114138558A (en) | Object storage method and device, electronic equipment and storage medium | |
| CN112559913A (en) | Data processing method and device, computing equipment and readable storage medium | |
| CN111753141B (en) | Data management method and related equipment | |
| CN111930684A (en) | Small file processing method, device and equipment based on HDFS (Hadoop distributed File System) and storage medium | |
| US8495025B2 (en) | Foldering by stable query | |
| CN114051162B (en) | Caching method and device based on play records | |
| CN109213760A (en) | The storage of high load business and search method of non-relation data storage | |
| CN114416676A (en) | Data processing method, device, equipment and storage medium | |
| CN113111138A (en) | Data processing method, device, computing equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |