CN117668881A

CN117668881A - Distributed file storage management method, device, platform and electronic equipment

Info

Publication number: CN117668881A
Application number: CN202311695548.8A
Authority: CN
Inventors: 赵鹏; 朱凤元; 胡东旭; 张珅; 邓丽钰
Original assignee: Du Xiaoman Technology Beijing Co Ltd
Current assignee: Du Xiaoman Technology Beijing Co Ltd
Priority date: 2023-12-11
Filing date: 2023-12-11
Publication date: 2024-03-08

Abstract

The embodiment of the application provides a distributed file storage management method, a device, a platform and electronic equipment, wherein the distributed file storage management method is applied to a preset distributed file management platform, a file storage proxy unit in the management platform is utilized to acquire a mounting point access request of a user, a target access mounting point created by the user according to requirements and a use authority of the user for a storage catalog application corresponding to the target access mounting point are received, the target access authority of the user is checked by means of authority information of each user recorded in a control unit in the management platform, and if the verification is passed, the file storage proxy unit forwards the mounting point access request to a corresponding target storage catalog. By selecting the embodiment of the application, the file storage agent unit and the control unit cooperate with each other, so that the defect of lack of authority management functions in the existing file storage technology is overcome, and data generated by users can be well ensured.

Description

Distributed file storage management method, device, platform and electronic equipment

Technical Field

The present disclosure relates to the field of database management, and in particular, to a method, an apparatus, a platform, and an electronic device for storing and managing a distributed file.

Background

In the internet field, users can generate massive data storage demands, and data generated by the users must be reliably stored, so that the users cannot be lost even in extreme fault scenes. Furthermore, the storage cost of the data generated by the user is as low as possible, so that the cost of the fees paid by the user for the generated data is reduced. In the related art, it is proposed to store data generated by a user by using a distributed storage technology, so as to achieve the effects of low cost and reliability.

However, the existing file storage technology has a rights management function deficiency, which easily causes serious security risks to data generated by users.

Disclosure of Invention

In view of this, the embodiments of the present application provide a method, an apparatus, and a platform electronic device for managing distributed file storage, so as to solve the problem that in the prior art, perfect rights management cannot be performed on distributed file storage.

In a first aspect, an embodiment of the present application provides a distributed file storage management method, where the method is applied to a preset distributed file management platform, and the preset distributed file management platform at least includes: the file storage agent unit and the control unit, the method comprises the following steps:

Acquiring a mounting point access request of a user by using the file storage proxy unit, wherein the mounting point access request carries a target access mounting point and target access rights, the target access mounting point is a mounting point created by the user according to storage requirements, the target access rights are the use rights of the user for target storage catalogues, and the target storage catalogues are storage catalogues corresponding to the target access mounting point;

and verifying the target access rights according to the rights information of each user recorded by the control unit, and if the verification is passed, forwarding the mount point access request to the target storage directory by the file storage proxy unit.

According to a first aspect, in a second possible embodiment, the file storage agent unit includes a protocol conversion module, where the protocol conversion module is configured to perform protocol conversion according to different storage systems, and the storage directory is a storage path of each file in the storage systems, and the method further includes:

acquiring a mounting point statistical result of each resource pool in each storage system according to the data transmission protocol of each storage system;

If the mounting point statistical result of each resource pool has a target resource pool exceeding a preset safety water line, scanning the mounting points in the target resource pool by using the control unit, and determining target mounting points to be migrated;

and migrating the target mounting point in the target resource pool to a new resource pool based on a preset mounting point migration rule.

According to a second possible embodiment of the first aspect, in a third possible embodiment, the scanning, by the control unit, mounting points in the target resource pool, and determining a target mounting point to be migrated includes:

scanning the storage data volume of each mounting point in the target resource pool, and sequencing the storage data volume of each mounting point according to the sequence from small to large to obtain a target mounting point sequence;

based on each stored data amount in the target mounting point sequence, determining that the target resource pool predicts the residual resource usage amount after moving out each mounting point;

and determining a mounting point corresponding to the predicted residual resource usage closest to the preset recommended water line as a target mounting point to be migrated.

According to a second possible embodiment of the first aspect, in a fourth possible embodiment, the new resource pool is determined by:

Based on the storage data volume of the target mounting point to be migrated, determining the predicted resource utilization rate of each resource pool in each storage system after the resource pools are migrated into the target mounting point;

and determining the resource pool with the predicted resource utilization rate closest to a preset recommended water line as the new resource pool.

According to a second possible embodiment of the first aspect, in a fifth possible embodiment, the scanning, by the control unit, mounting points in the target resource pool, and determining a target mounting point to be migrated includes:

and scanning the storage time length of the historical data stored by the mounting points in the target resource pool, and determining the mounting points with the storage time length of the stored historical data exceeding a preset storage time length threshold as target mounting points to be migrated.

According to a fifth possible embodiment of the first aspect, in a sixth possible embodiment, the migrating, based on a preset mount point migration rule, the target mount point in the target resource pool to a new resource pool includes:

and migrating the target mounting point to be migrated in the target resource pool to a low-speed resource pool, wherein the low-speed resource pool stores data in an object storage mode.

In a seventh possible embodiment according to the first aspect, the method further comprises:

creating a new mounting point in the new resource pool by using the control unit, and binding the new mounting point with a user bound with the target mounting point to be migrated;

and generating new target access right information based on the new mounting point, and updating the right information of the user in the control unit based on the new target access right information.

In a second aspect, an embodiment of the present application provides a distributed file storage management apparatus, where the apparatus is applied to a preset distributed file management platform, where the preset distributed file management platform at least includes: the device comprises a file storage agent unit and a control unit, wherein the device comprises:

the receiving module is used for acquiring a mounting point access request of a user by utilizing the file storage proxy unit, wherein the mounting point access request carries a target access mounting point and target access permission, the target access mounting point is a mounting point created by the user according to storage requirements, the target access permission is the use permission of the user for a target storage catalog application, and the target storage catalog is a storage catalog corresponding to the target access mounting point;

And the verification module is used for verifying the target access rights according to the rights information of each user recorded by the control unit, and if the verification is passed, the file storage proxy unit forwards the mount point access request to the target storage directory.

In a third aspect, an embodiment of the present application provides a distributed file storage management platform, where the platform includes:

the file storage proxy unit is used for providing a mounting service for a user and acquiring a mounting point access request of the user, wherein the mounting point access request carries a target access mounting point and target access rights, the target access mounting point is a mounting point created by the user according to storage requirements, the target access rights are the use rights of the user for target storage catalogues, and the target storage catalogues are storage catalogues corresponding to the target access mounting point;

and the control unit is used for checking the target access authority according to the authority information of each user recorded by the control unit, and if the target access authority passes the check, the file storage proxy unit forwards the mount point access request to the target storage directory.

With reference to the third aspect, in a second possible embodiment, the file storage agent unit includes a protocol conversion module, where the protocol conversion module is configured to perform protocol conversion according to different storage systems, and the storage directory is a storage path of each file in the storage systems;

the file storage agent platform is further used for acquiring mounting point statistics results of each resource pool in each storage system according to the data transmission protocol of each storage system; if the mounting point statistical result of each resource pool exists a target resource pool exceeding a preset safety water line;

the control unit is also used for scanning mounting points in the target resource pool and determining target mounting points to be migrated; and migrating the target mounting point in the target resource pool to a new resource pool based on a preset mounting point migration rule.

With reference to the second possible embodiment of the third aspect, in a third possible embodiment, the control unit is specifically configured to:

scanning the storage data volume of each mounting point in the target resource pool, and sequencing the storage data volume of each mounting point according to the sequence from small to large to obtain a target mounting point sequence; based on each stored data amount in the target mounting point sequence, determining that the target resource pool predicts the residual resource usage amount after moving out each mounting point;

In a fourth aspect, an embodiment of the present application provides an electronic device, where the electronic device includes:

a processor; and

a memory in which a program is stored,

wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the distributed file storage management method of the first aspect.

In a fifth aspect, embodiments of the present application provide a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the distributed file storage management method of the first aspect.

The beneficial effects of this application:

the embodiment of the application provides a distributed file storage management method, a device, a platform and electronic equipment, wherein the distributed file storage management method is applied to a preset distributed file management platform, a file storage proxy unit and a control unit are mainly included in the preset distributed file management platform, the method utilizes the file storage proxy unit to acquire a mounting point access request of a user, receives a target access mounting point created by the user according to requirements and a use permission of the user for a storage catalog application corresponding to the target access mounting point, checks the target access permission of the user by means of permission information of each user recorded in the control unit, and if the verification is passed, the file storage proxy unit forwards the mounting point access request to a corresponding target storage catalog. By selecting the embodiment of the application, the file storage agent unit and the control unit cooperate with each other, so that the defect of lack of authority management functions in the existing file storage technology is overcome, and data generated by users can be well ensured.

Drawings

Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments, with reference to the following drawings, wherein:

FIG. 1 is a schematic flow chart of a method for managing distributed file storage according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a logic structure of a distributed file storage management platform according to an embodiment of the present application;

FIG. 3 is a schematic flow chart of another method for distributed storage management according to an embodiment of the present disclosure;

FIG. 4 is a schematic flow chart of another method for distributed storage management according to an embodiment of the present disclosure;

FIG. 5 is a schematic structural diagram of a distributed storage management device according to an embodiment of the present disclosure;

fig. 6 shows a schematic logic structure of an electronic device according to an embodiment of the present application.

Detailed Description

Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present application are shown in the drawings, it is to be understood that the present application may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and examples of the present application are for illustrative purposes only and are not intended to limit the scope of the present application.

It should be understood that the various steps recited in the method embodiments of the present application may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present application is not limited in this respect.

The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below. It should be noted that the terms "first," "second," and the like herein are merely used for distinguishing between different devices, modules, or units and not for limiting the order or interdependence of the functions performed by such devices, modules, or units.

It should be noted that references to "one" or "a plurality" in this application are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be interpreted as "one or more" unless the context clearly indicates otherwise.

In the field of databases, a user can directly use a storage space provided by the distributed file storage as a local disk after mounting by providing a universal remote disk mounting interface, and data written into the storage space by the user can copy a plurality of copies in a storage system according to the preset copy number so as to exist on different servers, thereby guaranteeing the usability of the data written by the user. The server of the distributed file storage refers to computer equipment running a program, and data written by a user or the program can be run by providing hardware resources such as a CPU, a memory, a disk, a network interface, a GPU and the like.

As described in the background, massive amounts of data generated by users need to be stored reliably, and it needs to be ensured that no loss occurs in extreme fault scenarios. In addition, in the field of internet finance, there are a large number of business systems facing the C (Client) side directly, and when a user reads stored data, the response time of the business system is required to be short, that is, the requirement of the user on the response time is high, which depends on the file access speed of the storage system to be as fast as possible. Furthermore, the storage costs for the data generated by the user are required to be as low as possible, since massive amounts of data, even 1% saving in storage costs, means a considerable cost saving.

From the perspective of saving the storage cost, the existing distributed file storage technology reduces the cost and difficulty. Because the demands for both high reliability and high performance can be solved based on the distributed file storage technology, in order to maintain high performance, the existing distributed file storage technology generally adopts a multi-copy technology (i.e. the manner of copying multiple copies) to store data, which also means that the storage cost cannot be reduced further.

For example, in the technical background of the existing multi-copy technology, if a piece of original data needs at least 3 copies if high reliability is to be ensured, the storage resource space occupied by the original data is enlarged to 4 copies compared with the original 1 copy, and the storage resource space is enlarged by 3 times. In order to improve the access speed performance, the bottom storage needs to use a relatively expensive solid state disk, which means that the user data is continuously increased, and the cost is only higher and higher.

In addition, the existing open source distributed file storage technology has the defect of authority management function, and under the service scene of authority management, the files in the resource pool can be hung and read and written at will as long as the IP address of the resource pool for file storage is known, so that serious potential safety hazards are caused. In addition, there are often cases where multiple users mix with the same set of underlying resource pools, and a case where storage resources are preempted easily occurs between each other. If the bottom resource pools of each user are separated independently, great waste of storage resources is caused.

In view of this, in a first aspect, the present application provides a distributed file storage management method, where the method is applied to any electronic device with a distributed file storage management function, including, but not limited to, a personal mobile terminal, a computer, a server, or the like. Specifically, the distributed storage management method provided by the embodiment of the application is applied to a preset distributed file storage management platform, and the preset distributed file storage management platform at least comprises: a file proxy unit and a control unit. As shown in fig. 1, the method comprises the steps of:

s11, acquiring a mounting point access request of a user by using a file storage agent unit. The mounting point access request carries a target access mounting point and target access permission, wherein the target access mounting point is a mounting point created by a user according to storage requirements, the target access permission is the application permission of the user for a target storage catalog, and the target storage catalog is a storage catalog corresponding to the target access mounting point.

And S12, checking the target access rights according to the rights information of each user recorded by the control unit, and if the target access rights pass the checking, forwarding the mount point access request to the target storage catalog by the file storage proxy unit.

By selecting the embodiment of the application, the file storage agent unit and the control unit cooperate with each other to carry out security check on the access request of the mounting point submitted by the user, so that the gap of the lack of the authority management function in the existing file storage technology is overcome, and the data generated by the user can be well ensured.

In addition, the file proxy unit and the control unit in the embodiment of the application play a role of a bridge between the user and the mounting points, so that the resources of each mounting point can be scheduled in real time, flexible adjustment of the mounting points according to the conditions of the resources is facilitated, storage resources are saved, and storage cost is reduced.

The above steps S11 to S12 will be described in detail below:

in this embodiment of the present application, the file storage proxy unit is a file storage proxy logic module embedded in a preset distributed file management platform, and is configured to receive a mount point access request sent by a user, and schedule a mount point and a resource pool accessed by the mount point access request. As an example, the preset distributed file storage management platform may be as shown in fig. 2, and the file storage agent unit may be directly connected to a user to provide a mount service for the user, and the user may apply for using a resource pool in the storage system by sending a mount point access request. The file storage proxy unit dispatches the mount point access request to the corresponding resource pool for response based on the received mount point access request.

As an implementation mode, the file storage agent unit is different from the existing file storage software, the file storage agent unit can flexibly schedule the access request of the mounting point through a preset distributed file storage platform without installing additional file storage software, is applicable to the original mounting command, can directly mount the storage catalogue for use based on the original mounting command, has no difference in usage, and can schedule the storage resource pool under the condition of no sense of a user.

In addition, in some embodiments, the file storage agent unit includes a protocol conversion module for performing protocol conversion according to different storage systems. The types of resource pools provided by different manufacturers are different, the types are different, and the supported communication protocol types are also different due to the influence of factors such as different manufacturers, different operators, different storage cost and the like. In the embodiment of the application, the encoding and decoding rules of different protocols are preset in the file storage proxy unit, so that message interaction can be performed with different storage systems based on the communication protocol types supported by the corresponding storage systems. Specifically, in the embodiment of the present application, storage devices in a storage system may be divided based on a data storage manner, each storage device in a file storage manner is determined to be a high-speed resource pool, and each storage device stored in an object is determined to be a low-speed resource pool. Therefore, the back end can be supported to use a plurality of different storage products, and further the storage performance and the storage cost are balanced.

In this embodiment of the present application, the control unit is configured to manage authority information of all users, and is further configured to manage a resource quota in the resource pool, and further configured to manage a correspondence between a used storage mount point and a real resource pool at a back end of the mount point.

In the embodiment of the present application, the mount point refers to an entry directory of a disk file system, which is similar to a different disk partition and a corresponding storage file path in a windows system. As one embodiment, the target access mount point is a mount point created by a user according to storage requirements, i.e., a storage directory or a storage file path created by the user according to storage requirements. The target access rights indicate the usage rights applied by the user for the target storage directory, including several types: read-only, write-only, readable and writable, unreadable and non-writable. As one implementation mode, the target access right information is generated based on the target storage path and the unique identification of the user, so that the uniqueness of the access right information of each user can be ensured, and the accuracy of access right verification is further improved.

Based on this, when step S11 is executed, the file storage agent unit is used to obtain the mount point access request of the user, the file storage agent unit is used to receive the mount point access command or instruction sent by the user, and the information carried in the access command or instruction is analyzed to obtain the target access mount point to which the user applies to access and the authority possessed by the user. As an embodiment, when executing step S11, the file storage agent unit may further acquire authority information of all users from the control unit, and then execute step S12 to verify the target access authority of the user submitting the mount point access request. As another embodiment, when executing step S11, the file storage agent unit sends the target access rights obtained by analysis to the control unit, and when executing step S12, the control unit verifies the target access rights of the user submitting the mount point access request based on the rights information of all users stored by the control unit, and then returns the verification result to the file storage agent unit.

In the application, the file storage agent unit and the control unit embody the management work of the following layers when managing the distributed file storage system:

creation of mounting points, rights management, movement adjustment of data, and consistency verification of data among multiple resource pools.

The creation of mount points, rights management, movement adjustment of data, and consistency check of data among multiple resource pools will be described in detail below:

creation of mount points: after the user stores the mounted file, because the data required by the user is already stored in the storage file path corresponding to the mounted point, the mounted point is not easy to change, and basically, the data is stored by continuously adding subdirectories and files. The user needs to create the mounting point and then apply for the access authority of the mounting point. In the embodiment of the application, the control unit processes the mount point access request of the user through the following procedures:

the user initiates the creation of the mounting point through the communication interface provided by the notification unit. The user needs to give the required storage space, and the control unit preferentially creates the target access mounting point in the high-speed resource pool according to the required storage space. Before the target access mounting point is created, the control unit traverses all the high-speed resource pools through the file storage proxy unit to judge whether available storage space can meet the requirements of users.

Specifically, in some embodiments, the resource pool available to create the mount point is determined by:

according to the data transmission protocol of each storage system, acquiring the statistical result of the mounting points of each resource pool of each storage system;

and determining a resource pool with the statistical result of the mounting point at a safe water level as a resource pool which can be used for creating the mounting point.

As an implementation manner, if there are multiple high-speed resource pools currently, the available space below the safe water line can meet the user's requirement, then each high-speed resource pool can meet the user's requirement. At this time, in consideration of poor elasticity of storage products, the distributed file storage management method adopted in the application needs to reserve more redundant space as much as possible to cope with unexpected situations of data burst writing, so that a high-speed resource pool with the largest available space is selected to create a mounting point.

The safe water line refers to a safe storage resource threshold value capable of guaranteeing the data read-write rate and the storage cost of the resource pool. As one embodiment, the secure water line may be the maximum storage space that can be safely used. When the total amount of stored data exceeds the amount, a certain risk may occur in the storage resource pool, so that the data cannot be written continuously or even is lost.

As another implementation, if all the high-speed resource pools currently exist, the available space under the safe water level cannot meet the requirements of the users. At this time, the distributed file storage management method adopted in the application needs to consider the cost of the resource pool and the user requirement, and performs secondary evaluation on the available space of each resource pool, specifically, performs secondary evaluation through the following procedures:

step 1: and after the safe water level is reached, the available storage space of each high-speed resource pool is calculated, and the proportion of the high-speed resource pools which can meet the demands of users is determined. By way of example, assuming that resource pool A is 1TB of available space from the safety water line and the user's demand is 2TB, the resource pool may satisfy 50% of the user's demand.

Step 2: and selecting the resource pool with the largest available space from all resource pools meeting 50% of the user demands, and creating a corresponding mounting point.

Step 3: if all the resource pools can not meet 50% of the user requirements, the control unit automatically creates a new resource pool, and creates a corresponding mounting point in the new resource pool.

For rights management: after the corresponding mounting point is established, the mounting point has a one-to-one correspondence with a certain data directory or file directory in the resource pool, and the user can bind with the mounting point when applying for access rights. I.e. the mount point pre-data directory is bound. The specific flow is as follows:

The file storage agent unit receives the mount point access request message sent by the user, then creates a mount point based on the mount point access request message, performs user verification based on the unique identification information of the user, determines whether the user accords with the admitted user, if the user passes the verification, forwards the mount point access request of the user to the catalog corresponding to the mount point, binds the mount point with the catalog and the unique identification information of the user by the control unit, and stores the bound authority information to the rear end so as to perform the subsequent authority verification on the user accessing the mount point based on the stored authority information.

Because the data of the user is continuously written, a certain proportion of the data needs to be stored for a long time, and the total data can be in a continuously rising state. But the underlying resource pool cannot be extended wirelessly due to technical constraints. Therefore, the bottom layer can necessarily use a plurality of resource pools, and in the embodiment of the application, the control unit and the file storage agent unit are mutually matched, so that the mobile adjustment of data among the plurality of resource pools can be realized.

So adjust for movement of data: in some embodiments, as shown in FIG. 3, the movement adjustment of the data may be performed by:

S31, acquiring a mounting point statistical result of each resource pool in each storage system according to the data transmission protocol of each storage system.

S32, if the mounting point statistics result of each resource pool has a target resource pool exceeding a preset safety water line, scanning the mounting points in the target resource pool by using the control unit, and determining the target mounting points to be migrated.

S33, migrating the target mounting points in the target resource pool to a new resource pool based on a preset mounting point migration rule.

In this embodiment of the present application, the mounting point statistics result of each resource pool refers to the data amount stored in each resource pool under the storage directory corresponding to each mounting point. Specifically, as an implementation manner, when executing step S31, according to the data transmission protocols of different storage systems, a mounting point statistics instruction may be sent to each resource pool, the resource pool calculates the total data amount of the data stored under the directory corresponding to each mounting point based on the received statistics instruction, and the statistics result is fed back to the file storage agent unit.

Specifically, for the situation that the mounting point data in the resource pool normally grows, so that the resource pool is insufficient, when one resource pool which is expanded to the maximum state is used, the utilization rate of the resource pool exceeds the safety water line. For example, when the overall disk space has been used beyond the safe watermark threshold by 90%, it is indicated that the resources of the resource pool are no longer sufficient. At this time, the adjustment of the mounting point involves the movement of the data, and the removed data has a certain influence on the new resource Chi Ou, so as an implementation manner, when executing step S32, as shown in fig. 4, the target mounting point to be migrated and the new resource pool for migrating into the target mounting point may be determined through the following steps S32-1 to S32-5:

S32-1, scanning the storage data quantity of each mounting point in the target resource pool, and sequencing the storage data quantity of each mounting point according to the sequence from small to large to obtain a target mounting point sequence.

In the embodiment of the present application, two water lines are set for each resource pool: presetting a safe water level line and presetting a suggested water level line. The preset secure water line is as described above, and is the maximum storage space allowed to be used safely for the storage resource pool. The preset proposal water level line is an optimal water level line considering storage space redundancy and cost, and can ensure that a reasonable redundant storage space is reserved for coping with the data volume increased by users, and can also ensure that a resource pool can be effectively utilized, so that waste caused by a large amount of space limitation is avoided. Preferably, the memory space threshold corresponding to the preset suggested water line is smaller than the memory space threshold corresponding to the preset safe water line. For example, the preset recommended water line corresponds to a memory space threshold of 80% and the preset safety water line corresponds to a memory space threshold of 90%.

When a certain resource pool exceeds a preset safety water line, the control unit scans the data quantity of all mounting points in the resource pool, and then sorts the data according to the sequence from small to large or sorts the data according to the sequence from large to small to obtain a target mounting point sequence.

S32-2, determining the residual resource usage amount of the target resource pool after moving out of each mounting point based on each storage data amount in the target mounting point sequence.

In one embodiment, only one mounting point may be moved by default, so that, based on each stored data amount in the target mounting point sequence, the predicted remaining resource usage amount of the target resource pool after each mounting point in the target mounting point sequence is removed may be calculated one by one, and the predicted remaining resource usage amounts of the target resource pools corresponding to each mounting point may be compared.

S32-3, determining a mounting point corresponding to the predicted residual resource usage closest to the preset recommended water line as a target mounting point to be migrated.

And comparing the results of the predicted residual resource usage amounts corresponding to all the mounting points, and determining the mounting point corresponding to one preset residual resource usage amount closest to the recommended water line in the predicted residual resource usage amounts as a target mounting point to be migrated. Preferably, a mount point corresponding to the predicted remaining resource usage amount, which is closest to the recommended water line and which is not more than 5% floating up and down from the construction water line, among the predicted remaining resource usage amounts may be determined as the target mount point to be migrated.

In some embodiments, the new resource pool into which the target mount point to be migrated can be migrated may be determined through the following steps S32-4 and S32-5:

s32-4, determining the predicted resource utilization rate of each resource pool in each storage system after the resource pools are migrated into the target mounting points based on the storage data volume of each target mounting point to be migrated;

when executing step S32-4, the control unit may be used to scan the current data storage amount of each resource pool of each storage system with reference to step S31, and then calculate the predicted resource utilization rate of each resource pool after each target mount point to be migrated is migrated to each resource pool. As an implementation mode, the predicted resource utilization rate can be obtained by directly adding the stored data quantity corresponding to the target mounting point to be migrated on the basis of the existing data storage quantity of each resource pool.

Preferably, in order to ensure the normal operation of each resource pool migrating into the target mounting point to be migrated and save computing resources, before executing step S32-4, the control unit may be utilized to scan the current usage of each resource pool to obtain the current usage of each resource pool, and then screen out the priority resource pools whose current usage is located under the safe water line from each resource pool. And then calculating the predicted resource utilization rate of each priority resource pool after the target mounting point to be migrated is migrated to each priority resource pool.

S32-5, determining the resource pool with the predicted resource utilization rate closest to the preset recommended water line as a new resource pool for migrating into the target mounting point to be migrated.

Specifically, the control unit selects a resource pool closest to the suggested water line as a new resource pool to migrate into the target mounting point to be migrated by comparing the predicted resource utilization rates of the resource pools. Preferably, the resource pool closest to the suggested water line is selected, and the predicted resource usage does not float up and down by more than 5% from the suggested water line.

When executing step S33, as an implementation manner, the control unit may create a new mounting point in the screened new resource pool, and bind the new mounting point with the target mounting point to be migrated and the corresponding same user, and may specifically bind the new mounting point with the same user mounting point. And generating configuration information, sending the configuration information to a file storage proxy unit, and subsequently scheduling a subsequent writing request of a user to the created new mounting point by the file storage proxy unit according to the configuration information. And based on the new mounting point, generating new target access right information, and based on the new target access right information, updating the right information of the user in the control unit to ensure that the data written by the subsequent user can be smoothly written into the storage file corresponding to the new mounting point.

Further, the control unit operates to gradually move data in the original target mount point to be migrated to the created new mount point until the data in the target mount point to be migrated is emptied. At this time, the target mounting point is deleted, and the configuration information of the target mounting point is deleted in the file storage proxy unit, and only the new mounting point is reserved, and at this time, the whole data migration can be regarded as being finished.

By selecting the embodiment of the application, the control unit can scan the storage data quantity of the mounting points in each resource pool in real time or periodically, and timely migrate the mounting points with more tense storage resources, so that the problem of lacking resource management functions in the existing open-source distributed file storage is effectively solved, manual intervention is not needed, and labor cost is effectively saved.

Due to the data stored in the mounting points in the resource pools, there may be situations where part of the history data is no longer accessed frequently. To save storage costs, in some embodiments, this portion of historical data may be migrated by:

In one embodiment, the storage duration of the history data stored by the mounting point may be determined by the last time the stored data is read, and the duration from the last time the stored data of the mounting point is read to the current time of the system is determined as the storage duration of the history data stored by the mounting point.

As another embodiment, the storage duration of the history data stored by the mounting point may be determined by the writing time of the stored data, and the duration between the writing time of the data stored by the mounting point and the current time of the system is determined as the storage duration of the history data stored by the mounting point.

In another embodiment, the control unit scans the user data in each high-speed resource pool, determines the historical storage data that the effective time length between the writing time and the current time exceeds the first preset failure time length, and further determines the historical storage data that the last time of reading exceeds the second preset failure time length from the historical storage time length of the current time in the part of the historical storage data. And then determining the mounting point with the part of historical storage time exceeding the preset storage time threshold as the target mounting point to be migrated. Preferably, the first preset failure time period is 1 year, the second preset failure time period is 1 year, and the preset storage time period threshold value can be obtained based on the first preset failure time period and the second preset failure time period.

Based on this, when step S33 is executed, as another embodiment, the target mount point to be migrated in the target resource pool is migrated into the low-speed resource pool. As described above, the data storage method adopted in the low-speed resource pool is different from the file storage method adopted in the high-speed resource pool. Specifically, a new mounting point is created in the low-speed resource pool by the control unit, the mounting point is bound with a user by the control unit, configuration information is generated and sent to the file proxy unit, and data in the old mounting point is gradually migrated to the new mounting point.

Because the storage cost of the high-speed resource pool is higher than that of the low-speed resource pool, the historical data which is read or written by a person for a long time in the high-speed resource pool can be migrated to the low-speed resource pool by adopting the embodiment of the application, the storage cost of the data can be effectively saved, and the storage resource of the high-speed resource pool can be further ensured to be utilized maximally.

If the data belonging to the same user is distributed in multiple resource pools, for example, the data of the user a is distributed in 3 resource pools, one is the original resource pool, the other is a new mounting point when the mounting point moves as a whole, and the other is a mounting point in a low-speed resource pool used by historical data. At this time, there is a problem that data in a plurality of resource pools is not uniform. Therefore, for data consistency among multiple resource pools, the distributed file storage management method provided by the embodiment of the application further includes the following steps:

And the file storage proxy unit controls the data written by the user, determines a unique writing resource pool from the plurality of resource pools, and controls the writing of the data written by the user into the unique writing resource pool.

In response to a user's data read request, the user is controlled by the file storage agent unit to access each readable resource pool. In the process of transferring the data of the mounting points, the situation that two mounting points exist in certain data at the same time can occur, and the problem is solved based on the following steps:

step 1), responding to a read data request of a user, preferentially searching in the mounting point of the unique write-in resource pool bound for the read data request, and directly returning the searched data if the mounting point is searched.

Step 2), if the data is not found in the step 1), indicating that the data is migrated to the read-only mounting point, and searching in the read-only mounting point in the read-only resource pool. At this time, the self-reading mounting point for the mounting point to be moved is searched preferentially, and if the self-reading mounting point is found, the searched data is returned. If not found, searching is carried out on the mounting point of the low-speed resource pool.

Step 3), if the data is not found in both the step 1) and the step 2), the feedback read data request is wrong, and the retry is performed after correction.

In a second aspect, the present application provides a distributed file storage management apparatus, where the apparatus is applied to a preset distributed file management platform, and the preset distributed file management platform at least includes: file storage agent unit, control unit, as shown in fig. 5, the apparatus 50 includes:

the receiving module 501 is configured to obtain, by using a file storage proxy unit, a mount point access request of a user, where the mount point access request carries a target access mount point and a target access right, where the target access mount point is a mount point created by the user according to a storage requirement, the target access right is a use right applied by the user for a target storage directory, and the target storage directory is a storage directory corresponding to the target access mount point;

and the verification module 502 is configured to verify the target access rights according to the rights information of each user recorded by the control unit, and if the verification is passed, forward the mount point access request to the target storage directory by the file storage proxy unit.

The processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the user referred in the application all conform to the rules of relevant laws and regulations and do not violate the public order colloquial.

The names of messages or information interacted between the various devices in the embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of such messages or information.

In a third aspect, exemplary embodiments of the present application further provide a distributed file storage management platform, where the platform includes:

and the control unit is used for checking the target access rights according to the rights information of each user recorded by the control unit, and if the target access rights pass the checking, the file storage proxy unit forwards the mount point access request to the target storage directory.

In one possible embodiment, the file storage agent unit includes a protocol conversion module, configured to perform protocol conversion according to different storage systems, where the storage directory is a storage path of each file in the storage system;

The file storage agent platform is also used for acquiring the mounting point statistical result of each resource pool in each storage system according to the data transmission protocol of each storage system; if the mounting point statistical result of each resource pool exists a target resource pool exceeding a preset safety water line;

the control unit is also used for scanning mounting points in the target resource pool and determining target mounting points to be migrated; and migrating the target mounting points in the target resource pool to a new resource pool based on a preset mounting point migration rule.

In a possible embodiment, the control unit is specifically configured to:

scanning the storage data quantity of each mounting point in the target resource pool, and sequencing the storage data quantity of each mounting point according to the sequence from small to large to obtain a target mounting point sequence; determining the predicted residual resource usage amount of the target resource pool after the target mounting point sequence is moved out of each mounting point based on each stored data amount in the target mounting point sequence;

and when the predicted residual resource usage is closest to the preset recommended water line, determining the mounting point which is predicted to be moved out and corresponds to the predicted residual resource usage closest to the preset recommended water line as the target mounting point to be migrated.

In a fourth aspect, exemplary embodiments of the present application further provide an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor. The memory stores a computer program executable by the at least one processor for causing the electronic device to perform a method according to an embodiment of the present application when executed by the at least one processor.

In a fifth aspect, the present exemplary embodiments also provide a non-transitory computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is for causing the computer to perform a method according to the present embodiments.

In a sixth aspect, the exemplary embodiments of the present application also provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor of a computer, is for causing the computer to perform a method according to an embodiment of the present application.

Referring to fig. 6, a block diagram of an electronic device 800 that may be a server or a client of the present application, which is an example of a hardware device that may be applied to aspects of the present application, will now be described. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the application described and/or claimed herein.

As shown in fig. 6, the electronic device 600 includes a computing unit 601 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, ROM 602, and RAM 603 are connected to each other by a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

A number of components in the electronic device 600 are connected to the I/O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input numeric or character information and generate key signal inputs related to user settings and/or function controls of the electronic device. The output unit 607 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. Storage unit 608 may include, but is not limited to, magnetic disks, optical disks. The communication unit 609 allows the electronic device 600 to exchange information/data with other devices through a computer network, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth (TM) devices, wiFi devices, wiMax devices, cellular communication devices, and/or the like.

The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the foregoing distributed file storage management method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 600 via the ROM 602 and/or the communication unit 609. In some embodiments, the computing unit 601 may be configured to perform the aforementioned distributed file storage management method by any other suitable means (e.g., by means of firmware).

Program code for carrying out methods of the present application may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. The distributed file storage management method is characterized by being applied to a preset distributed file management platform, wherein the preset distributed file management platform at least comprises: the file storage agent unit and the control unit, the method comprises the following steps:

2. The method of claim 1, wherein the file storage agent unit comprises a protocol conversion module for performing protocol conversion according to different storage systems, the method further comprising:

3. The method of claim 2, wherein the scanning, with the control unit, mounting points in the target resource pool, and determining target mounting points to be migrated comprises:

4. The method of claim 2, wherein the new resource pool is determined by:

5. The method of claim 2, wherein the scanning, with the control unit, mounting points in the target resource pool, and determining target mounting points to be migrated comprises:

6. The method of claim 5, wherein the migrating the target mount point in the target resource pool to a new resource pool based on a preset mount point migration rule comprises:

7. The method according to any one of claims 2-6, further comprising:

8. A distributed file storage management apparatus, wherein the apparatus is applied to a preset distributed file management platform, and the preset distributed file management platform at least comprises: the device comprises a file storage agent unit and a control unit, wherein the device comprises:

9. A distributed file storage management platform, the platform comprising:

10. The platform of claim 9, wherein the file storage agent unit comprises a protocol conversion module for performing protocol conversion according to different storage systems;

11. The platform according to claim 10, characterized in that said control unit is in particular adapted to:

12. An electronic device, the electronic device comprising:

a processor; and

a memory in which a program is stored,

wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the method according to any of claims 1-7.

13. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-7.