CN107092686B - File management method and device based on cloud storage platform - Google Patents
File management method and device based on cloud storage platform Download PDFInfo
- Publication number
- CN107092686B CN107092686B CN201710269824.2A CN201710269824A CN107092686B CN 107092686 B CN107092686 B CN 107092686B CN 201710269824 A CN201710269824 A CN 201710269824A CN 107092686 B CN107092686 B CN 107092686B
- Authority
- CN
- China
- Prior art keywords
- file
- service
- user
- target
- sharing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
Abstract
The application provides a file management method and a device based on a cloud storage platform, wherein the method is applied to the cloud storage platform, and the cloud storage platform comprises a resource group consisting of a plurality of storage node devices; the storage resources in the resource group are divided into a plurality of sharing pools; wherein, different sharing pools correspond to different types of storage files; the method can comprise the following steps: receiving a service request aiming at obtaining a target file issued by a service system; wherein, the service request carries a user directory of the target file, and the user directory comprises a sharing pool code; analyzing the user directory to obtain the sharing pool code; searching a sharing pool matched with the obtained sharing pool code; and searching the target file in the searched sharing pool, and returning the target file to the service system for corresponding service processing. By using the cloud storage method provided by the application, the file retrieval efficiency can be improved.
Description
Technical Field
The application relates to the field of computer communication, in particular to a cloud storage technology.
Background
The cloud storage may be a system that integrates a large number of different types of storage node devices in a network to cooperatively work through functions such as cluster application, network technology, or a distributed file system, and provides data storage and service access functions to the outside.
Generally, cloud storage can provide an interface for data storage and service access for a service user, and the service user can save and access service files by calling the interface. Since a great amount of service files stored in a mixed manner, such as picture files and video files, are stored in the cloud storage node device, how to quickly and effectively search the service files required by the service user becomes an urgent problem in the industry.
Disclosure of Invention
In view of this, the present application provides a file management method and apparatus based on a cloud storage platform, so as to improve efficiency of file retrieval.
Specifically, the method is realized through the following technical scheme:
according to a first aspect of the present application, a file management method based on a cloud storage platform is provided, where the method is applied to the cloud storage platform, and the cloud storage platform includes a resource group composed of a plurality of storage node devices; the storage resources in the resource group are divided into a plurality of sharing pools; wherein, different sharing pools correspond to different types of storage files; the method comprises the following steps:
receiving a service request aiming at obtaining a target file issued by a service system; wherein, the service request carries a user directory of the target file, and the user directory comprises a sharing pool code;
analyzing the user directory to obtain the sharing pool code;
searching a sharing pool matched with the obtained sharing pool code;
and searching the target file in the searched sharing pool, and returning the target file to the service system for corresponding service processing.
According to a second aspect of the present application, a file management apparatus based on a cloud storage platform is provided, where the apparatus is applied to the cloud storage platform, and the cloud storage platform includes a resource group composed of a plurality of storage node devices; the storage resources in the resource group are divided into a plurality of sharing pools; wherein, different sharing pools correspond to different types of storage files; the device comprises:
a receiving unit, configured to receive a service request for acquiring a target file, where the service request is issued by a service system; wherein, the service request carries a user directory of the target file, and the user directory comprises a sharing pool code;
the analysis unit is used for analyzing the user directory to obtain the sharing pool code;
the searching unit is used for searching the sharing pool matched with the obtained sharing pool code;
and the returning unit is used for searching the target file in the searched sharing pool and returning the target file to the service system for corresponding service processing.
The application provides a file management method based on a cloud storage platform, which divides storage resources in a resource group consisting of different storage node devices into a plurality of sharing pools aiming at different file types. During retrieval, the cloud storage platform can receive a service request for acquiring a target file, which is issued by a service system. Wherein, the service request carries the user directory of the target file, and the user directory comprises a sharing pool code. The cloud storage platform can analyze the user directory, obtain the sharing pool code, and search for a sharing pool matched with the obtained sharing pool code. The cloud storage platform can search the target file in the searched sharing pool and return the target file to the service system for corresponding service processing.
The cloud storage platform can store the service files in a classified manner based on the sharing pool, and during retrieval, the cloud storage platform does not match each file based on the file name any more, but adopts a two-stage searching manner, firstly searches the sharing pool where the file with the same type as the target file is located based on the sharing pool code in the user directory, and then matches the file from the searched sharing pool based on the file name, so that the file retrieval efficiency is greatly improved, and the workload of storage node equipment brought by retrieval is reduced.
Drawings
FIG. 1 is a schematic diagram illustrating a related cloud storage platform for file retrieval according to an exemplary embodiment of the present application;
FIG. 2 is a network architecture diagram illustrating a file management method based on a cloud storage platform according to an exemplary embodiment of the present application;
FIG. 3 is a flow chart illustrating a cloud storage platform based file retrieval according to an exemplary embodiment of the present application;
FIG. 4 is a schematic diagram of a shared pool shown in an exemplary embodiment of the present application;
FIG. 5 is a schematic diagram illustrating a service file retrieval according to an exemplary embodiment of the present application;
FIG. 6 is a diagram illustrating a method for acquiring occupied resource capacity according to an exemplary embodiment of the present application;
fig. 7 is a hardware structure diagram of a device in which a cloud storage apparatus is located according to an exemplary embodiment of the present application;
fig. 8 is a block diagram of a cloud storage device according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
The cloud storage may be a system that integrates a large number of different types of storage node devices in a network to cooperatively work through functions such as cluster application, network technology, or a distributed file system, and provides data storage and service access functions to the outside.
Generally, cloud storage can provide an interface for data storage and service access for a service user, and the service user can save and access service files by calling the interface.
In related file management technologies based on cloud storage platforms, files stored in cloud storage are usually mixed for saving, such as mixed saving of picture files, video files or other types of files.
Based on the storage mode of mixed storage of different types of files, when the files are retrieved, retrieval can be carried out through the retrieval mode shown in figure 1. Fig. 1 is a schematic diagram illustrating a related cloud storage platform for file retrieval according to an exemplary embodiment of the present application.
The cloud storage platform may find a file name (s094000e.ts) from a file path (e.g., usrdir001/20150314/s094000e.ts) issued by a service user, match the file name with each file stored on each storage node device, and return the target file to the service user after matching the target file corresponding to the file name.
However, on one hand, when files of different file types are stored in a mixed manner, each file on different node devices needs to be retrieved to find out the service file required by the service user. For example, a file required by a service user is a picture file a, and due to the mixed storage of different types of files, the cloud storage platform needs to search each file on each storage node device, whether the file is a picture file or a video file, which greatly increases the workload of file retrieval and reduces the efficiency of file retrieval.
On the other hand, different types of files are stored in a mixed mode, so that different levels of disaster recovery processing are difficult to perform on the files of different file types.
In addition, the picture file is a small file, and when the picture file is a large file, files of different file types are stored in a mixed manner, so that the large file and the small file are stored in a mixed manner, when the occupied resource capacity of the storage node device is obtained, each file needs to be scanned respectively, and the occupied resource capacity of the storage node device can be calculated only by calculating the size of the occupied resource of each file. Because each file needs to be scanned, the size of the resource capacity occupied by each file is calculated, so that the efficiency of acquiring the occupied resource capacity is greatly reduced, and the workload of equipment is increased.
The application provides a file management method based on a cloud storage platform, which divides storage resources in a resource group consisting of different storage node devices into a plurality of sharing pools aiming at different file types. During retrieval, the cloud storage platform can receive a service request for acquiring a target file, which is issued by a service system. Wherein, the service request carries the user directory of the target file, and the user directory comprises a sharing pool code. The cloud storage platform can analyze the user directory, obtain the sharing pool code, and search for a sharing pool matched with the obtained sharing pool code. The cloud storage platform can search the target file in the searched sharing pool and return the target file to the service system for corresponding service processing.
On one hand, the cloud storage platform can store the service files in a classified manner based on the sharing pool, and during retrieval, the cloud storage platform can search the sharing pool where the files of the same type as the target file are located based on the sharing pool code in the user directory and then match the files from the searched sharing pool based on the file names instead of matching each file in a two-level retrieval manner, so that the file retrieval efficiency is greatly improved, and the workload of storage node equipment brought by retrieval is reduced.
On the other hand, when the cloud storage is performed, different types of files can be stored respectively based on the shared pool. When the occupied resource capacity is obtained, the capacity of each small file can not be scanned any more for the picture files, namely the small files, but the number of the occupied data segments of the sharing pool for the pictures can be directly calculated, so that the capacity occupation amount of the resources in the sharing pool can be obtained, and the efficiency of the occupied resource capacity is greatly improved.
In the third aspect, since files of different file types are stored separately, disaster tolerance levels for the files of different file types can be established.
Referring to fig. 2, fig. 2 is a network architecture diagram illustrating a file management method based on a cloud storage platform according to an exemplary embodiment of the present application. The network architecture comprises a cloud storage platform, a service system and front-end equipment.
The cloud storage platform can provide an interface for the service system, and the service system can realize storage and access of the service file by calling the interface of the cloud storage platform. The cloud storage platform can comprise a plurality of cloud storage node devices, and the storage node devices can form a resource group. In the embodiment of the application, in order to store different types of files respectively, resources on a resource group composed of storage node devices are divided into a plurality of sharing pools, and each sharing pool corresponds to different types of storage files.
The service system can be visual service system software, and can perform operations such as access and storage on service files on the cloud storage platform according to different service requirements. In the field of video monitoring, the service system can be a video monitoring platform and the like, and the video monitoring platform has the functions of managing a front-end camera, providing video playback for a user and the like. Here, the service system is merely exemplified and not particularly limited.
The hardware device on which the service system is installed may be a server, a server cluster, or the like, and is not particularly limited herein.
The front-end device is a device that performs operations such as data collection. In a scene in the field of video monitoring, the front-end device may be a front-end camera, such as IPC. In a live scene, the front-end device may be a terminal device of a live client, and the like. Here, the front-end device is merely exemplified and not particularly limited.
Under the network architecture, when the service file is stored, the cloud storage platform can send the user directory to the front-end device through the service system, and the front-end device writes the service file into the resource of the cloud storage platform specified by the user directory based on the user directory to which the front-end device belongs, and generates file path information.
When the service file is retrieved, after the cloud platform receives the file path information issued by the service system, the cloud platform can search the service file locally based on the file path information, and can return the searched service file to the service system.
Referring to fig. 3, fig. 3 is a flowchart illustrating a file retrieval process based on a cloud storage platform according to an exemplary embodiment of the present application. The method is applied to a cloud storage platform and can comprise steps 301 to 303.
Step 301: receiving a service request aiming at obtaining a target file issued by a service system; wherein, the service request carries a user directory of the target file, and the user directory comprises a sharing pool code;
step 302: analyzing the user directory to obtain the sharing pool code;
step 303: searching a sharing pool matched with the obtained sharing pool code;
step 304: and searching the target file in the searched sharing pool, and returning the target file to the service system for corresponding service processing.
It should be noted that the cloud storage method provided by the present application may be applied to any scene in which cloud storage is performed, for example, a video monitoring scene, a live broadcast scene, an e-commerce shopping scene, and the like, and a scene to which the cloud storage method provided by the present application is applied is not specifically limited herein.
Here, in order to better introduce the cloud storage method proposed by the present application, a video monitoring scene is taken as an example, and the cloud storage method proposed by the present application is described in detail in three aspects of configuration of a cloud storage platform, storage of a service file, and retrieval of the service file.
1) Configuration of cloud storage platform
The configuration of the shared pool is first described.
In the embodiment of the application, a plurality of storage node devices in the cloud storage platform can form a resource group. The resources in the resource group may be divided into a number of resource blocks, and the divided number of resource blocks may constitute a shared pool. Further, a shared pool is a set of several resource blocks.
During configuration, a cloud storage manager can input sharing pool information on a cloud storage platform, and after the sharing pool information is input, a sharing pool ID can be generated, for example 001, and the sharing pool ID is unique. Cloud storage administrators may then create a shared pool. When the method is implemented, the cloud storage platform can lead a plurality of resource blocks into the sharing pool, establish a mapping relation between the plurality of resource blocks and the ID of the sharing pool, and generate the sharing pool with the plurality of resource blocks.
The resource group may be composed of one storage node device or a plurality of storage node devices, and the form of the resource group is not particularly limited.
The mapping relationship between the shared pool and the resource block may be expressed in various forms, for example, a shared pool code may be added after the resource block ID to establish the mapping relationship between the resource block and the shared pool.
For example, the above shared pool code is 0010, and the resource block IDs of the resource blocks corresponding to the shared pool are res001, res002, res003, and the like. After the mapping relationship between the shared pool and the resource block is established, a shared pool code may be added after the resource block ID to represent the corresponding relationship between the resource block and the shared pool, for example, the resource block ID after the shared pool code is added may be res001_0010, res002_0010, res003_0010, and the like.
Of course, the mapping relationship between the shared pool and the resource blocks may also be represented by a mapping table. Here, this is an exemplary explanation of the mapping relationship between the resource block and the shared pool, and is not particularly limited. For example, as shown in fig. 4, fig. 4 is a schematic diagram of a shared pool according to an exemplary embodiment of the present application. The storage node device a, the storage node b and the storage node c may constitute a resource group, Res representing resource blocks in the resource group. The resources in a resource group are divided into resource blocks that make up a shared pool for different storage file types. Of course, fig. 4 is only an exemplary illustration of resource groups, shared pools, and the like, and the shared pools, resource groups, and the like are not specifically limited herein.
Second, the configuration of the service user is introduced.
The management personnel can configure user information of the service user on the cloud storage platform, and the user information of the service user can comprise a service user ID, a service user name, a target sharing pool allocated to the service user, a resource capacity size allocated to the service user, a file type of a file stored by the service user and the like.
The cloud storage platform can also issue the service user configuration to a service system. After receiving the configuration of the service user, the service system can record the user information of the service user therein, record the corresponding relationship between the service user and the belonging shared pool, and the like.
Finally, the generation of the shared pool code is introduced.
The cloud storage may find a target shared pool allocated for the service user based on the user information of the service user. The cloud storage platform can generate a sharing pool code of the target sharing pool according to the file type of the file stored by the service user and the sharing pool ID of the target sharing pool.
For example, assuming that the target sharing pool ID is 001, a bit of identification bit may be added after the target sharing pool ID, and the value of the identification bit may be used to indicate the file type of the file stored by the service user. It is assumed that 0 indicates that the file stored by the service user is a video file, and 1 indicates that the file stored by the service user is a picture file.
When the cloud storage platform reads that the file to be stored by the service user is a video file, and the shared pool ID of the target shared pool divided for the service user is 001, the shared pool code generated by the cloud storage platform may be 0010.
It should be noted that the generation method of the shared pool code proposed in the present application is not only to generate the shared pool code, but also to carry the file type of the file to be stored in the shared pool code.
A commonly used method for implementing the mapping relationship between the shared pool and the file types of the files stored therein may include establishing a mapping table between the shared pool and the file types. However, the mapping table is used to establish the corresponding relationship between the sharing pool and the file type, and during retrieval, the mapping table needs to be searched first to determine the sharing pool corresponding to the file to be retrieved.
The method and the device realize the correspondence between the shared pool and the file type through the shared pool code, and can directly identify the shared pool corresponding to the file to be retrieved through the identification bit for identifying the file type to be retrieved in the shared pool code without the step of searching a mapping table when the file to be retrieved is retrieved, thereby greatly reducing the load of a system and improving the retrieval efficiency.
2) Storage of business files
After the cloud storage platform receives a service user online message sent by a service system, the cloud storage platform can search a target sharing pool allocated to the target service user, and can create one or more corresponding user directories for the target sharing pool based on a sharing pool code of the target sharing pool.
For example, when the file type to be stored by the service user is a picture file and the target sharing pool code is 0011, the cloud storage platform may create a user directory for the target sharing pool according to the user ID of the service user and the sharing pool code.
For example, the user directory may be in the format of 1179861310 (hash value of the user ID hashed) _0011 (shared pool code)
When the file type to be stored by the service user is a video file and the code of the target sharing pool is 0010, the cloud storage platform may create a plurality of user directories for the target sharing pool according to the user ID of the service user, a plurality of device IDs of a plurality of front-end devices corresponding to the user and the sharing pool code.
For example, the format of the user directory may be HTS _ HC121_51_003 (camera ID) #825060717 (hash value obtained by hashing the user ID) _0010 (shared pool encoding).
After the user directory is generated, the cloud storage platform can send the user directory generated for the target sharing pool to the service system, so that the service system sends the user directory to the front-end camera corresponding to the service user.
For example, when the user directory is a user directory for a picture file, the cloud storage platform may send the user directory to the service system, and the service system may send the user directory to all front-end cameras corresponding to the service user.
When the user directory is a user directory for the video file, the cloud storage platform can issue the user directories to the service system, so that the service system issues the user directories to the front-end cameras corresponding to the front-end camera IDs respectively based on the front-end camera ID information carried in the user directory.
After the front-end camera receives the user directory, when the service file is stored, the service file can be sent to the sharing pool corresponding to the user directory for storage based on the user directory, and path information of the service file is generated. The path information of the service file may include information such as a user directory, a file name, and the like. Here, the path information of the service file is merely exemplary and is not specifically limited.
For example, for a picture file, the path information of the service file may be 1179861310_0011/20150314/s094000e.ts, where 1179861310_0011 is a user directory, 20150314 is time information, and s094000e.ts is a file name.
For a video file, the path information of the above-described service file may be HTS _ HC121_51_003#825060717_ 0010/20150314/s094000e.ts. Here, HTS _ HC121_51_003#825060717_0010 is a user directory, 20150314 is time information, and s094000e.ts is a file name.
After the path information of the service file is generated, the cloud storage platform can return the path information of the service file to a service system or a front-end camera.
3) Retrieval of service documents
In the embodiment of the application, when retrieving the service file, the cloud storage platform may use a two-stage retrieval method, and first, based on the sharing pool code in the user directory, searches for the sharing pool corresponding to the file of the service file to be retrieved, and then based on the file name, matches the file from the found sharing pool, instead of matching each file, thereby greatly improving the efficiency of file retrieval, and reducing the workload of the storage node device caused by retrieval.
When the method is implemented, after a service request for acquiring the target file sent by a service system is received, the cloud storage platform can acquire the path information of the target file carried in the service request. Then, the cloud storage platform may analyze the path information to obtain information such as a user directory and a file name in the path information.
The cloud storage platform can analyze the user directory aiming at the target file and acquire the sharing pool code in the user directory.
The cloud storage platform may search for a shared pool corresponding to the shared pool code based on the obtained shared pool code. After finding the sharing pool corresponding to the sharing pool code, the cloud storage platform may search the sharing pool for a target file corresponding to the file name of the target file.
When the method is implemented, after the sharing pool corresponding to the sharing pool code is found, the cloud storage platform can search the resource block corresponding to the sharing pool based on the mapping relation between the sharing pool and the resource block, and search the target file corresponding to the file name of the target file in the searched resource block.
For example, the above example is still taken as an example in which the shared pool code is added after the resource block ID to indicate the mapping relationship between the resource block and the shared pool.
When searching for the resource block corresponding to the shared pool, the cloud storage platform may search for the resource block carrying the shared pool code based on the shared pool code of the shared pool. After the resource block carrying the sharing pool code is found, the cloud storage platform can find the target file corresponding to the file name of the target file in the resource block.
After the target file is found, the cloud storage platform can read the target file and return the target file to the service system, so that the service system performs corresponding service processing based on the target file.
For example, referring to fig. 5, fig. 5 is a schematic diagram illustrating a service file retrieval according to an exemplary embodiment of the present application.
Assume that the path information of the target file carried in the service request is HTS _ HC121_51_003#825060717_ 0010/20150314/s094000e.ts. The cloud storage platform may parse the path information, and obtain the user directory in the path information, such as information of HTS _ HC121_51_003#825060717_0010, file name s094000e.ts, and the like.
Then, the cloud storage platform may parse the user directory, and obtain a shared pool code, such as 0010, carried in the user directory. The cloud storage platform may find a shared pool corresponding to the shared pool code based on the shared pool code.
After finding the shared pool as shown in fig. 5(a), the cloud storage platform may find a target file matching the file name in the resources of the shared pool based on the file name of the target file, and return the found target file to the service system, so that the service system performs corresponding service processing.
In addition, in the embodiment of the application, due to the adoption of a shared pool mechanism, different types of files can be independently stored. Therefore, managers can configure different priorities for the sharing pools, the disaster tolerance levels of the service files stored in the sharing pools with different priorities are different, and the cloud storage platform can perform disaster tolerance backup with different levels on the service files stored in the cloud storage platform based on the priorities of the sharing pools.
For example, a picture file is more important than a video file for a traffic police user, and the picture file in the shared pool for storing the picture file for the traffic police user can be backed up on one or more storage node devices for disaster recovery processing.
In the embodiment of the application, the files of different types are classified and stored through the shared pool mechanism, so that the large file and the small file can be stored independently. When the occupied resource capacity of the storage node equipment is calculated, the occupied resource capacity can be obtained according to different calculation methods of the large file and the small file.
In the embodiment of the present application, an implementation is provided in which a shared pool corresponding to a picture file corresponds to a unique user directory.
On one hand, one sharing pool corresponds to one user, each layer of directory can not be scanned any more, the occupied capacity of the whole equipment is obtained by obtaining the resource capacity occupied by the files under each layer of directory, and the efficiency of obtaining the occupied resource capacity of the storage node equipment is greatly improved due to the fact that the number of directory scanning is reduced.
On the other hand, for small files, such as picture files, which are files of a standard size generally, when the small files are stored, the cloud storage platform can store the small files into data segments of a fixed capacity and a fixed size, which are the same as the small files, and when the small files are scanned, the size of the occupied resource capacity can be counted directly by calculating the number of the occupied data segments and multiplying the number by the size of the data segments, so that the rate of acquiring the occupied resource capacity of the small files is greatly increased, the rate of acquiring the occupied resource capacity of the storage device storing the large files and the small files is increased, and the workload of scanning the occupied resource capacity by the storage node device is reduced.
For example, as shown in fig. 6, fig. 6 is a schematic diagram illustrating a method for acquiring occupied resource capacity according to an exemplary embodiment of the present application.
Let T denote the directory quota capacity, U directory already occupies resource capacity. Assuming that the quota capacity of the shared pool allocated by the service user for storing the picture file is 1T, and there are two storage devices under the shared pool, a resource with a size of 512G needs to be allocated on each storage device, and a data fragment slice existing on each resource is 512 × 1024/64 — 8192 slices.
For a picture file, the picture file may be stored on slice. In obtaining the used capacity of the user directory, assuming that 500 slices are used in scanning to Res1 and 610 slices are used in Res2, U is T-64 (500+ 610).
In addition, in the embodiment of the application, a unique user directory corresponding to each shared pool is provided. The capacity of resources occupied by scanning the service files under each level of directory is reduced, so that the acquisition rate of the occupied resource capacity is greatly improved.
The application provides a file management method based on a cloud storage platform, which divides storage resources in a resource group consisting of different storage node devices into a plurality of sharing pools aiming at different file types. During retrieval, the cloud storage platform can receive a service request for acquiring a target file, which is issued by a service system. Wherein, the service request carries the user directory of the target file, and the user directory comprises a sharing pool code. The cloud storage platform can analyze the user directory, obtain the sharing pool code, and search for a sharing pool matched with the obtained sharing pool code. The cloud storage platform can search the target file in the searched sharing pool and return the target file to the service system for corresponding service processing.
On one hand, the cloud storage platform can store the service files in a classified manner based on the sharing pool, and during retrieval, the cloud storage platform can search the sharing pool where the files of the same type as the target file are located based on the sharing pool code in the user directory in a two-stage searching manner, and then match the files from the searched sharing pool based on the file names instead of matching each file, so that the file retrieval efficiency is greatly improved, and the workload of storage node equipment brought by retrieval is reduced.
On the other hand, when the cloud storage is performed, different types of files can be stored respectively based on the shared pool. When the occupied resource capacity is obtained, the capacity of each small file can not be scanned any more for the picture files, namely the small files, but the number of the occupied data segments of the sharing pool for the pictures can be directly calculated, so that the capacity occupation amount of the resources in the sharing pool can be obtained, and the efficiency of the occupied resource capacity is greatly improved.
In the third aspect, since files of different file types are stored separately, disaster tolerance levels for the files of different file types can be established.
Corresponding to the embodiment of the file management method based on the cloud storage platform, the application also provides an embodiment of a file management device based on the cloud storage platform.
The embodiment of the file management device based on the cloud storage platform can be applied to the cloud storage platform. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. Taking a software implementation as an example, as a device in a logical sense, a processor of a cloud storage platform where the device is located reads corresponding computer program instructions in a nonvolatile memory into an internal memory for running. In terms of hardware, as shown in fig. 7, the present application is a hardware structure diagram of a cloud storage platform where a file management device based on the cloud storage platform is located, except for the processor, the memory, the network output interface, and the nonvolatile memory shown in fig. 7, a cloud storage platform where the device is located in the embodiment may also include other hardware according to the actual function of the cloud storage platform, which is not described again.
Referring to fig. 8, fig. 8 is a block diagram illustrating a file management apparatus based on a cloud storage platform according to an exemplary embodiment of the present application. The cloud storage device is applied to a cloud storage platform, and the cloud storage platform comprises a resource group consisting of a plurality of storage node devices; the storage resources in the resource group are divided into a plurality of sharing pools; wherein, different sharing pools correspond to different types of storage files; the apparatus may comprise: a receiving unit 810, a parsing unit 820, a lookup unit 830 and a returning unit 840.
The receiving unit 810 is configured to receive a service request for acquiring a target file, which is issued by a service system; wherein, the service request carries a user directory of the target file, and the user directory comprises a sharing pool code;
an analyzing unit 820, configured to analyze the user directory to obtain the shared pool code;
a searching unit 830, configured to search for a shared pool that matches the obtained shared pool code;
a returning unit 840, configured to search the target file in the searched shared pool, and return the target file to the service system for performing corresponding service processing.
In an optional implementation manner, the file management apparatus based on the cloud storage platform may further include a generating unit 850, configured to obtain user information of a preconfigured service user; the user information comprises the file type of the file to be stored by the service user and a target sharing pool allocated to the service user, and a sharing pool code of the target sharing pool is generated based on the file type of the file to be stored by the service user and the sharing pool ID of the target sharing pool allocated to the service user.
In another optional implementation manner, the file management apparatus based on the cloud storage platform may further include a sending unit 860, configured to search, when any target service user is online, a target sharing pool allocated to the target service user; based on the target sharing pool code, creating a corresponding user directory for the target sharing pool; and sending the user directory to the front-end equipment corresponding to the service user, so that the front-end equipment stores the service file with the file type corresponding to the user directory to the target sharing pool based on the user directory.
In another optional implementation manner, the shared pools have different priorities, the priorities of the shared pools are different, and the disaster tolerance levels corresponding to the shared pools are different.
In another optional implementation manner, each shared pool corresponds to a unique user directory; alternatively, each shared pool corresponding to at least a picture file corresponds to a unique user directory.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (10)
1. A file management method based on a cloud storage platform is characterized in that the method is applied to the cloud storage platform, and the cloud storage platform comprises a resource group consisting of a plurality of storage node devices; the storage resources in the resource group are divided into a plurality of sharing pools; wherein, different sharing pools correspond to different types of storage files; the method comprises the following steps:
receiving a service request aiming at obtaining a target file issued by a service system; wherein, the service request carries a user directory of the target file, and the user directory comprises a sharing pool code; the sharing pool code comprises a file type of a storage file;
analyzing the user directory to obtain the sharing pool code;
searching a sharing pool matched with the obtained sharing pool code;
and searching the target file in the searched sharing pool, and returning the target file to the service system.
2. The method of claim 1, further comprising:
acquiring user information of a preconfigured service user; the user information comprises the file type of the file to be stored by the service user and a target sharing pool allocated to the service user;
and generating a sharing pool code of the target sharing pool based on the file type of the file to be stored by the service user and the sharing pool ID of the target sharing pool allocated to the service user.
3. The method of claim 2, further comprising:
when any target service user is online, searching a target sharing pool allocated to the target service user;
based on the target sharing pool code, creating a corresponding user directory for the target sharing pool;
and sending the user directory to the front-end equipment corresponding to the service user, so that the front-end equipment stores the service file with the file type corresponding to the user directory to the target sharing pool based on the user directory.
4. The method of claim 1, wherein the shared pools have different priorities, and wherein the shared pools have different disaster tolerance levels.
5. The method of claim 1, wherein each shared pool corresponds to a unique user directory; alternatively, each shared pool corresponding to at least a picture file corresponds to a unique user directory.
6. The file management device based on the cloud storage platform is characterized by being applied to the cloud storage platform, wherein the cloud storage platform comprises a resource group consisting of a plurality of storage node devices; the storage resources in the resource group are divided into a plurality of sharing pools; wherein, different sharing pools correspond to different types of storage files; the device comprises:
a receiving unit, configured to receive a service request for acquiring a target file, where the service request is issued by a service system; wherein, the service request carries a user directory of the target file, and the user directory comprises a sharing pool code; the sharing pool code comprises a file type of a storage file;
the analysis unit is used for analyzing the user directory to obtain the sharing pool code;
the searching unit is used for searching the sharing pool matched with the obtained sharing pool code;
and the returning unit is used for searching the target file in the searched sharing pool and returning the target file to the service system for corresponding service processing.
7. The apparatus of claim 6, further comprising:
the generating unit is used for acquiring the user information of the pre-configured service user; the user information comprises the file type of the file to be stored by the service user and a target sharing pool allocated to the service user, and a sharing pool code of the target sharing pool is generated based on the file type of the file to be stored by the service user and the sharing pool ID of the target sharing pool allocated to the service user.
8. The apparatus of claim 7, further comprising:
a sending unit, configured to search a target sharing pool allocated to a target service user when any target service user is online; based on the target sharing pool code, creating a corresponding user directory for the target sharing pool; and sending the user directory to the front-end equipment corresponding to the service user, so that the front-end equipment stores the service file with the file type corresponding to the user directory to the target sharing pool based on the user directory.
9. The apparatus of claim 6, wherein the shared pools have different priorities, and wherein the shared pools have different priorities and the disaster tolerance levels corresponding to the shared pools are different.
10. The apparatus of claim 6, wherein each shared pool corresponds to a unique user directory; alternatively, each shared pool corresponding to at least a picture file corresponds to a unique user directory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710269824.2A CN107092686B (en) | 2017-04-24 | 2017-04-24 | File management method and device based on cloud storage platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710269824.2A CN107092686B (en) | 2017-04-24 | 2017-04-24 | File management method and device based on cloud storage platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107092686A CN107092686A (en) | 2017-08-25 |
| CN107092686B true CN107092686B (en) | 2020-04-10 |
Family
ID=59637697
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710269824.2A Active CN107092686B (en) | 2017-04-24 | 2017-04-24 | File management method and device based on cloud storage platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107092686B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107943423A (en) * | 2017-12-07 | 2018-04-20 | 郑州云海信息技术有限公司 | The management method and computer-readable recording medium of storage resource in cloud system |
| CN109241053B (en) * | 2018-07-27 | 2022-08-23 | 创新先进技术有限公司 | Identification code allocation method, device and server |
| CN109669790A (en) * | 2018-12-17 | 2019-04-23 | 北京锐安科技有限公司 | Data sharing method, device, shared platform and storage medium based on cloud platform |
| CN111125475A (en) * | 2019-12-26 | 2020-05-08 | 深圳前海环融联易信息科技服务有限公司 | Shared image management method and device, computer equipment and storage medium |
| CN111881305B (en) * | 2020-06-23 | 2024-03-01 | 博泰车联网科技(上海)股份有限公司 | Scanning method and related equipment |
| CN112084250A (en) * | 2020-09-15 | 2020-12-15 | 深圳市宝能投资集团有限公司 | Data storage method, data query method and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101354726A (en) * | 2008-09-17 | 2009-01-28 | 中国科学院计算技术研究所 | Method for managing memory metadata of cluster file system |
| CN103823760A (en) * | 2014-03-03 | 2014-05-28 | 山西科泰微技术有限公司 | High-efficiency data storage and analysis method |
| CN104660690A (en) * | 2015-02-06 | 2015-05-27 | 中国农业大学 | Cloud video service monitoring system |
| CN104991742A (en) * | 2015-07-01 | 2015-10-21 | 北京奇虎科技有限公司 | Cloud resource classifying and storing method and apparatus |
| CN105005611A (en) * | 2015-07-10 | 2015-10-28 | 中国海洋大学 | File management system and file management method |
| CN105204782A (en) * | 2015-10-13 | 2015-12-30 | 中国联合网络通信集团有限公司 | Data storage achieving method and device |
| CN105224392A (en) * | 2015-10-13 | 2016-01-06 | 中国联合网络通信集团有限公司 | A kind of virtual computing resource quota management method and platform |
-
2017
- 2017-04-24 CN CN201710269824.2A patent/CN107092686B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101354726A (en) * | 2008-09-17 | 2009-01-28 | 中国科学院计算技术研究所 | Method for managing memory metadata of cluster file system |
| CN103823760A (en) * | 2014-03-03 | 2014-05-28 | 山西科泰微技术有限公司 | High-efficiency data storage and analysis method |
| CN104660690A (en) * | 2015-02-06 | 2015-05-27 | 中国农业大学 | Cloud video service monitoring system |
| CN104991742A (en) * | 2015-07-01 | 2015-10-21 | 北京奇虎科技有限公司 | Cloud resource classifying and storing method and apparatus |
| CN105005611A (en) * | 2015-07-10 | 2015-10-28 | 中国海洋大学 | File management system and file management method |
| CN105204782A (en) * | 2015-10-13 | 2015-12-30 | 中国联合网络通信集团有限公司 | Data storage achieving method and device |
| CN105224392A (en) * | 2015-10-13 | 2016-01-06 | 中国联合网络通信集团有限公司 | A kind of virtual computing resource quota management method and platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107092686A (en) | 2017-08-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107092686B (en) | File management method and device based on cloud storage platform | |
| CN107943951B (en) | Method and system for retrieving block chain service information | |
| CN106060442B (en) | Video storage method, device and system | |
| US20180232889A1 (en) | Generating and reviewing motion metadata | |
| CN112632129B (en) | Code stream data management method, device and storage medium | |
| CN109600410B (en) | Data storage system and method | |
| CN108228709B (en) | Data storage method and system, electronic device, program, and medium | |
| CN113938524B (en) | Method and system for monitoring sensitive information leakage of Internet of things terminal based on flow agent | |
| CN112788366B (en) | Video processing method and device | |
| CN110543584B (en) | Method, device, processing server and storage medium for establishing face index | |
| CN110740293A (en) | data transmission method, system, device and computer storage medium | |
| CN109672721B (en) | Media file pushing method and device, server and computer readable storage medium | |
| CN111026709A (en) | Data processing method and device based on cluster access | |
| CN112134719A (en) | Method and system for analyzing base station security log | |
| CN113473166A (en) | Data storage system and method | |
| CN107122136B (en) | Capacity obtaining method and device | |
| CN112632124A (en) | Multimedia information acquisition method, device, system, storage medium and electronic device | |
| CN110909072B (en) | Data table establishment method, device and equipment | |
| CN112002130B (en) | Data management method, device, server and computer readable storage medium | |
| CN113612811B (en) | Method, system, equipment and medium for client mounting in multiple channels | |
| CN110209853B (en) | Image searching method, device and equipment for vehicle | |
| CN111722783B (en) | Data storage method and device | |
| CN113760907A (en) | Data uniqueness identification method in database | |
| CN111061719B (en) | Data collection method, device, equipment and storage medium | |
| CN115373831A (en) | Data processing method, device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20221104 Address after: 250101 4th floor, building 5, zone A2, Hanyu Jingu, Shunhua South Road, high tech Zone, Jinan City, Shandong Province Patentee after: Jinan Yushi Intelligent Technology Co.,Ltd. Address before: 310051 Zhejiang Jiangling Hangzhou Road, Binjiang District, Jiangling, 88, No. 10 South Block 1-11. Patentee before: ZHEJIANG UNIVIEW TECHNOLOGIES Co.,Ltd. |