CN110908965A - Object storage management method, device, equipment and storage medium - Google Patents
Object storage management method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN110908965A CN110908965A CN201911082318.8A CN201911082318A CN110908965A CN 110908965 A CN110908965 A CN 110908965A CN 201911082318 A CN201911082318 A CN 201911082318A CN 110908965 A CN110908965 A CN 110908965A
- Authority
- CN
- China
- Prior art keywords
- target file
- file
- object storage
- memory
- storage cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 52
- 238000000034 method Methods 0.000 claims abstract description 18
- 238000004590 computer program Methods 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 6
- 230000002411 adverse Effects 0.000 abstract description 7
- 230000000694 effects Effects 0.000 abstract description 7
- 230000005540 biological transmission Effects 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000005055 memory storage Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1737—Details of further file system functions for reducing power consumption or coping with limited storage space, e.g. in mobile devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1727—Details of free space management performed by the file system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0625—Power saving in storage systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an object storage management method, a device, equipment and a storage medium, wherein the method comprises the following steps: receiving an access request sent by a client, and determining a file corresponding to the access request as a target file; if the target file exists in the memory, returning the target file in the memory to the client; the memory stores files with access frequencies greater than corresponding frequency thresholds; if the target file does not exist in the memory, whether the file size of the target file is smaller than a corresponding file threshold value is judged, if yes, the target file in the object storage cluster is returned to the client, and if not, the address of the target file in the object storage cluster is returned to the client so as to indicate the client to acquire the target file from the corresponding address in the object storage cluster. The adverse effect on the performance of the object storage cluster caused by the occurrence of frequent access operations is avoided to a certain extent, and the reliability and the availability of the object storage service are improved.
Description
Technical Field
The present invention relates to the field of object storage technologies, and in particular, to an object storage management method, an object storage management apparatus, an object storage management device, and a storage medium.
Background
In the cloud computing era, a large amount of static resources of various audio, video and websites are stored in an object storage cluster; the access heat of different static resources is uneven, when resources with high access heat exist, especially small files with high heat, a large number of requests similar to 4K random reading can be generated, and the performance of an object storage cluster, especially the performance of a high-speed mechanical disk cluster, is seriously influenced; that is, the performance of the object storage cluster is severely affected by frequent access operations in the prior art.
Disclosure of Invention
The invention aims to provide an object storage management method, device, equipment and storage medium, which can avoid adverse effects on the performance of an object storage cluster caused by frequent access operations to a certain extent.
In order to achieve the above purpose, the invention provides the following technical scheme:
an object storage management method, comprising:
receiving an access request sent by a client, and determining a file corresponding to the access request as a target file;
if the target file exists in the memory, returning the target file in the memory to the client; the memory stores files with access frequencies greater than corresponding frequency thresholds;
if the target file does not exist in the memory, judging whether the file size of the target file is smaller than a corresponding file threshold value, if so, returning the target file in the object storage cluster to the client, and if not, returning the address of the target file in the object storage cluster to the client so as to instruct the client to acquire the target file from the corresponding address in the object storage cluster.
Preferably, the method further comprises the following steps:
if the target file does not exist in the memory and the file size of the target file is smaller than the corresponding file threshold, judging whether the access frequency of the target file is larger than the corresponding frequency threshold, if so, copying the target file in the object storage cluster to the memory, and if not, determining that the target file does not need to be cached to the memory; and if the target file does not exist in the memory and the file size of the target file is not smaller than the corresponding file threshold value, determining that the target file does not need to be cached in the memory.
Preferably, after copying the target file in the object storage cluster to the memory, the method further includes:
respectively calculating a check value of the target file in the memory and a check value of the target file in the object storage cluster, judging whether the two check values are consistent, if so, determining that the copying is finished, if not, returning to execute the step of copying the target file in the object storage cluster to the memory until the number of times of returning to execute the step of copying the target file in the object storage cluster to the memory reaches a threshold number of times.
Preferably, the determining whether the target file exists in the memory includes:
calling a cache table, inquiring whether the cache table contains a file identifier carried by the access request, if so, determining that the target file exists in the memory, and if not, determining that the target file does not exist in the memory.
Preferably, before returning the address of the target file in the object storage cluster to the client, the method further includes:
judging whether other access requests are not processed currently, if so, returning the target file in the object storage cluster to the client, and if not, executing the step of returning the address of the target file in the object storage cluster to the client.
Preferably, the method further comprises the following steps:
if any file existing in the memory and in the object storage cluster is changed in the memory or the object storage cluster, synchronizing the change to the object storage cluster or the memory where the any file is not changed.
An object storage management apparatus, comprising:
a receiving module to: receiving an access request sent by a client, and determining a file corresponding to the access request as a target file;
a first processing module to: if the target file exists in the memory, returning the target file in the memory to the client; the memory stores files with access frequencies greater than corresponding frequency thresholds;
a second processing module to: if the target file does not exist in the memory, judging whether the file size of the target file is smaller than a corresponding file threshold value, if so, returning the target file in the object storage cluster to the client, and if not, returning the address of the target file in the object storage cluster to the client so as to instruct the client to acquire the target file from the corresponding address in the object storage cluster.
Preferably, the object storage management device and the object storage cluster are independent of each other.
An object storage management apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the object storage management method as described in any one of the above when executing said computer program.
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the object storage management method according to any one of the preceding claims.
The invention provides an object storage management method, device, equipment and storage medium, wherein the method comprises the following steps: receiving an access request sent by a client, and determining a file corresponding to the access request as a target file; if the target file exists in the memory, returning the target file in the memory to the client; the memory stores files with access frequencies greater than corresponding frequency thresholds; if the target file does not exist in the memory, judging whether the file size of the target file is smaller than a corresponding file threshold value, if so, returning the target file in the object storage cluster to the client, and if not, returning the address of the target file in the object storage cluster to the client so as to instruct the client to acquire the target file from the corresponding address in the object storage cluster. In the technical scheme disclosed by the application, files with high access frequency are stored by the memory, so that the files can be acquired from the memory, and compared with the files acquired from the object storage cluster, the files acquired from the memory obviously improve the file access rate and reduce the access times to the object storage cluster, thereby avoiding adverse effects on the performance of the object storage cluster due to frequent access operations to a certain extent; in addition, for the files which are not stored in the memory, if the files are smaller, the files are directly acquired from the object storage cluster, and if the files are larger, the files are prevented from occupying relevant resources for a longer time by the larger files, and further adverse effects on other access operations are avoided, so that the client is instructed to acquire the files from the object storage cluster, and the effective processing on the access request is further ensured; therefore, the reliability and the availability of the object storage service are improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a first flowchart of an object storage management method according to an embodiment of the present invention;
fig. 2 is a second flowchart of an object storage management method according to an embodiment of the present invention;
fig. 3 is a flowchart of a cache file in an object storage management method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an object storage management apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a flowchart of an object storage management method according to an embodiment of the present invention is shown, where the method includes:
s11: and receiving an access request sent by a client, and determining that a file corresponding to the access request is a target file.
Object storage is an object-based discrete file storage system, widely used in storage applications for audio, video and other static resources in the IT industry. It should be noted that, an execution subject of the object storage management method provided in the embodiment of the present invention may be a corresponding object storage management apparatus, and the apparatus may be understood as an access agent for implementing the object storage management method, where the access agent may be disposed at a server (the server is an end that provides a service for a client, relative to the client), so that only an address of the access agent needs to be provided to the client, and the client accesses the access agent to enable the access agent to implement the corresponding object storage management method, which is transparent to a user; in the embodiment of the present application, an execution subject of the object storage management method is specifically described as an access agent.
When a user or other applications and the like need to access a file (or data) in an object storage cluster, an access request may be sent to an access proxy through a client, where the access request may carry a file identifier of the file to be accessed, such as a file name, a file number, and the like, so that the access proxy may uniquely determine the corresponding file based on the file identifier, and at this time, the file corresponding to the access request, that is, the file to which the file identifier carried in the access request belongs, may be determined to be a target file.
S12: if the target file exists in the memory, returning the target file in the memory to the client; and the memory stores files with access frequencies greater than corresponding frequency thresholds.
After determining the target file, it is first determined whether the target file exists in a memory, where the memory is a memory local to the access proxy and may be a memory storage cluster. If the target file exists in the memory, the target file can be directly obtained from the memory and returned to the client, so that the obtaining speed of the target file can be greatly improved compared with the obtaining of the target file from the object storage cluster, and the response speed of the access request of the client is further improved. The files with the access frequency greater than the corresponding frequency threshold are stored in the memory, the frequency threshold can be set according to actual needs, and the files with the access frequency greater than the corresponding frequency threshold are files with higher heat, namely files which are likely to be frequently accessed; and compared with the data access to the object storage cluster, the access speed of the data access to the memory is obviously higher.
In addition, the access frequency may refer to the total number of times of accessing a certain file within a preset time period, or may be a quotient of the total number of times of accessing a certain file within the preset time period and the preset time period, and of course, other settings may be performed according to actual needs, which are within the protection scope of the present invention. If the access frequency of a certain file is greater than the corresponding frequency threshold, the access heat of the file is high, and the operation of accessing the file is frequent, otherwise, the access heat of the file is low, and the operation of accessing the file is not frequent.
S13: if the target file does not exist in the memory, whether the file size of the target file is smaller than a corresponding file threshold value is judged, if yes, the target file in the object storage cluster is returned to the client, and if not, the address of the target file in the object storage cluster is returned to the client so as to indicate the client to acquire the target file from the corresponding address in the object storage cluster.
If the target file does not exist in the memory, judging whether the file size of the target file is smaller than a corresponding file threshold, wherein the file threshold can be set according to actual needs, such as 55k, and the like, and it needs to be noted that the setting of the file threshold may have great difference in different scenes; if the target file is a small file, the access agent directly accesses the object storage cluster to obtain the target file, and then returns the target file to the client; if the target file is not a small file, the access proxy needs to occupy more resources for a longer time to acquire the target file from the target storage cluster, such as control resources and data transmission resources of the access proxy, and at this time, in order to avoid the influence of the longer-time occupation of the resources on other files needing frequent access, the address of the target file is returned to the client, so that the client directly accesses the target storage cluster based on the address and acquires the target file in the target storage cluster, thereby further ensuring the quick response to the file needing frequent access through the redirection mode (HTTP 302).
In the technical scheme disclosed by the application, files with high access frequency are stored by the memory, so that the files can be acquired from the memory, and compared with the files acquired from the object storage cluster, the files acquired from the memory obviously improve the file access rate and reduce the access times to the object storage cluster, thereby avoiding adverse effects on the performance of the object storage cluster due to frequent access operations to a certain extent; in addition, for the files which are not stored in the memory, if the files are smaller, the files are directly acquired from the object storage cluster, and if the files are larger, the files are prevented from occupying relevant resources for a longer time by the larger files, and further adverse effects on other access operations are avoided, so that the client is instructed to acquire the files from the object storage cluster, and the effective processing on the access request is further ensured; therefore, the reliability and the availability of the object storage service are improved.
The object storage management method provided by the embodiment of the invention can further comprise the following steps:
if the target file does not exist in the memory and the file size of the target file is smaller than the corresponding file threshold, judging whether the access frequency of the target file is larger than the corresponding frequency threshold, if so, copying the target file in the object storage cluster to the memory, and if not, determining that the target file does not need to be cached to the memory; and if the target file does not exist in the memory and the file size of the target file is not smaller than the corresponding file threshold value, determining that the target file does not need to be cached in the memory.
When an access request for any file is received and it is determined that the any file does not exist in the memory, it may be determined whether the any file needs to be copied to the memory based on the file size and the access frequency of the any file. Specifically, when any file is a target file, if the target file does not exist in the memory, determining whether the file size of the target file is smaller than a corresponding file threshold, if the determination result is yes, the target file is a small file, at this time, it needs to be determined whether the target file needs to be copied into the memory based on the access frequency of the target file, if the access frequency of the target file is high (the access frequency is greater than the corresponding frequency threshold), the operation of accessing the target file is more frequent, at this time, in order to reduce the access to the target storage cluster, the target file may be copied into the memory; if the file size of the target file is not smaller than the file threshold and/or the access frequency of the target file is low (the access frequency is not larger than the corresponding frequency threshold), it is indicated that the target file is neither a small file, and the operation of accessing the target file is not frequent, and at this time, the target file does not need to be copied to the memory. In addition, in this embodiment, in addition to considering the access frequency of the target file, the file size of the target file is also considered when determining whether the target file needs to be copied into the memory, because if the target file is a small file, it is necessary to perform random read-write in each time of acquiring the small file from the object storage cluster, which takes a long time in addressing, and acquiring a larger file from the object storage cluster can be generally achieved by sequential read-write, which does not need to take a long time to achieve the positioning of the corresponding file, therefore, in this embodiment, only the small file with a high access frequency is copied into the memory, which can reduce the number of random read-write in the object storage cluster, thereby reducing adverse effects on the object storage cluster and accelerating the access rate of the small file.
The object storage management method provided in the embodiment of the present invention may further include, after copying the target file in the object storage cluster to the memory:
respectively calculating a check value of a target file in the memory and a check value of a target file in the object storage cluster, judging whether the two check values are consistent, if so, determining that the copying is finished, if not, returning to execute the step of copying the target file in the object storage cluster to the memory until the number of times of returning to execute the step of copying the target file in the object storage cluster to the memory reaches a threshold number of times.
It should be noted that, in order to ensure the validity of the file cached in the memory, in this embodiment, whether the file changes in the transmission process may also be determined by comparing whether the check value of the version of the file in the object storage cluster and the check value of the version of the file in the memory are consistent, so as to ensure the validity of the file cached in the memory. In addition, in order to avoid that the file transmission process is changed due to other reasons (such as a problem occurring in data transmission resources), and cannot be influenced by the reasons through retransmission, in this embodiment, when the number of times of file retransmission (the number of times of returning to the step of copying the target file in the object storage cluster into the memory reaches the number threshold), even if the file transmission process is still changed, the file retransmission is not performed, but corresponding prompt information can be output, so that the outside can analyze and repair the reason of outputting the existing problem, and the like, thereby avoiding resource and time waste caused by performing the invalid step indefinitely. The number threshold may be set according to actual needs, for example, may be set for 3 times, 5 times, and the like; the calculation of the file check value may be a calculation of the file check value by using a hash algorithm, and of course, other settings may be performed according to actual needs, which are within the protection scope of the present invention.
The object storage management method provided in the embodiment of the present invention determines whether a target file exists in a memory, and may include:
and calling a cache table, inquiring whether the cache table contains a file identifier carried by the access request, if so, determining that the target file exists in the memory, and if not, determining that the target file does not exist in the memory.
It should be noted that, in this embodiment, a cache table may be set, where file identifiers of all files stored in the memory are cached in the cache table, so that whether a target file exists in the memory may be determined in a table lookup manner, that is, if a file identifier of the target file exists in the cache table, the target file exists in the memory, and if a file identifier of the target file does not exist in the cache table, the target file does not exist in the memory; therefore, the method for determining whether the corresponding file exists in the memory through the table lookup saves the time required by determining whether the file exists in the memory and improves the implementation efficiency of the method compared with the method for traversing and searching the file in the memory if the file is not determined to exist in the memory. In addition, when the file in the memory changes, the cache table needs to be updated correspondingly, that is, the file identifier of the file added to the memory needs to be added to the cache table, and the file identifier of the file deleted from the memory needs to be deleted from the cache table, so as to ensure the consistency between the file identifier in the cache table and the file in the memory.
The object storage management method provided in the embodiment of the present invention may further include, before returning an address of the target file in the object storage cluster to the client:
and judging whether other access requests are not processed currently, if so, returning the target file in the object storage cluster to the client, and if not, executing the step of returning the address of the target file in the object storage cluster to the client.
It should be noted that, after the target file is determined to be a large file, if the access proxy does not currently have other access requests to be processed, it is indicated that the acquisition of the large file in the object storage cluster also does not temporarily affect other access requests, so that the target file can be directly acquired from the object storage cluster in a scene in which the target file is urgently acquired, and the target file is returned to the client, so that the method is applicable to different application scenes on the premise of not affecting other access requests as much as possible, and has certain universality.
The object storage management method provided by the embodiment of the invention can further comprise the following steps:
if any file existing in the memory and in the object storage cluster is changed in the memory or in the object storage cluster, synchronizing the change to the object storage cluster or the memory where any file is not changed.
It should be noted that, in order to ensure consistency of the stored files, when any file in the memory changes, the change needs to be synchronized to the any file in the object storage cluster, and when any file in the object storage cluster changes, the change needs to be synchronized to the any file in the memory, so that files existing in the memory and the object storage cluster at the same time can be kept consistent, and validity of the files is further ensured.
In addition, in a specific application scenario, a flowchart of the object storage management method according to the embodiment of the present application may be as shown in fig. 2, and specifically includes:
1) receiving an access request sent by a client;
2) inquiring a cache table to determine whether a file identifier carried by the access request exists in the cache table;
3) if yes, the target file is obtained by the memory storage cluster and returned to the client;
4) if the target file does not exist, the address of the target file is returned to the client when the target file is a larger file, and the target file is obtained by the object storage cluster and returned to the client when the target file is a smaller file.
In addition, when the target file needs to be copied into the memory, it is also necessary to determine whether the space in the memory is full, if so, the least recently used file is replaced with the target file by using an LRU (least recently used) algorithm to realize storage of the target file, and if the space in the memory is not full, the target file may be directly stored into the memory, so that it is ensured that the files stored in the memory are all files that are most frequently accessed, specifically, a flowchart for caching the files may be as shown in fig. 3, and may include the following steps:
1) judging whether the file size of the file is smaller than a file threshold value or not so as to determine whether caching is needed or not;
2) judging whether the access frequency of the file is greater than a frequency threshold value or not so as to determine whether caching is needed or not;
3) if the fact that caching is needed is determined, preparing to copy the corresponding file into a memory;
4) if the memory is not fully occupied, the corresponding file is directly cached, and if the memory is fully occupied, the LRU algorithm is used for replacement;
5) comparing check values, and keeping consistency of the memory and the object storage cluster files;
6) and updating the cache table.
The method and the device use the memory as the preposed cache of the object storage cluster, and can judge whether the cache is needed or not and automatically finish the caching process according to the size and the access frequency of the file during the file caching; when the file is accessed, the file can be automatically judged to be obtained from the memory or the object storage cluster, and the file accessing is transparent to the user. Therefore, the memory is used for accelerating the object storage service, the influence of frequently reading the small files with high heat on the object storage cluster is relieved, and the IOPS performance of the object storage system can be improved to a certain extent.
An embodiment of the present invention further provides an object storage management apparatus, as shown in fig. 4, which may include:
a receiving module 11, configured to: receiving an access request sent by a client, and determining a file corresponding to the access request as a target file;
a first processing module 12 configured to: if the target file exists in the memory, returning the target file in the memory to the client; the memory stores files with access frequencies greater than corresponding frequency thresholds;
a second processing module 13, configured to: if the target file does not exist in the memory, whether the file size of the target file is smaller than a corresponding file threshold value is judged, if yes, the target file in the object storage cluster is returned to the client, and if not, the address of the target file in the object storage cluster is returned to the client so as to indicate the client to acquire the target file from the corresponding address in the object storage cluster.
According to the object storage management device provided by the embodiment of the invention, the object storage management device and the object storage cluster can be mutually independent.
In this embodiment, the object storage device (or access agent) and the object storage cluster may be independent from each other, that is, the object storage management device and the object storage cluster are decoupled and may be deployed separately; therefore, the object storage management device can manage a plurality of object storage clusters and is convenient for the unified management of different object storage clusters.
The object storage management apparatus provided in the embodiment of the present invention may further include:
a cache module to: if the target file does not exist in the memory and the file size of the target file is smaller than the corresponding file threshold, judging whether the access frequency of the target file is larger than the corresponding frequency threshold, if so, copying the target file in the object storage cluster to the memory, and if not, determining that the target file does not need to be cached to the memory; and if the target file does not exist in the memory and the file size of the target file is not smaller than the corresponding file threshold value, determining that the target file does not need to be cached in the memory.
The object storage management apparatus provided in the embodiment of the present invention may further include:
a verification module to: after the target files in the object storage cluster are copied into the memory, respectively calculating a check value of the target files in the memory and a check value of the target files in the object storage cluster, judging whether the two check values are consistent, if so, determining that the copying is finished, if not, returning to execute the step of copying the target files in the object storage cluster into the memory until the number of times of returning to execute the step of copying the target files in the object storage cluster into the memory reaches a time threshold.
In an object storage management apparatus provided in an embodiment of the present invention, each of the first processing module and the second processing module may include:
a query unit to: and calling a cache table, inquiring whether the cache table contains a file identifier carried by the access request, if so, determining that the target file exists in the memory, and if not, determining that the target file does not exist in the memory.
The object storage management apparatus provided in the embodiment of the present invention may further include:
a determination module configured to: before returning the address of the target file in the object storage cluster to the client, judging whether other access requests are not processed currently, if so, returning the target file in the object storage cluster to the client, and if not, executing the step of returning the address of the target file in the object storage cluster to the client.
The object storage management apparatus provided in the embodiment of the present invention may further include:
a synchronization module to: if any file existing in the memory and in the object storage cluster is changed in the memory or in the object storage cluster, synchronizing the change to the object storage cluster or the memory where any file is not changed.
An embodiment of the present invention further provides an object storage management device, which may include:
a memory for storing a computer program;
a processor for implementing the steps of the object storage management method as any one of the above when executing the computer program.
The embodiment of the invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the object storage management method can be implemented.
It should be noted that, for the description of the relevant parts in the object storage management apparatus, the device and the storage medium provided in the embodiment of the present invention, reference is made to the detailed description of the corresponding parts in the object storage management method provided in the embodiment of the present invention, and details are not described herein again. In addition, parts of the above technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail, so as to avoid redundant description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. An object storage management method, comprising:
receiving an access request sent by a client, and determining a file corresponding to the access request as a target file;
if the target file exists in the memory, returning the target file in the memory to the client; the memory stores files with access frequencies greater than corresponding frequency thresholds;
if the target file does not exist in the memory, judging whether the file size of the target file is smaller than a corresponding file threshold value, if so, returning the target file in the object storage cluster to the client, and if not, returning the address of the target file in the object storage cluster to the client so as to instruct the client to acquire the target file from the corresponding address in the object storage cluster.
2. The method of claim 1, further comprising:
if the target file does not exist in the memory and the file size of the target file is smaller than the corresponding file threshold, judging whether the access frequency of the target file is larger than the corresponding frequency threshold, if so, copying the target file in the object storage cluster to the memory, and if not, determining that the target file does not need to be cached to the memory; and if the target file does not exist in the memory and the file size of the target file is not smaller than the corresponding file threshold value, determining that the target file does not need to be cached in the memory.
3. The method of claim 2, wherein after copying the target file in the object storage cluster into the memory, further comprising:
respectively calculating a check value of the target file in the memory and a check value of the target file in the object storage cluster, judging whether the two check values are consistent, if so, determining that the copying is finished, if not, returning to execute the step of copying the target file in the object storage cluster to the memory until the number of times of returning to execute the step of copying the target file in the object storage cluster to the memory reaches a threshold number of times.
4. The method of claim 3, wherein determining whether the target file exists in the memory comprises:
calling a cache table, inquiring whether the cache table contains a file identifier carried by the access request, if so, determining that the target file exists in the memory, and if not, determining that the target file does not exist in the memory.
5. The method of claim 4, before returning the address of the target file in the object storage cluster to the client, further comprising:
judging whether other access requests are not processed currently, if so, returning the target file in the object storage cluster to the client, and if not, executing the step of returning the address of the target file in the object storage cluster to the client.
6. The method of claim 5, further comprising:
if any file existing in the memory and in the object storage cluster is changed in the memory or the object storage cluster, synchronizing the change to the object storage cluster or the memory where the any file is not changed.
7. An object storage management apparatus, comprising:
a receiving module to: receiving an access request sent by a client, and determining a file corresponding to the access request as a target file;
a first processing module to: if the target file exists in the memory, returning the target file in the memory to the client; the memory stores files with access frequencies greater than corresponding frequency thresholds;
a second processing module to: if the target file does not exist in the memory, judging whether the file size of the target file is smaller than a corresponding file threshold value, if so, returning the target file in the object storage cluster to the client, and if not, returning the address of the target file in the object storage cluster to the client so as to instruct the client to acquire the target file from the corresponding address in the object storage cluster.
8. The apparatus of claim 7, wherein the object storage management apparatus is independent of the object storage cluster.
9. An object storage management apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the object storage management method according to any one of claims 1 to 6 when executing said computer program.
10. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the object storage management method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911082318.8A CN110908965A (en) | 2019-11-07 | 2019-11-07 | Object storage management method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911082318.8A CN110908965A (en) | 2019-11-07 | 2019-11-07 | Object storage management method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110908965A true CN110908965A (en) | 2020-03-24 |
Family
ID=69816532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911082318.8A Pending CN110908965A (en) | 2019-11-07 | 2019-11-07 | Object storage management method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110908965A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111596972A (en) * | 2020-05-15 | 2020-08-28 | 北京小米松果电子有限公司 | Neural network model storage method, loading method, device, equipment and storage medium |
| CN113242185A (en) * | 2021-05-28 | 2021-08-10 | 广州科语机器人有限公司 | Map data transmission method, cleaning robot, equipment and storage medium |
| CN113704200A (en) * | 2021-11-01 | 2021-11-26 | 北京国科环宇科技股份有限公司 | Data storage method, device, equipment and storage medium |
| CN113992658A (en) * | 2021-10-28 | 2022-01-28 | 科大讯飞股份有限公司 | Data transmission method and related device, equipment, system and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070288526A1 (en) * | 2006-06-08 | 2007-12-13 | Emc Corporation | Method and apparatus for processing a database replica |
| CN101510219A (en) * | 2009-03-31 | 2009-08-19 | 成都市华为赛门铁克科技有限公司 | File data accessing method, apparatus and system |
| CN105677904A (en) * | 2016-02-04 | 2016-06-15 | 杭州数梦工场科技有限公司 | Distributed file system based small file storage method and device |
| CN106775446A (en) * | 2016-11-11 | 2017-05-31 | 中国人民解放军国防科学技术大学 | Based on the distributed file system small documents access method that solid state hard disc accelerates |
| CN107341267A (en) * | 2017-07-24 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of distributed file system access method and platform |
| CN107368608A (en) * | 2017-08-07 | 2017-11-21 | 杭州电子科技大学 | The HDFS small documents buffer memory management methods of algorithm are replaced based on ARC |
| CN110191194A (en) * | 2019-06-13 | 2019-08-30 | 华中科技大学 | A kind of Distributed File System Data transmission method and system based on RDMA network |
-
2019
- 2019-11-07 CN CN201911082318.8A patent/CN110908965A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070288526A1 (en) * | 2006-06-08 | 2007-12-13 | Emc Corporation | Method and apparatus for processing a database replica |
| CN101510219A (en) * | 2009-03-31 | 2009-08-19 | 成都市华为赛门铁克科技有限公司 | File data accessing method, apparatus and system |
| CN105677904A (en) * | 2016-02-04 | 2016-06-15 | 杭州数梦工场科技有限公司 | Distributed file system based small file storage method and device |
| CN106775446A (en) * | 2016-11-11 | 2017-05-31 | 中国人民解放军国防科学技术大学 | Based on the distributed file system small documents access method that solid state hard disc accelerates |
| CN107341267A (en) * | 2017-07-24 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of distributed file system access method and platform |
| CN107368608A (en) * | 2017-08-07 | 2017-11-21 | 杭州电子科技大学 | The HDFS small documents buffer memory management methods of algorithm are replaced based on ARC |
| CN110191194A (en) * | 2019-06-13 | 2019-08-30 | 华中科技大学 | A kind of Distributed File System Data transmission method and system based on RDMA network |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111596972A (en) * | 2020-05-15 | 2020-08-28 | 北京小米松果电子有限公司 | Neural network model storage method, loading method, device, equipment and storage medium |
| CN111596972B (en) * | 2020-05-15 | 2023-09-26 | 北京小米松果电子有限公司 | Neural network model storage method, loading method, device, equipment and storage medium |
| CN113242185A (en) * | 2021-05-28 | 2021-08-10 | 广州科语机器人有限公司 | Map data transmission method, cleaning robot, equipment and storage medium |
| CN113992658A (en) * | 2021-10-28 | 2022-01-28 | 科大讯飞股份有限公司 | Data transmission method and related device, equipment, system and storage medium |
| CN113992658B (en) * | 2021-10-28 | 2023-09-26 | 科大讯飞股份有限公司 | Data transmission method, related device, equipment, system and storage medium |
| CN113704200A (en) * | 2021-11-01 | 2021-11-26 | 北京国科环宇科技股份有限公司 | Data storage method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110908965A (en) | Object storage management method, device, equipment and storage medium | |
| US8935482B2 (en) | Method, system and server of removing a distributed caching object | |
| US8463846B2 (en) | File bundling for cache servers of content delivery networks | |
| US10715622B2 (en) | Systems and methods for accelerating object stores with distributed caching | |
| US10275347B2 (en) | System, method and computer program product for managing caches | |
| CN106302595B (en) | Method and equipment for carrying out health check on server | |
| US20140081911A1 (en) | Optimizing automatic deletion of backup files | |
| US20190347167A1 (en) | Primary Node-Standby Node Data Transmission Method, Control Node, and Database System | |
| US10942867B2 (en) | Client-side caching for deduplication data protection and storage systems | |
| CN108540510B (en) | Cloud host creation method and device and cloud service system | |
| CN109582649B (en) | Metadata storage method, device and equipment and readable storage medium | |
| JP5817558B2 (en) | Information processing apparatus, distributed processing system, cache management program, and distributed processing method | |
| CN109522043B (en) | Method and device for managing configuration data and storage medium | |
| CN113032335A (en) | File access method, device, equipment and storage medium | |
| US9928174B1 (en) | Consistent caching | |
| CN105610917B (en) | Method and system for realizing synchronous data repair in system | |
| CN113687781A (en) | Method, device, equipment and medium for pulling up thermal data | |
| CN109165078B (en) | Virtual distributed server and access method thereof | |
| CN112995723A (en) | EPG data management method, server and readable storage medium | |
| CN111367921A (en) | Data object refreshing method and device | |
| CN114064725A (en) | Data processing method, device, equipment and storage medium | |
| CN108073660B (en) | Method and system for realizing cache data synchronization | |
| CN112732667A (en) | Usability enhancing method and system for distributed file system | |
| CN111597477A (en) | Cache cluster optimization method, equipment and storage medium | |
| CN113448971A (en) | Data updating method based on distributed system, computing node and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200324 |