CN115495012A - Resource processing method, device, platform, equipment and storage medium - Google Patents

Resource processing method, device, platform, equipment and storage medium Download PDF

Info

Publication number
CN115495012A
CN115495012A CN202210916127.2A CN202210916127A CN115495012A CN 115495012 A CN115495012 A CN 115495012A CN 202210916127 A CN202210916127 A CN 202210916127A CN 115495012 A CN115495012 A CN 115495012A
Authority
CN
China
Prior art keywords
resource
storage space
level
level storage
heat
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210916127.2A
Other languages
Chinese (zh)
Inventor
林瑶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN202210916127.2A priority Critical patent/CN115495012A/en
Publication of CN115495012A publication Critical patent/CN115495012A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608Saving storage space on storage systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638Organizing or formatting or addressing of data
    • G06F3/0643Management of files
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638Organizing or formatting or addressing of data
    • G06F3/0644Management of space entities, e.g. partitions, extents, pools
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0668Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671In-line storage system
    • G06F3/0673Single storage device
    • G06F3/068Hybrid storage device

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The disclosure provides a resource processing method, a resource processing device, a resource processing platform, a resource processing device and a storage medium, and relates to the technical fields of cloud computing, data storage and the like. The specific implementation scheme is as follows: responding to a resource request, and acquiring a resource corresponding to the resource request; calculating the heat degree of the resource; based on the heat degree of the resource, performing storage processing on the resource in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced. The technology disclosed by the invention can effectively improve the rationality of resource storage, and further can effectively improve the storage efficiency of resources.

Description

Resource processing method, device, platform, equipment and storage medium
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to the technical fields of cloud computing and data storage, and in particular, to a resource processing method, apparatus, platform, device, and storage medium.
Background
As the conventional internet gradually changes to the mobile internet, a Content Delivery Network (CDN) is used as a basic service of the internet and will meet an explosion period of demand. The mobile internet industry will also become the main target market for future CDNs. Under this great trend, it is the core competitiveness of CDN service providers to improve the overall service capability of the CDN and provide more specialized acceleration services.
In the prior art, CDN service providers use Solid State Disk (SSD) or Serial Advanced Technology Attachment (SATA) hard disks to store content resources. The former has good performance but is expensive and space-saving, while the latter is the opposite. In the prior art, many CDN service providers store some resources of important content in the SSD and other resources of other content in the SATA.
Disclosure of Invention
The disclosure provides a resource processing method, a device, a platform, equipment and a storage medium.
According to an aspect of the present disclosure, there is provided a resource processing method, including:
responding to a resource request, and acquiring a resource corresponding to the resource request;
calculating the heat degree of the resource;
based on the heat degree of the resource, performing storage processing on the resource in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced.
According to another aspect of the present disclosure, there is provided a resource management method including:
receiving a resource application request sent by a first resource processing device; the resource application request carries a storage space identifier;
allocating a storage space for the first resource processing device in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space based on the resource application request; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced.
According to still another aspect of the present disclosure, there is provided a resource processing apparatus including:
the acquisition module is used for responding to a resource request and acquiring a resource corresponding to the resource request;
the computing module is used for computing the heat degree of the resource;
the processing module is used for performing storage processing on the resources in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space based on the heat degree of the resources; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced.
According to still another aspect of the present disclosure, there is provided a resource management apparatus including:
the receiving module is used for receiving a resource application request sent by a first resource processing device; the resource application request carries a storage space identifier;
the allocation module is used for allocating a storage space for the first resource processing device in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space based on the resource application request; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced.
According to yet another aspect of the present disclosure, there is provided a content distribution network platform including at least one resource processing apparatus employing the aspect as described above, and a resource management apparatus of the aspect as described above, and a storage framework including a primary storage space, a secondary storage space, and a tertiary storage space.
According to still another aspect of the present disclosure, there is provided an electronic device including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method of the aspects and any possible implementation described above.
According to yet another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of the aspects and any possible implementation as described above.
According to yet another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method of the aspects and any possible implementation as described above.
According to the technology disclosed by the invention, the rationality of resource storage can be effectively improved, and the storage efficiency of resources can be effectively improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure;
FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure;
FIG. 3 is a schematic view of a storage frame according to the present embodiment;
FIG. 4 is a schematic diagram of a resource hot queue according to the present embodiment;
FIG. 5 is a schematic illustration of a third embodiment according to the present disclosure;
FIG. 6 is a schematic illustration of a fourth embodiment according to the present disclosure;
FIG. 7 is a schematic diagram according to a fifth embodiment of the present disclosure;
FIG. 8 is a schematic diagram according to a sixth embodiment of the present disclosure;
FIG. 9 is a schematic diagram according to a seventh embodiment of the present disclosure;
fig. 10 is a schematic structural diagram of a CDN platform provided in the present embodiment;
FIG. 11 is a schematic diagram according to an eighth embodiment of the present disclosure;
FIG. 12 is a schematic illustration according to a ninth embodiment of the present disclosure;
FIG. 13 is a schematic diagram according to a tenth embodiment of the present disclosure;
FIG. 14 is a schematic illustration according to an eleventh embodiment of the present disclosure;
FIG. 15 is a block diagram of an electronic device used to implement methods of embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It is to be understood that the described embodiments are only a few, and not all, of the disclosed embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
It should be noted that the terminal device involved in the embodiments of the present disclosure may include, but is not limited to, a mobile phone, a Personal Digital Assistant (PDA), a wireless handheld device, a Tablet Computer (Tablet Computer), and other intelligent devices; the display device may include, but is not limited to, a personal computer, a television, and the like having a display function.
In addition, the term "and/or" herein is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
In the prior art, a resource storage policy provided by a CDN service provider stores some resources of more important content in SSD and other resources of other content in SATA, for example. In this storage method, the resources are fixedly stored in the storage space and do not change with factors such as access frequency. For example, for very frequently accessed resources in SATA, each access still requires the acquisition of resources by accessing SATA. The resource reading speed is very slow; for the resources which are rarely accessed in the SSD, the resources are still stored in the SSD, and the space of the SSD is wasted. Therefore, the storage processing manner of the prior art resource is not reasonable.
FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure; as shown in fig. 1, the present embodiment provides a resource processing method, which can be applied in a resource processing apparatus, and specifically includes the following steps:
s101, responding to a resource request, and acquiring a resource corresponding to the resource request;
s102, calculating the heat of resources;
s103, storing the resources in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space based on the heat of the resources; the storage spaces in the first-level storage space, the second-level storage space and the third-level storage space are gradually increased, and the resource reading speed is gradually reduced.
The resource processing device of this embodiment may be a resource processing device in a CDN platform, and is configured to perform relevant resource processing based on a resource request. The resource processing apparatus may correspond to a work process.
The resource request in this embodiment may be a resource request inside a service provider corresponding to the CDN platform, or may be a resource request outside the service provider.
The CDN platform of this embodiment further includes a storage framework of a first-level storage space, a second-level storage space, and a third-level storage space. After receiving the resource request, the resource processing device may return the resource to respond to the resource request after acquiring the corresponding resource based on the resource request. And then further calculating the heat degree of the resource, and performing storage processing on the resource in the storage frame based on the heat degree of the resource. That is, in this embodiment, the resource may be based on the heat degree
The resource processing method of this embodiment may perform storage processing on the resource in a storage framework including a first-level storage space, a second-level storage space, and a third-level storage space based on the heat degree of the resource. Compared with the prior art, the storage processing of the resources refers to the heat degree of the resources, so that the resources are stored in a more reasonable position in the storage frame, the rationality of the resource storage is improved, and the storage efficiency of the resources can be effectively improved.
FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure; the resource processing method of this embodiment further introduces the technical solution of the present disclosure in more detail based on the technical solution of the embodiment shown in fig. 1, and as shown in fig. 2, the resource processing method of this embodiment may specifically include the following steps:
s201, responding to a resource request, and detecting whether a corresponding resource exists in a first-level storage space; if yes, returning the resource to respond to the resource request, and ending; if not, go to step S202;
s202, detecting whether corresponding resources exist in the second-level storage space; if yes, go to step S203; if not, go to step S207;
s203, acquiring the resource from the second-level storage space, and returning to respond to the resource request; executing the step S204;
fig. 3 is a schematic view of a storage frame according to this embodiment. As shown in fig. 3, a storage frame including a first-level storage space, a second-level storage space, and a third-level storage space is taken as an example, where the storage spaces in the first-level storage space, the second-level storage space, and the third-level storage space are gradually increased, and the resource reading speed is gradually decreased. For example, the first-level storage space may be a memory (mem), for short, the second-level storage space may be implemented by using an SSD storage medium, and the third-level storage space may be implemented by using an SATA mechanical hard disk. Where mem may be 32g 6, ssd 1t 3, sata 8. The storage framework can be used in a CDN platform for the resource processing device to perform storage processing on the resource. .
S204, calculating the heat of the resources based on at least one of the latest accessed time of the resources, the accessed frequency within a preset time length and the space occupation amount of the resources; step S205 is executed;
in this embodiment, the calculated heat of the resource is a relative heat, and the heat of the resource may be calculated together based on parameters such as the latest access time of the resource, the frequency of access within a preset time period, and the space occupation amount of the resource. The more parameters are referred to during calculation, the more reasonable and accurate the heat degree of the calculated resources. The occupied amount of the resource is used to identify the size of the storage space occupied by the resource when storing, and may be equal to the size of the resource in the attribute information of the resource.
Further optionally, a weight may be configured for each type of parameter, and the heat of the resource is calculated based on a weighted summation manner by combining the values of the parameters. And in order to make the heat degree of each resource have contrast, the heat degree of each resource can be normalized between 0 and 1.
S205, detecting whether a resource heat queue to which the identification information of the resource belongs changes or not based on the heat of the resource; if yes, go to step S206; otherwise, if no change occurs, no operation is executed for the moment, and the operation is finished.
S206, updating the identification information of the resources from the first resource heat queue to a second resource heat queue, wherein the heat of the resources in the second resource heat queue is higher than that of the resources in the first resource heat queue; and (6) ending.
In this embodiment, the second-level storage space corresponds to at least two resource heat queues; each resource heat queue comprises resource identification information of different heat intervals.
For example, as shown in fig. 4, a plurality of resource hot queues may be correspondingly arranged in the second-level storage space, and each Section corresponds to one resource hot queue. For example, 4 resource heat queues are set in the second-level storage space. Based on this, the resource heat intervals corresponding to the 4 resource heat queues may be [ 0,0.25 ], [ 0.25,0.5 ], [ 0.5,0.75 ], and [ 0.75,1 ], respectively.
Assuming that the heat of the resource is originally 0.2, after the resource is accessed this time, the calculated heat is 0.4, and at this time, the identification information of the resource needs to be updated from the resource heat queue of the heat interval [ 0,0.25 ] to the resource heat queue of the heat interval [ 0.25,0.5 ]. Alternatively, when the resource hot queue to the hot interval [ 0.25,0.5 ] is updated, the resource hot queue may be directly inserted into the head of the resource hot queue.
Optionally, the 4 resource heat queues are divided into equal hot intervals, and in practical application, the resource heat queues may be divided according to the density of the resource heat, for example, if the number of resources in a certain hot interval is large, the corresponding hot interval may be divided into smaller ones, and if the number of resources in some hot intervals is small, the corresponding hot interval may be divided into larger ones appropriately, so as to accommodate more resources. That is, the resource heat queue may be dynamically divided by the resource management device according to the heat condition of the resource stored in the second-level storage space.
S207, detecting whether corresponding resources exist in the third-level storage space; if yes, go to step S208; if not, go to step S213;
s208, acquiring the resource from the third-level storage space, and returning to respond to the resource request; step S209 is executed; s209, calculating the heat of the resource; executing step S210;
the calculation method in step S204 is not described herein again.
S210, detecting whether the heat degree of the resource reaches a preset heat degree threshold value or not based on the heat degree of the resource; if yes, go to step S211; otherwise, if not, executing no operation for the moment, and ending.
The preset heat threshold of this embodiment may be set according to the heat of the resource in the second-level storage space. For example, a heat value greater than the minimum heat of the resource in the second-level storage space may be taken as the preset heat threshold. Or may be set according to practical experience, and is not limited herein.
S211, storing the resources to a second-level storage space; executing the step S212;
s212, based on the resource heat, inserting the identification information of the resource into the corresponding resource heat queue, and ending.
For example, the identification information for the resource may be inserted into the head of the corresponding resource hot queue.
By the method, the resources can be dynamically stored in the storage frame instead of being fixed and unchanged in storage position, so that the resources are stored more reasonably, and the hit rate of the resources can be improved.
S213, obtaining the resource from the source station of the resource library, and returning to respond to the resource request; step S214 is executed;
by adopting the mode of the embodiment, when the resources are requested in the storage frame comprising the first-level storage space, the second-level storage space and the third-level storage space, whether the corresponding resources exist in the first-level storage space, the second-level storage space and the third-level storage space can be detected in sequence, if so, the resources can be directly acquired for resource response, and the resource response speed can be effectively improved. And if the storage frame does not have the resource corresponding to the resource request, the corresponding resource can still be acquired from the source station of the resource library, so as to ensure that the resource request can be responded.
Specifically, the resource request may carry information of a resource pool corresponding to the resource, and the corresponding resource may be acquired from a source station of the resource pool according to the information of the resource pool.
S214, storing the resource into a first-level storage space; and simultaneously, storing the resource into a second-level storage space or a third-level storage space according to a preset storage strategy.
The resource is stored in the first-level storage space by applying for the storage space from the resource management device. Specifically, a resource management device is applied for a storage space equal to the space occupation amount of the resource in the first-level storage space, and after the storage space of the space occupation amount of the resource allocated by the resource management device is acquired, the resource is stored in the storage space allocated within the first-level storage space. This configuration ensures that the resource management apparatus can efficiently manage the storage space and the resources stored in the storage architecture. The occupied amount of the resource is used to identify the size of the storage space occupied by the resource when storing, and may be equal to the size of the resource in the attribute information of the resource.
Because the first-level storage space is a cache, the Input/Output (I/O) is fastest, and the response is fastest when a corresponding resource request is received. In this embodiment, after the resource is obtained from the source station of the resource library for the first time, the resource is stored in the first-level storage space, so that when the resource is accessed again, a quick response can be achieved, and the stand-alone capability of the CDN platform is improved. In addition, because the memory of the first-level storage space is limited, the resource may be eliminated quickly, and therefore, the resource also needs to be stored in the second-level storage space or the third-level storage space to prevent the resource from being acquired from the second-level storage space or the third-level storage space after the resource in the memory fails, so as to improve the response speed of the resource request.
It should be noted that, in this embodiment, the resource is stored in the first-level storage space, and the accessed time of the resource needs to be recorded, so that the resource is eliminated and used when the first-level storage space is full.
The accessed resources can be stored in a storage frame of the CDN platform, wherein the storage frame comprises a first-level storage space, a second-level storage space and a third-level storage space.
By adopting the technical scheme, the resource processing method can dynamically store and process the resources in the storage frame comprising the first-level storage space, the second-level storage space and the third-level storage space based on the heat of the resources, so that the resources are stored at a more reasonable position in the storage frame, the rationality of resource storage is improved, and further the hit rate of the resources during resource request and the response capability of the resource request can be effectively improved.
FIG. 5 is a schematic diagram according to a third embodiment of the present disclosure; the resource processing method of this embodiment further introduces the technical solution of the present disclosure in more detail on the basis of the technical solution of the embodiment shown in fig. 2, as shown in fig. 5, specifically includes the following steps:
s501, detecting whether the space occupation amount of the resources is smaller than a preset threshold value, if so, executing a step S502; otherwise, go to step S506;
s502, detecting whether a free resource block exists in a second-stage storage space; if yes, go to step S503; if not, go to step S504;
s503, storing the resources into a first resource block of a second-level storage space; the first resource block is applied to a resource management device in advance; step S505 is executed;
s504, applying for resource blocks in the second-level storage space to the resource management device, and executing the step S503;
it should be noted that, the resource blocks of the second-level storage space of the current resource processing apparatus are obtained by applying in advance, which can ensure that the current resource processing apparatus can have a storage space to store in time when it wants to store resources, and can effectively improve the resource storage efficiency.
In this embodiment, each time a resource is to be stored, it is first detected through step S502 whether a free resource block exists in the second-level storage space of the current resource processing apparatus. Free resource blocks here may refer to completely unused resource blocks. That is, the resource block currently used by the resource processing device is the last resource block, and if the resource block is used up, there is no resource block available for the resource processing device in the second-level storage space. Therefore, in order to improve the storage efficiency, the resource block needs to be applied in advance.
Optionally, in an embodiment of the present disclosure, the step S502 may not be performed after the step S501, and the detection may be performed once every preset time period. At this time, when it is detected that the space occupation amount of the resource is less than the preset threshold, step S503 may be directly performed. And the resource block applied for in step S504 is usually not the first resource block of step S503.
Of course, optionally, for special cases, if the resource processing device does not apply for the resource block in time, the resource block may also be applied in real time during storage, and at this time, the resource block applied in step S504 may also be the first resource block stored in step S603.
S505, based on the resource heat, inserting the identification information of the resource into the corresponding resource heat queue, and ending;
on the premise of the same bandwidth request, the request of the small file is divided into a plurality of times, and if the small file is stored in the SATA, the IO of the SATA is occupied; in the embodiment, the resources with the space occupation smaller than the preset threshold are stored in the second-level storage space, so that when the resources of the small files are accessed, the corresponding resources can be acquired from the second-level storage space to respond, and the pressure of the SATA can be relieved. The single machine capability is improved.
S506, detecting whether the heat of the resources reaches a preset threshold value or not, and if so, returning to the step S502; if not, go to step S507;
alternatively, the process may return to step S503 when the heat of the resource reaches a preset predetermined threshold or more.
S507, detecting whether an idle resource block exists in the third-level storage space; if yes, go to step S508; if not, go to step S509;
s508, storing the resources into a second resource block of a third-level storage space, and ending; the second resource block is applied to a resource management device in advance;
s509, applying for a resource block in a third-level storage space from the resource management device, and executing the step S508;
the storage processing method for storing the resource in the third-level storage space in steps S507-S509 is the same as the storage processing method for storing the resource in the second-level storage space in steps S502-S504, and reference may also be made to the related description of the above steps, which is not repeated herein. That is, the resource blocks of the current resource processing device in the third-level storage space are also obtained in a pre-application manner, so that the current resource processing device can be ensured to have the storage space to store in time when the resource processing device wants to store the resource, and the resource storage efficiency can be effectively improved. For example, the resource processing method of this embodiment may specifically be a specific implementation scheme of "storing the resource into the second-level storage space or the third-level storage space according to a preset storage policy" in the step S214 illustrated in fig. 2.
Or step S211 of the above-mentioned embodiment shown in fig. 2 can also be implemented by using steps S502 to S504 of this embodiment.
The resource processing method of the embodiment can store the resource with the space occupation smaller than the preset threshold value in the second-level storage space, so as to improve the single machine capability. And storing the resources with the heat degree higher than the preset threshold value in the second-level storage space and storing the resources with the heat degree lower than the preset threshold value in the third-level storage space based on the heat degree of the resources, so that the resources can be reasonably stored based on the heat degree, and the hit rate of the resources in resource request and the response capability of the resource request can be effectively improved.
FIG. 6 is a schematic diagram according to a fourth embodiment of the present disclosure; the resource processing method of the present embodiment is further described in detail based on the technical solutions of the above embodiments. As shown in fig. 6, the resource processing method of this embodiment may specifically include the following steps:
s601, receiving a resource elimination request sent by a resource management device; the resource elimination request carries an identifier of a storage space and an identifier of an elimination resource block;
and S602, based on the identification of the storage space and the identification of the eliminated resource blocks, carrying out elimination processing on the eliminated resource blocks.
In this embodiment, in a storage frame including a first-level storage space, a second-level storage space, and a third-level storage space, the first-level storage space is stored in the form of resources, and the second-level storage space and the third-level storage space are allocated to the resource processing device in the form of resource blocks, so that the resource processing device stores the resources in the applied resource blocks. Therefore, when the data is eliminated, the second-level storage space and the third-level storage space are also eliminated in the form of resource blocks.
For example, when the resource blocks in the second-level storage space are eliminated, the corresponding step S602 performs elimination processing on the eliminated resource blocks based on the identifier of the storage space and the identifier of the eliminated resource blocks, which may specifically include the following steps:
(1) If the eliminated resource block is determined to be in the second-level storage space according to the identification of the storage space, whether the resource in the eliminated resource block needs to be written back is detected based on at least two resource heat queues and the heat of the resource in the eliminated resource block; if the writing is needed, executing the step (2);
for example, in this embodiment, the resource that needs to write back the resource may be set based on the resource hot queue. Because the resource heat queues are set according to the heat intervals of the resources, the resources in one, two or more resource heat queues with higher heat intervals of the resources need to be written back when the stored resource blocks are eliminated. In the concrete implementation, whether a resource heat queue to which each resource in the eliminated resource block belongs is in a queue needing to write back the resource is detected, and if so, the corresponding resource needs to write back when the resource block is eliminated. If a resource is in the queue that does not need to write back the resource, the resource processing device directly releases the resource without writing back again.
(2) Acquiring information of the write-back resource; executing the step (3);
(3) Releasing the resources in the eliminated resource blocks; executing the step (4);
(4) And based on the information of the write-back resource, the write-back resource is stored into the second-level storage space again.
Because the second-level storage space adopts the SSD, when the resource blocks in the SSD are eliminated, the resources with high heat degree can be eliminated, so when the resource blocks in the SSD are eliminated, the resources with high heat degree can be written back to the SSD again in order to avoid eliminating the resources with high heat degree. The write-back process is the same as the process of normally storing resources into the SSD in principle, and the resource blocks which are applied in advance are directly written in. And no idle resource block is available, and the resource management device is applied in advance.
For example, when a resource block in the third-level storage space is eliminated, the corresponding step S602 performs elimination processing on the eliminated resource block based on the identifier of the storage space and the identifier of the eliminated resource block, which may specifically include the following steps:
and if the eliminated resource blocks are determined to be in the third-level storage space according to the identification of the storage space, releasing the resources in the eliminated resource blocks.
Because the SATA storage medium adopted by the third-level storage space realizes storage, the problem of write back does not exist, and resources in the eliminated resource blocks can be directly released.
In addition, for the elimination of the resources in the first-level storage space, a resource elimination request sent by the resource management device can be directly received; the resource elimination request carries an identifier of the storage space and an identifier of the eliminated resource; based on the identification of the storage space and the identification of the eliminated resources, the eliminated resources are eliminated, and the method is very simple to realize.
By adopting the technical scheme, the resource processing method of the embodiment can eliminate the resources when the storage space is insufficient so as to vacate the storage space, store other resources and realize more reasonable utilization of the multi-stage storage space of the storage frame. And for the second-level storage space, the resources with high heat can be written back during elimination, and the hit rate and the response capability during resource request are improved.
FIG. 7 is a schematic diagram according to a fifth embodiment of the present disclosure; the embodiment provides a resource management method, which may be applied in a resource management device, and specifically includes the following steps:
s701, receiving a resource application request sent by a first resource processing device; the resource application request carries a storage space identifier;
s702, based on the resource application request, allocating a storage space for the first resource processing device in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space; the storage spaces in the first-level storage space, the second-level storage space and the third-level storage space are gradually increased, and the resource reading speed is gradually reduced.
The present embodiment describes the technical solution of the present disclosure on the resource management device side. In this embodiment, when the resource processing apparatus stores the resource, the used storage space is obtained by applying the application method. In this embodiment, a first resource processing device is taken as an example to apply for a resource from a resource management device. Since there are three levels of storage spaces in the storage framework, the first resource processing device needs to carry a storage space identifier when sending a resource application request, so as to specify the location of the storage space that is desired to be applied.
Specifically, the resource management device allocates a storage space for the first resource processing device in a storage framework including a first-level storage space, a second-level storage space, and a third-level storage space based on the resource application request. The storage frame of this embodiment and the first level storage space, the second level storage space, and the third level storage space included therein refer to the related descriptions of the above embodiments, and are not described herein again.
The resource management method of this embodiment may allocate a storage space for the first resource processing device in a storage framework including a first-level storage space, a second-level storage space, and a third-level storage space based on the resource application request, and may implement effective and reasonable allocation of the storage space.
FIG. 8 is a schematic diagram according to a sixth embodiment of the present disclosure; the resource management method of this embodiment further introduces the technical solution of the present disclosure in more detail on the basis of the technical solution of the embodiment shown in fig. 7. As shown in fig. 8, the resource management method of this embodiment may specifically include the following steps:
s801, receiving a resource application request sent by a first resource processing device; the resource application request carries the identifier of the first-level storage space and the space occupation of the resource to be stored;
s802, detecting whether the residual storage space in the first-level storage space is smaller than the space occupation amount of the resource, if so, executing a step S803; if not, go to step S805;
s803, acquiring an identifier of a eliminated resource based on a preset first elimination strategy; executing step S804;
the preset first elimination strategy of the present embodiment may be implemented based on Least Recently Used (LRU) or Least Recently Used (LFU).
For example, the resource with the longest last access time may be selected for eviction based on LRU. Or based on the LFU, selecting the resource with the least access frequency in a preset time period for elimination.
Optionally, during elimination, the space occupation amount of the resource and the remaining storage space in the first-level storage space may be further referred to determine an identifier of the eliminated resource, so that after the eliminated resource is eliminated, a storage space larger than the resource space occupation amount can be vacated and allocated to the first resource processing apparatus for the first resource processing apparatus to store the resource.
In this embodiment, according to an actual situation, the identifier of the eliminated resource may be one, two, or more, so as to make the storage space that is larger than the resource space occupation amount in the first-level storage space be finally vacated.
S804, sending a first resource elimination request to a second resource processing device corresponding to the eliminated resource identifier, wherein the first resource elimination request carries the eliminated resource identifier so that the second resource processing device can release the storage space of the eliminated resource; executing step S805;
the second resource processing device may be different from the first resource processing device, or may be the same as the first resource processing device.
Correspondingly, after receiving the first resource elimination request, the second resource processing device directly releases the storage space corresponding to the identification of the eliminated resource in the first-level storage space.
S805, based on the space occupation amount of the resource, allocating a storage space in the first-level storage space for the resource to be stored by the first resource processing device, and ending.
By adopting the above technical solution, the resource management method of this embodiment can realize reasonable and effective allocation of storage space for the first resource processing device in the first-level storage space; and when the first-stage storage space is insufficient, other resources can be reasonably and effectively eliminated to realize the allocation of space for the currently applied resources, unreasonable resources in the first-stage storage space can be effectively removed, more effective resources can be stored, the hit rate of the first-stage storage space is improved, and the reasonable utilization of a storage frame is realized.
FIG. 9 is a schematic diagram according to a seventh embodiment of the present disclosure; the resource management method of this embodiment further introduces the technical solution of the present disclosure in more detail on the basis of the technical solution of the embodiment shown in fig. 7. As shown in fig. 9, the resource management method of this embodiment may specifically include the following steps:
s901, receiving a resource application request sent by a first resource processing device; the resource application request carries an identifier of a second-level storage space or a third-level storage space;
s902, detecting whether the second-level storage space or the third-level storage space has allocable resource blocks, and if not, executing the step S903; if not, go to step S906;
s903, acquiring elimination parameters of each resource block in the second-level storage space or the third-level storage space; executing step S904;
in this embodiment, the elimination mechanism of the second-level storage space or the third-level storage space is the same as that of the third-level storage space.
However, the obsolete parameter settings of the resource blocks in the second level storage space and the third level storage space are different. For example, in the second-level storage space, the elimination parameter of each resource block may be set based on the current time and the creation time of each resource block. Specifically, the inverse of the time difference between the current time and the creation time of the resource block may be taken as the elimination parameter of the resource block. In this way, the elimination parameters of all resource blocks can be set in the range of 0-1, and comparison is convenient. The smaller the elimination parameter set in this way, the more the current resource block should be eliminated. Conversely, the larger the elimination parameter is, the more the current resource block should not be eliminated.
The elimination parameters of each resource block in the third-level storage space can be obtained based on the current time and the last accessed time of the last accessed resource in each resource block. That is, if a plurality of resources are included in one resource block, only the last accessed resource is referred to at this time, and the last accessed time of the last accessed resource is referred to. Specifically, the inverse of the time difference between the current time and the last accessed time of the last accessed resource in the resource block may be taken as the elimination parameter of the resource block. In this way, the elimination parameters of all resource blocks can be set in the range of 0-1, and comparison is convenient. The smaller the elimination parameter set in this way, the more the current resource block should be eliminated. Conversely, the larger the elimination parameter is, the less the current resource block should be eliminated.
S904, acquiring the identification of the eliminated resource block based on a preset second elimination strategy and elimination parameters of each resource block; executing step S905;
for example, according to the second elimination strategy, the resource block with the smallest elimination parameter may be selected for elimination, and the identifier of the resource block with the smallest elimination parameter is correspondingly obtained as the identifier of the eliminated resource block.
S905, sending a second resource elimination request to a third resource processing device corresponding to the eliminated resource block identifier, wherein the second resource elimination request carries the eliminated resource block identifier, so that the third resource processing device can release the storage space of the eliminated resource block; executing step S906;
the third resource processing device may be different from the first resource processing device, or may be the same as the first resource processing device.
And S906, allocating resource blocks to the first resource processing device in the second-level storage space or the third-level storage space, and ending.
Based on the above, it can be known that in the second-level storage space, i.e., the storage space of the SSD, the elimination is performed in a manner of complying with First Input First Output (FIFO). However, since the resource with high heat in the resource block is eliminated in this way, in order to avoid eliminating the resource with high heat, the embodiment shown in fig. 6 can be adopted to write back the resource with high heat, so as to improve the hit rate of the resource request.
By adopting the above technical solution, the resource management method of this embodiment can realize reasonable and effective allocation of storage space for the first resource processing device in the second-level storage space and the third-level storage space; and when the second-level storage space and the third-level storage space are insufficient, other resources can be reasonably and effectively eliminated to realize the allocation of space for the currently applied resources, unreasonable resources in the second-level storage space and the third-level storage space can be effectively removed, more effective resources can be stored, the hit rate of the second-level storage space and the third-level storage space is improved, and the reasonable utilization of a storage frame is realized.
Fig. 10 is a schematic architecture diagram of a CDN platform provided in this embodiment, and as shown in fig. 10, a CDN platform 1000 may include at least one resource processing device 1001, a resource management device 1002, and a storage frame 1003 including a first-level storage space, a second-level storage space, and a third-level storage space.
Preferably, the present embodiment may include at least two resource processing devices 1001, each resource processing device 1001 corresponds to a resource service of one channel, for example, if only two resource processing devices 1001 are included, one resource processing device 1001 may process an internal resource service request, and another resource processing device 1001 may process an external resource service request. Alternatively, if a plurality of resource processing devices 1001 are provided, different clients, different areas, or specified resource service requests of different ranges may be processed by different resource processing devices 1001.
Compared with the prior art in which only one resource processing device exists, the architecture of the present embodiment, which employs at least two resource processing devices 1001, can not affect the resource processing of other resource processing devices 1001 when one resource processing device 1001 fails, and can improve the resource processing efficiency of CDN platform 1000.
The resource processing apparatus 1001 of this embodiment may be referred to as a work process, the resource management apparatus 1002 may be referred to as a master process, and after receiving a resource request, according to the method embodiment, when it is necessary to apply for a storage space in a first-level storage space, a second-level storage space, or a third-level storage space in the storage frame 1003 from the master process, the work process applies for the storage space from the master process according to the method of the embodiment. And when the master process detects that the storage space in the first-level storage space, the second-level storage space or the third-level storage space is insufficient, unreasonable storage space can be eliminated to allocate the storage space for the work process. For details, reference may be made to the technical solutions of the embodiments shown in fig. 1 to fig. 9, which are not described herein again.
FIG. 11 is a schematic diagram according to an eighth embodiment of the present disclosure; as shown in fig. 11, the present embodiment provides a resource processing apparatus 1100, including:
an obtaining module 1101, configured to obtain, in response to a resource request, a resource corresponding to the resource request;
a calculating module 1102, configured to calculate a heat degree of the resource;
a processing module 1103, configured to perform storage processing on the resource in a storage framework including a first-level storage space, a second-level storage space, and a third-level storage space, based on the heat of the resource; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced.
The resource processing apparatus 1100 of this embodiment implements the principle and technical effect of implementing resource processing by using the modules, which are the same as the implementation of the related method embodiments described above, and details of the related method embodiments may be referred to and are not described herein again.
FIG. 12 is a schematic illustration according to a ninth embodiment of the present disclosure; as shown in fig. 12, the present embodiment provides a resource processing apparatus 1200, including: the modules with the same name and the same function shown in the above fig. 11: an acquisition module 1201, a calculation module 1202 and a processing module 1203.
In this embodiment, the calculating module 1202 is configured to:
and calculating the heat degree of the resource based on at least one of the latest accessed time of the resource, the accessed frequency within a preset time length and the space occupation amount of the resource.
In an embodiment of the present disclosure, an obtaining module 1201 for
And acquiring the resource from a source station of a resource library.
In one embodiment of the present disclosure, the processing module 1203 is configured to:
storing the resource into a first-level storage space;
and simultaneously, storing the resources into the second-level storage space or the third-level storage space according to a preset storage strategy.
In an embodiment of the present disclosure, the processing module 1203 is configured to:
and if the space occupation amount of the resources is smaller than a preset threshold value, storing the resources to the second-level storage space.
In an embodiment of the present disclosure, the processing module 1203 is further configured to:
if the heat degree of the resource reaches a preset heat degree threshold value, storing the resource into the second-level storage space; or
And if the heat degree of the resource does not reach the preset heat degree threshold value, storing the resource into the third-level storage space.
In an embodiment of the present disclosure, the processing module 1203 is configured to:
and storing the resources in the first-level storage space in a mode of applying for a storage space to a resource management device.
In an embodiment of the present disclosure, the processing module 1203 is configured to:
storing the resource into a first resource block of the second level storage space; the first resource block is applied to a resource management device in advance;
before storing the resource into the first resource block of the second level storage space, the method further comprises:
detecting whether idle resource blocks exist in the second-level storage space or not;
and if the resource blocks do not exist, applying for the resource blocks in the second-level storage space from the resource management device.
In an embodiment of the present disclosure, the processing module 1203 is further configured to:
based on the heat degree of the resource, inserting the identification information of the resource into a corresponding resource heat degree queue; the second-level storage space corresponds to at least two resource heat queues; each resource hot queue comprises resource identification information of different hot intervals.
In an embodiment of the present disclosure, the processing module 1203 is further configured to:
storing the resource into a second resource block of the third level of storage space; the second resource block is applied to a resource management device in advance;
the processing module 1203 is further configured to:
detecting whether idle resource blocks exist in the third-level storage space;
and if the resource blocks do not exist, applying for the resource blocks in the third-level storage space to the resource management device.
In an embodiment of the present disclosure, the obtaining module 1201 is configured to:
and acquiring the resource from the third-level storage space.
In an embodiment of the present disclosure, the processing module 1203 is configured to:
detecting whether the heat degree of the resource reaches a preset heat degree threshold value or not based on the heat degree of the resource;
if so, storing the resource to the second-level storage space;
and inserting the identification information of the resource into a corresponding resource heat queue based on the heat of the resource.
In an embodiment of the present disclosure, the obtaining module 1201 is configured to:
and acquiring the resource from the second-level storage space.
In an embodiment of the present disclosure, the processing module 1203 is configured to:
and updating the identification information of the resource from the first resource heat queue to a second resource heat queue based on the heat of the resource, wherein the heat of the resource in the second resource heat queue is higher than that of the resource in the first resource heat queue.
As shown in fig. 12, in an embodiment of the present disclosure, the resource processing apparatus 1200 further includes:
a receiving module 1204, configured to receive a resource elimination request sent by a resource management apparatus; the resource elimination request carries an identifier of a storage space and an identifier of an elimination resource block;
the processing module 1203 is further configured to perform elimination processing on the eliminated resource blocks based on the identifier of the storage space and the identifier of the eliminated resource blocks.
In one embodiment of the disclosure, the processing module 1203 is configured to:
if the eliminated resource block is determined to be in the second-level storage space according to the identifier of the storage space, detecting whether the resource in the eliminated resource block needs to be written back or not based on the at least two resource heat queues and the heat of the resource in the eliminated resource block;
if the write is needed, the information of the write-back resource is obtained;
releasing the resources in the eliminated resource blocks;
and based on the information of the write-back resource, the write-back resource is stored into the second-stage storage space again.
In an embodiment of the present disclosure, the processing module 1203 is configured to:
and if the eliminated resource blocks are determined to be in the third-level storage space according to the identification of the storage space, releasing the resources in the eliminated resource blocks.
The resource processing apparatus 1200 of this embodiment implements the principle and technical effect of implementing resource processing by using the modules, which are the same as the implementation of the related method embodiment described above, and details of the related method embodiment may be referred to and are not described herein again.
FIG. 13 is a schematic diagram according to a tenth embodiment of the present disclosure; as shown in fig. 13, the present embodiment provides a resource management apparatus 1300, including:
a receiving module 1301, configured to receive a resource application request sent by a first resource processing apparatus; the resource application request carries a storage space identifier;
an allocating module 1302, configured to allocate a storage space for the first resource processing apparatus in a storage framework including a first-level storage space, a second-level storage space, and a third-level storage space based on the resource application request; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced.
The resource management apparatus 1300 of this embodiment implements the principle and technical effect of implementing resource management by using the modules, which are the same as the implementation of the related method embodiments described above, and details of the related method embodiments may be referred to and are not described herein again.
FIG. 14 is a schematic illustration according to an eleventh embodiment of the present disclosure; as shown in fig. 14, the present embodiment provides a resource management apparatus 1400, which includes: the modules with the same name and the same function shown in fig. 13: a receiving module 1401 and an assigning module 1402.
In this embodiment, the allocating module 1402 is configured to: when the storage space identifier is the identifier of the first-level storage space, the resource application request also carries the space occupation amount of the resource to be stored, and based on the space occupation amount of the resource, the storage space is allocated for the resource to be stored by the first resource processing device in the first-level storage space.
As shown in fig. 14, in one embodiment of the present disclosure, the resource management device 1400 further includes:
an obtaining module 1403, configured to obtain an identifier of a eliminated resource based on a preset first elimination policy if the remaining storage space in the first-stage storage space is smaller than the space occupation amount of the resource;
a sending module 1404, configured to send a first resource elimination request to a first resource processing apparatus corresponding to the eliminated resource identifier, where the first resource elimination request carries the eliminated resource identifier, so that the first resource processing apparatus releases the storage space of the eliminated resource.
In one embodiment of the present disclosure, the assignment module 1402 is configured to: and when the storage space identifier is the identifier of the second-level storage space or the third-level storage space, allocating resource blocks for the resources in the second-level storage space or the third-level storage space.
In an embodiment of the present disclosure, the obtaining module 1403 is further configured to:
if the second-level storage space or the third-level storage space does not have allocable resource blocks, acquiring elimination parameters of the resource blocks in the second-level storage space or the third-level storage space;
acquiring the identification of the eliminated resource blocks based on a preset second elimination strategy and elimination parameters of each resource block;
in an embodiment of the present disclosure, the sending module 1404 is further configured to send a second resource elimination request to a second resource processing apparatus corresponding to the identifier of the eliminated resource block, where the second resource elimination request carries the identifier of the eliminated resource block, so that the second resource processing apparatus releases the storage space of the eliminated resource block.
In an embodiment of the present disclosure, the elimination parameter of each resource block in the second-level storage space is obtained based on the current time and the creation time of each resource block; and/or the elimination parameter of each resource block in the third-level storage space is obtained based on the current moment and the last accessed moment of the last accessed resource in each resource block.
The resource management apparatus 1400 of this embodiment implements the principle and technical effect of implementing resource management by using the above modules, which are the same as the implementation of the related method embodiment, and reference may be made to the description of the related method embodiment for details, which are not described herein again.
As shown in fig. 10, this embodiment further provides a content distribution network platform 1000, where the resource processing device 1001 may adopt the resource processing device of the embodiment shown in fig. 11 or fig. 12, and the resource management device 1002 may adopt the resource management device shown in fig. 13 or fig. 14. The resource processing apparatus 1001 and the resource management apparatus 1002 may implement the resource processing method or the resource management method according to the embodiments shown in fig. 1 to fig. 10 to implement the related resource processing or resource management, and refer to the description of the related embodiments in detail, which is not described herein again.
In the technical scheme of the disclosure, the acquisition, storage, application and the like of the personal information of the related user all accord with the regulations of related laws and regulations, and do not violate the good customs of the public order.
The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.
FIG. 15 shows a schematic block diagram of an example electronic device 1500 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 15, the apparatus 1500 includes a computing unit 1501 which can perform various appropriate actions and processes in accordance with a computer program stored in a Read Only Memory (ROM) 1502 or a computer program loaded from a storage unit 1508 into a Random Access Memory (RAM) 1503. In the RAM 1503, various programs and data necessary for the operation of the device 1500 can also be stored. The calculation unit 1501, the ROM 1502, and the RAM 1503 are connected to each other by a bus 1504. An input/output (I/O) interface 1505 is also connected to bus 1504.
Various components in device 1500 connect to I/O interface 1505, including: an input unit 1506 such as a keyboard, a mouse, or the like; an output unit 1507 such as various types of displays, speakers, and the like; a storage unit 1508, such as a magnetic disk, optical disk, or the like; and a communication unit 1509 such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 1509 allows the device 1500 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The computing unit 1501 may be various general and/or special purpose processing components having processing and computing capabilities. Some examples of the computation unit 1501 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computation chips, various computation units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 1501 executes the respective methods and processes described above, such as the above-described methods of the present disclosure. For example, in some embodiments, the above-described methods of the present disclosure may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 1508. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 1500 via the ROM 1502 and/or the communication unit 1509. When the computer program is loaded into the RAM 1503 and executed by the computing unit 1501, one or more steps of the above-described method of the present disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 1501 may be configured in any other suitable manner (e.g., by means of firmware) to perform the above-described methods of the present disclosure.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.
The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims (50)

1. A method of resource processing, comprising:
responding to a resource request, and acquiring a resource corresponding to the resource request;
calculating the heat degree of the resource;
based on the heat degree of the resource, performing storage processing on the resource in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced.
2. The method of claim 1, wherein calculating the heat of the resource comprises:
and calculating the heat degree of the resource based on at least one of the latest accessed time of the resource, the accessed frequency within a preset time length and the space occupation amount of the resource.
3. The method according to claim 1 or 2, wherein obtaining the resource corresponding to the resource request comprises:
and acquiring the resource from a source station of a resource library.
4. The method of claim 3, wherein performing storage processing on the resource within a storage framework comprising a first level of storage space, a second level of storage space, and a third level of storage space based on a heat of the resource comprises:
storing the resource into a first-level storage space;
and simultaneously, storing the resources into the second-level storage space or the third-level storage space according to a preset storage strategy.
5. The method of claim 4, wherein storing the resource in the second level storage space or the third level storage space according to a preset storage policy comprises:
and if the space occupation amount of the resources is smaller than a preset threshold value, storing the resources into the second-level storage space.
6. The method of claim 5, wherein storing the resource in the second level storage space or the third level storage space according to a preset storage policy further comprises:
if the heat degree of the resource reaches a preset heat degree threshold value, storing the resource into the second-level storage space; or alternatively
And if the heat degree of the resource does not reach the preset heat degree threshold value, storing the resource into the third-level storage space.
7. The method of any of claims 4-6, wherein storing the resource into a first level of storage space comprises:
and storing the resources in the first-level storage space in a mode of applying for a storage space to a resource management device.
8. The method of any of claims 4-6, wherein storing the resource in the second level of storage space comprises:
storing the resource into a first resource block of the second level storage space; the first resource block is applied to a resource management device in advance;
before storing the resource into the first resource block of the second level storage space, the method further comprises:
detecting whether idle resource blocks exist in the second-level storage space;
and if the resource blocks do not exist, applying for the resource blocks in the second-level storage space from the resource management device.
9. The method of any of claims 4-6, wherein performing storage processing on the resource within a storage framework comprising a first level storage space, a second level storage space, and a third level storage space based on a heat of the resource, further comprises:
based on the heat degree of the resource, inserting the identification information of the resource into a corresponding resource heat degree queue; the second-level storage space corresponds to at least two resource heat queues; each resource hot queue comprises resource identification information of different hot intervals.
10. The method of any of claims 4-6, wherein storing the resource into the third level of storage space comprises:
storing the resource into a second resource block of the third level of storage space; the second resource block is applied to a resource management device in advance;
before storing the resource into the second resource block of the third level of storage space, the method further comprises:
detecting whether idle resource blocks exist in the third-stage storage space or not;
and if the resource blocks do not exist, applying for the resource blocks in the third-level storage space from the resource management device.
11. The method according to claim 1 or 2, wherein obtaining the resource corresponding to the resource request comprises:
and acquiring the resource from the third-level storage space.
12. The method of claim 11, wherein performing storage processing on the resource within a storage framework comprising a first level of storage space, a second level of storage space, and a third level of storage space based on a heat of the resource comprises:
detecting whether the heat degree of the resource reaches a preset heat degree threshold value or not based on the heat degree of the resource;
if so, storing the resource to the second-level storage space;
and inserting the identification information of the resource into a corresponding resource heat queue based on the heat of the resource.
13. The method according to claim 1 or 2, wherein obtaining the resource corresponding to the resource request comprises:
and acquiring the resource from the second-level storage space.
14. The method of claim 12, wherein performing storage processing on the resource within a storage framework comprising a first level storage space, a second level storage space, and a third level storage space based on the heat of the resource comprises:
and updating the identification information of the resource from the first resource heat queue to a second resource heat queue based on the heat of the resource, wherein the heat of the resource in the second resource heat queue is higher than that of the resource in the first resource heat queue.
15. The method according to claim 1 or 2, wherein the method further comprises:
receiving a resource elimination request sent by a resource management device; the resource elimination request carries an identifier of a storage space and an identifier of an elimination resource block;
and based on the identification of the storage space and the identification of the eliminated resource blocks, carrying out elimination treatment on the eliminated resource blocks.
16. The method of claim 15, wherein de-selecting the de-selected resource blocks based on the identification of the storage space and the identification of the de-selected resource blocks comprises:
if the eliminated resource block is determined to be in the second-level storage space according to the identifier of the storage space, detecting whether the resource in the eliminated resource block needs to be written back or not based on the at least two resource heat queues and the heat of the resource in the eliminated resource block;
if the write is needed, the information of the write-back resource is obtained;
releasing the resources in the eliminated resource blocks;
and based on the information of the write-back resource, the write-back resource is stored into the second-stage storage space again.
17. The method of claim 15, wherein de-selecting the de-selected resource blocks based on the identification of the storage space and the identification of the de-selected resource blocks comprises:
and if the eliminated resource blocks are determined to be in the third-level storage space according to the identification of the storage space, releasing the resources in the eliminated resource blocks.
18. A method of resource management, comprising:
receiving a resource application request sent by a first resource processing device; the resource application request carries a storage space identifier;
allocating a storage space for the first resource processing device in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space based on the resource application request; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced.
19. The method according to claim 18, wherein when the storage space identifier is an identifier of the first-level storage space, the resource application request further carries a space occupation amount of a resource to be stored; based on the resource application request, allocating a storage space for the resource processing device in a storage framework comprising a first-level storage space, a second-level storage space and a third-level storage space, comprising:
and allocating a storage space for the resource to be stored by the first resource processing device in the first-level storage space based on the space occupation amount of the resource.
20. The method of claim 19, wherein prior to allocating storage space for the resource within the first level of storage space based on the space occupancy of the resource, further comprising:
if the residual storage space in the first-stage storage space is smaller than the space occupation of the resources, acquiring identifiers of eliminated resources based on a preset first elimination strategy;
and sending a first resource elimination request to a first resource processing device corresponding to the eliminated resource identifier, wherein the first resource elimination request carries the eliminated resource identifier, so that the first resource processing device can release the storage space of the eliminated resource.
21. The method of claim 18, wherein allocating storage space for the resource processing device within a storage framework comprising a first level storage space, a second level storage space, and a third level storage space based on the resource application request when the storage space identification is an identification of the second level storage space or the third level storage space comprises:
and allocating resource blocks for the resources in the second-level storage space or the third-level storage space.
22. The method of claim 21, wherein prior to allocating resource blocks for the resources within the second level storage space or the third level storage space, the method further comprises:
if the second-level storage space or the third-level storage space does not have allocable resource blocks, acquiring elimination parameters of the resource blocks in the second-level storage space or the third-level storage space;
acquiring an identification of eliminated resource blocks based on a preset second elimination strategy and elimination parameters of each resource block;
and sending a second resource elimination request to a second resource processing device corresponding to the eliminated resource block identifier, wherein the second resource elimination request carries the eliminated resource block identifier, so that the second resource processing device can release the storage space of the eliminated resource block.
23. The method of claim 22, wherein the elimination parameter for each of the resource blocks in the second level of storage space is obtained based on a current time and a creation time of each of the resource blocks; and/or the elimination parameter of each resource block in the third-level storage space is obtained based on the current moment and the last accessed moment of the last accessed resource in each resource block.
24. A resource processing apparatus comprising:
the acquisition module is used for responding to a resource request and acquiring a resource corresponding to the resource request;
the computing module is used for computing the heat degree of the resource;
the processing module is used for performing storage processing on the resources in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space based on the heat degree of the resources; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced.
25. The apparatus of claim 24, wherein the means for calculating is configured to:
and calculating the heat degree of the resource based on at least one of the latest accessed time of the resource, the accessed frequency within a preset time length and the space occupation amount of the resource.
26. The apparatus of claim 24 or 25, wherein the obtaining means is configured to obtain the data from the wireless device
And acquiring the resource from a source station of a resource library.
27. The apparatus of claim 26, wherein the processing module is configured to:
storing the resource into a first-level storage space;
and simultaneously, storing the resources into the second-level storage space or the third-level storage space according to a preset storage strategy.
28. The apparatus of claim 27, wherein the processing module is configured to:
and if the space occupation amount of the resources is smaller than a preset threshold value, storing the resources into the second-level storage space.
29. The apparatus of claim 28, wherein the processing module is further configured to:
if the heat degree of the resource reaches a preset heat degree threshold value, storing the resource into the second-level storage space; or
And if the heat degree of the resource does not reach the preset heat degree threshold value, storing the resource into the third-level storage space.
30. The apparatus of any one of claims 27-29, wherein the processing module is configured to:
and storing the resources in the first-level storage space in a mode of applying for a storage space to a resource management device.
31. The apparatus of any one of claims 27-29, wherein the processing module is configured to:
storing the resource into a first resource block of the second level storage space; the first resource block is applied to a resource management device in advance;
before storing the resource into the first resource block of the second level storage space, the method further comprises:
detecting whether idle resource blocks exist in the second-level storage space;
and if the resource blocks do not exist, applying for the resource blocks in the second-level storage space from the resource management device.
32. The apparatus of any of claims 27-29, wherein the processing module is further configured to:
based on the heat degree of the resource, inserting the identification information of the resource into a corresponding resource heat degree queue; the second-level storage space corresponds to at least two resource heat queues; each resource hot degree queue comprises resource identification information of different hot degree intervals.
33. The apparatus of any of claims 27-29, wherein the processing module is further configured to:
storing the resource into a second resource block of the third level of storage space; the second resource block is applied to a resource management device in advance;
the processing module is further configured to:
detecting whether idle resource blocks exist in the third-level storage space;
and if the resource blocks do not exist, applying for the resource blocks in the third-level storage space from the resource management device.
34. The apparatus of claim 24 or 25, wherein the obtaining means is configured to:
and acquiring the resource from the third-level storage space.
35. The apparatus of claim 34, wherein the processing module is configured to:
detecting whether the heat degree of the resource reaches a preset heat degree threshold value or not based on the heat degree of the resource;
if so, storing the resource to the second-level storage space;
and inserting the identification information of the resource into a corresponding resource heat queue based on the heat of the resource.
36. The apparatus of claim 24 or 25, wherein the obtaining means is configured to:
and acquiring the resource from the second-level storage space.
37. The apparatus of claim 35, wherein the processing module is configured to:
and updating the identification information of the resource from the first resource heat queue to a second resource heat queue based on the heat of the resource, wherein the heat of the resource in the second resource heat queue is higher than that of the resource in the first resource heat queue.
38. The apparatus of claim 24 or 25, wherein the apparatus further comprises:
the receiving module is used for receiving a resource elimination request sent by the resource management device; the resource elimination request carries an identifier of a storage space and an identifier of an elimination resource block;
and the processing module is also used for carrying out elimination processing on the eliminated resource blocks based on the identification of the storage space and the identification of the eliminated resource blocks.
39. The apparatus of claim 38, wherein the processing module is configured to:
if the eliminated resource block is determined to be in the second-level storage space according to the identifier of the storage space, detecting whether the resource in the eliminated resource block needs to be written back or not based on the at least two resource heat queues and the heat of the resource in the eliminated resource block;
if the write is needed, the information of the write-back resource is obtained;
releasing the resources in the eliminated resource blocks;
and based on the information of the write-back resource, the write-back resource is stored into the second-stage storage space again.
40. The apparatus of claim 38, the processing module to:
and if the eliminated resource blocks are determined to be in the third-level storage space according to the identification of the storage space, releasing the resources in the eliminated resource blocks.
41. A resource management apparatus, comprising:
the receiving module is used for receiving a resource application request sent by a first resource processing device; the resource application request carries a storage space identifier;
the allocation module is used for allocating a storage space for the first resource processing device in a storage frame comprising a first-level storage space, a second-level storage space and a third-level storage space based on the resource application request; the storage spaces in the first-stage storage space, the second-stage storage space and the third-stage storage space are gradually increased, and the resource reading speed is gradually reduced.
42. The apparatus of claim 41, wherein the means for assigning is configured to: when the storage space identifier is the identifier of the first-level storage space, the resource application request also carries the space occupation of the resource to be stored, and based on the space occupation of the resource, the first-level storage space is internally provided with the resource allocation storage space to be stored by the first resource processing device.
43. The apparatus of claim 43, wherein the apparatus further comprises:
the acquisition module is used for acquiring an identifier of the eliminated resource based on a preset first elimination strategy if the residual storage space in the first-stage storage space is smaller than the space occupation amount of the resource;
and the sending module is used for sending a first resource elimination request to a first resource processing device corresponding to the eliminated resource identifier, wherein the first resource elimination request carries the eliminated resource identifier so that the first resource processing device can release the storage space of the eliminated resource.
44. The apparatus of claim 43, wherein the assignment module is configured to: and when the storage space identifier is the identifier of the second-level storage space or the third-level storage space, allocating resource blocks for the resources in the second-level storage space or the third-level storage space.
45. The apparatus of claim 44, wherein the means for obtaining is further configured to:
if the second-level storage space or the third-level storage space does not have allocable resource blocks, acquiring elimination parameters of the resource blocks in the second-level storage space or the third-level storage space;
acquiring the identification of the eliminated resource blocks based on a preset second elimination strategy and elimination parameters of each resource block;
the sending module is further configured to send a second resource elimination request to a second resource processing device corresponding to the identifier of the eliminated resource block, where the second resource elimination request carries the identifier of the eliminated resource block, so that the second resource processing device releases the storage space of the eliminated resource block.
46. The apparatus of claim 45, wherein the elimination parameters for each of the resource blocks in the second level of storage space are obtained based on a current time and a time of creation of each of the resource blocks; and/or the elimination parameter of each resource block in the third-level storage space is acquired based on the current time and the last accessed time of the last accessed resource in each resource block.
47. A content distribution network platform comprising at least one resource handling means according to any of claims 24 to 40 and a resource management means according to any of claims 41 to 46, and a storage framework comprising a primary storage space, a secondary storage space and a tertiary storage space.
48. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-17 or 18-23.
49. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any of claims 1-17 or 18-23.
50. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-17 or 18-23.
CN202210916127.2A 2022-08-01 2022-08-01 Resource processing method, device, platform, equipment and storage medium Pending CN115495012A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210916127.2A CN115495012A (en) 2022-08-01 2022-08-01 Resource processing method, device, platform, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210916127.2A CN115495012A (en) 2022-08-01 2022-08-01 Resource processing method, device, platform, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN115495012A true CN115495012A (en) 2022-12-20

Family

ID=84466104

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210916127.2A Pending CN115495012A (en) 2022-08-01 2022-08-01 Resource processing method, device, platform, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115495012A (en)

Similar Documents

Publication Publication Date Title
JP7127010B2 (en) Resource allocation methods, apparatus, electronic equipment, computer readable media and computer programs
CN109947668B (en) Method and device for storing data
CN105205014B (en) A kind of date storage method and device
CN111786895A (en) Method and apparatus for dynamic global current limiting
CN110598138A (en) Cache-based processing method and device
CN107729570B (en) Data migration method and device for server
CN111858067B (en) Data processing method and device
CN111737168A (en) Cache system, cache processing method, device, equipment and medium
CN114155026A (en) Resource allocation method, device, server and storage medium
CN113760982A (en) Data processing method and device
CN113778644A (en) Task processing method, device, equipment and storage medium
CN108205559B (en) Data management method and equipment thereof
CN112615795A (en) Flow control method and device, electronic equipment, storage medium and product
CN112527509A (en) Resource allocation method and device, electronic equipment and storage medium
CN116633879A (en) Data packet receiving method, device, equipment and storage medium
CN116594563A (en) Distributed storage capacity expansion method and device, electronic equipment and storage medium
CN115495012A (en) Resource processing method, device, platform, equipment and storage medium
CN115858147A (en) Cost modeling method and device
CN115438007A (en) File merging method and device, electronic equipment and medium
CN114564149A (en) Data storage method, device, equipment and storage medium
CN113778645A (en) Task scheduling method, device and equipment based on edge calculation and storage medium
CN108984431B (en) Method and apparatus for flushing stale caches
CN111090633A (en) Small file aggregation method, device and equipment of distributed file system
CN115391042B (en) Resource allocation method and device, electronic equipment and storage medium
CN115442432B (en) Control method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination