CN115190181B - Resource management method and system based on cloud management - Google Patents

Resource management method and system based on cloud management Download PDF

Info

Publication number
CN115190181B
CN115190181B CN202211087646.9A CN202211087646A CN115190181B CN 115190181 B CN115190181 B CN 115190181B CN 202211087646 A CN202211087646 A CN 202211087646A CN 115190181 B CN115190181 B CN 115190181B
Authority
CN
China
Prior art keywords
metadata
data
node set
node
screening
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211087646.9A
Other languages
Chinese (zh)
Other versions
CN115190181A (en
Inventor
刘凤屿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ruizhi Technology Group Co ltd
Original Assignee
Ruizhi Technology Group Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ruizhi Technology Group Co ltd filed Critical Ruizhi Technology Group Co ltd
Priority to CN202211087646.9A priority Critical patent/CN115190181B/en
Publication of CN115190181A publication Critical patent/CN115190181A/en
Application granted granted Critical
Publication of CN115190181B publication Critical patent/CN115190181B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network

Abstract

The application discloses a resource management method and a system thereof based on cloud management, wherein the resource management method based on the cloud management specifically comprises the following steps: acquiring a resource management request; setting the priority of the resources according to the resource management request; in response to the completion of the setting of the priority of the resources, screening the resources; and managing the screened resources. According to the method and the system, resources in the data grid can be managed in a grading mode, different management strategies are provided, so that the resource management is more reasonable, meanwhile, important metadata and data copies are widely and reasonably distributed on different nodes through specific management strategies, the access delay can be effectively reduced, and the data access efficiency is greatly improved.

Description

Resource management method and system based on cloud management
Technical Field
The present application relates to the field of data processing, and in particular, to a resource management method and system based on cloud management.
Background
With the continuous improvement of computer performance and the rapid development of network communication technology, the application requirements are increasingly developing towards high performance, large scale, diversity and multiple functions, and geographically distributed and heterogeneous high performance resources, storage resources, data resources and other special resources are required to be connected through a high-speed network to realize high-performance joint computation and jointly complete important application problems, namely the grid technology. The network technology is developed in the directions of data grids, information grids, knowledge grids and the like, and the final aim of the data grids is to establish the architecture and environment of integrated storage, management, access, transmission and service of mass data in a heterogeneous distribution environment so as to better solve the problem that the mass data is difficult to organize and process. The cloud technology used in combination with the concept of "data grid" provides great help for resource management of the grid, but when the cloud technology is used for managing resources of the data grid, the resources of the data grid are often not uniformly managed in consideration of whether the resources are important, and the unimportant resources are also subjected to excessive management, so that management of management resources is wasted.
Therefore, how to provide a method for reasonably managing resources in a grid is an urgent problem to be solved by those skilled in the art.
Disclosure of Invention
The application provides a resource management method based on cloud management, which specifically comprises the following steps: acquiring a resource management request; setting the priority of the resources according to the resource management request; in response to the completion of the setting of the priority of the resources, screening the resources; and managing the screened resources.
As above, among other things, resource management requests include management of metadata and/or management of copies of data.
As above, the setting of the priority of the resource according to the resource management request specifically includes the following sub-steps: determining the access amount of the metadata and the data copy; if the access amount of the metadata is larger than that of the data copy, setting the priority of the metadata to be higher than that of the data copy; and if the access amount of the metadata is less than that of the data copy, setting the priority of the data copy to be higher than that of the metadata.
As above, among others, further comprising: if the access amount of the metadata is equal to the access amount of the data copy, determining the total cost required for accessing the metadata and the data copy; and setting the priority of the metadata and the data copy according to the total cost required by the metadata and the data copy.
As above, the screening of the resource in response to the completion of the setting of the priority of the resource specifically includes: screening the data copies; and screening the metadata.
The screening of the data copies includes dividing the plurality of nodes storing the data copies into a plurality of first node sets, where each first node set includes a plurality of nodes for storing the data copies, and a node in each first node set includes one or more data copies.
A resource management system based on cloud management specifically comprises a request acquisition unit, a resource priority setting unit, a resource screening unit and a resource management unit; a request acquisition unit, configured to acquire a resource management request; a resource priority setting unit, configured to set a priority of a resource according to the resource management request; the resource screening unit is used for screening resources; and the resource management unit is used for managing the screened resources.
The method for setting the resource priority comprises the following steps of setting a resource priority by a resource priority setting unit, wherein the resource priority setting unit comprises an access quantity determining module and a cost determining module; the access quantity determining module is used for determining the access quantity of the metadata and the data copy; and the cost determining module is used for determining the total cost required for accessing the metadata and the data copy if the access amount of the metadata is equal to the access amount of the data copy.
The resource screening unit specifically includes a data copy screening module and a metadata screening module; the data copy screening module is used for screening the data copies; the metadata screening module is used for screening the metadata.
The data copy screening module comprises a first node set partitioning module, configured to partition a node storing a data copy into a first node set.
The application has the following beneficial effects:
(1) The resource management method and the resource management system can carry out hierarchical management on the resources in the data grid, and provide different management strategies, so that the resource management is more reasonable.
(2) In the resource management process, through a primary management strategy, metadata and data copies which are considered as important are widely and reasonably distributed on different nodes, so that access delay can be effectively reduced, and the data access efficiency is greatly improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings.
Fig. 1 is a flowchart of a resource management method based on cloud management according to an embodiment of the present application;
fig. 2 is an internal structural diagram of a resource management system based on cloud management according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application are clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The cloud management is a 'cloud technology' in the prior art, resources in a data grid are comprehensively considered by utilizing the cloud technology, the resources in the grid are reasonably managed, and the response rate of the resources can be greatly improved.
Example one
As shown in fig. 1, a resource management method based on a cloud management platform provided by the present application specifically includes the following steps:
step S110: and acquiring a resource management request.
In a data grid, in order to improve data reliability, a common method is to place metadata and data copies in different nodes, and storing a plurality of data copies in a plurality of independent nodes will effectively improve the chance of obtaining at least one data copy, and storing a plurality of metadata in a plurality of independent nodes will facilitate reading the metadata, so managing the metadata and the data copies is a goal of improving the service quality of a data grid system. The present embodiment mainly manages metadata and data copies in the metadata grid.
And when the metadata and/or the data copy carried in the data grid exceed a specified threshold, triggering a resource management mechanism to acquire a resource management request.
Step S120: and setting the priority of the resources according to the resource management request.
Specifically, in the present embodiment, since the data and the data duplicates are mainly managed, theoretically, since the data duplicates are data obtained by copying metadata, the priority of the metadata is greater than that of the data duplicates, but in order to manage the metadata and the data duplicates more strictly, the metadata and the data duplicates are studied more carefully, so as to determine the priority between the metadata and the data duplicates. Wherein the step S120 specifically includes the following substeps:
step S1201: the amount of access to the metadata and the data copy is determined.
Wherein if the access amount of the metadata is greater than the access amount (i.e., the number of accesses) of the data copy, the priority of the metadata is set to be higher than the priority of the data copy, and step S130 is performed. If the access amount of the metadata is smaller than that of the data copy, the priority of the data copy is set to be higher than that of the metadata, and step S130 is performed.
If the access amount of the metadata is equal to the access amount of the data copy, step S1202 is performed.
Step S1202: the total cost required to access the metadata and the data copy is determined.
In particular, the total cost required to access the metadata therein
Figure 396622DEST_PATH_IMAGE001
The concrete expression is as follows:
Figure 676294DEST_PATH_IMAGE002
wherein
Figure 148864DEST_PATH_IMAGE003
Representing the number of times a user accesses the node i that stores the metadata, m representing the total number of nodes in the data grid that store the metadata,
Figure 996734DEST_PATH_IMAGE004
indicating the routing cost required for node i to reach the node closest to itself,
Figure 656385DEST_PATH_IMAGE005
representing coefficients of metadata stored in the data grid, the coefficients being predominantly derived from nodes storing metadataIt is decided that, in general, the more nodes that store metadata, the smaller the coefficient, and vice versa, where
Figure 615114DEST_PATH_IMAGE005
The specific numerical value of (a) is set by a worker in advance, and is not limited herein.
Total cost details required to access a copy of data
Figure 535666DEST_PATH_IMAGE006
Expressed as:
Figure 820017DEST_PATH_IMAGE007
wherein
Figure 966964DEST_PATH_IMAGE008
Representing the number of times a user accesses the node j to store a copy of the data, n representing the total number of nodes in the data grid that store copies of the data,
Figure 729384DEST_PATH_IMAGE009
indicating the required routing cost of node j to the node closest to itself,
Figure 645387DEST_PATH_IMAGE010
a coefficient representing the metadata stored in the data grid, the coefficient being determined primarily by the nodes storing the metadata, the more nodes storing the metadata the smaller the coefficient, and the larger the coefficient is, wherein
Figure 835060DEST_PATH_IMAGE010
The specific numerical value of (a) is set by a worker in advance, and is not limited herein.
Step S1203: and setting the priority of the metadata and the data copy according to the total cost required by the metadata and the data copy.
If the total cost
Figure 95402DEST_PATH_IMAGE011
If the user access amount is the same, it indicates that it is more costly for the system to access the metadata, for example, more communication resources are needed, in which case it is easier to access the data copy, so the priority of the data copy is set higher than that of the metadata, if the total cost is the same
Figure 395934DEST_PATH_IMAGE012
And if the user access amount is the same, setting the priority of the metadata to be higher than that of the data copy.
Step S130: and responding to the completion of the setting of the priority of the resources, and screening the resources.
In the data grid, both metadata and data copies can be represented by data sequences, one data sequence represents one metadata or data copy, the metadata and data copies stored in the data grid only contain one metadata or data copy, and if all data copies are managed, management resources may be wasted, so that after the data copies and the metadata are prioritized, the metadata and the data copies need to be screened respectively.
The step S130 specifically includes the following sub-steps:
step S1301: and screening the data copies.
Step S1302: and screening the metadata.
Wherein step S1301 and step S1302 may be performed simultaneously.
The step S1301 specifically includes the following substeps:
step S13011: and dividing the nodes for storing the data copies into a first node set.
The data copies can be stored in a plurality of nodes, and some nodes are not provided with the data copies, so that time cost is wasted if all nodes in the data grid are traversed to screen the data copies, and therefore the node set is divided firstly.
The method specifically divides a plurality of nodes for storing data copies into a plurality of first node sets, each first node set comprises a plurality of nodes for storing data copies, and each node in each first node set comprises one or more data copies.
Step S13012: and screening the data copies according to the first node set.
Specifically, in the embodiment, first, a screening standard value of the data copy is calculated, and screening of the data copy is performed according to the size of the value.
Wherein the screening criterion value of the data copy in each first node set
Figure 697602DEST_PATH_IMAGE013
The concrete expression is as follows:
Figure 792597DEST_PATH_IMAGE014
wherein
Figure 914137DEST_PATH_IMAGE015
Representing the memory space occupied by the data copy v in the first node set, R representing the total storage space of the current first node set, N representing the number of times that all data copies in the current first node set are stored in the historical time period,
Figure 142993DEST_PATH_IMAGE016
the number of times that the data copy v is stored in the historical time period is represented, and G represents the number of data copies stored in the node set.
If screening the standard value
Figure 33588DEST_PATH_IMAGE013
If the value is larger than the designated threshold value, the more important the data copies in the current node set are, the data copies in the current first node set are all reserved in the current first node set, and if the value is the screening standard value
Figure 565064DEST_PATH_IMAGE013
If the data copy number is less than the specified threshold value, the partial data copy in the first node set is considered not to beImportantly, the data copies in the first node set that are accessed less than a specified threshold number of times are considered insignificant copies and the current first node set is removed.
The screening of the data copies in each first set of nodes is performed according to the above method.
The step S1302 specifically includes the following sub-steps:
step S13021: and dividing the nodes storing the metadata into a second node set.
The nodes storing the metadata are divided into a plurality of second node sets, each second node set comprises a plurality of nodes used for storing the metadata, and each node can store one or more metadata.
Step S13022: and screening the metadata according to the second node set.
Wherein the filtering criterion value of the metadata in each second node set
Figure 908320DEST_PATH_IMAGE017
The concrete expression is as follows:
Figure 816234DEST_PATH_IMAGE018
wherein
Figure 390697DEST_PATH_IMAGE019
Representing the memory space occupied by the metadata u in the current second node set, E representing the total storage space of the current second node set, H representing the number of times that all metadata in the current second node set are stored in the historical time period,
Figure 420970DEST_PATH_IMAGE020
and D represents the number of times the metadata u is stored in the historical time period, and the quantity of the metadata stored in the current second node in a centralized manner.
If screening the standard value
Figure 346463DEST_PATH_IMAGE017
If the value is larger than the specified threshold value, the description shows thatThe metadata in the current second node set is important, the metadata in the current second node set is all reserved to the current second node set, and if a standard value is screened out
Figure 651542DEST_PATH_IMAGE017
And if the number of times of access to the second node set is less than the specified threshold, the metadata in the second node set is considered to be unimportant, and the metadata with the number of times of access to the second node set less than the specified threshold is considered to be unimportant data and the current second node set is removed.
The filtering of the metadata in each second set of nodes is performed according to the above method.
Step S140: and managing the screened resources.
The screened metadata and the screened data copy are managed in the following specific mode: and performing a primary management strategy on the metadata and the data copies which are screened and retained in the node set, and performing a secondary management strategy on the metadata and the data copies which are removed from the node set.
Specifically, the primary management policy is a distribution policy of performing a data copy by one node on data copies reserved in the plurality of first node sets, and a distribution policy of performing a metadata by one node on metadata reserved in the plurality of second node sets.
Specifically, in order to ensure that the metadata in each node set can be well stored later and that the metadata can respond to the subsequent operations that need to be performed on the metadata in time, the following operations are specifically performed in the primary management policy:
and checking the condition that each node in each first node set stores the data copies, performing data dispersion operation on the node storing a plurality of data copies in one node, and specifically migrating the other data copies except for one data copy stored in the node to other nodes of the data grid, so that only one data copy is stored in each node in the data grid.
And checking the condition that each node in each second node set stores the metadata, performing data dispersion operation on the nodes storing a plurality of metadata in one node, and specifically migrating the other metadata except for one metadata stored in the node to other nodes of the data grid, so that only one metadata is stored in each node.
Through a primary management strategy, metadata and data copies which are considered to be important are distributed on different nodes widely and reasonably, so that access delay can be effectively reduced, and the access efficiency of data is greatly improved.
The secondary management policy specifically performs the following operations:
and integrating a plurality of metadata and data copies which are removed in the first node set and the second node set, specifically integrating the removed metadata together, and integrating the removed data copies together.
And putting the integrated metadata into a plurality of nodes of the data grid, wherein each node stores a plurality of metadata. And putting the integrated multiple data copies into nodes of the data grid, wherein each node stores multiple data copies. Wherein the nodes into which the metadata and the data copies are placed are distinct nodes.
Through the secondary management strategy, a plurality of metadata which are not considered to be important can be distributed on different nodes respectively, occupation of the nodes is reduced, and integrated metadata and data copies can be responded.
Example two
As shown in fig. 2, the resource management system based on cloud management provided by the present application specifically includes a request obtaining unit 210, a resource priority setting unit 220, a resource screening unit 230, and a resource management unit 240.
Wherein the request obtaining unit 210 is configured to obtain the resource management request.
The resource priority setting unit 220 is connected to the request obtaining unit 210, and is configured to perform priority setting on resources according to the resource management request.
The resource priority setting unit 220 includes an access amount determining module and a cost determining module.
The access amount determining module is used for determining the access amount of the metadata and the data copy.
The cost determining module is connected with the access quantity determining module and is used for determining the total cost required for accessing the metadata and the data copy if the access quantity of the metadata is equal to the access quantity of the data copy.
The resource screening unit 230 is connected to the resource priority setting unit 220, and is configured to perform resource screening.
The resource screening unit 230 specifically includes a data copy screening module and a metadata screening module.
The data copy screening module is used for screening the data copies.
The data copy screening module comprises a first node set dividing module and a first screening module.
The first node set partitioning module is used for partitioning the nodes storing the data copies into a first node set.
The first screening module is connected with the first node set and used for screening the data copies according to the first node set.
The metadata screening module is used for screening the metadata.
The metadata screening module comprises a second node set partitioning module and a second screening module.
And the second node set partitioning module is used for partitioning the nodes storing the metadata into a second node set.
The second screening module is connected with the second node set partitioning module and is used for screening the metadata according to the second node set.
The resource management unit 240 is connected to the resource screening unit 230, and is configured to manage the screened resources.
The application has the following beneficial effects:
(1) The resource management method and the resource management system can carry out hierarchical management on the resources in the data grid, and provide different management strategies, so that the resource management is more reasonable.
(2) In the resource management process, the primary management strategy is adopted, so that metadata and data copies which are considered as important are widely and reasonably distributed on different nodes, the access delay can be effectively reduced, and the data access efficiency is greatly improved.
Although the present application has been described with reference to examples, which are intended to be illustrative only and not to be limiting of the application, changes, additions and/or deletions may be made to the embodiments without departing from the scope of the application.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (7)

1. A resource management method based on cloud management is characterized by comprising the following steps:
acquiring a resource management request;
setting the priority of the resources according to the resource management request;
in response to the completion of the setting of the priority of the resources, screening the resources;
managing the screened resources;
responding to the completion of the setting of the priority of the resources, and screening the resources, which specifically comprises the following steps:
dividing a plurality of nodes for storing data copies into a plurality of first node sets, wherein each first node set comprises a plurality of nodes for storing the data copies, and the nodes in each first node set comprise one or more data copies; dividing a plurality of nodes for storing metadata into a plurality of second node sets, wherein each second node set comprises a plurality of nodes for storing the metadata, and each node can store one or more metadata;
screening the data copies, namely firstly calculating a screening standard value of the data copies, and screening the data copies according to the value;
wherein the screening criterion value of the data copy in each first node set
Figure 69087DEST_PATH_IMAGE001
The concrete expression is as follows:
Figure 318934DEST_PATH_IMAGE002
wherein
Figure 149487DEST_PATH_IMAGE003
Representing the memory space occupied by the data copy v in the first node set, R representing the total storage space of the current first node set, N representing the number of times that all data copies in the current first node set are stored in the historical time period,
Figure 392249DEST_PATH_IMAGE004
representing the number of times that the data copy v is stored in the historical time period, and G representing the number of data copies stored in the node set;
if screening the standard value
Figure 54175DEST_PATH_IMAGE001
If the value is larger than the designated threshold value, all the data copies in the current first node set are reserved in the current first node set, and if the value is larger than the designated threshold value, the standard value is screened
Figure 724191DEST_PATH_IMAGE001
If the number of times of access to the first node set is less than the designated threshold value, removing the data copy which is accessed to the first node set less than the designated threshold value from the first node set;
screening the metadata, namely firstly calculating a screening standard value of the metadata, and screening the metadata according to the value;
wherein the filtering standard value of the metadata in each second node set
Figure 42039DEST_PATH_IMAGE005
The concrete expression is as follows:
Figure 567787DEST_PATH_IMAGE006
wherein
Figure 84219DEST_PATH_IMAGE007
Representing the memory space occupied by the metadata u in the current second node set, E representing the total storage space of the current second node set, H representing the number of times that all metadata in the current second node set are stored in the historical time period,
Figure 190715DEST_PATH_IMAGE008
representing the number of times that the metadata u are stored in the historical time period, and D representing the number of metadata which are stored in the current second node in a centralized manner;
if screening the standard value
Figure 792598DEST_PATH_IMAGE005
If the value is larger than the designated threshold value, all metadata in the current second node set are reserved in the current second node set, and if the value is larger than the designated threshold value, the standard value is screened
Figure 580425DEST_PATH_IMAGE005
If the number of times of access to the second node set is less than the specified threshold, removing the metadata of which the number of times of access to the second node set is less than the specified threshold from the second node set;
wherein managing the screened resources comprises:
performing a primary management strategy on the metadata and the data copies which are screened and retained in the node set;
performing a secondary management strategy on the metadata and the data copies removed from the node set;
the primary management policy comprises: checking the condition that each node in each first node set stores the data copies, and if the condition that a plurality of data copies are stored in one node exists, migrating the other data copies stored in the node except for one data copy to other nodes of the data grid; checking the condition that each node in each second node set stores metadata, and if the condition that a plurality of metadata are stored in one node exists, migrating the other metadata stored in the node except for one metadata to other nodes of the data grid;
the secondary management policy comprises: integrating a plurality of metadata which are removed from the second node set, and placing the integrated metadata into a plurality of nodes of a data grid, wherein each node stores a plurality of metadata; integrating a plurality of data copies which are removed in the first node set, placing the integrated data copies into nodes of a data grid, storing a plurality of data copies in each node, and placing the nodes into which metadata and the data copies are placed into different nodes.
2. The method for resource management based on cloud management according to claim 1, wherein the resource management request comprises management of metadata and/or management of a data copy.
3. The resource management method based on cloud management as claimed in claim 2, wherein the setting of the priority of the resource according to the resource management request specifically includes the following sub-steps:
determining the access amount of metadata and data copies;
if the access amount of the metadata is larger than that of the data copy, setting the priority of the metadata to be higher than that of the data copy;
and if the access amount of the metadata is smaller than that of the data copy, setting the priority of the data copy to be higher than that of the metadata.
4. The cloud management-based resource management method of claim 1,
further comprising:
if the access amount of the metadata is equal to the access amount of the data copy, determining the total cost required for accessing the metadata and the data copy;
and setting the priority of the metadata and the data copy according to the total cost required by the metadata and the data copy.
5. A resource management system based on cloud management is characterized by specifically comprising a request acquisition unit, a resource priority setting unit, a resource screening unit and a resource management unit;
a request acquisition unit, configured to acquire a resource management request;
a resource priority setting unit, configured to set a priority of the resource according to the resource management request;
the resource screening unit is used for screening resources;
the resource management unit is used for managing the screened resources;
the resource screening unit specifically comprises:
dividing a plurality of nodes for storing data copies into a plurality of first node sets, wherein each first node set comprises a plurality of nodes for storing the data copies, and the node in each first node set comprises one or more data copies; dividing a plurality of nodes for storing metadata into a plurality of second node sets, wherein each second node set comprises a plurality of nodes for storing the metadata, and each node can store one or more metadata;
screening the data copies, namely firstly calculating a screening standard value of the data copies, and screening the data copies according to the value;
wherein the screening criterion value of the data copy in each first node set
Figure 702096DEST_PATH_IMAGE001
The concrete expression is as follows:
Figure 979493DEST_PATH_IMAGE002
wherein
Figure 803093DEST_PATH_IMAGE003
Representing the memory occupied by the data copy v in the first node setSpace, R represents the total storage space of the current first node set, N represents the number of times all data copies in the current first node set have been deposited in the historical time period,
Figure 394611DEST_PATH_IMAGE004
representing the number of times that the data copy v is stored in the historical time period, and G representing the number of data copies stored in the node set;
screening the standard value
Figure 885636DEST_PATH_IMAGE001
If the value is larger than the designated threshold value, all the data copies in the current first node set are reserved in the current first node set, and if the value is larger than the designated threshold value, the standard value is screened
Figure 819088DEST_PATH_IMAGE001
If the number of times of access to the first node set is less than the specified threshold value, removing the data copy with the number of times of access less than the specified threshold value from the first node set;
screening the metadata, namely firstly calculating a screening standard value of the metadata, and screening the metadata according to the value;
wherein the filtering standard value of the metadata in each second node set
Figure 598825DEST_PATH_IMAGE005
The concrete expression is as follows:
Figure 525192DEST_PATH_IMAGE006
wherein
Figure 870723DEST_PATH_IMAGE007
Representing the memory space occupied by the metadata u in the current second node set, E representing the total storage space of the current second node set, H representing the number of times that all metadata in the current second node set are stored in the historical time period,
Figure 489923DEST_PATH_IMAGE008
representing the number of times that the metadata u are stored in the historical time period, and D representing the number of metadata which are stored in the current second node in a centralized manner;
if screening the standard value
Figure 491377DEST_PATH_IMAGE005
If the value is larger than the designated threshold value, all metadata in the current second node set are reserved in the current second node set, and if the value is larger than the designated threshold value, the standard value is screened
Figure 972168DEST_PATH_IMAGE005
If the number of times of access to the second node set is less than the specified threshold, removing the metadata of which the number of times of access to the second node set is less than the specified threshold from the second node set;
the resource management unit is used for managing the screened resources and comprises the following steps:
performing a primary management strategy on the metadata and the data copies which are screened and retained in the node set;
performing a secondary management strategy on the metadata and the data copies removed from the node set;
the primary management policy comprises: checking the condition that each node in each first node set stores the data copies, and if the condition that a plurality of data copies are stored in one node exists, migrating the other data copies stored in the node except for one data copy to other nodes of the data grid; checking the condition that each node in each second node set stores metadata, and if the condition that a plurality of metadata are stored in one node exists, migrating the other metadata stored in the node except for one metadata to other nodes of the data grid;
the secondary management policy comprises: integrating a plurality of metadata which are removed in the second node set, and placing the integrated metadata into a plurality of nodes of a data grid, wherein each node stores a plurality of metadata; integrating a plurality of data copies which are removed in the first node set, placing the integrated data copies into nodes of a data grid, storing a plurality of data copies in each node, and placing the nodes into which metadata and the data copies are placed into different nodes.
6. The cloud management-based resource management system of claim 5, wherein the resource priority setting unit includes an access amount determining module, a cost determining module;
the access quantity determining module is used for determining the access quantity of the metadata and the data copy;
and the cost determination module is used for determining the total cost required for accessing the metadata and the data copy if the access amount of the metadata is equal to the access amount of the data copy.
7. The cloud management-based resource management system of claim 5, wherein the resource filtering unit specifically comprises a data copy filtering module and a metadata filtering module;
the data copy screening module is used for screening the data copy;
the metadata screening module is used for screening the metadata.
CN202211087646.9A 2022-09-07 2022-09-07 Resource management method and system based on cloud management Active CN115190181B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211087646.9A CN115190181B (en) 2022-09-07 2022-09-07 Resource management method and system based on cloud management

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211087646.9A CN115190181B (en) 2022-09-07 2022-09-07 Resource management method and system based on cloud management

Publications (2)

Publication Number Publication Date
CN115190181A CN115190181A (en) 2022-10-14
CN115190181B true CN115190181B (en) 2023-02-17

Family

ID=83522302

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211087646.9A Active CN115190181B (en) 2022-09-07 2022-09-07 Resource management method and system based on cloud management

Country Status (1)

Country Link
CN (1) CN115190181B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101504663A (en) * 2009-03-17 2009-08-12 北京大学 Swarm intelligence based spatial data copy self-adapting distribution method
CN101753625A (en) * 2009-12-28 2010-06-23 北京理工大学 Method for deployment of copy service and copy establishment in peer-to-peer network environment
CN101800768A (en) * 2010-01-19 2010-08-11 南京邮电大学 Gridding data transcription generation method based on storage alliance subset partition
CN104142953A (en) * 2013-05-10 2014-11-12 中国电信股份有限公司 Metadata dynamic management method and system
WO2015145454A1 (en) * 2014-03-28 2015-10-01 Hewlett-Packard Development Company, L.P. Data file hoarding
CN105574153A (en) * 2015-12-16 2016-05-11 南京信息工程大学 Transcript placement method based on file heat analysis and K-means
CN110232049A (en) * 2019-06-12 2019-09-13 腾讯科技(深圳)有限公司 A kind of metadata cache management method and device
CN112947860A (en) * 2021-03-03 2021-06-11 成都信息工程大学 Hierarchical storage and scheduling method of distributed data copies

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11513708B2 (en) * 2020-08-25 2022-11-29 Commvault Systems, Inc. Optimized deduplication based on backup frequency in a distributed data storage system

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101504663A (en) * 2009-03-17 2009-08-12 北京大学 Swarm intelligence based spatial data copy self-adapting distribution method
CN101753625A (en) * 2009-12-28 2010-06-23 北京理工大学 Method for deployment of copy service and copy establishment in peer-to-peer network environment
CN101800768A (en) * 2010-01-19 2010-08-11 南京邮电大学 Gridding data transcription generation method based on storage alliance subset partition
CN104142953A (en) * 2013-05-10 2014-11-12 中国电信股份有限公司 Metadata dynamic management method and system
WO2015145454A1 (en) * 2014-03-28 2015-10-01 Hewlett-Packard Development Company, L.P. Data file hoarding
CN105574153A (en) * 2015-12-16 2016-05-11 南京信息工程大学 Transcript placement method based on file heat analysis and K-means
CN110232049A (en) * 2019-06-12 2019-09-13 腾讯科技(深圳)有限公司 A kind of metadata cache management method and device
CN112947860A (en) * 2021-03-03 2021-06-11 成都信息工程大学 Hierarchical storage and scheduling method of distributed data copies

Also Published As

Publication number Publication date
CN115190181A (en) 2022-10-14

Similar Documents

Publication Publication Date Title
CN103106152B (en) Based on the data dispatching method of level storage medium
CN104025057B (en) Collaborative storage management
US10929341B2 (en) Iterative object scanning for information lifecycle management
CN104615606A (en) Hadoop distributed file system and management method thereof
CN103077197A (en) Data storing method and device
CN107766159A (en) A kind of metadata management method, device and computer-readable recording medium
CN111381928B (en) Virtual machine migration method, cloud computing management platform and storage medium
CN111522636A (en) Application container adjusting method, application container adjusting system, computer readable medium and terminal device
US20130054930A1 (en) Method, system and program product for storing downloadable content on a plurality of enterprise storage system (ess) cells
CN108415962A (en) A kind of cloud storage system
CN111399761A (en) Storage resource allocation method, device and equipment and storage medium
CN107453948A (en) The storage method and system of a kind of network measurement data
CN115190181B (en) Resource management method and system based on cloud management
CN112711564B (en) Merging processing method and related equipment
CN113177033A (en) Log data storage method and device, electronic equipment and medium
CN109298949B (en) Resource scheduling system of distributed file system
CN115604294A (en) Method and device for managing storage resources
CN115878625A (en) Data processing method and device and electronic equipment
CN114666334A (en) Node management method and system
CN111722783B (en) Data storage method and device
CN112883239A (en) Resource allocation method and device, computer equipment and storage medium
CN114385081B (en) Disk protection method for kafka cluster and related equipment
CN117041258B (en) BIM engineering information management system
WO2023226257A1 (en) Method for determining optimal computing and storage path of computing power network, and monitoring device
CN116414620A (en) Data backup method, system, device and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant