CN113360284A - Resource management method, device and equipment - Google Patents

Resource management method, device and equipment Download PDF

Info

Publication number
CN113360284A
CN113360284A CN202110626971.7A CN202110626971A CN113360284A CN 113360284 A CN113360284 A CN 113360284A CN 202110626971 A CN202110626971 A CN 202110626971A CN 113360284 A CN113360284 A CN 113360284A
Authority
CN
China
Prior art keywords
resource
tag
information
resources
label
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110626971.7A
Other languages
Chinese (zh)
Inventor
黄山
尹强
王和平
刘有
杨峙岳
邸帅
卢道和
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
WeBank Co Ltd
Original Assignee
WeBank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by WeBank Co Ltd filed Critical WeBank Co Ltd
Priority to CN202110626971.7A priority Critical patent/CN113360284A/en
Publication of CN113360284A publication Critical patent/CN113360284A/en
Priority to PCT/CN2021/132313 priority patent/WO2022252510A1/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061Partitioning or combining of resources

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention discloses a resource management method, a device and equipment, wherein the method comprises the following steps: receiving a resource application request of a resource user, and matching each piece of first tag information in a first tag group carried by the resource application request with tag information associated with each preset resource to obtain a first matching resource successfully matched with the first tag group; detecting whether the number of usable resources of each first matching resource is not less than the number of application resources in the resource application request; if yes, determining resource application of the resource user, and locking the available resource quantity of each first matched resource according to the applied resource quantity. The invention identifies and divides resources through the label information, thereby improving the flexibility of resource allocation.

Description

Resource management method, device and equipment
Technical Field
The invention relates to the technical field of big data, in particular to a resource management method, a resource management device and resource management equipment.
Background
The yann (yetanotherresourceremotator) is a resource management system, and is responsible for the unified management and allocation of all resources in a cluster through one resource manager. Specifically, the resources in the cluster are divided according to the dimension of the queue, which queue each node belongs to is specified in advance, a resource user (user) can specify which queue the resource needs to use when applying for the resource, and if the number of the resources currently applied by the user does not exceed the limit, the application can be successful. The resource management method can only fix the granularity of management and control in advance, such as the dimensions of queues, users and the like, and cannot support dynamic adjustment and insertion of other granularities according to actual requirements, so that after a resource allocation strategy is formulated, resources cannot be segmented and summarized at any time, and the resource management mode is not flexible enough.
Disclosure of Invention
The invention mainly aims to provide a resource management method, a resource management device and resource management equipment, and aims to solve the technical problems that the existing resource management system cannot divide and sum resources at any time, and the resource management mode is not flexible enough.
In order to achieve the above object, the present invention provides a resource management method, including the following steps:
receiving a resource application request of a resource user, and respectively matching each piece of first tag information in a first tag group carried by the resource application request with each piece of preset tag information to obtain a target tag number associated with the preset tag information successfully matched with the first tag information;
taking resources associated with the target tag number in all preset resources as candidate resources, and detecting whether all matchable preset tag information associated with the candidate resources has successfully matched tag information in the first tag group;
if the resource application requests exist, the candidate resources are used as first matching resources which are successfully matched with the first label group, and whether the number of usable resources of each first matching resource is not less than the number of application resources in the resource application requests is detected;
if yes, determining resource application of the resource user, and locking the available resource quantity of each first matched resource according to the applied resource quantity.
Optionally, the step of matching each piece of first tag information in the first tag group carried by the resource application request with each piece of preset tag information respectively to obtain a target tag number associated with the preset tag information that is successfully matched with each piece of the first tag information correspondingly includes:
extracting each tag value in each piece of first tag information in a first tag group carried by the resource application request respectively;
searching each label value in a preset quick query table, and acquiring a label number associated with the searched label value in the preset quick query table;
and if the label number is found in a preset label table, taking the label number as a target label number, wherein one line of record in the preset label table comprises a piece of preset label information and a label number associated with the preset label information, and one line of record in the preset fast query table comprises a label value in the piece of preset label information and a label number associated with the preset label information.
Optionally, the step of locking the number of usable resources of each first matching resource according to the number of applied resources further includes:
adding a new resource, and taking the number of the applied resources as the number of the locking resources of the newly added resource;
and respectively associating the newly added resource with each piece of first label information in the first label group.
Optionally, after the step of locking the number of usable resources of each of the first matching resources according to the number of application resources, the method further includes:
receiving resource use report information of the resource user, and matching each piece of second tag information in a second tag group carried by the resource use information with tag information associated with each preset resource to obtain a second matching resource successfully matched with the second tag group, wherein the second tag group is the same as the first tag group;
detecting whether the number of the locked resources of each second matched resource is not less than the number of the actually used resources in the resource use reporting information;
if yes, the number of the locking resources in each second matching resource is converted into the number of the used resources according to the number of the actually used resources.
Optionally, after the step of converting the number of locked resources in each item of the second matching resources into the number of used resources according to the number of actually used resources, the method further includes:
receiving a resource recovery request of the resource user, and matching each piece of third tag information in a third tag group carried by the resource recovery request with tag information associated with each preset resource to obtain a third matching resource successfully matched with the third tag group, wherein the third tag group and the first tag information group;
detecting whether the used resource quantity of each third matching resource is not less than the resource quantity in the resource recycling request;
and if so, recycling the used resource quantity in each third matching resource according to the recycling resource quantity.
Optionally, if so, after the step of recovering the used resource quantity in each third matching resource according to the recovered resource quantity, the method further includes:
detecting whether the used resource quantity of a target resource in the third matching resource, which has an association relation with the user tag information, is zero;
and if so, deleting the target resource.
Optionally, the method further comprises:
adding a resource according to resource quantity information carried by a resource registration request, and associating each piece of fourth label information carried by the resource registration request with the added resource respectively, wherein the resource registration request is initiated by a resource provider or generated according to resource use limitation information set by a resource user, and when the resource registration request is generated according to the resource use limitation information, each piece of fourth label information is generated according to limitation conditions in the resource use limitation information, and the resource quantity information is generated according to the limitation resource quantity in the resource use limitation information.
Optionally, the method further comprises:
receiving a resource offline request, and matching each piece of fifth label information carried by the resource offline request with label information associated with each preset resource to search resources to be offline in each preset resource, wherein the resources to be offline are resources associated with label information which is the same as the corresponding label information of each piece of fifth label information;
and if the resource to be offline is found, deleting the resource quantity information of the resource to be offline and the associated information between the resource to be offline and the label information so as to offline the resource to be offline.
In order to achieve the above object, the present invention further provides a resource management device, including:
the receiving module is used for receiving a resource application request of a resource user, and matching each piece of first tag information in a first tag group carried by the resource application request with each piece of preset tag information respectively to obtain a target tag number associated with the preset tag information successfully matched with the first tag information;
the first detection module is used for taking resources related to the target tag number in all preset resources as candidate resources and detecting whether matched preset tag information related to the candidate resources exists in the first tag group or not;
a second detection module, configured to, if both of the candidate resources exist, use the candidate resource as a first matching resource successfully matched with the first tag group, and detect whether the number of available resources of each first matching resource is not less than the number of application resources in the resource application request;
and the determining module is used for determining the resource application of the resource user if the resource application is successful, and locking the available resource quantity of each first matched resource according to the applied resource quantity.
In order to achieve the above object, the present invention further provides a resource management device, including: a memory, a processor and a resource management program stored on the memory and executable on the processor, the resource management program when executed by the processor implementing the steps of the resource management method as described above.
Furthermore, to achieve the above object, the present invention further provides a computer readable storage medium, which stores a resource management program, and when the resource management program is executed by a processor, the resource management program implements the steps of the resource management method as described above.
Compared with the existing resource management method which only can lead the granularity of management and control to be fixed in advance and can not support dynamic adjustment and insertion of other granularities according to actual requirements, in the invention, through associating each resource with the tag information, and through carrying the tag group comprising one or more tag information in the resource application request when the resource user applies for the resource, the resource manager matches the tag information in the tag group with the tag information associated with each resource to obtain the matched resource matched with the tag group, and through the resource application of the resource user when the number of the available resource of the matched resource is not less than the number of the resource applied in the resource application request, and locking the available resource of the matched resource according to the number of the applied resource, the identification and the resource division through the tag information are realized, and the resource can be divided from different granularities through the flexible setting of the tag information, in addition, resources are allocated in a tag matching mode in the resource application process, so that a resource manager can re-divide or sum the resources by changing tag information associated with the resources at any time, and the flexibility of resource allocation is improved.
Drawings
FIG. 1 is a schematic diagram of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a resource management method according to a first embodiment of the present invention;
FIG. 3 is a table diagram of a resource and a tag according to an embodiment of the present invention;
FIG. 4 is a functional block diagram of a resource management device according to a preferred embodiment of the invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1, fig. 1 is a schematic device structure diagram of a hardware operating environment according to an embodiment of the present invention.
It should be noted that, the resource management device in the embodiment of the present invention may be a smart phone, a personal computer, a server, and the like, and is not limited herein.
As shown in fig. 1, the resource management device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the device architecture shown in fig. 1 does not constitute a limitation of the resource management device and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a resource management program. An operating system is a program that manages and controls the hardware and software resources of a device, supporting the operation of resource management programs, as well as other software or programs. In the device shown in fig. 1, the user interface 1003 is mainly used for data communication with a client; the network interface 1004 is mainly used for establishing communication connection with a server; and the processor 1001 may be configured to invoke a resource manager stored in the memory 1005 and perform the following operations:
receiving a resource application request of a resource user, and respectively matching each piece of first tag information in a first tag group carried by the resource application request with each piece of preset tag information to obtain a target tag number associated with the preset tag information successfully matched with the first tag information;
taking resources associated with the target tag number in all preset resources as candidate resources, and detecting whether all matchable preset tag information associated with the candidate resources has successfully matched tag information in the first tag group;
if the resource application requests exist, the candidate resources are used as first matching resources which are successfully matched with the first label group, and whether the number of usable resources of each first matching resource is not less than the number of application resources in the resource application requests is detected;
if yes, determining resource application of the resource user, and locking the available resource quantity of each first matched resource according to the applied resource quantity.
Further, the step of matching each piece of first tag information in the first tag group carried by the resource application request with each piece of preset tag information respectively to obtain a target tag number associated with the preset tag information which is successfully matched with each piece of first tag information correspondingly includes:
extracting each tag value in each piece of first tag information in a first tag group carried by the resource application request respectively;
searching each label value in a preset quick query table, and acquiring a label number associated with the searched label value in the preset quick query table;
and if the label number is found in a preset label table, taking the label number as a target label number, wherein one line of record in the preset label table comprises a piece of preset label information and a label number associated with the preset label information, and one line of record in the preset fast query table comprises a label value in the piece of preset label information and a label number associated with the preset label information.
Further, after the step of locking the number of available resources of each first matching resource according to the number of requested resources, the processor 1001 may be further configured to call a resource management program stored in the memory 1005 to perform the following operations:
adding a new resource, and taking the number of the applied resources as the number of the locking resources of the newly added resource;
and respectively associating the newly added resource with each piece of first label information in the first label group.
Further, after the step of locking the number of usable resources of each first matching resource according to the number of application resources, the processor 1001 may be further configured to call a resource management program stored in the memory 1005, and perform the following operations:
receiving resource use report information of the resource user, and matching each piece of second tag information in a second tag group carried by the resource use information with tag information associated with each preset resource to obtain a second matching resource successfully matched with the second tag group, wherein the second tag group is the same as the first tag group;
detecting whether the number of the locked resources of each second matched resource is not less than the number of the actually used resources in the resource use reporting information;
if yes, the number of the locking resources in each second matching resource is converted into the number of the used resources according to the number of the actually used resources.
Further, the processor 1001 after the step of converting the number of locked resources in each of the second matching resources into the number of used resources according to the number of actually used resources may be further configured to call a resource management program stored in the memory 1005, and perform the following operations:
receiving a resource recovery request of the resource user, and matching each piece of third tag information in a third tag group carried by the resource recovery request with tag information associated with each preset resource to obtain a third matching resource successfully matched with the third tag group, wherein the third tag group and the first tag information group;
detecting whether the used resource quantity of each third matching resource is not less than the resource quantity in the resource recycling request;
and if so, recycling the used resource quantity in each third matching resource according to the recycling resource quantity.
Further, if yes, after the step of recovering the used resource quantity in each third matching resource according to the recovered resource quantity, the processor 1001 may be further configured to call a resource management program stored in the memory 1005, and perform the following operations:
detecting whether the used resource quantity of a target resource in the third matching resource, which has an association relation with the user tag information, is zero;
and if so, deleting the target resource.
Further, the processor 1001 may be further configured to call a resource management program stored in the memory 1005, and perform the following operations:
adding a resource according to resource quantity information carried by a resource registration request, and associating each piece of fourth label information carried by the resource registration request with the added resource respectively, wherein the resource registration request is initiated by a resource provider or generated according to resource use limitation information set by a resource user, and when the resource registration request is generated according to the resource use limitation information, each piece of fourth label information is generated according to limitation conditions in the resource use limitation information, and the resource quantity information is generated according to the limitation resource quantity in the resource use limitation information.
Further, the processor 1001 may be further configured to call a resource management program stored in the memory 1005, and perform the following operations:
receiving a resource offline request, and matching each piece of fifth label information carried by the resource offline request with label information associated with each preset resource to search resources to be offline in each preset resource, wherein the resources to be offline are resources associated with label information which is the same as the corresponding label information of each piece of fifth label information;
and if the resource to be offline is found, deleting the resource quantity information of the resource to be offline and the associated information between the resource to be offline and the label information so as to offline the resource to be offline.
Based on the above structure, various embodiments of the resource management method are proposed.
Referring to fig. 2, fig. 2 is a flowchart illustrating a resource management method according to a first embodiment of the present invention.
While a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different than that shown. The execution subject of each embodiment of the resource management method of the present invention may be a device such as a smart phone, a personal computer, a server, etc., and for convenience of description, a resource manager (or simply referred to as a manager) is used as the execution subject in the following embodiments for explanation. In this embodiment, the resource management method includes:
step S10, receiving a resource application request of a resource user, and respectively matching each piece of first label information in a first label group carried by the resource application request with each piece of preset label information to obtain a target label number associated with the preset label information which is successfully matched with the first label information correspondingly;
in this embodiment, the resource refers to a CPU resource, a memory resource, a hard disk resource, a network resource, and the like required by the running of the computer program. The resource manager may pre-record a plurality of resources, that is, each preset resource, and may represent the size of the resource by a piece of resource quantity information, for example, the resource quantity information may include a resource type, a maximum resource quantity, a minimum resource quantity, a used resource quantity, a remaining resource quantity (or referred to as a usable resource quantity), and a locked resource quantity. The resource types can include resources such as a CPU, a memory, a hard disk, a network and the like; the maximum resource quantity is the total resource quantity of the resource; the minimum resource quantity is a critical value of the applicable resource, and when the residual resource quantity is less than the minimum resource quantity, the resource can not be allocated to the resource user; the amount of used resources, i.e. the amount of resources that have been allocated to the use of the resource user; the number of remaining resources, i.e., the number of resources that have not been allocated and locked; the amount of locked resources is the amount of resources that are allocated to resource users but that have not actually been used. The manager can distinguish each resource by the resource number. In advance, one or more labels may be marked on each resource in the resource manager, that is, each resource may be associated with one or more pieces of label information to identify the resource. In other embodiments, the resource provider may report its own resource to the resource manager, and the report information carries the resource quantity information and the tag information, and the resource manager adds a new resource according to the resource quantity information in the report information, and associates the new resource with the tag information carried in the report information.
Wherein a piece of tag information may include a tag name and at least one tag value. Tag names refer to a descriptive vocabulary (e.g., color, category, height, etc.) that is formed to identify the characteristics of a transaction after abstracting the attributes that exist across multiple things. The tag value refers to the representation of a property that a transaction actually possesses (e.g., red, blue, large, small, high, medium, low, etc.). Thus, a smaller range can be matched from a group of things by specification of a series of tags. The tag name referred to in this embodiment may be any descriptive word that can be abstracted from attributes that exist for various resource providers or various resource consumers, such as engine type (engineType), service alias (serveranias), cluster name (cluster), user (user), and so forth. That is, the resource can be identified by any attribute of the provider or user of the resource, wherein the attribute of the provider of the resource can be used to define who provides the resource, and the attribute of the user of the resource can be used to define who can use the resource.
The resource user may send a resource application request to the resource manager when needing to use the resource, where the resource user is an object, such as a process, that needs to use a certain type of resource, and the type of the resource user is not limited in this embodiment. The resource application request of the resource user can carry at least one piece of tag information, and the combination of each piece of tag information is called a tag group because a plurality of pieces of tag information can be carried. In the following description, for the sake of distinction, the tag group and the tag information carried in the resource application request are referred to as a first tag group and a first tag information, respectively. The tag information that the resource user needs to carry when generating the resource application request can be preset, or the administrator can configure the tag information that the resource user needs to carry when generating the resource application request for each resource user as required at any time. Similarly, the tag information carried in the resource application request can be used to identify the resource user himself, or can be used to identify the resource provider.
The manager receives a resource application request of a resource user, extracts a first tag group in the resource application request, matches each piece of first tag information in the first tag group with tag information associated with each preset resource, and finds out a resource successfully matched with the first tag group. The resource successfully matched with the first tag group is hereinafter referred to as a first matching resource. According to different specific application scenes, the judgment rules for whether the label group is successfully matched with the resource can be different; for example, each piece of tag information associated with one resource may correspond to the same piece of tag information in the first tag group, that is, it may be determined that the matching between the resource and the first tag group is successful; or, the same tag information can be found in the first tag group for each piece of tag information associated with one resource, that is, it can be determined that the matching between the resource and the first tag group is successful, that is, the tag information associated with the resource is included in the first tag group, but the tag information in the first tag group may not necessarily be the tag information associated with the resource. It can be understood that, in this embodiment, the determination rule of whether the tag group and the resource are successfully matched is not limited.
In an embodiment, a determination rule for determining whether a tag group and a resource are successfully matched is specifically provided, that is, when each piece of tag information associated with a resource can find the same tag information in the tag group, it can be determined that the resource and the tag group are successfully matched.
According to the judgment rule, the resource manager can extract the first tag group carried in the resource application request, and match each piece of first tag information in the first tag group with each piece of preset tag information respectively. Specifically, in an embodiment, the resource manager may compare each piece of first tag information with each piece of preset tag information one by one, to determine whether there is preset tag information that is the same as the first tag information, where the two pieces of tag information are the same, that is, the tag name and the tag value are both correspondingly the same; if the preset tag information identical to the first tag information exists, it can be determined that the preset tag information is successfully matched with the first tag information, and the tag number (or referred to as tag ID) associated with the preset tag information is used as the target tag number. In other embodiments, the resource manager may also compare each tag value in each piece of first tag information with the tag values of each piece of preset tag information one by one, to determine whether the tag value is the same as the tag value in the first tag information, and if so, take the tag number of the preset tag information corresponding to the same tag value as the target tag number; in this case, when a plurality of tag values are included in one piece of first tag information, matching using the first tag information may match a plurality of target tag numbers.
And the resource manager respectively matches the first label information to obtain target label numbers associated with the preset label information successfully matched with some of the first label information. And if the matching of the first label information is successful, the number of the finally obtained target labels is multiple. In an embodiment, if each piece of first tag information is not successfully matched, the resource manager may determine that the resource application does not pass, that is, the resource matched with the tag in the resource application request is not found.
The preset tag information is tag information associated with each resource and recorded in the resource manager in advance. Each piece of preset label information is distinguished by adopting a label number, namely each piece of preset label information is associated with a unique label number, and the label information is associated with the resource in a way of associating the label number with the resource number. Specifically, in an embodiment, a tag table may be used to store each piece of preset tag information, and a row of records in the tag table includes a tag number that is associated with a piece of preset tag information and the preset tag information; storing each resource by adopting a resource table, wherein one row of record in the resource table comprises a resource number of one resource and resource quantity information of the resource; and storing the association relationship between the resources and the tags by adopting a resource tag association table, wherein a row of records in the resource tag association table comprises a tag number and a resource number. It is understood that the same piece of tag information may be associated with multiple resources, and one resource may also be associated with different pieces of tag information.
Further, in an embodiment, the resource manager may set tag information not participating in matching in advance, or the resource user marks first tag information not participating in matching in the resource application request, and the resource manager may not match the first tag information not participating in matching when matching. For example, a resource user generally carries an engine instance tag information when applying for a resource, and the engine instance tag information is unique for each resource user and generally not associated with the resource, so the engine instance tag information can be set not to participate in matching, thereby reducing unnecessary computation.
Step S20, taking the resource associated with the target label number in each preset resource as a candidate resource, and detecting whether each matchable preset label information associated with the candidate resource has successfully matched label information in the first label group;
after the resource manager obtains the target tag number through matching, the resource manager may use the resource associated with the target tag number as a candidate resource. Specifically, in an embodiment, the resource tag association table may be searched, the record where the target tag number is located may be searched, and the resource number stored in the record in association with the target tag number is extracted, where the resource number is the resource number of the candidate resource.
After the candidate resource is obtained, the resource manager may first find each piece of the preset tag information that may be matched, which is associated with the candidate resource. In the multiple preset tag information associated with one resource, tag information that does not participate in matching may be preset, for example, the engine manager tag information mostly indicates a machine and a port where a resource provider is located, and a resource user does not need to specify a range of the machine and the port every time of application, so the tag information may be set not to participate in matching. The resource manager detects whether each piece of matching preset tag information associated with the candidate resource has successfully matched tag information in the first tag group. The matching of the preset tag information in the first tag group is successful, namely, one piece of tag information in the first tag group is the same as the preset tag information; the first tag group is used for storing a piece of preset tag information, and the preset tag information is matched with the second tag group.
In an embodiment, the resource manager may first search the resource tag association table, find a record in which the resource number of the candidate resource is located, and extract the tag number stored in the record in association with the resource number. It can be understood that when one resource is associated with a plurality of pieces of label information, a plurality of label numbers can be found according to the resource numbers of the candidate resources. And according to each found label number, finding the record of the label number in the label table, and extracting the label information stored in the record in association with the label number, namely extracting each piece of preset label information associated with the candidate resource. And for each piece of preset tag information which is associated with the candidate resource and can participate in matching, comparing the preset tag information with each piece of first tag information in the first tag group respectively, and determining whether the tag information which is the same as the preset tag information exists in the first tag group.
Step S30, if both of them exist, using the candidate resource as a first matching resource successfully matched with the first tag group, and detecting whether the number of usable resources of each first matching resource is not less than the number of application resources in the resource application request;
if the matching success tag information exists in the first tag group for each piece of matching preset tag information associated with the candidate resource, the resource manager may determine that the candidate resource is successfully matched with the first tag group, that is, the candidate resource is used as the first matching resource. It should be noted that, when there are multiple candidate resources, it is determined whether the matching with the first tag group is successful for each candidate resource according to the above steps.
The first matching resources successfully matched with the first tag group may have one or more items, and the resource manager may detect whether the number of available resources of each item of the first matching resources is not less than the number of application resources in the resource application request after determining the first matching resources. The resource manager records the number of the usable resources of each resource, when the resource manager is not applied, the number of the usable resources is the maximum number of the resource, when the resource manager is applied for a part, the number of the usable resources is correspondingly reduced, and the number of the used resources is correspondingly increased. The resource application request of the resource user may carry the amount of the resource to be applied, that is, the amount of the resource to be applied, for example, 10 CPUs and 100G memories are required.
Step S40, if yes, determining a resource application by the resource user, and locking the number of the available resources of each first matching resource according to the number of the resource application.
If the resource manager detects that the number of the available resources of each first matching resource is not less than the number of the application resources, it indicates that each first matching resource can be allocated to the resource user for use. After determining that the application is successful, the resource user can lock the available resource quantity of each first matching resource according to the application resource quantity. Specifically, the locking may be performed by subtracting the number of application resources from the number of available resources of the first matching resource and adding the number of locking resources to the number of application resources, so as to indicate that the part of resources is locked and other resource users cannot apply for any more. After the resource user is informed that the application is passed, the requested resource can be actually used.
Further, in an embodiment, if the resource manager detects that the number of the available resources of each first matching resource is not all smaller than the number of the application resources, that is, the number of the available resources of a part of the first matching resources is smaller than the number of the application resources, it indicates that the part of the first matching resources may not be allocated to the resource user for use, and at this time, the resource manager may determine that the resource application does not pass through the resource user. Or, in other embodiments, if the number of available resources of a part of the first matching resources is less than the number of application resources, and the number of available resources of another part of the first matching resources is not less than the number of application resources, the resource manager may first determine whether the part of the first matching resources, the number of which is not less than the number of application resources, can be applied independently; if so, the resource manager can determine a passing application and feed back the label information associated with the part of the first matched resources to the resource user initiating the resource application request so as to allow the resource user to actually use the resources associated with the label information; the resource manager may lock the available resource amount of the portion of the first matching resource only according to the requested resource amount. It should be noted that, it may be preset in the resource manager whether each resource can be applied for separately.
Compared with the existing resource management method which only can lead the granularity of management and control to be fixed in advance and cannot support dynamic adjustment and insertion of other granularities according to actual requirements, in the embodiment, through associating each resource with the label information, and through the fact that a resource user carries the label group comprising one or more label information in a resource application request when applying for the resource, the resource manager matches the label information in the label group with the label information associated with each resource to obtain a matched resource matched with the label group, and through the resource application of the resource user when the number of the available resources of the matched resource is not less than the number of the applied resources in the resource application request, and locking the available resources of the matched resource according to the number of the applied resources, the purpose of identifying and dividing the resource through the label information is realized, and the resource can be divided from different granularities through flexible setting of the label information, in addition, resources are allocated in a tag matching mode in the resource application process, so that a resource manager can re-divide or sum the resources by changing tag information associated with the resources at any time, and the flexibility of resource allocation is improved.
Moreover, because the scenes targeted by the clusters are different, the resource allocation strategies are often greatly different, the definitions of resource attributes by resource users and providers are not uniform and are not in the same dimension (for example, the Yarn resource provider stands in the dimension of the queue to control the resources, the user only concerns the CPU and memory information of a single node single process and needs to distinguish the resources of the driver and the executor, wherein the resources of the driver are not managed in the queue of Yarn, which is very complex), the existing resource management method can only manage the resources of one cluster at a time, and each cluster needs to deploy one own resource manager, so that the operation cost is increased, and the efficiency of resource allocation is reduced. In the embodiment, the resources are divided in a mode of marking the resources, and the resources are allocated in a mode of matching the labels, so that the resources of different clusters can be identified and allocated by the resource manager in a mode of marking the labels, and therefore, the resource management of a plurality of clusters can be realized by adopting one resource manager without deploying a plurality of resource managers, the operation cost is reduced, and meanwhile, the efficiency of cross-cluster resource allocation is improved.
Further, based on the first embodiment, a second embodiment of the resource management method of the present invention is provided, and in this embodiment, the step S10 of matching each piece of first tag information in a first tag group carried by the resource application request with each piece of preset tag information respectively to obtain a target tag number associated with the preset tag information that is successfully matched with the first tag information correspondingly includes:
step S101, for each piece of first label information in a first label group carried by the resource application request, respectively extracting each label value in each piece of first label information;
in one embodiment, to improve the efficiency of tag matching and improve the efficiency of resource management, a fast lookup table may be used to backup each piece of tag information.
Specifically, a tag table may be used to store each piece of preset tag information, and a row of records in the tag table includes a tag number of the preset tag information associated with the preset tag information. The tag values in the tag information are typically stored in a serialized format (e.g., json format), where a tag value includes a name and a content, and where the serialized format is not easily retrievable. For example, one engine type tag information includes a tag name of "engineType" and two tag values, the name and content of one tag value are "engineType" and "io", respectively, the name and content of the other tag value are "version" and "1", respectively, the two tag values together form the tag of the engine type, and the json format is used to indicate that the two tag values are "{" engineType ": io", "version": 1 ".
To facilitate retrieval and comparison, a fast lookup table is used to store a copy of the tag information in the tag table. And one row of records in the quick query table comprises one label value in one piece of preset label information and a label number associated with the preset label information, and the label number in the quick query table is unified with the label number used in the label table. It is understood that when one tag information includes a plurality of tag values, it is split into a plurality of rows of records in the fast lookup table. For example, for two tag values in the above-mentioned engine type tag information, assuming that the tag number of the engine type tag information is 1, the record of the tag value of the engine type tag information in the fast lookup table is:
tag ID Name of value Content of value
1 engineType io
1 version 1
In one embodiment, the resource table, the tag table, the fast lookup table, and the resource tag association table may be constructed in the manner as shown in fig. 3. In this case, a row of records in the tag table may further include a number field of a value, that is, the tag information indicating the row of records includes several tag values, for example, the number of values of the above-mentioned engine type tag information is 2.
After receiving the resource application request, the resource manager may extract each tag value in each piece of first tag information in the first tag group carried by the resource application request.
Step S102, searching each label value in a preset quick query table, and acquiring a label number associated with the searched label value in the preset quick query table;
after extracting the tag values, the resource manager searches each tag value in the fast lookup table, that is, searches in the fast lookup table under the condition of the name of the value and the content of the value in the tag value, finds one or more rows of records that are the same as the name of the value and the content of the value, and obtains the tag number (hereinafter, referred to as a candidate number) in the found records.
Step S103, if the tag number is found in a preset tag table, the tag number is used as a target tag number, wherein one line of record in the preset tag table comprises a piece of preset tag information and the tag number associated with the preset tag information, and one line of record in the preset fast query table comprises a tag value in the piece of preset tag information and the tag number associated with the preset tag information.
And searching candidate numbers in the label table, and if the candidate numbers are found, taking the candidate numbers as target label numbers. If the candidate number is not found, it indicates that the tag information corresponding to the tag number has been deleted in the tag table, but may not be deleted in the fast lookup table in time, and at this time, the candidate number may be discarded, and it is considered that the target tag number is not matched. It is understood that a plurality of tag values may be included in a tag information, and a plurality of tag numbers may be found in a fast lookup table for a tag value, so that there may be a plurality of target tag numbers found finally.
In the embodiment, the tag table is used for storing the tag numbers and the tag information in an associated manner, a fast lookup table is used for backing up and storing the tag values in each tag information, the tag values are searched through the fast lookup table, and then the tag information is matched, so that the matching efficiency of the tag information is improved, and the resource management efficiency is improved.
Further, based on the first and/or second embodiments, a third embodiment of the resource management method according to the present invention is proposed, in this embodiment, after step S40, the method further includes:
step S50, adding a new resource, and using the applied resource quantity as the locking resource quantity of the added resource;
step S60, associating the new resource with each of the first tag information in the first tag group.
In this embodiment, after the resource manager applies for the resource by the resource user, the resource manager may record the resource quantity applied by the resource user, so as to manage the resource quantity in the following actual use and resource recovery of the resource.
Specifically, the resource user may carry a tag information capable of uniquely identifying the resource user in the resource application request, that is, the first tag group includes a first tag information for uniquely identifying the resource user, and generally, different characteristic attributes of each resource user are adopted as the tag information playing a role of unique identification, for example, the engine instance tag information of the resource user.
The resource manager can newly add a resource after the resource application of the resource user, and uses the application resource quantity in the resource application request as the locking resource quantity of the newly added resource, and associates the newly added resource with each piece of first tag information in the first tag group respectively. Specifically, in an embodiment, the resource manager may add a new row of records in the resource table, automatically generate the resource number of the record according to the current ordering, fill the applied resource number as the data of the field of the locked resource number in the record, and set the other fields to null; the label number associated with each piece of first label information in the first label group is found in the label table or the quick query table, and if the first label information is not recorded in the label table, the record of the first label information can be newly added in the label table and the quick query table; and respectively storing the label number of each piece of first label information and the resource number of the newly added resource in a resource label association table in an association manner, so as to respectively associate the newly added resource and each piece of first label information in the first label group.
That is, the newly added resource is not a real available resource, but the number of resources applied by the resource user is recorded in this way, and since the resource is associated with the tag information for uniquely identifying the resource user, other resource users will not match the resource when applying for the resource, and will not be mistakenly used as an available resource.
Further, based on the first, second and/or third embodiments, a fourth embodiment of the resource management method of the present invention is proposed, and in this embodiment, after step S40, the method further includes:
step S70, receiving resource usage report information of the resource user, and matching each piece of second tag information in a second tag group carried by the resource usage information with tag information associated with each preset resource to obtain a second matching resource successfully matched with the second tag group, where the second tag group is the same as the first tag group;
after determining that the resource application is passed, the resource user can actually use the resource according to the tag information and the application resource quantity in the resource application request, and after actually using the resource, the resource user can send resource use report information to the resource manager. The resource usage reporting information may carry a second tag group, where the second tag group includes one or more pieces of second tag information, and the resource usage reporting information may further include the number of resources actually used by the resource user (hereinafter, referred to as the number of actually used resources). It should be noted that the second tag group is the same as the first tag group, that is, when the resource user reports the actually used resource, the tag information carried in the resource usage report information is the same as the tag information carried in the resource application request when the resource user applies for the resource.
After receiving the resource usage report information, the resource manager may match each piece of second tag information in the second tag group with tag information associated with each preset resource, to obtain a second matching resource successfully matched with the second tag group. The process of matching the second tag information with the tag information associated with each preset resource is similar to the process of matching the first tag information with the tag information associated with each preset resource in the above embodiments, and details are not repeated here. Since the second tag group is the same as the first tag group, the second matching resource is the same as the first matching resource when no new resource item is added between the resource user making the resource application request and reporting the resource usage reporting information. If the resource user adds a resource item for recording the resource application amount of the resource user after providing the resource application request, the resource item is associated with each piece of first tag information in the first tag group, so that the resource item for recording the resource application amount of the resource user can be obtained by matching each piece of second tag information in the second tag group, besides the resource item which is the same as the first matching resource, by matching.
Step S80, detecting whether the number of the locked resources of each second matching resource is not less than the number of the actually used resources in the resource usage report information;
after the resource manager obtains each second matching resource through matching, it may detect whether the number of the locked resource of each second matching resource is not less than the number of the actually used resource in the resource usage report information. It can be understood that, after the resource user applies for the resource, the resource is actually used according to the number of the applied resource in the resource application request, that is, the actually used resource does not exceed the number of the applied resource, and the resource manager locks the available resource of the first matching resource according to the number of the applied resource, so that under normal conditions, the number of the locked resources in each second matching resource is not less than the number of the actually used resource, and if the number of the locked resources of one second matching resource is less than the number of the actually used resource, it indicates that the number of the actually used resource reported by the resource user is incorrect.
Step S90, if yes, converting the number of the locked resources in each second matching resource into the number of the used resources according to the number of the actually used resources.
If the resource manager determines that the number of the locked resources of each second matching resource is not less than the number of the actually used resources in the resource use report information, it indicates that the number of the actually used resources reported by the resource user is correct, and at this time, the resource manager may convert the number of the locked resources of each second matching resource into the number of the used resources according to the number of the actually used resources. In particular, the resource manager may subtract the actual number of used resources from the number of locked resources and add the actual number of used resources to complete the conversion.
It can be understood that, when the second matching resource includes a resource item for recording the resource amount requested by the resource user, the resource amount actually used by the resource user is recorded by converting the locked resource amount of the resource item into the used resource amount. In one embodiment, the resource user can report the number of actually used resources to the resource manager for multiple times, and the tag information carried by each time is the same, so that the resource manager can find the same resource item each time, and update the resource number information in the resource item, so as to accurately record the use condition of each resource, and facilitate accurate management of the resources.
If the number of the locked resources of each second matching resource is not less than the number of the actually used resources, that is, the number of the locked resources of a part of the second matching resources is less than the number of the actually used resources, it indicates that the number of the actually used resources carried in the resource use report information by the resource user is wrong, and at this time, the resource manager may feed back a wrong alarm to the resource user and does not process the resource use report information.
Further, after the step S90, the method further includes:
step A10, receiving a resource recovery request of the resource user, and matching each piece of third tag information in a third tag group carried by the resource recovery request with tag information associated with each preset resource to obtain a third matching resource successfully matched with the third tag group, wherein the third tag group and the first tag information group;
after the resource user finishes using the resource, the resource user may send a resource recycling request to the resource manager, where the resource recycling request may carry a third tag group, the third tag group includes one or more pieces of third tag information, and the resource recycling request may further include a resource quantity (hereinafter referred to as a recycled resource quantity) requested to be recycled by the resource user. It should be noted that the third tag group is the same as the second tag group, that is, when the resource user requests to recover the resource, the tag information carried in the resource recovery request is the same as the tag information carried in the resource application request when the resource user applies for the resource.
After receiving the resource recycling request, the resource manager may match each piece of third tag information in the third tag group with tag information associated with each preset resource, to obtain a third matching resource successfully matched with the third tag group. The process of matching the third tag information with the tag information associated with each preset resource is similar to the process of matching the first tag information with the tag information associated with each preset resource in the above embodiments, and details are not repeated here. Similarly, since the third tag group is identical to the first tag group, the third matching resource is identical to the first matching resource when there is no new resource item between the resource application request and the resource recovery request made by the resource user. If the resource user adds a resource item for recording the resource application amount of the resource user after providing the resource application request, the resource item is associated with each piece of first tag information in the first tag group, so that by matching each piece of third tag information in the third tag group, the resource item for recording the resource application amount of the resource user can be obtained by matching in addition to the resource item which is the same as the first matching resource.
Step A20, detecting whether the used resource quantity of each third matching resource is not less than the resource quantity in the recovery request;
after the resource manager obtains the third matching resources through matching, it may detect whether the used resource quantity of each third matching resource is not less than the quantity of the recycling resource in the resource recycling request. It can be understood that, after the resource user applies for the resource, the resource is actually used according to the number of the application resources in the resource application request, that is, the actually used resource does not exceed the number of the application resources, and the resource manager converts the number of the locked resources of the second matching resource into the number of the used resources according to the number of the actually used resource, so that under normal conditions, the number of the used resources in each third matching resource is not less than the number of the recycled resources, and if the number of the used resources of one third matching resource is less than the number of the recycled resources, it indicates that the number of the recycled resources reported by the resource user is incorrect.
And a step a30, if yes, recycling the used resource quantity in each third matching resource according to the recycled resource quantity.
If the resource manager determines that the used resource quantity of each third matching resource is not less than the recycled resource quantity in the resource recycling request, it indicates that the recycled resource quantity reported by the resource user is correct, and at this time, the resource manager can recycle the used resource quantity in each third matching resource according to the recycled resource quantity. Specifically, the resource manager may subtract the number of the used resources from the number of the recycled resources, and add the number of the usable resources to the number of the recycled resources to complete resource recycling, so as to facilitate the resource user or other resource users to perform the next resource application.
It is understood that, when the third matching resource includes a resource item for recording the resource amount requested by the resource user, the resource amount recovered by the resource user is recorded by recovering the used resource amount of the resource item. In an embodiment, a resource user can send a resource recovery request to a resource manager for multiple times, and tag information carried by the resource user is the same every time, so that the resource manager can find the same resource item every time, and then update resource quantity information in the resource item, so as to accurately record the use condition of each resource, and facilitate accurate management of the resource.
If the used resource quantity of each third matching resource is not less than the recycling resource quantity, that is, the used resource quantity of some third matching resources is less than the recycling resource quantity, it indicates that the recycling resource quantity carried in the resource recycling request by the resource user is wrong, and at this time, the resource manager may feed back a wrong alarm to the resource user and does not process the resource recycling request.
Further, in an embodiment, after the step a30, the method further includes:
step A40, detecting whether the used resource quantity of the target resource in the third matching resource which has the association relation with the user label information is zero;
and step A50, if yes, deleting the target resource.
When the resource manager adds a resource item for recording the resource application quantity of the resource user after providing the resource application request, and the resource manager performs matching according to the third tag group in the resource recovery request, the third matching resource obtained by matching includes the resource item, and the resource item is a resource associated with the user tag information and the only one of the third matching resources. The resource manager can find the resource which has the incidence relation with the user label information from the third matching resource after receiving the resource recovery request and recovering the used resource in the third matching resource according to the quantity of the recovered resource, wherein the resource item is a resource item for recording the quantity of the resource applied by the resource user, and the resource is called as a target resource for distinguishing; the resource manager detects whether the used resource quantity of the target resource is zero; if yes, the resource manager may delete the target resource, for example, from the resource table, and may delete the association between the target resource and the tag information.
In this embodiment, the resource manager searches for corresponding resource items in a tag matching manner by using the resource user to carry the same tag information in the resource application request, the resource usage report information, and the resource recovery request, and updates the resource quantity information of the resource items accordingly, so that the resource manager can manage resources through the tag information in the whole resource management process of resource application, resource usage report, and resource recovery, and the effect of flexibly dividing resources through the tag information can run through the whole resource management process, thereby further improving the flexibility of resource management.
Further, based on the first, second, third and/or fourth embodiments, a fifth embodiment of the resource management method of the present invention is provided, in this embodiment, the method further includes:
step A60, adding a resource according to the resource quantity information carried by the resource registration request, and associating each piece of fourth label information carried by the resource registration request with the added resource, wherein the resource registration request is initiated by a resource provider or generated according to resource usage restriction information set for a resource user, and when the resource registration request is generated according to the resource usage restriction information, each piece of fourth label information is generated according to the restriction condition in the resource usage restriction information, and the resource quantity information is generated according to the restricted resource quantity in the resource usage restriction information.
In this embodiment, the resource provider may send a resource registration request to the resource manager, or an operation of the administrator setting the resource usage restriction information for the resource user may trigger the resource registration request in the resource manager. The resource registration request carries a piece of resource quantity information and one or more pieces of label information (referred to as fourth label information to indicate distinction), the resource quantity information indicates the resource quantity of the resource to be registered, and the label information is the label information used for marking the resource. After receiving the resource registration request, the resource manager adds a new resource according to the resource quantity information carried in the request, that is, the resource quantity information is used as the resource quantity information of the new resource, and associates each piece of fourth tag information with the new resource respectively.
Specifically, in an embodiment, the resource manager may add a new row of records in the resource table, automatically generate the resource number of the record according to the current ordering, and correspondingly fill the resource quantity information in the resource registration request into each field of the record; the resource manager can search whether the label information identical to the fourth label information exists in the label table or the quick query table, and if the label information identical to the fourth label information does not exist in the label table or the quick query table, a record of the fourth label information is newly added in the label table and the quick query table; and respectively storing the label number of each piece of fourth label information in association with the resource number of the newly added resource in the resource label management table, thereby completing the registration of the resource in the resource registration request.
When the resource registration request is initiated by a resource provider, the resource items added according to the resource registration request are actually available resources, and in this case, the resource quantity information carried in the resource registration request generally includes information of fields of resource type, maximum resource quantity, minimum resource quantity, used resource quantity, remaining resource quantity (or called available resource quantity), and locked resource quantity.
When the resource registration request is generated according to the resource usage restriction information set for the resource user, the resource item added according to the resource registration request is not an actually available resource, but is restricted for the resource user conforming to the tag information of the resource item in the tag information matching process. Or, in this embodiment, the restriction information on the resource user is embodied in a virtual resource manner, so that the resource manager only needs to execute the logic for matching the tag information in the processes of resource application, resource usage information reporting, and resource recovery, and does not need to specially execute the logic for determining whether the resource applied by the resource user is restricted, thereby simplifying the management process of the resource manager and improving the resource management efficiency. In this case, each piece of the fourth label information in the resource registration request may be generated according to the restriction condition in the resource usage restriction information, and the resource amount information in the resource registration request may be generated according to the restricted resource amount in the resource usage restriction information. Generally, when the constraint condition includes a plurality of conditions that are all satisfied, a piece of tag information may be generated according to each condition, and then the pieces of tag information are combined into a piece of tag information, that is, a plurality of tag names are combined into one tag name by means of character string splicing, and a plurality of tag values are connected in series to serve as a tag value of one piece of tag information. The resource type for which the limited resource quantity is directed is used as the resource type in the resource registration request, the limited resource quantity is used as the usable resource quantity of the resource in the resource registration request, and the other fields can be set to be null.
Further, in an embodiment, the method further comprises:
step A70, receiving a resource offline request, and matching each piece of fifth label information carried by the resource offline request with label information associated with each preset resource to search for a resource to be offline in each preset resource, wherein the resource to be offline is a resource associated with label information corresponding to the same label information as each piece of fifth label information;
step A80, if the resource to be offline is found, deleting the resource quantity information of the resource to be offline and the associated information between the resource to be offline and the label information, so as to offline the resource to be offline.
The resource provider may send a resource logoff request to the resource manager. After receiving the resource offline request, the resource manager can perform offline on the resource according to the resource offline request. Specifically, the resource offline request carries one or more pieces of tag information, which is hereinafter referred to as fifth tag information to show differences. And the resource manager matches each piece of fifth label information in the resource offline request with label information associated with each preset resource to find out whether the offline resource is to be offline. Specifically, when the label information associated with a resource is completely corresponding to the label information in the resource offline request, the resource is the resource to be offline; if the resource to be offline is found, the resource manager can delete the resource quantity information of the resource to be offline and delete the association relationship between the resource to be offline and the label information, thereby completing the offline of the resource. If the resource to be offline is not found, that is, if no resource is associated with all the fifth tag information in advance, the resource manager may feed back an error alarm to the sender of the resource offline request, or may not process the resource offline request.
In one embodiment, the processes of resource registration, resource application, and resource recovery are illustrated.
1. Assume that a DEV cluster currently exists in which there is a machine named provider01 that can provide 50 CPUs and 500G of memory that communicates with an application such as a resource manager or computing storage through an engine manager of the 8080 port. It acts as a resource provider and submits its own resource-related information to the resource manager for storage. The machine sends an RPC message to a resource manager, and data in a Json format is used for marking operation types, label information and resource quantity information, wherein the specific format is as follows:
{"RegisterResource":{"labels":[{"key":"cluster","value":"{"clusterName":"DEV"}"},{"key":"emInstance","value":"{"instance":"provider01:8080","serviceNa me":"engineConnManager"}"}],"resources":{…}}}。
A. after receiving the message, the resource manager recognizes the message as a resource registration request according to the 'register resource', reads the labels column to obtain the label in the json format, and obtains the label information marked for the provider after analyzing the key and the value in the json format:
Figure BDA0003101751330000241
these tag information are stored in the tag table in the manner described above, respectively, assuming that their row IDs are 001 and 002, respectively. Here, the tag description is a field added for easy understanding, and the field is not actually stored in the tag table.
B. The resource amount information of the provider is stored as one row in the resource table, and the row ID is assumed to be 001. (the storage format of the resource itself is not limited in this embodiment, and is not described again).
C. In the association table of tag information and resource, there will be the following records:
tag ID Resource ID
001 001
002 001
2. Suppose that in a DEV cluster there is also a yarn queue named IDE that can provide 20 CPUs and 100G of queue memory. The yann resource provider sends an RPC message to the resource manager, and the operation type, the label information and the resource quantity information of the yann queue are indicated by data in a Json format, wherein the specific format is as follows:
{"RegisterResource":{"labels":[{"key":"cluster","value":"{"clusterName":"DEV"}"},{"key":"yarn","value":"{"url":"0.0.0.0:8899","queueName":"IDE"}"}],"re sources":{…}}}。
compared with the resource provider in step 1, this yann queue is also on the Dev cluster, so their cluster labels are the same, except that in step 1, an engine manager represents a local resource of a machine, so it carries an eminstant label indicating its machine name and port; in the second step is a yarn queue, so it carries a yarn tag that indicates the service interface address and queue name for that yarn.
A. The json is resolved by the same procedure as described in step 1, and the label information printed on the provider is:
Figure BDA0003101751330000251
where the cluster tag multiplexes the record with an ID of 001, the record for the other tag is assumed to be 003.
B. The resource amount information of the provider is stored as one row in the resource table, and the row ID is assumed to be 002.
C. In the association table of tag information and resource, there will be the following records:
tag ID Resource ID
001 002
003 002
3. Assuming that the user hduiser 001 is set, it can use only 5 CPUs and 5G memory at most when starting the spark engine on behalf of the BI. (the limitation of resource usage, in fact, is also analogous to a provider, but not the actual provider, but the two do not have to be differentiated when dealing with the logic of the resource application).
A. The tag information generated according to this setting is:
Figure BDA0003101751330000252
since it is a combination of tags that cannot be separately broken down for integration with the resource, it generates a combined tag as follows:
Figure BDA0003101751330000253
B. a resource is newly added to the resource table based on this setting, assuming that the row ID is 003.
C. In the association table of tag information and resource, there will be the following records:
tag ID Resource ID
004 003
4. At this point, a resource user begins to apply for resources from the resource manager. Suppose it is a user called hduiser 001, ready to start a spark engine on the 10001 port of machine consumer01 on behalf of the BI, ready to apply for the resources of the DEV or SIT environment, in the number of 10 CPUs, 10G memory, and 2 CPUs, 2G memory of the IDE queue.
The resource user sends RPC message to the resource manager, and the specific format is as follows:
{"RequestResource":{"labels":[{"Key":"user","value":"{"userName":"hduser001"}"},{"Key":"engineInstance","value":"{"instance":"consumer01:10001","serv iceName":"engineConn"}"},{"Key":"creator","value":"{"creator":"BI"}"},{"Key":"engineType","value":"{"engineType":"spark","version":"*"}"},{"Key":"Yarn","value":"{"url":"*","queueName":"IDE"}"},{"Key":"cluster","value":"{"clusterName":"DEV"}"},{"Key":"cluster","value":"{"clusterName":"SIT"}"}],"resources":{…}}}
the format is similar to the format of the resource registration described above, except that it begins with "RequestResource" to identify messages that are resource applications, labels is the label carried by the application, and resources is the resource that the user is preparing to apply for.
A. The tag information that this resource application will carry is as follows:
Figure BDA0003101751330000261
two cluster labels are brought about because two clusters of resources can be applied for.
B. After receiving the resource application RPC message, the resource manager screens out the resource meeting the conditions according to the tag information, specifically, traverses each value name and value content of each tag, finds the corresponding tag and resource, and finds a large candidate resource pool, such as:
taking a user label as an example, a label of { "userName": hduiser 001"} is carried in the resource application.
a. Then first, a record is found from the fast lookup table (this ID is 004 according to the assumption of step 3) conditioned on [ the name of the value is "userName", and the content of the value is "hduiser 001" ];
b. then, according to the recorded label ID, finding out the label which is associated with the ID and has a foreign key from the label table, namely the combined label in step 3;
c. finally, after the queried label is confirmed to be really stored, the corresponding resource ID is found through the resource label association table (according to the assumption in step 3, the ID is 003), and then the resource which is associated with the ID and has the foreign key is found from the resource table, namely the resource corresponding to the user label is matched.
And in the resource application, the set resources in the step 3 can be matched through the tags of the user, the creator or the engineType. And matching the resources of the resource provider in the steps 1 and 2 through the cluster label. The resources in step 2 can also be matched by means of a yarn tag. Not to mention possibilities here.
C. And matching the labels associated with each resource in the candidate resources, and determining whether the matched items can be found in the labels of the resource application.
For example, the resource matched by the user tag in the previous step has a user tag, a creator tag and an engineType tag, and the tags and values must coexist in the tags carried in the resource application to be considered as the successfully matched resource. In the above example, it can be seen that the resource can be successfully matched.
② there is also a case that the matching can not be successful: if the resource provided in step 3 is added with an additional tag of engineType (hive) during reporting, however, the tag is not carried by the resource application of this time, and thus cannot be matched by the application of this time. When the resource provided in step 3 is matched with the tag of the engineType hive, the resource can be offline first, and when the resource is re-registered again, the allocation policy of the resource can be adjusted by adding the tag of the engineType hive to the registration RPC message.
D. The label of the resource, in the reporting stage, may choose not to participate in the comparison of this step, such as the label of the emittance, mostly indicates the machine and the port where the resource provider is located, and does not need to specify the range of the machine and the port every time the application is made. However, if the resource application initiative is provided with an emInstance tag, the corresponding resource can still be added into the candidate resource pool in the previous step through the tag.
E. And after finding the matching item, comparing the number of the application resources with the number of the available resources (such as comparing the number of the CPUs and the number of the memories). If the available resources of all the matched resources are not less than the number of the applied resources, the application is passed, and the resource manager locks the resources according to the tags to wait for actual use:
adding a record in the resource table according to the applied resource quantity to obtain the ID of the record.
And recording and associating the resource record ID with all the tag IDs carried by the resource application in an association table. (since this tag is unique to each engine, the resource recorded here is not mistakenly used as an available resource.)
F. And after the resources are used, the resource manager recovers the resources.
Firstly, before a resource user (often a process) is closed, the resource user informs a resource manager to recycle resources through an RPC message, and the format is as follows:
{ "releaseresources": { "labels": … (the label content is consistent with that transmitted when applying for the resource) ], "resources": … } }
And secondly, after receiving the resource recycling application, the resource manager finds the ID of each label according to the key and the value of each label, finds the associated resource record, subtracts the resource quantity carried in the RPC message from the used quantity of the resource, and updates the resource record.
And deleting the resource records corresponding to the engineInstance labels during recovery.
In addition, an embodiment of the present invention further provides a resource management apparatus, and referring to fig. 4, the apparatus includes:
the receiving module 10 is configured to receive a resource application request of a resource user, match each piece of first tag information in a first tag group carried in the resource application request with each piece of preset tag information, and obtain a target tag number associated with the preset tag information that is successfully matched with the first tag information;
a first detecting module 20, configured to use a resource associated with the target tag number in each preset resource as a candidate resource, and detect whether each matchable preset tag information associated with the candidate resource has successfully matched tag information in the first tag group;
a second detecting module 30, configured to, if both of the candidate resources exist, use the candidate resource as a first matching resource successfully matched with the first tag group, and detect whether the number of available resources of each first matching resource is not less than the number of application resources in the resource application request;
and the determining module 40 is configured to determine a resource application passing through the resource user if the resource application is successful, and lock the available resource quantity of each item of the first matching resource according to the application resource quantity.
Further, the receiving module 10 includes:
an extracting unit, configured to extract, for each piece of first tag information in a first tag group carried by the resource application request, each tag value in each piece of first tag information;
the searching unit is used for searching each label value in a preset quick query table and acquiring a label number associated with the searched label value in the preset quick query table;
the determining unit is configured to use the tag number as a target tag number if the tag number is found in a preset tag table, where a line of records in the preset tag table includes a piece of preset tag information and a tag number associated with the preset tag information, and a line of records in the preset fast lookup table includes a tag value in the piece of preset tag information and a tag number associated with the preset tag information.
Further, the first tag group includes a piece of user tag information for uniquely identifying the resource user, and the apparatus further includes:
the first adding module is used for newly adding a resource and taking the quantity of the applied resource as the quantity of the locking resource of the newly added resource;
and the association module is used for associating the newly added resource with each piece of first label information in the first label group respectively.
Further, the receiving module 10 is further configured to: receiving resource use report information of the resource user, and matching each piece of second tag information in a second tag group carried by the resource use information with tag information associated with each preset resource to obtain a second matching resource successfully matched with the second tag group, wherein the second tag group is the same as the first tag group;
the second detection module 30 is further configured to: detecting whether the number of the locked resources of each second matched resource is not less than the number of the actually used resources in the resource use reporting information;
the device further comprises:
and if so, converting the number of the locked resources in each second matching resource into the number of the used resources according to the number of the actually used resources.
Further, the receiving module 10 is further configured to: receiving a resource recovery request of the resource user, and matching each piece of third tag information in a third tag group carried by the resource recovery request with tag information associated with each preset resource to obtain a third matching resource successfully matched with the third tag group, wherein the third tag group and the first tag information group;
the second detection module 30 is further configured to: detecting whether the used resource quantity of each third matching resource is not less than the resource quantity in the resource recycling request;
the device further comprises:
and if so, recovering the used resource quantity in each third matching resource according to the recovered resource quantity.
Further, the detection module 30 is further configured to: detecting whether the used resource quantity of a target resource in the third matching resource, which has an association relation with the user tag information, is zero;
the device further comprises:
and the deleting module is used for deleting the target resource if the target resource is the same as the target resource.
Further, the apparatus further comprises:
a second adding module, configured to add a resource according to resource quantity information carried in a resource registration request, and associate each piece of fourth tag information carried in the resource registration request with the added resource, respectively, where the resource registration request is initiated by a resource provider or generated according to resource usage restriction information set for a resource user, and when the resource registration request is generated according to the resource usage restriction information, each piece of fourth tag information is generated according to a restriction condition in the resource usage restriction information, and the resource quantity information is generated according to a restricted resource quantity in the resource usage restriction information.
Further, the receiving module 10 is further configured to: receiving a resource offline request, and matching each piece of fifth label information carried by the resource offline request with label information associated with each preset resource to search resources to be offline in each preset resource, wherein the resources to be offline are resources associated with label information which is the same as the corresponding label information of each piece of fifth label information;
the device further comprises:
and the offline module is used for deleting the resource quantity information of the resource to be offline and the associated information between the resource to be offline and the label information if the resource to be offline is found, so as to perform offline on the resource to be offline.
The specific implementation of the resource management apparatus of the present invention is basically the same as the embodiments of the resource management method, and is not described herein again.
In addition, an embodiment of the present invention further provides a computer-readable storage medium, where a resource management program is stored on the storage medium, and when executed by a processor, the resource management program implements the steps of the resource management method as described below.
The embodiments of the resource management device and the computer-readable storage medium of the present invention can refer to the embodiments of the resource management method of the present invention, and are not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims (10)

1. A method for resource management, the method comprising the steps of:
receiving a resource application request of a resource user, and respectively matching each piece of first tag information in a first tag group carried by the resource application request with each piece of preset tag information to obtain a target tag number associated with the preset tag information successfully matched with the first tag information;
taking resources associated with the target tag number in all preset resources as candidate resources, and detecting whether all matchable preset tag information associated with the candidate resources has successfully matched tag information in the first tag group;
if the resource application requests exist, the candidate resources are used as first matching resources which are successfully matched with the first label group, and whether the number of usable resources of each first matching resource is not less than the number of application resources in the resource application requests is detected;
if yes, determining resource application of the resource user, and locking the available resource quantity of each first matched resource according to the applied resource quantity.
2. The resource management method according to claim 1, wherein the step of matching each piece of first tag information in the first tag group carried by the resource application request with each piece of preset tag information respectively to obtain a target tag number associated with the preset tag information that is successfully matched with the corresponding first tag information comprises:
extracting each tag value in each piece of first tag information in a first tag group carried by the resource application request respectively;
searching each label value in a preset quick query table, and acquiring a label number associated with the searched label value in the preset quick query table;
and if the label number is found in a preset label table, taking the label number as a target label number, wherein one line of record in the preset label table comprises a piece of preset label information and a label number associated with the preset label information, and one line of record in the preset fast query table comprises a label value in the piece of preset label information and a label number associated with the preset label information.
3. The method according to any one of claims 1 or 2, wherein said first tag group includes a user tag information for uniquely identifying said resource user, and said step of locking the number of resources available for each of said first matching resources according to said requested number of resources further comprises:
adding a new resource, and taking the number of the applied resources as the number of the locking resources of the newly added resource;
and respectively associating the newly added resource with each piece of first label information in the first label group.
4. The method for resource management as claimed in claim 3, wherein after the step of locking the number of usable resources of each of the first matching resources according to the number of requested resources, further comprising:
receiving resource use report information of the resource user, and matching each piece of second tag information in a second tag group carried by the resource use information with tag information associated with each preset resource to obtain a second matching resource successfully matched with the second tag group, wherein the second tag group is the same as the first tag group;
detecting whether the number of the locked resources of each second matched resource is not less than the number of the actually used resources in the resource use reporting information;
if yes, the number of the locking resources in each second matching resource is converted into the number of the used resources according to the number of the actually used resources.
5. The method for resource management according to claim 4, wherein after the step of converting the number of locked resources in each of the second matching resources into the number of used resources according to the number of actually used resources, further comprising:
receiving a resource recovery request of the resource user, and matching each piece of third tag information in a third tag group carried by the resource recovery request with tag information associated with each preset resource to obtain a third matching resource successfully matched with the third tag group, wherein the third tag group and the first tag information group;
detecting whether the used resource quantity of each third matching resource is not less than the resource quantity in the resource recycling request;
and if so, recycling the used resource quantity in each third matching resource according to the recycling resource quantity.
6. The resource management method according to claim 5, wherein if yes, after the step of recycling the used resource amount in each of the third matching resources according to the recycled resource amount, the method further comprises:
detecting whether the used resource quantity of a target resource in the third matching resource, which has an association relation with the user tag information, is zero;
and if so, deleting the target resource.
7. The method for resource management according to any of claims 1 or 2, wherein the method further comprises:
adding a resource according to resource quantity information carried by a resource registration request, and associating each piece of fourth label information carried by the resource registration request with the added resource respectively, wherein the resource registration request is initiated by a resource provider or generated according to resource use limitation information set by a resource user, and when the resource registration request is generated according to the resource use limitation information, each piece of fourth label information is generated according to limitation conditions in the resource use limitation information, and the resource quantity information is generated according to the limitation resource quantity in the resource use limitation information.
8. The method for resource management according to any of claims 1 or 2, wherein the method further comprises:
receiving a resource offline request, and matching each piece of fifth label information carried by the resource offline request with label information associated with each preset resource to search resources to be offline in each preset resource, wherein the resources to be offline are resources associated with label information which is the same as the corresponding label information of each piece of fifth label information;
and if the resource to be offline is found, deleting the resource quantity information of the resource to be offline and the associated information between the resource to be offline and the label information so as to offline the resource to be offline.
9. An apparatus for resource management, the apparatus comprising:
the receiving module is used for receiving a resource application request of a resource user, and matching each piece of first tag information in a first tag group carried by the resource application request with each piece of preset tag information respectively to obtain a target tag number associated with the preset tag information successfully matched with the first tag information;
the first detection module is used for taking resources related to the target tag number in all preset resources as candidate resources and detecting whether matched preset tag information related to the candidate resources exists in the first tag group or not;
a second detection module, configured to, if both of the candidate resources exist, use the candidate resource as a first matching resource successfully matched with the first tag group, and detect whether the number of available resources of each first matching resource is not less than the number of application resources in the resource application request;
and the determining module is used for determining the resource application of the resource user if the resource application is successful, and locking the available resource quantity of each first matched resource according to the applied resource quantity.
10. A resource management device, characterized in that the resource management device comprises: memory, a processor and a resource management program stored on the memory and executable on the processor, the resource management program when executed by the processor implementing the steps of the resource management method according to any of claims 1 to 8.
CN202110626971.7A 2021-06-04 2021-06-04 Resource management method, device and equipment Pending CN113360284A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202110626971.7A CN113360284A (en) 2021-06-04 2021-06-04 Resource management method, device and equipment
PCT/CN2021/132313 WO2022252510A1 (en) 2021-06-04 2021-11-23 Resource management method, apparatus and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110626971.7A CN113360284A (en) 2021-06-04 2021-06-04 Resource management method, device and equipment

Publications (1)

Publication Number Publication Date
CN113360284A true CN113360284A (en) 2021-09-07

Family

ID=77532397

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110626971.7A Pending CN113360284A (en) 2021-06-04 2021-06-04 Resource management method, device and equipment

Country Status (2)

Country Link
CN (1) CN113360284A (en)
WO (1) WO2022252510A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114666410A (en) * 2022-02-14 2022-06-24 深圳爱捷云科技有限公司 Resource calling method and device, electronic equipment and storage medium
WO2022252510A1 (en) * 2021-06-04 2022-12-08 深圳前海微众银行股份有限公司 Resource management method, apparatus and device
CN117009060A (en) * 2023-09-27 2023-11-07 腾讯科技(深圳)有限公司 Resource scheduling method, device, equipment and storage medium

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9832078B2 (en) * 2015-01-06 2017-11-28 Netapp, Inc. Stateless resource management
CN109150815B (en) * 2017-06-28 2021-11-23 阿里巴巴集团控股有限公司 Resource processing method, device and machine readable medium
CN109375992A (en) * 2018-08-17 2019-02-22 华为技术有限公司 A kind of resource regulating method and device
CN109213573A (en) * 2018-09-14 2019-01-15 珠海国芯云科技有限公司 The equipment blocking method and device of virtual desktop based on container
CN111221644B (en) * 2018-11-27 2023-06-13 阿里巴巴集团控股有限公司 Resource scheduling method, device and equipment
CN112651739A (en) * 2020-12-23 2021-04-13 微民保险代理有限公司 Resource transfer method, device, electronic equipment and storage medium
CN113360284A (en) * 2021-06-04 2021-09-07 深圳前海微众银行股份有限公司 Resource management method, device and equipment

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022252510A1 (en) * 2021-06-04 2022-12-08 深圳前海微众银行股份有限公司 Resource management method, apparatus and device
CN114666410A (en) * 2022-02-14 2022-06-24 深圳爱捷云科技有限公司 Resource calling method and device, electronic equipment and storage medium
CN117009060A (en) * 2023-09-27 2023-11-07 腾讯科技(深圳)有限公司 Resource scheduling method, device, equipment and storage medium
CN117009060B (en) * 2023-09-27 2024-01-12 腾讯科技(深圳)有限公司 Resource scheduling method, device, equipment and storage medium

Also Published As

Publication number Publication date
WO2022252510A1 (en) 2022-12-08

Similar Documents

Publication Publication Date Title
CN113360284A (en) Resource management method, device and equipment
CN106951430B (en) Account table query method and device
US10044522B1 (en) Tree-oriented configuration management service
US7269593B2 (en) Data processing apparatus and method
US10437584B2 (en) Web client command infrastructure integration into a rich client application
US9785687B2 (en) System and method for transparent multi key-value weighted attributed connection using uni-tag connection pools
WO2019154353A1 (en) System running parameter query method, matching method and apparatus, and node device
WO2019100635A1 (en) Editing method and apparatus for automated test script, terminal device and storage medium
US20090265301A1 (en) Database Object Update Order Determination
CN107515879B (en) Method and electronic equipment for document retrieval
JP5146020B2 (en) Information processing apparatus, resource identification program, and resource identification method
CN105893509A (en) Marking and explaining system and method for large-data analysis model
CN110134681B (en) Data storage and query method and device, computer equipment and storage medium
CN112000696B (en) Framework for mirror image list query of multi-framework cloud platform mirror image warehouse
US8290950B2 (en) Identifying locale-specific data based on a total ordering of supported locales
US11151088B2 (en) Systems and methods for verifying performance of a modification request in a database system
CN101211361A (en) Information processing apparatus, information processing system, and information processing method
CN113792274A (en) Information management method, management system and storage medium
CN110321246B (en) Fault processing method and device
CN107045466B (en) Service data auditing method, device and system
CN114816170A (en) Rule scheme generation method, device, medium and electronic equipment
US20090254896A1 (en) Simplifying Bundling Of Fixes Used To Resolve Errors In Pre-Installed Software
CN113448493B (en) Method, electronic device and computer readable medium for backing up data
US8793263B2 (en) Avoiding duplicate and equivalent data objects in service registries
CN112685474A (en) Application management method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination