CN116244063A

CN116244063A - Resource pool creation and data processing method and device

Info

Publication number: CN116244063A
Application number: CN202211621199.0A
Authority: CN
Inventors: 王高飞; 聂磊; 冷家冰
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-12-16
Filing date: 2022-12-16
Publication date: 2023-06-09

Abstract

The present disclosure provides a resource pool creation and data processing method. Relates to the technical fields of artificial intelligence such as intelligent industry, deep learning, computer vision and the like, and can be used for scenes such as industrial quality inspection and the like. The creation method comprises the following steps: acquiring resource information of machines in a machine cluster; dividing the resource information according to the resource category to obtain at least one resource pool; and splitting the resource information in at least one resource pool to obtain at least one piece of sub-resource information. The processing method comprises the following steps: determining a target resource pool according to the resource category of hardware resources required by processing the data to be processed; acquiring target sub-resource information from a target resource pool, and determining a target machine; transmitting the data to be processed and the target sub-resource information to a target machine; and receiving the returned data processing result. The processing method comprises the following steps: receiving data to be processed and target sub-resource information sent by a server; using a hardware resource corresponding to the target sub-resource information to run a target processing model to process the data to be processed; and sending the data processing result to the server.

Description

Resource pool creation and data processing method and device

Technical Field

The disclosure relates to the technical field of computers, in particular to the technical field of artificial intelligence such as cloud service, big data, deep learning and the like. Provided are a resource pool creation and data processing method, a device, an electronic device and a readable storage medium.

Background

With the rapid development of artificial intelligence technology, it is becoming more and more common to perform data quality inspection tasks, data detection tasks, data identification tasks and the like by using a neural network model. The prior art generally adopts a machine cluster comprising a plurality of machines to execute different tasks, and how to manage the machine cluster comprising the plurality of machines to better execute the tasks is a technical problem to be solved.

Disclosure of Invention

According to a first aspect of the present disclosure, there is provided a resource pool creation method including: acquiring resource information of machines in a machine cluster; dividing the resource information according to the resource category to obtain at least one resource pool; and dividing the resource information in the at least one resource pool to obtain at least one piece of sub-resource information, wherein different pieces of sub-resource information in the same resource pool have the same resource attribute.

According to a second aspect of the present disclosure, there is provided a data processing method comprising: acquiring data to be processed; determining a target resource pool according to the resource category of hardware resources required by processing the data to be processed; acquiring target sub-resource information from the target resource pool, and determining a target machine corresponding to the target sub-resource information; transmitting the data to be processed and the target sub-resource information to the target machine, so that the target machine uses a hardware resource corresponding to the target sub-resource information to run a target processing model to process the data to be processed; and receiving a data processing result returned by the target machine.

According to a third aspect of the present disclosure, there is provided a data processing method comprising: receiving data to be processed and target sub-resource information sent by a server; using a hardware resource operation target processing model corresponding to the target sub-resource information to process the data to be processed; and sending the data processing result of the data to be processed to the server.

According to a fourth aspect of the present disclosure, there is provided a resource pool creation apparatus including: the first acquisition unit is used for acquiring the resource information of the machines in the machine cluster; the dividing unit is used for dividing the resource information according to the resource category to obtain at least one resource pool; the splitting unit is used for splitting the resource information in the at least one resource pool to obtain at least one piece of sub-resource information, wherein different piece of sub-resource information in the same resource pool has the same resource attribute.

According to a fifth aspect of the present disclosure, there is provided a data processing apparatus comprising: the second acquisition unit is used for acquiring data to be processed; the determining unit is used for determining a target resource pool according to the resource category of the hardware resource required by processing the data to be processed; the first processing unit is used for acquiring target sub-resource information from the target resource pool and determining a target machine corresponding to the target sub-resource information; the first sending unit is used for sending the data to be processed and the target sub-resource information to the target machine so that the target machine can use a hardware resource corresponding to the target sub-resource information to run a target processing model to process the data to be processed; and the first receiving unit is used for receiving the data processing result returned by the target machine.

According to a sixth aspect of the present disclosure, there is provided a data processing apparatus comprising: the second receiving unit is used for receiving the data to be processed and the target sub-resource information sent by the server; the second processing unit is used for operating a target processing model to process the data to be processed by using the hardware resources corresponding to the target sub-resource information; and the third sending unit is used for sending the data processing result of the data to be processed to the server.

According to a seventh aspect of the present disclosure, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described above.

According to an eighth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method as described above.

According to a ninth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a method as described above.

According to the technical scheme, the management of the hardware resources of the machine cluster can be facilitated, so that the allocation flexibility of the hardware resources is improved, the utilization rate of the hardware resources is improved, and the allocation accuracy of the hardware resources and the data processing efficiency in data processing are improved.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure;

FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure;

FIG. 3 is a schematic diagram according to a third embodiment of the present disclosure;

FIG. 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

FIG. 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

FIG. 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

FIG. 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

FIG. 8 is a schematic diagram according to an eighth embodiment of the present disclosure;

FIG. 9 is a schematic diagram according to a ninth embodiment of the present disclosure;

FIG. 10 is a schematic diagram according to a tenth embodiment of the present disclosure;

FIG. 11 is a schematic illustration according to an eleventh embodiment of the present disclosure;

fig. 12 is a block diagram of an electronic device used to implement a resource pool creation or data processing method of an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a schematic diagram according to a first embodiment of the present disclosure. As shown in fig. 1, the resource pool creation method of the present embodiment is applied to a server, and specifically includes the following steps:

s101, acquiring resource information of machines in a machine cluster;

s102, dividing the resource information according to the resource category to obtain at least one resource pool;

s103, splitting the resource information in the at least one resource pool to obtain at least one piece of sub-resource information, wherein different piece of sub-resource information in the same resource pool has the same resource attribute.

According to the resource pool creation method, the hardware resources which can be provided by the machines in the machine cluster are divided based on the resource types, the purpose of isolating different hardware resources is achieved, and the sources of the hardware resources are not needed to be considered, so that the hardware resources of the machine cluster are convenient to manage, the resource information in the same resource pool is further divided into the sub-resource information with the same resource attribute, the allocation flexibility of the hardware resources can be improved, and the utilization rate of the hardware resources is further improved.

The execution main body of the resource pool creation method of the embodiment is a server for controlling the machine cluster to run; the machine cluster includes a plurality of machines, each machine capable of providing a different class of hardware resources, the machines processing data using respective hardware resources to run a process model, e.g., to run a respective process model for data quality inspection, data identification, etc.

The resource information obtained in the step S101 corresponds to a hardware resource that can be provided by a machine, and may include resource category information, resource identification information, resource attribute information, and the like; the resource category information is used for representing resource categories of hardware resources, such as AI computing power resources, CPU computing power resources, memory resources, network IO resources, disk IO resources and the like; the resource identification information is used to represent a machine that provides the hardware resource, such as a name of the machine, a location of the machine, etc.; the resource attribute information is used to indicate the size, number, etc. of the hardware resources, for example, how many GB the memory contains, the number of CPUs, etc.

In the embodiment, after executing S101 to obtain resource information, executing S102 to divide the resource information according to the resource category to obtain at least one resource pool; wherein different resource pools correspond to different resource categories.

In the embodiment, when executing S102, the resource information belonging to the same resource category may be divided into one resource pool according to the resource category information, so as to obtain at least one resource pool, where different resource pools include resource information corresponding to different resource categories.

The embodiment may further include the following after executing S102 to obtain at least one resource pool: calculating affinities between resource information in at least one resource pool; dividing resource information in at least one resource pool according to the calculation result of the affinity to obtain at least one sub-resource pool; the resource information in the same sub-resource pool may have the same resource identification information or the same resource attribute information.

In this embodiment, when executing S102 to calculate affinities between resource information in at least one resource pool, affinities between resource information may be calculated according to resource attribute information and/or resource identification information; the larger the calculation result of affinity, the greater the probability that two resource information in the resource pool have the same resource attribute or the same resource identification.

That is, in this embodiment, by dividing the resource information in the resource pools twice, each resource pool includes at least one sub-resource pool, and the resource information in the same sub-resource pool has a higher affinity (for example, is located in the same machine or has the same attribute, etc.), so that the accuracy of dividing the resource information is improved, and the accuracy in allocating the hardware resources can be correspondingly improved.

In this embodiment, after executing S102 to obtain at least one resource pool, executing S103 to segment resource information in the at least one resource pool to obtain at least one sub-resource information; wherein different sub-resource information located in the same resource pool (or in the same sub-resource pool) has the same resource attribute.

In this embodiment, when executing S103 to segment the resource information in at least one resource pool to obtain at least one sub-resource information, the optional manner may be: determining a resource category corresponding to the at least one resource pool; and splitting the resource information in at least one resource pool by using a splitting mode corresponding to the determined resource category to obtain at least one piece of sub-resource information.

It can be understood that, compared with the resource information from which the sub-resource information is obtained in S103, the sub-resource information obtained in this embodiment only changes the corresponding resource attribute (for example, if the resource attribute corresponding to the resource information is 100GB, the resource attribute corresponding to the sub-resource information may be 20 GB), and does not change the resource category information and the resource identification information.

In this embodiment, when executing S103 to segment the resource information using the segmentation method corresponding to the resource category, the resource information may be segmented into a plurality of sub-resource information having the same size (e.g., memory size), or the resource information may be segmented into a plurality of sub-resource information having the same number (e.g., CPU number).

That is, in this embodiment, by splitting the resource information in the resource pool, the information contained in the resource pool has smaller granularity, and sub-resource information with smaller granularity is more convenient for allocation of hardware resources, so that allocation flexibility of hardware resources can be enhanced, and utilization rate of hardware resources can be improved.

The present embodiment may further include the following after S103 is executed: acquiring resource information of machines added into a machine cluster as first resource information; determining a first resource category of the first resource information; the first resource information is added to a resource pool corresponding to the first resource category.

The present embodiment may further include the following after S103 is executed: acquiring resource information of machines leaving the machine cluster as second resource information; determining a second resource category of the second resource information; and deleting the second resource information in the resource pool corresponding to the second resource category.

That is, the method of creating the resource pool according to the resource types achieves the purpose of adding resource information to the resource pool or deleting resource information from the resource pool through different resource types, and improves the management convenience of the resource pool, so that hardware resources provided by newly added or newly removed machines can be managed more conveniently.

In this embodiment, when the first resource information is added to the resource pool of the first resource class, affinity calculation with other resource information may also be performed, so that the first resource information is added to the corresponding sub-resource pool, and after the first resource information is added to the resource pool, the first resource information may be further segmented, where a specific process is as described above and will not be described herein.

Fig. 2 is a schematic diagram according to a second embodiment of the present disclosure. As shown in fig. 2, the data processing method of the present embodiment is applied to a server, and specifically includes the following steps:

s201, acquiring data to be processed;

s202, determining a target resource pool according to the resource category of hardware resources required by processing the data to be processed;

s203, acquiring target sub-resource information from the target resource pool, and determining a target machine corresponding to the target sub-resource information;

S204, the data to be processed and the target sub-resource information are sent to the target machine, so that the target machine uses a hardware resource operation target processing model corresponding to the target sub-resource information to process the data to be processed;

s205, receiving a data processing result returned by the target machine.

According to the data processing method, after the target resource pool is determined according to the resource type of the hardware resource required for processing the data to be processed, target sub-resource information is acquired from the target resource pool, a target machine is determined, the data to be processed and the target sub-resource information are further sent to the target machine, and a data processing result returned by the target machine is received.

The execution main body of the data processing method of the embodiment is a server for controlling the operation of the machine cluster; the machine cluster comprises a plurality of machines, each machine is capable of providing a different class of hardware resources, through which the machine performs a corresponding data processing task, such as running a corresponding neural network model for data quality inspection, data identification, etc.

In the embodiment, when S201 is executed to acquire data to be processed, the data input by the input end may be used as the data to be processed, or the data selected by the input end on the network may be used as the data to be processed; the data type of the data to be processed is not limited in this embodiment, and may be text data, picture data, audio data, and the like.

After the step S201 is executed to obtain the data to be processed, the data information of the data to be processed may also be obtained, where the data information may be used to determine hardware resources required for processing the data to be processed, and may also be used to determine a processing model to be operated by a machine; the embodiment executes the data information acquired in S201, including the data type, the data number, the task information, and the like of the data to be processed.

The task information obtained in S201 may include task types, such as a picture quality inspection task, a text recognition task, an audio recognition task, a picture detection task, and the like, and may also include a Completion Time (CT) of the task; the acquired task information can be used for determining the resource category on the one hand, and can be used for determining a processing model to be operated by a machine on the other hand, if the task information is a picture quality inspection task, the machine can operate the quality inspection model to inspect the data to be processed, and if the task information is a text recognition task, the machine can operate the text recognition model to recognize the data to be processed.

That is, in the case that different processing models can be run in the machine, by acquiring the data information of the data to be processed, the machine runs the processing model corresponding to the data information, so that the waste of hardware resources can be avoided, and the processing accuracy of the data can be improved.

It may be understood that the data to be processed acquired in the step S201 may be divided according to data groups, where each data group corresponds to different task information, that is, the data to be processed corresponding to different task information may be acquired in the step S201.

In this embodiment, after executing S101 to obtain the data to be processed, executing S202 determines the target resource pool according to the resource category of the hardware resource required for processing the data to be processed.

In this embodiment, when executing S202, determining the target resource pool according to the resource category of the hardware resource required for processing the data to be processed, the optional implementation manner may be: according to the data information of the data to be processed, determining the resource category of hardware resources required for processing the data to be processed, for example, the resource category corresponding to the picture data is GPU computing resources, the resource category corresponding to the text recognition task is memory resources and the like; and taking the resource pool corresponding to the determined resource category as a target resource pool.

Because different resource pools correspond to different resource categories in the embodiment, the purpose of determining the target resource pool from the resource pools can be achieved after the resource categories corresponding to the data to be processed are determined, so that the determination steps of the target resource pool are simplified, and the determination efficiency of the target resource pool is improved.

After determining the target resource pool in S202, the present embodiment executes S203 to acquire target sub-resource information from the target resource pool and determine a target machine corresponding to the target sub-resource information.

In the embodiment, when S203 is executed to acquire the target sub-resource information from the target resource pool, the following optional implementation manners may be adopted: determining a target sub-resource pool from the target resource pool according to the data information of the data to be processed; the target sub-resource information is obtained from the determined target sub-resource pool, and the determined target sub-resource information can meet the hardware resources required for processing the data to be processed.

In this embodiment, when determining the target sub-resource pool from the target resource pool according to the data to be processed in S203, the sub-resource pool matched with the data information of the data to be processed in at least one sub-resource pool may be used as the target sub-resource pool, for example, the data to be processed needs to be processed by using the CPU computing power resource of 5GHz, and the sub-resource pool corresponding to the CPU computing power resource of 5GHz is used as the target sub-resource pool in S203.

That is, in this embodiment, by acquiring the target sub-resource information from the target sub-resource pool, it is ensured that multiple target sub-resource information have the same resource identifier or resource attribute, so that the accuracy of allocation of hardware resources is improved, and the data to be processed is processed by the hardware resources having the same resource identifier or resource attribute, so that the processing efficiency of the data is further improved.

In the embodiment, when S203 is executed to acquire the target sub-resource information from the target resource pool, the following optional implementation manners may be adopted: determining a resource allocation model corresponding to the target resource pool, wherein different resource pools correspond to different resource allocation models; acquiring resource pool information of a target resource pool, running information of a machine cluster and data information of data to be processed; inputting the acquired resource pool information, operation information and data information into a resource allocation model; and acquiring target sub-resource information from the target resource pool according to the sub-resource information prediction result output by the resource allocation model.

The resource pool information obtained in S203 may be information such as sub-resource information contained in the target resource pool, the number of resources corresponding to the sub-resource information, affinity between the sub-resource information, and resource capacity corresponding to the sub-resource information; if the target resource pool in the present embodiment is a sub-resource pool, the resource pool information acquired in S203 may not include affinities between sub-resource information, where the sub-resource information, the number of resources and the capacity of resources corresponding to the sub-resource information, and the like only correspond to the target sub-resource pool.

The execution of the operation information obtained in S203 in this embodiment may be information such as resource utilization rate of a machine cluster, number of waiting machines, number of operating machines, and time consumption of operation of the machines; the data information acquired in S203 may be information such as task CT, task type, and data amount of the data to be processed.

It can be understood that the resource allocation model used in executing S203 in this embodiment is obtained by training in advance, and the resource allocation model can output the sub-resource information prediction result corresponding to the data to be processed according to the input resource pool information, the operation information and the data information.

When the resource allocation model is a reinforcement learning model, in the embodiment, when S203 is executed, the resource allocation benefit is obtained, and then the resource allocation benefit and the three kinds of information are input into the resource allocation model together, so as to obtain a sub-resource information prediction result output by the resource allocation model; the resource allocation benefit in this embodiment may be preset, and the data to be processed with different data information may correspond to different resource allocation benefits.

In addition, in this embodiment, after executing S203 to obtain the target sub-resource information from the target resource pool according to the sub-resource information prediction result output by the resource allocation model, the actual allocation benefit of the resource allocation model during this allocation may be further obtained, and then the obtained actual allocation benefit and the sub-resource information prediction result are used to update the resource allocation model, so as to continuously improve the accuracy of the prediction result output by the resource allocation model.

That is, in this embodiment, through the resource allocation model corresponding to different target resource pools (the target resource pool may be a resource pool or a sub-resource pool in the resource pool), corresponding hardware resources are allocated to the data to be processed, that is, target sub-resource information corresponding to the data to be processed is determined, so that the machine cluster can operate with optimal performance based on the allocation result, and the accuracy of resource allocation is further improved.

After the target sub-resource information is acquired in S203, the embodiment may determine the target machine according to the resource identification information in the target sub-resource information, such as the machine name or the machine location.

After determining the target machine in S203, the present embodiment executes S204 to send the data to be processed and the target sub-resource information to the target machine, so that the target machine uses the hardware resource corresponding to the target sub-resource information to run the target processing model to process the data to be processed.

In the case that the target machine includes multiple processing models, the present embodiment may further send the data information of the data to be processed to the target machine together when executing S204, so that the target machine executes the target processing model to process the data to be processed using the hardware resource corresponding to the target sub-resource information after determining the target processing model according to the data information.

That is, in this embodiment, by sending the target sub-resource information to the target machine, the target machine operates the target processing model corresponding to the data information according to the hardware resource corresponding to the resource attribute information in the target sub-resource information, so as to process the data to be processed, thereby avoiding the waste of the hardware resource and improving the processing efficiency of the data.

After executing S204 to send the data to be processed and the target sub-resource information to the target machine, the embodiment executes S205 to receive the data processing result returned by the target machine.

The data processing result received in S205 may include quality data of the data to be processed, including feature vectors and/or quality scores, in addition to task execution results (e.g., quality inspection results, identification results, etc.) of the data to be processed.

After executing S205 to receive the data processing result, the embodiment may further send the task execution result in the received data processing result to the input end, so as to display the data processing result in the input end, for example, display a picture quality inspection result, a text recognition result, and the like.

Fig. 3 is a schematic diagram according to a third embodiment of the present disclosure. As shown in fig. 3, after executing S205 "receive data processing result returned by the target machine", the present embodiment may further include the following steps:

S301, acquiring the resource utilization rate of the target resource pool;

s302, selecting target data from the data to be processed under the condition that the resource utilization rate meets the preset utilization rate requirement;

s303, sending the target data and the data processing result of the target data to the target machine, so that the target machine updates the target processing model according to the target data and the data processing result of the target data.

That is, in this embodiment, when it is determined that the resource usage rate of the target resource pool meets the preset usage rate requirement, the target processing model is updated in a data backflow manner, so that the problem that the processing performance of the processing model is poor due to a small amount of data in the early stage can be solved, and the accuracy of the data processing result output by the target processing model is further improved.

In the present embodiment, when S301 is executed, the ratio of the currently allocated sub-resource information to the total sub-resource information included in the target resource pool may be used as the resource usage rate.

In the embodiment, when executing S302, it may be determined that the resource usage rate meets the preset usage rate requirement under the condition that it is determined that the resource usage rate is less than or equal to the preset usage rate threshold.

In the present embodiment, when executing S302 to select target data from data to be processed, the following manner may be adopted: obtaining a quality score of the data to be processed from a data processing result of the data to be processed; and selecting target data from the data to be processed according to the quality score of the data to be processed.

In the embodiment, when S302 is executed and the target data is selected from the data to be processed according to the quality score of the data to be processed, the data to be processed with the quality score lower than or equal to the preset score threshold value can be used as the target data; the data to be processed may be classified according to the quality score, and the data to be processed with the classification result being a preset data category may be used as target data, for example, the picture may be classified into an OK category or an NG category, and then the picture belonging to the NG category may be used as target data.

In the embodiment, when executing S302 to select the target data from the data to be processed, the following manner may be adopted: acquiring a feature vector of the data to be processed from a data processing result of the data to be processed; calculating the similarity between the feature vector and a preset feature vector; and taking the data to be processed, of which the similarity calculation result exceeds a preset similarity threshold value, as target data.

After executing S302 to send the target data and the data processing result of the target data to the target machine, the target machine performs iterative training on the target processing model according to the received target data and the data processing result, so as to complete updating of the target processing model.

Fig. 4 is a schematic diagram according to a fourth embodiment of the present disclosure. As shown in fig. 4, the data processing method of the present embodiment is applied to machines in a machine cluster, and specifically includes the following steps:

s401, receiving data to be processed and target sub-resource information sent by a server;

s402, using a hardware resource operation target processing model corresponding to the target sub-resource information to process the data to be processed;

s403, sending the data processing result of the data to be processed to the server.

The execution main body of the data processing method in this embodiment is a machine located in a machine cluster, after receiving the data to be processed and the target sub-resource information sent by the server, the target processing model is operated to process the data to be processed by using the hardware resource corresponding to the target sub-resource information, and finally the data processing result is sent to the server.

In executing S401, the present embodiment may receive data information transmitted by the server in addition to the data to be processed and the target sub-resource information, and the data information may be used to determine a target processing model in the processing models.

The embodiment executes the target processing model used in S402, which is composed of a backbone network, a task head network and a quality head network; the main network is used for extracting characteristics of input data; the task head network is used for outputting task execution results (such as quality inspection results, identification results, detection results and the like) according to the characteristics extracted by the backbone network; the quality head network is used for outputting quality scores and/or feature vectors according to the features extracted by the backbone network.

The target processing model used in the execution of S402 in this embodiment may be obtained by training in the following manner: acquiring a training set, wherein the training set comprises a plurality of training samples and task labeling results of the plurality of training samples; according to the task labeling result of the training sample, quality labeling data of the training sample, such as quality scores or feature vectors, are obtained; constructing a neural network model comprising a backbone network, a task head network and a quality head network, wherein the backbone network and the task head network form an initial network, and the initial network can output a task execution result according to input data; inputting the training sample into a neural network model to obtain quality prediction data output by the neural network model; calculating a loss function value according to the quality labeling data and the quality prediction data of the training sample; and adjusting parameters of the quality head network according to the calculated loss function value until the quality head network converges to obtain a target processing model.

That is, in this embodiment, the training set for training the target processing model is automatically created by the existing training set, so that the training cost of the neural network model including the quality head network is reduced, and only the parameters of the quality head network are updated, so that the training speed of the target processing model can be improved.

In the embodiment, when the step S402 is executed to obtain quality labeling data according to the task labeling result of the training sample, a quality score can be obtained according to the task labeling result, and the quality score is used as the quality labeling data; the training samples can be classified according to the quality scores, and classification categories of the training samples are used as quality labeling data.

In this embodiment, when executing S402 to obtain the quality score according to the task labeling result, the following calculation formula may be adopted:

in the formula: score represents a quality score; a, a _i Representing the weight coefficient; c _i And representing factor information, wherein the factor information is a task labeling result of a training sample, and taking a quality inspection task as an example, the factor information can be the defect number, defect category, defect area, defect position and the like of the picture, and different factor information corresponds to different weight coefficients.

It can be understood that, if the quality score obtained according to the above formula in this embodiment classifies the training samples, a larger difference between the weight coefficients corresponding to different factor information may be set to better classify the training samples.

The present embodiment executes the data processing result transmitted to the server in S403, and includes quality data in addition to the task execution result.

The present embodiment may further include the following after executing S403 to send the data processing result of the data to be processed to the server: receiving target data and a data processing result of the target data sent by a server; and updating the target processing model according to the target data and the data processing result of the target data.

In this embodiment, when S403 is executed to update the target processing model according to the target data and the data processing result of the target data, the optional implementation manner may be: inputting the target data into a target processing model to obtain a task execution prediction result and quality prediction data which are output by the target processing model; calculating a loss function value according to the data processing result, the task execution prediction result and the quality prediction data; and adjusting parameters of the target processing model according to the calculated loss function value.

When the loss function value is calculated in the execution 403, the embodiment may calculate a first loss function value between the task execution result and the task execution prediction result in the data processing result and a second loss function value between the quality data and the quality prediction data in the data processing result, and adjust the parameters of the target processing model according to the first loss function value and the second loss function value.

That is, the machine of this embodiment may update the target processing model according to the target data returned by the server, so that the target processing model has more and more training samples for training, thereby improving accuracy of the target processing model when processing the data.

It may be understood that, the machine cluster where the machine of the embodiment is located manages the machine in a decentralizing manner, and each machine stores topology information of other machines (for example, processing model information that can be run by other machines, hardware resource information that can be provided by other machines, etc.) in a broadcasting manner; when a change (addition or deletion) occurs to a machine in the machine cluster, the topology information stored by each machine is updated again in a broadcast manner.

According to the embodiment, the machine is managed in a decentralizing mode, when one or more machines fail, the data processed by the failed machine is transferred to other normal machines for processing, so that the disaster tolerance capability of the machine cluster is enhanced, and the stability of data processing is improved.

Fig. 5 is a schematic diagram according to a fifth embodiment of the present disclosure. A system diagram of resource pool creation and data processing is shown in fig. 5, and in fig. 5, quality inspection of circuit boards produced by a production line is illustrated as an example: the left side is a picture of the circuit board produced by different production lines; the machine cluster comprises machines 1 to n, and different machines provide different hardware resources, such as AI computing resources, CPU computing resources, memory resources and the like; dividing the resource information of the machine into a plurality of resource pools according to the resource category, such as a resource pool 1 corresponding to AI computing power resources, a resource pool 2 corresponding to CPU computing power resources, a resource pool 3 corresponding to memory resources, a resource pool 4 corresponding to network IO resources and a resource pool 5 corresponding to disk IO resources; managing resource pools of different resource categories (for example, adding resource information to the resource pools or deleting the resource information) through a resource management module; corresponding hardware resources are allocated for processing the data to be processed through a reinforcement learning module (such as a resource allocation model); the computing module is used for running a target processing model (different target processing models correspond to different service functions) according to the allocated hardware resources by the machine to process data to be processed (for example, running a picture quality inspection model to inspect the quality of a circuit board picture); the function management module is used for determining whether to start a data reflow function according to the utilization rate of the resource pool so as to update a target processing model in the machine.

Fig. 6 is a schematic diagram according to a sixth embodiment of the present disclosure. Fig. 6 shows a structure diagram of a target processing model in the present embodiment: the target processing model consists of an original model and a quality head network, wherein the original model consists of a backbone network and a task head network, and the original model can output a task execution result according to input data to be processed; the quality head network in the target processing model can output quality scores and feature vectors according to the input data to be processed.

Fig. 7 is a schematic diagram according to a seventh embodiment of the present disclosure. A flow chart of resource pool creation in this embodiment is shown in fig. 7: the hardware layer is a machine contained in a machine cluster, for example, a machine 1 to a machine n; the hardware resource layer is the resource information of the hardware resource provided by each machine; the uniform resource pool layer is a resource pool obtained by dividing the resource information of hardware resources provided by the machine according to the resource category; a resource pool consisting of resource information corresponding to resource class 1 provided by different machines is shown in fig. 7; the embodiment may further segment the resource information in the resource pool, and each small square represents sub-resource information with the same attribute (e.g., the same size).

Fig. 8 is a schematic diagram according to an eighth embodiment of the present disclosure. A schematic diagram of this embodiment when performing resource allocation is shown in fig. 8: the resource requester and the resource daemon are used for representing different processes of the server when data processing is performed, and a target resource pool corresponds to one resource daemon process; the resource requester process is used for sending data to be processed to the server and determining a target resource pool; after determining the target resource pool, the server allocates hardware resources for the data to be processed by using a resource daemon corresponding to the target resource pool, namely, acquires target sub-resource information, and the resource daemon can run a resource allocation model to acquire the target sub-resource information; wherein, the blocks with larger gray values represent the allocated sub-resource information, and the blocks with smaller gray values represent the unallocated sub-resource information.

Fig. 9 is a schematic diagram according to a ninth embodiment of the present disclosure. As shown in fig. 9, the resource pool creation device 900 of the present embodiment is applied to a server, and includes:

the first obtaining unit 901 is configured to obtain resource information of a machine in the machine cluster;

a dividing unit 902, configured to divide the resource information according to a resource class, so as to obtain at least one resource pool;

The splitting unit 903 is configured to split the resource information in the at least one resource pool to obtain at least one sub-resource information, where different sub-resource information in the same resource pool has the same resource attribute.

The resource information acquired by the first acquiring unit 901 corresponds to a hardware resource that can be provided by a machine, and may include resource category information, resource identification information, resource attribute information, and the like.

In this embodiment, after the first obtaining unit 901 obtains the resource information, the dividing unit 902 divides the resource information according to the resource category to obtain at least one resource pool; wherein different resource pools correspond to different resource categories.

The dividing unit 902 may divide the resource information belonging to the same resource category into one resource pool according to the resource category information, thereby obtaining at least one resource pool, where different resource pools contain resource information corresponding to different resource categories.

After the partitioning unit 902 obtains at least one resource pool, the following may be further included: calculating affinities between resource information in at least one resource pool; dividing resource information in at least one resource pool according to the calculation result of the affinity to obtain at least one sub-resource pool; the resource information in the same sub-resource pool may have the same resource identification information or the same resource attribute information.

When the dividing unit 902 calculates affinities between the resource information in at least one resource pool, the affinities between the resource information may be calculated according to the resource attribute information and/or the resource identification information; the larger the calculation result of affinity, the greater the probability that two resource information have the same resource attribute or the same resource identity.

That is, the dividing unit 902 divides the resource information in the resource pools twice, so that each resource pool includes at least one sub-resource pool, and the resource information in the same sub-resource pool has a higher affinity (for example, is located in the same machine or has the same attribute, etc.), thereby improving the accuracy of dividing the resource information and correspondingly improving the accuracy when allocating the hardware resources.

In this embodiment, after at least one resource pool is obtained by the dividing unit 902, the splitting unit 903 splits the resource information in the at least one resource pool to obtain at least one sub-resource information; wherein different sub-resource information located in the same resource pool (or in the same sub-resource pool) has the same resource attribute.

The splitting unit 903 may split the resource information in at least one resource pool to obtain at least one sub-resource information in the following optional manners: determining a resource category corresponding to the at least one resource pool; and splitting the resource information in at least one resource pool by using a splitting mode corresponding to the determined resource category to obtain at least one piece of sub-resource information.

It can be understood that, compared with the resource information from which the sub-resource information is obtained, the sub-resource information obtained by the splitting unit 903 only changes the corresponding resource attribute, and does not change the resource category information and the resource identification information.

When the splitting unit 903 splits the resource information by using the splitting manner corresponding to the resource category, the resource information may be split into a plurality of sub-resource information having the same size, or the resource information may be split into a plurality of sub-resource information having the same number.

That is, the splitting unit 903 splits the resource information in the resource pool, so that the information contained in the resource pool has smaller granularity, and the sub-resource information with smaller granularity is more convenient for the allocation of the hardware resources, thereby enhancing the flexibility of the allocation of the hardware resources and improving the utilization rate of the hardware resources.

The resource pool creation apparatus 900 of the present embodiment may further include an adding unit 904 configured to perform: acquiring resource information of machines added into a machine cluster as first resource information; determining a first resource category of the first resource information; the first resource information is added to a resource pool corresponding to the first resource category.

The resource pool creation apparatus 900 of the present embodiment may further include a deletion unit 905 for performing: acquiring resource information of machines leaving the machine cluster as second resource information; determining a second resource category of the second resource information; and deleting the second resource information in the resource pool corresponding to the second resource category.

The adding unit 904 may further perform affinity calculation with other resource information when adding the first resource information to the resource pool of the first resource class, so as to add the first resource information to the corresponding sub-resource pool, and may further segment the first resource information after adding the first resource information to the resource pool, as described above, which is not described herein.

Fig. 10 is a schematic diagram according to a tenth embodiment of the present disclosure. As shown in fig. 10, the data processing apparatus 1000 of the present embodiment is applied to a server, and includes:

A second acquiring unit 1001 configured to acquire data to be processed;

a determining unit 1002, configured to determine a target resource pool according to a resource class of a hardware resource required for processing the data to be processed;

a first processing unit 1003, configured to acquire target sub-resource information from the target resource pool, and determine a target machine corresponding to the target sub-resource information;

a first sending unit 1004, configured to send the data to be processed and the target sub-resource information to the target machine, so that the target machine uses a hardware resource corresponding to the sub-target resource information to run a target processing model to process the data to be processed;

the first receiving unit 1005 is configured to receive a data processing result returned by the target machine.

When the second obtaining unit 1001 obtains data to be processed, the data input by the input end may be used as the data to be processed, or the data selected by the input end on the network may be used as the data to be processed; the data type of the data to be processed is not limited in this embodiment, and may be text data, picture data, audio data, and the like.

The second obtaining unit 1001 may also obtain data information of the data to be processed after obtaining the data to be processed, where the data information may be used to determine hardware resources required for processing the data to be processed, may also be used to determine a processing model to be run by a machine, and the like; the embodiment executes the data information acquired in S201, including the data type, the data number, the task information, and the like of the data to be processed.

The task information acquired by the second acquiring unit 1001 may include a task type, for example, a picture quality inspection task, a text recognition task, an audio recognition task, a picture detection task, and the like, and may further include a completion time of the task; the acquired task information can be used on the one hand for determining the resource class and on the other hand also for machine determination of the process model to be run.

That is, in the case where the second obtaining unit 1001 can run different processing models in the machine, by obtaining the data information of the data to be processed, the machine is made to run the processing model corresponding to the data information, so that the waste of hardware resources can be avoided, and the processing accuracy of the data can be improved.

It may be understood that the data to be processed acquired by the second acquiring unit 1001 may be divided in data groups, each data group corresponding to different task information, i.e., the second acquiring unit 1001 may acquire the data to be processed corresponding to different task information.

In the present embodiment, after the data to be processed is acquired by the second acquisition unit 1001, the target resource pool is determined by the determination unit 1002 according to the resource class of the hardware resource required for processing the data to be processed.

When determining the target resource pool according to the resource category of the hardware resource required for processing the data to be processed, the determining unit 1002 may adopt the following alternative implementation manners: determining the resource category of hardware resources required for processing the data to be processed according to the data information of the data to be processed; and taking the resource pool corresponding to the determined resource category as a target resource pool.

Since different resource pools correspond to different resource categories, the determining unit 1002 can achieve the purpose of determining the target resource pool from the resource pools after determining the resource category corresponding to the data to be processed, thereby simplifying the determining step of the target resource pool and improving the determining efficiency of the target resource pool.

After the determination unit 1002 determines the target resource pool, the first processing unit 1003 acquires target sub-resource information from the target resource pool, and determines the target machine corresponding to the target sub-resource information.

When the first processing unit 1003 obtains the target sub-resource information from the target resource pool, alternative implementation manners may be: determining a target sub-resource pool from the target resource pool according to the data information of the data to be processed; and acquiring target sub-resource information from the determined target sub-resource pool.

When determining the target sub-resource pool from the target resource pool according to the data to be processed, the first processing unit 1003 may use, as the target sub-resource pool, a sub-resource pool that matches with the data information of the data to be processed in at least one sub-resource pool.

That is, the first processing unit 1003 can ensure that the plurality of target sub-resource information has the same resource identifier or resource attribute by acquiring the target sub-resource information from the target sub-resource pool, thereby improving the accuracy of allocation of the hardware resource, and further improving the processing efficiency of the data by processing the data to be processed by the hardware resource having the same resource identifier or resource attribute.

When the first processing unit 1003 obtains the target sub-resource information from the target resource pool, alternative implementation manners may be: determining a resource allocation model corresponding to the target resource pool; acquiring resource pool information of a target resource pool, running information of a machine cluster and data information of data to be processed; inputting the acquired resource pool information, operation information and data information into a resource allocation model; and acquiring target sub-resource information from the target resource pool according to the sub-resource information prediction result output by the resource allocation model.

The resource pool information acquired by the first processing unit 1003 may be information such as sub-resource information included in the target resource pool, the number of resources corresponding to the sub-resource information, affinity between the sub-resource information, and resource capacity corresponding to the sub-resource information; if the target resource pool in the present embodiment is a sub-resource pool, the resource pool information acquired by the first processing unit 1003 may not include affinities between sub-resource information, where the sub-resource information, the number of resources and the resource capacity corresponding to the sub-resource information, and the like only correspond to the target sub-resource pool.

The operation information acquired by the first processing unit 1003 may be information such as resource utilization rate of a machine cluster, the number of waiting machines, the number of operating machines, and time consumption of operation of the machines; the data information acquired by the first processing unit 1003 may be information such as task CT, task type, and data amount of the data to be processed.

It may be appreciated that the resource allocation model used by the first processing unit 1003 is pre-trained, and the resource allocation model may output a sub-resource information prediction result corresponding to the data to be processed according to the input resource pool information, the operation information and the data information.

In the case that the resource allocation model is a reinforcement learning model, the first processing unit 1003 may further acquire resource allocation benefits, and further input the resource allocation benefits and the three kinds of information into the resource allocation model together, so as to obtain a sub-resource information prediction result output by the resource allocation model; the resource allocation benefit in this embodiment may be preset, and the data to be processed with different data information may correspond to different resource allocation benefits.

In addition, after obtaining the target sub-resource information from the target resource pool according to the sub-resource information prediction result output by the resource allocation model, the first processing unit 1003 may further obtain the actual allocation benefit of the resource allocation model during the current allocation, and further update the resource allocation model by using the obtained actual allocation benefit and the sub-resource information prediction result, so as to continuously improve the accuracy of the prediction result output by the resource allocation model.

That is, the first processing unit 1003 allocates corresponding hardware resources for the data to be processed through the resource allocation model corresponding to different target resource pools (the target resource pool may be a resource pool or a sub-resource pool in the resource pool), that is, determines target sub-resource information corresponding to the data to be processed, so that the machine cluster can operate with optimal performance based on the allocation result, and further improves accuracy of resource allocation.

After acquiring the target sub-resource information, the first processing unit 1003 may determine the target machine according to the resource identification information in the target sub-resource information, such as the machine name or the machine location.

The present embodiment, after the target machine is determined by the first processing unit 1003, transmits the data to be processed and the target sub-resource information to the target machine by the first transmitting unit 1004 so that the target machine runs the target processing model to process the data to be processed using the hardware resources corresponding to the target sub-resource information.

In the case that the target machine includes multiple processing models, the first sending unit 1004 may also send the data information of the data to be processed to the target machine together, so that the target machine may use the hardware resource corresponding to the target sub-resource information to run the target processing model to process the data to be processed after determining the target processing model according to the data information.

That is, the first sending unit 1004 sends the target sub-resource information to the target machine, so that the target machine runs the target processing model corresponding to the data information according to the hardware resource corresponding to the resource attribute information in the target sub-resource information, so as to process the data to be processed, avoid the waste of the hardware resource, and improve the processing efficiency of the data.

In this embodiment, after the first sending unit 1004 sends the data to be processed and the target sub-resource information to the target machine, the first receiving unit 1005 receives the data processing result returned by the target machine.

The data processing result received by the first receiving unit 1005 may include quality data of the data to be processed, which includes a feature vector and/or a quality score, in addition to a task execution result (e.g., a quality inspection result, a recognition result, etc.) of the data to be processed.

The first receiving unit 1005 may further send the task execution result in the received data processing result to the input terminal after receiving the data processing result, so as to display the data processing result in the input terminal, for example, display a picture quality inspection result, a text recognition result, and the like.

The data processing apparatus 1000 of the present embodiment may further include a second transmitting unit 1006 for performing the following: after receiving a data processing result returned by the target machine, acquiring the resource utilization rate of the target resource pool; under the condition that the resource utilization rate meets the preset utilization rate requirement, selecting target data from the data to be processed; and sending the target data and the data processing result of the target data to the target machine so that the target machine updates the target processing model according to the target data and the data processing result of the target data.

That is, the second sending unit 1006 updates the target processing model in a data reflux manner under the condition that it is determined that the resource usage rate of the target resource pool meets the preset usage rate requirement, so that the problem of poor processing performance of the processing model caused by a small amount of data in the early stage can be solved, and further, the accuracy of the data processing result output by the target processing model is improved.

The second transmitting unit 1006 may use the ratio of the target resource pool between the currently allocated sub-resource information and the total sub-resource information included as the resource usage rate.

The second sending unit 1006 may determine that the resource usage meets the preset usage requirement when determining that the resource usage is less than or equal to the preset usage threshold.

The second transmitting unit 1006 may select the target data from the data to be processed in the following manner: obtaining a quality score of the data to be processed from a data processing result of the data to be processed; and selecting target data from the data to be processed according to the quality score of the data to be processed.

The second sending unit 1006 may use, as the target data, the data to be processed having a quality score lower than or equal to a preset score threshold when selecting the target data from the data to be processed according to the quality score of the data to be processed; the data to be processed can be classified according to the quality score, and the classification result is the data to be processed of the preset data category and is used as target data.

The second transmitting unit 1006 may also adopt the following manner when selecting target data from the data to be processed: acquiring a feature vector of the data to be processed from a data processing result of the data to be processed; calculating the similarity between the feature vector and a preset feature vector; and taking the data to be processed, of which the similarity calculation result exceeds a preset similarity threshold value, as target data.

After the second sending unit 1006 sends the target data and the data processing result of the target data to the target machine, the target machine performs iterative training on the target processing model according to the received target data and the data processing result, so as to complete updating of the target processing model.

Fig. 11 is a schematic diagram according to an eleventh embodiment of the present disclosure. As shown in fig. 11, the data processing apparatus 1100 of the present embodiment is applied to a machine located in a machine cluster, and includes:

a second receiving unit 1101, configured to receive data to be processed and target sub-resource information sent by a server;

a second processing unit 1102, configured to process the data to be processed by using a hardware resource operation target processing model corresponding to the target sub-resource information;

a third sending unit 1103 is configured to send a data processing result of the data to be processed to the server.

The second receiving unit 1101 may receive data information transmitted by the server, which may be used to determine a target process model among the process models, in addition to the data to be processed and the target sub-resource information.

The target processing model used by the second processing unit 1102 is composed of a backbone network, a task head network and a quality head network; the main network is used for extracting characteristics of input data; the task head network is used for outputting task execution results (such as quality inspection results, identification results, detection results and the like) according to the characteristics extracted by the backbone network; the quality head network is used for outputting quality scores and/or feature vectors according to the features extracted by the backbone network.

The data processing apparatus 1100 of the present embodiment may further include a training unit 1104 for training to obtain a target processing model in the following manner: acquiring a training set, wherein the training set comprises a plurality of training samples and task labeling results of the plurality of training samples; obtaining quality marking data of the training sample according to the task marking result of the training sample; constructing a neural network model comprising a backbone network, a task head network and a quality head network; inputting the training sample into a neural network model to obtain quality prediction data output by the neural network model; calculating a loss function value according to the quality labeling data and the quality prediction data of the training sample; and adjusting parameters of the quality head network according to the calculated loss function value until the quality head network converges to obtain a target processing model.

That is, the training unit 1104 automatically creates a training set for training the target processing model through the existing training set, reduces the training cost of the neural network model including the quality head network, and updates only the parameters of the quality head network, so that the training speed of the target processing model can be improved.

When the training unit 1104 obtains quality labeling data according to the task labeling result of the training sample, the quality score can be obtained according to the task labeling result, and the quality score is used as the quality labeling data; the training samples can be classified according to the quality scores, and classification categories of the training samples are used as quality labeling data.

When the training unit 1104 obtains the quality score according to the task labeling result, the following calculation formula may be adopted:

in the formula: score represents a quality score; a, a _i Representing the weight coefficient; c _i Representation factorThe factor information is a task labeling result of the training sample, taking quality inspection task as an example, and the factor information can be defect number, defect category, defect area, defect position and the like of the picture, and different factor information corresponds to different weight coefficients.

The third transmission unit 1103 includes quality data in addition to the task execution result in the data processing result transmitted to the server.

The data processing apparatus 1100 of the present embodiment may further include an updating unit 1105 for performing: after the data processing result of the data to be processed is sent to the server, receiving the target data sent by the server and the data processing result of the target data; and updating the target processing model according to the target data and the data processing result of the target data.

When updating the target processing model according to the target data and the data processing result of the target data, the updating unit 1105 may adopt the following alternative implementation manners: inputting the target data into a target processing model to obtain a task execution prediction result and quality prediction data which are output by the target processing model; calculating a loss function value according to the data processing result, the task execution prediction result and the quality prediction data; and adjusting parameters of the target processing model according to the calculated loss function value.

When calculating the loss function value, the updating unit 1105 may calculate a first loss function value between the task execution result and the task execution prediction result in the data processing result, and a second loss function value between the quality data and the quality prediction data in the data processing result, respectively, and adjust the parameters of the target processing model according to the first loss function value and the second loss function value.

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the related user personal information all conform to the regulations of related laws and regulations, and the public sequence is not violated.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

As shown in fig. 12, is a block diagram of an electronic device of a resource pool creation or data processing method according to an embodiment of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 12, the apparatus 1200 includes a computing unit 1201, which may perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a Random Access Memory (RAM) 1203. In the RAM1203, various programs and data required for the operation of the device 1200 may also be stored. The computing unit 1201, the ROM1202, and the RAM1203 are connected to each other via a bus 1204. An input/output (I/O) interface 1205 is also connected to the bus 1204.

Various components in device 1200 are connected to I/O interface 1205, including: an input unit 1206 such as a keyboard, mouse, etc.; an output unit 1207 such as various types of displays, speakers, and the like; a storage unit 1208 such as a magnetic disk, an optical disk, or the like; and a communication unit 1209, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks.

The computing unit 1201 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 1201 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as resource pool creation or data processing methods. For example, in some embodiments, the resource pool creation or data processing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 1208.

In some embodiments, part or all of the computer program may be loaded and/or installed onto device 1200 via ROM1202 and/or communication unit 1209. When a computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the resource pool creation or data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the resource pool creation or data processing method in any other suitable manner (e.g., by means of firmware).

Various implementations of the systems and techniques described here can be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable resource pool creation or data processing apparatus, such that the program code, when executed by the processor or controller, causes the implementation of the functions/operations specified in the flowchart and/or block diagram block or blocks. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a presentation device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for presenting information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service ("Virtual Private Server" or simply "VPS") are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. A resource pool creation method, comprising:

acquiring resource information of machines in a machine cluster;

dividing the resource information according to the resource category to obtain at least one resource pool;

and dividing the resource information in the at least one resource pool to obtain at least one piece of sub-resource information, wherein different pieces of sub-resource information in the same resource pool have the same resource attribute.

2. The method of claim 1, further comprising,

after obtaining the at least one resource pool, calculating affinities between resource information in the at least one resource pool;

dividing the resource information in the at least one resource pool according to the calculation result of the affinity to obtain at least one sub-resource pool.

3. The method of claim 1, wherein the dividing the resource information in the at least one resource pool to obtain at least one sub-resource information comprises:

determining a resource category corresponding to the at least one resource pool;

and dividing the resource information in the at least one resource pool by using a dividing mode corresponding to the resource category to obtain the at least one sub-resource information.

4. The method of claim 1, further comprising,

Acquiring resource information of machines added into the machine cluster as first resource information;

determining a first resource category of the first resource information;

and adding the first resource information to a resource pool corresponding to the first resource category.

5. The method of claim 1, further comprising,

acquiring resource information of machines leaving the machine cluster as second resource information;

determining a second resource category of the second resource information;

and deleting the second resource information in the resource pool corresponding to the second resource category.

6. A data processing method, comprising:

acquiring data to be processed;

determining a target resource pool according to the resource category of hardware resources required by processing the data to be processed;

acquiring target sub-resource information from the target resource pool, and determining a target machine corresponding to the target sub-resource information;

transmitting the data to be processed and the target sub-resource information to the target machine, so that the target machine uses a hardware resource corresponding to the target sub-resource information to run a target processing model to process the data to be processed;

and receiving a data processing result returned by the target machine.

7. The method of claim 6, wherein the determining a target resource pool based on a resource class of hardware resources required to process the data to be processed comprises:

determining the resource category of hardware resources required for processing the data to be processed according to the data information of the data to be processed;

and taking the resource pool corresponding to the resource category as the target resource pool.

8. The method of claim 6, wherein the obtaining target sub-resource information from the target resource pool comprises:

determining a target sub-resource pool from the target resource pool according to the data information of the data to be processed;

and acquiring the target sub-resource information from the target sub-resource pool.

9. The method of claim 6, wherein the obtaining target sub-resource information from the target resource pool comprises:

determining a resource allocation model corresponding to the target resource pool;

acquiring resource pool information of the target resource pool, running information of the machine cluster and data information of the data to be processed;

inputting the resource pool information, the operation information and the data information into the resource allocation model;

And acquiring the target sub-resource information from the target resource pool according to a sub-resource information prediction result output by the resource allocation model.

10. The method of claim 6, further comprising,

after receiving a data processing result returned by the target machine, acquiring the resource utilization rate of the target resource pool;

under the condition that the resource utilization rate meets the preset utilization rate requirement, selecting target data from the data to be processed;

and sending the target data and the data processing result of the target data to the target machine so that the target machine updates the target processing model according to the data processing result of the target data and the target data.

11. The method of claim 10, wherein the selecting target data from the data to be processed comprises:

acquiring a quality score of the data to be processed from a data processing result of the data to be processed;

and selecting the target data from the data to be processed according to the quality score.

12. The method of claim 10, wherein the selecting target data from the data to be processed comprises:

Acquiring a feature vector of the data to be processed from a data processing result of the data to be processed;

calculating the similarity between the feature vector and a preset feature vector;

and taking the data to be processed, of which the similarity calculation result exceeds a preset similarity threshold value, as the target data.

13. A data processing method, comprising:

receiving data to be processed and target sub-resource information sent by a server;

using a hardware resource operation target processing model corresponding to the target sub-resource information to process the data to be processed;

and sending the data processing result of the data to be processed to the server.

14. The method of claim 13, wherein the target processing model is trained by:

acquiring a training set, wherein the training set comprises a plurality of training samples and task labeling results of the plurality of training samples;

obtaining quality marking data of a training sample according to a task marking result of the training sample;

constructing a neural network model comprising a backbone network, a task head network and a quality head network;

inputting the training sample into the neural network model to obtain quality prediction data output by the neural network model;

Calculating a loss function value according to the quality labeling data and the quality prediction data;

and adjusting parameters of the quality head network according to the loss function value until the quality head network converges to obtain the target processing model.

15. The method of claim 13, further comprising,

after the data processing result of the data to be processed is sent to the server, receiving the target data sent by the server and the data processing result of the target data;

and updating the target processing model according to the target data and the data processing result of the target data.

16. The method of claim 15, wherein the updating the target processing model based on the target data and the data processing results of the target data comprises:

inputting the target data into the target processing model to obtain a task execution prediction result and quality prediction data which are output by the target processing model;

calculating a loss function value according to the data processing result, the task execution prediction result and the quality prediction data;

and adjusting parameters of the target processing model according to the loss function value.

17. A resource pool creation apparatus comprising:

the first acquisition unit is used for acquiring the resource information of the machines in the machine cluster;

the dividing unit is used for dividing the resource information according to the resource category to obtain at least one resource pool;

the splitting unit is used for splitting the resource information in the at least one resource pool to obtain at least one piece of sub-resource information, wherein different piece of sub-resource information in the same resource pool has the same resource attribute.

18. A data processing apparatus comprising:

the second acquisition unit is used for acquiring data to be processed;

the determining unit is used for determining a target resource pool according to the resource category of the hardware resource required by processing the data to be processed;

the first processing unit is used for acquiring target sub-resource information from the target resource pool and determining a target machine corresponding to the target sub-resource information;

the first sending unit is used for sending the data to be processed and the target sub-resource information to the target machine so that the target machine can use a hardware resource corresponding to the target sub-resource information to run a target processing model to process the data to be processed;

and the first receiving unit is used for receiving the data processing result returned by the target machine.

19. A data processing apparatus comprising:

the second receiving unit is used for receiving the data to be processed and the target sub-resource information sent by the server;

the second processing unit is used for operating a target processing model to process the data to be processed by using the hardware resources corresponding to the target sub-resource information;

and the third sending unit is used for sending the data processing result of the data to be processed to the server.

20. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-16.

21. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-16.

22. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-16.