CN117873712A - Scientific computing task processing method, device and computing system - Google Patents

Scientific computing task processing method, device and computing system Download PDF

Info

Publication number
CN117873712A
CN117873712A CN202311791754.9A CN202311791754A CN117873712A CN 117873712 A CN117873712 A CN 117873712A CN 202311791754 A CN202311791754 A CN 202311791754A CN 117873712 A CN117873712 A CN 117873712A
Authority
CN
China
Prior art keywords
data point
computing
node
task
storage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311791754.9A
Other languages
Chinese (zh)
Inventor
郭璟
雍安睿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Silang Technology Co ltd
Original Assignee
Shanghai Silang Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Silang Technology Co ltd filed Critical Shanghai Silang Technology Co ltd
Priority to CN202311791754.9A priority Critical patent/CN117873712A/en
Publication of CN117873712A publication Critical patent/CN117873712A/en
Pending legal-status Critical Current

Links

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a processing method, a device and a computing system for scientific computing tasks, and belongs to the field of data processing. The method comprises the following steps: determining a computing node of a preset computing task, wherein the preset computing task is a data point-data point computing task; storing attribute data of a data point and a calculation position pointer of the data point to a storage node of the data point, wherein the calculation position pointer of the data point is generated according to the node position of a calculation node of a preset calculation task participated by the data point; whenever the storage position of a data point changes, reporting the storage position of the changed data point to a computing node pointed by a computing position pointer of the data point; the computing node updates the storage location pointer for the data point after receiving the storage location for the data point. According to the invention, the data point storage positions with changed positions are automatically reported to the corresponding computing nodes, so that the computing nodes can conveniently and rapidly extract the data point attribute data required by the computing task, and the processing efficiency of the scientific computing task is improved.

Description

Scientific computing task processing method, device and computing system
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for processing a scientific computing task, and a computing system.
Background
The scientific calculation task is a calculation task which needs to be realized in the process of developing scientific research practice, and has the characteristics of large calculation amount, high processing difficulty and the like. At present, a distributed processing mode is generally adopted to process scientific computing tasks, wherein a huge scientific computing task is divided into a large number of subtasks, and a plurality of processing nodes process corresponding subtasks in parallel.
Data point-data point computing tasks are a common type of subtask that is used to process the functional relationship between two data points. In the prior art, when a data point-data point type calculation task is processed, a processing node of the calculation task acquires attribute data of data points required by calculation and then performs data processing.
In carrying out the invention, the inventors have found that the prior art has at least the following problems: when the processing nodes of the computing task acquire the attribute data of the data points, the processing nodes need to be traversed to search the attribute data of the data points required by computation, so that the processing efficiency of the data point-data point computing task is low, and the processing efficiency of the whole scientific computing task is reduced.
Disclosure of Invention
In order to solve the problems in the prior art, the embodiment of the invention provides a processing method, a device, a computing system, computing equipment and a computer readable storage medium for scientific computing tasks. The technical scheme is as follows:
In a first aspect, a method for processing a scientific computing task is provided, where the method includes:
determining a computing node of a preset computing task aiming at any preset computing task contained in a scientific computing task; the preset calculation task is a data point-data point calculation task;
storing attribute data of any data point contained in a scientific calculation task and a calculation position pointer of the data point to a storage node of the data point; the computing position pointer of the data point is generated according to the node position of the computing node of the preset computing task participated by the data point;
whenever the storage position of a data point changes, reporting the storage position of the changed data point to a calculation node pointed by a calculation position pointer of the data point;
after receiving the storage position of the data point, the computing node updates the storage position pointer of the data point.
In an alternative embodiment, before storing the attribute data of any data point included in the scientific computing task and the calculated position pointer of the data point to the storage node of the data point, the method further includes: dividing the problem space of the scientific computing task into a plurality of subspaces;
Establishing a mapping relation between each subspace and each storage node; for any data point contained in a scientific calculation task, determining a target subspace in which the data point is located, and taking a storage node with a mapping relation with the target subspace as a storage node of the data point.
In an alternative embodiment, the dividing the problem space of the scientific computing task into a plurality of subspaces further comprises: for any space dimension in the problem space, determining the number of storage nodes corresponding to the space dimension; and uniformly segmenting the space dimension of the corresponding problem space according to the number of storage nodes corresponding to each space dimension, so that the segmentation segment number of the problem space in each space dimension is consistent with the number of storage nodes corresponding to the corresponding space dimension.
In an alternative embodiment, the method further comprises: determining a migration data point at which the location migration occurs at the beginning of the new time step; determining a target subspace where the migration data point is located after migration, and taking a storage node with a mapping relation with the target subspace as a storage node of the migration data point; and migrating the attribute data of the migration data point and the calculation position pointer to a storage node of the migration data point.
In an alternative embodiment, the storing node that stores the attribute data of the data point and the calculated position pointer of the data point to the data point further includes: generating storage information of the data point according to the attribute data of the data point, the calculated position pointer of the data point and the storage position of the data point; the stored information for the data point is stored in a storage node for the data point.
In an alternative embodiment, the method further comprises: the storage location in the stored information for a data point is updated each time the storage location for the data point changes.
In an optional implementation manner, the computing node for determining any preset computing task included in the scientific computing task further includes: and determining the computing nodes of each preset computing task contained in the scientific computing task according to the resource overhead of the preset computing task, the number of the computing nodes, the load of the computing nodes and/or the processing capacity of the computing nodes.
In an optional implementation manner, the computing node for determining any preset computing task included in the scientific computing task further includes: in the static allocation stage, allocating preset calculation tasks with the same quantity to each calculation node; in the dynamic allocation stage, different numbers of preset computing tasks are allocated to each computing node according to the resource overhead of the preset computing tasks which are not allocated, the load of each computing node at present and/or the residual processing capacity of each computing node at present.
In an alternative embodiment, the method further comprises: the method comprises the steps that a computing node determines a target preset computing task to be executed currently, and determines data points associated with the target preset computing task; the computing node extracts attribute data of the associated data point from the storage node pointed by the storage position pointer of the associated data point; and the computing node processes a target preset computing task based on the extracted attribute data.
In an optional implementation manner, after the computing node performs the processing of the target preset computing task based on the extracted attribute data, the method further includes: the computing node obtains a processing result of a target preset computing task, and returns the processing result to a storage node pointed by a storage position pointer of a data point associated with the target preset computing task.
In a second aspect, there is provided a processing apparatus for scientific computing tasks, the apparatus comprising:
the computing node distribution module is used for determining the computing node of a preset computing task aiming at any preset computing task contained in a scientific computing task; the preset calculation task is a data point-data point calculation task;
The storage node distribution module is used for storing attribute data of any data point contained in the scientific calculation task and a calculation position pointer of the data point to a storage node of the data point; the computing position pointer of the data point is generated according to the node position of the computing node of the preset computing task participated by the data point;
the position reporting module is used for reporting the changed storage position of the data point to a calculation node pointed by a calculation position pointer of the data point every time the storage position of the data point is changed; and after the computing node receives the storage position of the data point, updating the storage position pointer of the data point.
In an alternative embodiment, the apparatus further comprises: the subspace dividing module is used for dividing the problem space of the scientific computing task into a plurality of subspaces; establishing a mapping relation between each subspace and each storage node;
the storage node allocation module is used for: for any data point contained in a scientific calculation task, determining a target subspace in which the data point is located, and taking a storage node with a mapping relation with the target subspace as a storage node of the data point.
In an alternative embodiment, the subspace partitioning module is configured to: for any space dimension in the problem space, determining the number of storage nodes corresponding to the space dimension; and uniformly segmenting the space dimension of the corresponding problem space according to the number of storage nodes corresponding to each space dimension, so that the segmentation segment number of the problem space in each space dimension is consistent with the number of storage nodes corresponding to the corresponding space dimension.
In an alternative embodiment, the apparatus further comprises: the migration module is used for determining migration data points with position migration when a new time step starts; determining a target subspace where the migration data point is located after migration, and taking a storage node with a mapping relation with the target subspace as a storage node of the migration data point; and migrating the attribute data of the migration data point and the calculation position pointer to a storage node of the migration data point.
In an alternative embodiment, the storage node allocation module is configured to: generating storage information of the data point according to the attribute data of the data point, the calculated position pointer of the data point and the storage position of the data point; the stored information for the data point is stored in a storage node for the data point.
In an alternative embodiment, the apparatus further comprises: and the updating module is used for updating the storage position in the storage information of the data point every time the storage position of the data point changes.
In an alternative embodiment, the computing node allocation module is configured to: and determining the computing nodes of each preset computing task contained in the scientific computing task according to the resource overhead of the preset computing task, the number of the computing nodes, the load of the computing nodes and/or the processing capacity of the computing nodes.
In an alternative embodiment, the computing node allocation module is configured to: in the static allocation stage, allocating preset calculation tasks with the same quantity to each calculation node; in the dynamic allocation stage, different numbers of preset computing tasks are allocated to each computing node according to the resource overhead of the preset computing tasks which are not allocated, the load of each computing node at present and/or the residual processing capacity of each computing node at present.
In an alternative embodiment, a computing node determines a target preset computing task to be currently executed, and determines data points associated with the target preset computing task; the computing node extracts attribute data of the associated data point from the storage node pointed by the storage position pointer of the associated data point; and the computing node processes a target preset computing task based on the extracted attribute data.
In an alternative embodiment, a computing node obtains a processing result of a target preset computing task, and returns the processing result to a storage node pointed by a storage location pointer of a data point associated with the target preset computing task.
In a third aspect, a computing system is provided, including a processing device for the scientific computing task and a plurality of processing nodes;
the processing node is used as a computing node of a preset computing task in the processing device of the scientific computing task; and/or the processing node is used as a storage node of data points in the processing device of the scientific calculation task.
In a fourth aspect, there is provided a computing device comprising: a processor;
the processor is used for executing the operation corresponding to the processing method of the scientific calculation task.
In a fifth aspect, a computer readable storage medium is provided, where at least one executable instruction is stored in the storage medium, where the executable instruction causes a processor to perform operations corresponding to a processing method of the above-mentioned scientific computing task.
The technical scheme provided by the embodiment of the invention has the beneficial effects that: by recording the calculation position pointer of the data point in the storage node of the data point, the storage position of the data point with position change can be reported to the calculation node pointed by the calculation position pointer in time, the calculation node generates the storage position pointer of the data point according to the reported storage position, the storage position of the attribute data of the data point required to be calculated can be determined when a preset calculation task is processed, the time for acquiring the attribute data of the data point is greatly shortened, and the execution efficiency of the scientific calculation task is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a method for processing a scientific computing task according to an embodiment of the invention;
fig. 2 is a flow chart of a method for processing a scientific computing task according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of a computing task list according to a second embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating a problem space segmentation according to a second embodiment of the present invention;
FIG. 5 is a diagram of data points stored according to a second embodiment of the present invention;
FIG. 6 is a migration diagram of data points according to a second embodiment of the present invention;
fig. 7 is a flow chart of a method for processing a scientific computing task according to a third embodiment of the present invention;
fig. 8 is a schematic structural diagram of a processing device for scientific computing tasks according to a fourth embodiment of the present invention;
Fig. 9 is a schematic structural diagram of a computing device according to a fifth embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
First, some technical knowledge involved in the embodiments of the present application will be briefly described.
Scientific computing tasks are computing tasks that need to be implemented in the course of developing scientific research practices, which are often combined with numerical simulation to enable the processing of scientific questions, e.g., scientific computing tasks may include molecular dynamics simulation tasks, hydrodynamic simulation tasks, and so forth.
Since the processing amount of a scientific computing task is generally huge, a huge scientific computing task is generally divided into a plurality of subtasks in the actual processing process. Data point-data point computing tasks are a common subtask. Wherein a data point is a basic data unit in a scientific computing task that can be considered as the smallest analyte in the scientific computing task, e.g., if the scientific computing task is a molecular dynamics simulation, then the data point corresponds to an atom or molecule in the molecular dynamics; if the scientific computational task is a smooth particle hydrodynamic simulation, then the data points correspond to particles in the particle hydrodynamic, and so on. The data point-data point calculation task is a task for processing the action relationship between two data points, for example, the data point-data point calculation task may be to calculate the action force between two atoms, and so on.
Each data point-data point type calculation task after the scientific calculation task is divided can be distributed to different processing nodes in the distributed system for parallel execution, so that the processing efficiency of the scientific calculation task is improved. The processing node has both a computing function and a storage function, so that the processing node can perform data processing of computing tasks and can also perform storage of attribute data of data points.
In the context of a computing task in which a plurality of processing nodes process data points-data points, there is one embodiment as follows: for any data point-data point type calculation task, a processing node storing attribute data of one data point serves as a calculation node of the calculation task, namely, the processing node gathers attribute data of each data point required by the calculation task and processes the attribute data based on the attribute data of the gathered data point.
In an actual implementation process, some data points will change in storage positions during the processing of the scientific computing task, that is, the storage positions of the data points are not fixed, the processing node only records identification information of the data points related to the computing task, where the identification information is a number allocated to the data points in the whole scientific computing task, for example, each data point may be encoded in a mode of 1, 2 … …, and the like. Thus, when the processing nodes gather the attribute data of each data point required by the computing task, each processing node in the whole distributed system often needs to be traversed to determine which processing node the attribute data of the data point related to the computing task is stored in. However, when there are many processing nodes, it takes a lot of time to extract attribute data, thereby reducing the processing efficiency of the entire scientific computing task.
In view of this, the present application sets up the calculation position pointer at the storage node of data point, sets up the storage position pointer at the calculation node of data point, and the reporting object of storage position of data point is confirmed according to the calculation position pointer by the storage node, confirms the storage position of the attribute data of data point according to the storage position pointer by the calculation node to the calculation node can be fast according to the attribute data of data point that storage position pointer draws calculation task required, promotes scientific calculation task's processing efficiency.
The implementation of the present application is explained in detail below by means of various examples.
Example 1
The embodiment of the invention provides a flow diagram of a processing method of a scientific computing task, referring to fig. 1, the method flow comprises the following steps:
step 101: determining a computing node of a preset computing task aiming at any preset computing task contained in a scientific computing task; the preset calculation task is a data point-data point calculation task.
The preset calculation task is a subtask obtained by splitting a scientific calculation task, and the type of the preset calculation task is a data point-data point calculation task. The embodiment of the invention is mainly used for optimizing the processing process of scientific computing tasks which can be divided into a plurality of preset computing tasks.
In a specific implementation process, for any preset computing task obtained after splitting a scientific computing task, a fixed computing node is allocated to the preset computing task, wherein the computing node is a processing node for processing the preset computing task. The embodiment of the invention adopts a distributed processing mode to process scientific calculation tasks, so that a plurality of processing nodes can process the scientific calculation tasks, the preset calculation tasks to be processed by each processing node can be determined through the multiple implementation of the step, namely, the calculation mapping relation between each processing node and each preset calculation task is established, and if the processing node and the preset calculation task establish the calculation mapping relation, the processing node is the calculation node of the preset calculation task. The method for distributing the computing nodes to each preset computing task is not limited in the embodiment of the invention. For example, corresponding task lists may be allocated to different processing nodes according to the load of the processing nodes, or random allocation, or other allocation modes, where the task lists include preset computing tasks that need to be processed by the corresponding processing nodes.
Compared with the processing mode that the processing node storing the attribute data of one data point is used as the computing node of the computing task in the prior art, the computing node of each preset computing task is not required to be distributed in each non-first time step of the scientific computing task, so that the overall execution time of the scientific computing task is shortened, and the execution efficiency of the scientific computing task is improved.
Step 102: storing attribute data of any data point contained in a scientific calculation task and a calculation position pointer of the data point into a storage node of the data point; the computing position pointer of the data point is generated according to the node position of the computing node of the preset computing task participated by the data point.
Each data point is respectively provided with a storage node for storing attribute data of the data point, namely, the data point and a processing node establish a storage mapping relation, and if a certain data point and a certain processing node establish a storage mapping relation, the processing node is indicated to be the storage node of the data point.
In the prior art, attribute data of information points are stored only in storage nodes. Unlike the prior art, the storage node according to the embodiment of the invention stores the calculated position pointers of the data points in addition to the attribute data of the data points.
The calculated position pointer of the data point is generated according to the node position of the calculated node of the preset calculation task participated by the data point. Specifically, for any one data point, determining each preset computing task participated in by the data point, and determining the computing node of each preset computing task participated in by the data point by searching the mapping relation between the preset computing task and the computing node, wherein a computing position pointer can be generated according to the node position of the computing node of any one preset computing task participated in by the data point, and each computing position pointer points to the corresponding computing node, so that each computing position pointer of the data point can be obtained.
It should be appreciated herein that since a data point may participate in one or more predetermined computing tasks, the computed position pointer for a data point may correspond to one or more. For example, the Data points associated with the preset computing Task task_1 are data_1 and data_2, the computing Node for processing the preset computing Task task_1 is node_1, the Data points associated with the preset computing Task task_2 are data_1 and data_3, the computing Node for processing the preset computing Task task_2 is node_2, the computing position pointer p_1 is generated according to the Node position of node_1, the computing position pointer p_2 is generated according to the Node position of node_2, the computing position pointer of data_1 comprises p_1 and p_2, the computing position pointer of data_2 comprises p_1, and the computing position pointer of data_3 comprises p_3.
In some cases, the storage node of a data point and the computing node of the preset computing task in which the data point participates may be different processing nodes: for example, when the Data points data_1 and data_2 participate in the preset computing Task task_1, the attribute Data of the Data point data_1 is stored in the processing Node node_1, the attribute Data of the Data point data_2 is stored in the processing Node node_2, the processing Node node_3 is a computing Node for processing the preset computing Task task_1, and the storage nodes of the Data point data_1 (the processing Node node_1) and the Data point data_2 (the processing Node node_2) are different processing nodes from the computing Node of the preset computing Task task_1 (the processing Node node_3) in which the Data point data_1 participates; in other cases, the storage node of the data point and the computing node of the preset computing task in which the data point participates may be the same processing node: for example, when the Data points data_1 and data_2 participate in the preset computing Task task_1, the attribute Data of the Data point data_1 is stored in the processing Node node_1, the attribute Data of the Data point data_2 is stored in the processing Node node_2, the processing Node node_1 is a computing Node for processing the preset computing Task task_1, and the storage Node (processing Node node_1) of the Data point data_1 is the same as the computing Node (processing Node node_1) of the preset computing Task task_1 that the storage Node participates in.
Step 103: whenever the storage position of a data point changes, the changed storage position of the data point is reported to a computing node pointed by a computing position pointer of the data point.
When the storage position of a data point changes, the storage node or other equipment can actively report the storage position of the changed data point. In reporting, the changed storage node stores the calculated position pointer of the data point, so that the reporting object of this time can be determined through the calculated position pointer, namely, the calculated node pointed by the calculated position pointer of the data point is the reporting object, and the changed storage position of the data point is reported to the calculated node pointed by the calculated position pointer of the data point.
The reporting timer comprises a step of reporting the latest storage position when the storage node is not stored with the attribute data of the data point in the initial state, wherein the step of initially storing the attribute data of the information point and the calculated position pointer in the initial state after initial storage and migration storage is performed, and the step of reporting the latest storage position is performed when the storage node is capable of being regarded as the change of the storage position of the data point after the storage node is used for initially storing the attribute data of the data point; or, the migration storage represents the attribute data and the calculation position pointer of the information point which are not stored for the first time, namely, after the data point is migrated in the problem space of the scientific calculation task in the execution process of the scientific calculation task, the attribute data and the calculation position pointer of the data point need to be migrated from one storage node to the current storage node. Thus, each time a data point is reported is the current latest storage location.
Step 104: after the computing node receives the storage location of the data point, the storage location pointer of the data point is updated.
Each computing node is also recorded with a storage position pointer of each data point associated with a preset computing task which is responsible for processing by the computing node, and the storage position pointer of the data point points to a storage node currently storing attribute data of the data point, so that the computing node can quickly determine the storage position of the attribute data of the data point according to the storage position pointer of the data point.
In the implementation process, after the computing node receives the storage position of the data point reported by the storage node, a storage position pointer is generated by using the storage position, and the storage position pointer is used as the latest storage position pointer of the data point.
The implementation of the present embodiment is explained in detail below with a specific example: dividing a scientific calculation Task into N preset calculation tasks, wherein the preset calculation Task task_1 is a Task for calculating acting forces in Data points data_1 and data_2, a calculation Node distributed for the preset calculation Task task_1 is node_1, a calculation position pointer generated according to the Node position of the node_1 is PJ_1, attribute Data of the data_1 and the PJ_1 are associated and stored in a storage Node node_2, the attribute Data of the data_2 and the PJ_1 are associated and stored in the storage Node node_3, the node_2 or other equipment can report the Node position of the node_2 and the Data point identifier of the data_1 to the node_1 pointed by the PJ_1, the node_1 generates the storage position pointer PC_1 of the data_1 according to the Node position of the node_2, and establishes a mapping relation between the data_1 and the PC_1; node_3 or other devices can report the Node position of node_3 and the Data point identifier of data_2 to node_1 pointed by PJ_1, node_1 generates the storage position pointer PC_2 of data_2 according to the Node position of node_3, and establishes the mapping relation between data_2 and PC_2. Thus, when processing task_1, node_1 can determine the storage location of the attribute Data of data_1 according to pc_1, and determine the storage location of the attribute Data of data_2 according to pc_2. Furthermore, if the storage location of data_1 is migrated from node_2 to node_4, node_4 or other devices can report the Node location of node_4 and the Data point identifier of data_1 to node_1 pointed to by pj_1, and node_1 updates the storage location pointer pc_1 of data_1.
In summary, according to the processing method for a scientific computing task provided by the embodiment of the invention, by recording the computing position pointer of the data point in the storage node of the data point, the storage position of the data point with position change can be timely reported to the computing node pointed by the computing position pointer, and the computing node generates the storage position pointer of the data point according to the reported storage position, so that the storage position of the attribute data currently stored with the data point required for computing can be determined when the preset computing task is processed, the time for acquiring the attribute data of the data point is greatly reduced, and the execution efficiency of the scientific computing task is improved.
Example two
The embodiment of the invention provides a flow diagram of a processing method of a scientific computing task, referring to fig. 2, the method flow comprises the following steps:
step 201: and distributing corresponding computing nodes for each preset computing task contained in the scientific computing task.
And distributing corresponding computing nodes for each preset computing task contained in the scientific computing task, wherein each computing node can obtain a corresponding computing task list, and storing the corresponding computing task list in the computing node. The computing task list records the related information of the preset computing task which needs to be processed by the computing node. The information related to the preset computing task may include a data point identification of the data point associated with the preset computing task, and so on. Taking the computing task list shown in fig. 3 as an example, the computing task list is a computing Node x_y_z The method comprises the steps of storing a calculation task list, wherein the calculation task list comprises M preset calculation tasks: { T Ele1_1 -T Ele1_2 }、{T Ele2_1 -T Ele2_2 }…{T Elei_1 -T Elei_2 }…{T EleM-1_1 -T EleM-1_2 }、{T EleM_1 -T ElM2_2 An ith preset computing task { T }, where the ith preset computing task is Elei_1 -T Elei_2 }, which represents the data point T Elei_1 T and T Elei_2 Calculation task of processing, T Elei_1 T and T Elei_2 Identified for the corresponding data point. And the M preset computing tasks are in the computing Node x_y_z And executing.
In an alternative embodiment, the allocation of computing nodes may be performed by: and determining the computing nodes of each preset computing task contained in the scientific computing task according to the resource overhead of the preset computing task, the number of the computing nodes, the load of the computing nodes and/or the processing capacity of the computing nodes. The resource expense of the preset computing task is the amount of resources consumed for processing the preset computing task; the number of compute nodes is typically the number of compute nodes that can process tasks in parallel; the load of the computing node is the task processing amount born by the computing node; the processing capacity of the computing node is the processing capacity of the computing node to the task, and the processing capacity can be determined by the CPU size, the memory size, and the like. The distribution of the computing nodes is carried out by presetting the resource cost of the computing tasks, the number of the computing nodes, the load of the computing nodes and/or the processing capacity of the computing nodes, so that the load balance of the whole computing nodes can be realized, and the overload of some computing nodes and/or the resource waste of some computing nodes are avoided.
In yet another alternative embodiment, the process of assigning computing nodes to preset computing tasks may be divided into two phases, a static assignment phase and a dynamic assignment phase. In the specific implementation process, firstly, a static allocation stage is entered, in the static allocation stage, preset calculation tasks with the same quantity are allocated to each calculation node, for example, 5 preset calculation tasks can be firstly allocated to each calculation node in average, and after the static allocation stage is finished, the subsequent steps such as task execution can be started in time so as to improve the execution efficiency of scientific calculation tasks; after the static allocation phase is finished, a dynamic allocation phase is entered, wherein the dynamic allocation phase can be asynchronously and parallelly executed with other subsequent steps, in the dynamic allocation phase, preset calculation tasks with different numbers are allocated to each calculation node according to the resource cost of the preset calculation tasks which are not allocated, the load of each calculation node at present and/or the residual processing capacity of each calculation node at present, for example, if a certain calculation node has higher current residual processing capacity (such as lower CPU utilization rate, etc.), more preset calculation tasks or preset calculation tasks with larger required resource cost can be selected from the preset calculation tasks which are not allocated to the calculation node; if the current load of a certain computing node is higher, less preset computing tasks or the preset computing tasks with less required resource expenditure can be selected from the preset computing tasks which are not distributed, and the preset computing tasks are distributed to the computing node. The method comprises the steps of carrying out average distribution in a static distribution stage, carrying out unequal distribution in a dynamic distribution stage according to the actual condition of the current computing node, and improving the processing efficiency of scientific computing tasks on the basis of guaranteeing the balance of the computing nodes.
Step 202: the problem space of the scientific calculation task is divided into a plurality of subspaces, and the mapping relation between each subspace and each storage node is established.
The problem space of the scientific computing task may be a physical space on which the scientific computing task is executed, for example, the problem space may be a one-dimensional space, a two-dimensional space, or a three-dimensional space, etc., and in general, the problem space of the scientific computing task is a three-dimensional space. All data points contained in the scientific calculation task are distributed in the problem space, each data point has a corresponding space position in the problem space, the scientific calculation task is taken as a hydrodynamic simulation task as an example, the problem space is a three-dimensional space, each particle contained in the hydrodynamic simulation task is distributed in the three-dimensional space, each particle is a data point, and each particle corresponds to a space position.
And dividing the problem space of the scientific calculation task to obtain a plurality of subspaces, wherein each subspace has a corresponding position range, and if the spatial position of a certain data point is in the position range, the data point is distributed in the subspace. In this embodiment, the number of subspaces included in the problem space is identical to the number of storage nodes, and corresponding storage nodes are allocated to each subspace, so that a mapping relationship between the screwdriver space and the storage nodes is established, each storage node is used for storing attribute data of data points in the corresponding subspace, and the subspaces can establish a one-to-one mapping relationship with the storage nodes.
In an optional questionIn the problem space division mode, the problem space is divided according to the distribution characteristics of the storage nodes, so that the scientific calculation task can be conveniently processed. In a specific implementation process, the storage nodes generally have a certain location distribution characteristic, for example, in a three-dimensional space, the storage nodes may be arranged correspondingly to obtain the distribution characteristic of the storage nodes, and a plurality of storage nodes may exist in each space dimension. When the method is used for dividing the problem space, the number of storage nodes corresponding to the space dimension can be determined according to any space dimension in the problem space, and the space dimension of the problem space is uniformly divided according to the number of storage nodes corresponding to each space dimension, so that the dividing segment number of the problem space in each space dimension is consistent with the number of storage nodes corresponding to the corresponding space dimension. For example, the problem space includes three spatial dimensions X, Y and Z, the problem space being L in length in the three spatial dimensions X, Y and Z, respectively x 、L y L and L z The number of storage nodes in the three spatial dimensions X, Y and Z are N x 、N y N z Uniformly segmenting the problem space in the space dimension X, wherein each segment has the length L x /N x The number of the segmentation segments is N x The method comprises the steps of carrying out a first treatment on the surface of the Uniformly segmenting the problem space in the space dimension Y, wherein each segment has a length L y /N y The number of the segmentation segments is N y The method comprises the steps of carrying out a first treatment on the surface of the Uniformly segmenting the problem space in a space dimension Z, wherein each segment has a length L z /N z The number of the segmentation segments is N z 。
Taking fig. 4 as an example, the problem space has lengths of L in three spatial dimensions X, Y and Z, respectively x 、L y L and L z The number of storage nodes in the three spatial dimensions X, Y and Z is 5, 5 and 3, respectively, then the problem space is equally divided into 5 parts in the X dimension, 5 parts in the Y dimension, and 3 parts in the X dimension, resulting in 5X 3 subspaces, each of which may or may not contain data points. And establishing a mapping relation between each subspace and the storage node.
It should be understood that the division manner of the problem space is not limited to the above-mentioned one, and the embodiment of the present invention does not limit the division manner of the problem space. For example, the problem space may also be irregularly divided, i.e., the subspaces are not exactly uniform in size. For example, the data points can be divided according to the data point concentration, the area with sparse data point distribution in the problem space is divided by adopting larger granularity, and the area with dense data point distribution in the problem space is divided by adopting smaller granularity, so that the size of the subspace of the sparse data point distribution area is larger than that of the subspace of the dense data point distribution area.
Step 203: for any data point contained in a scientific calculation task, determining a target subspace of the data point, taking a storage node with a mapping relation with the target subspace as a storage node of the data point, and storing attribute data of the data point and a calculation position pointer of the data point into the storage node of the data point.
After the problem space is divided to obtain subspaces, for each data point, the subspace where the data point is located is the target subspace of the data point, and then the storage node with the mapping relation with the target subspace is used as the storage node of the data point according to the mapping relation between the constructed subspace and the storage node. Wherein the storage node of the data point is the current most current storage node of the data point.
After determining the storage node of the data point, generating storage information of the data point according to the attribute data of the data point, the calculated position pointer of the data point and the node position of the storage node of the data point, and storing the storage information of the data point into the storage node of the data point. Taking fig. 5 as an example, the stored information of the data points mainly includes three types: attribute data of data points, such as data point intrinsic attribute 1, data point intrinsic attribute 2 … data point intrinsic attribute n shown in fig. 5; data point storage locations, such as the data point storage locations (Node x_y_z ) The method comprises the steps of carrying out a first treatment on the surface of the Calculated position pointer for data pointThe calculation unit position pointer p of the calculation task 1 as shown in fig. 5 1 (Node x1_y1_z1 ) Calculating unit position pointer p of calculation task 2 2 (Node x2_y2_z2 ). Wherein the attribute data may include: physical location, velocity, particle type, force magnitude, etc.
Step 204: whenever the storage location of a data point changes, the changed storage location of the data point is reported to the computing node pointed by the computing location pointer of the data point.
In practical implementation, in some scientific computing tasks, particles or atoms represented by data points may migrate in the problem space, so that data migration may be performed at the beginning of each time step. Specifically, when a new time step starts, determining a migration data point where position migration occurs, determining a target subspace where the migration data point is located after migration, taking a storage node with a mapping relation with the target subspace as a storage node of the migration data point, and migrating attribute data of the migration data point and a calculated position pointer to the storage node of the migration data point, thereby realizing data migration of the data point.
Taking fig. 6 as an example, at time T1, data point i is located in subspace 4_1_1, and the storage Node of data point i is Node 4_1_1 The data point j is located in the subspace 4_4_1, and the storage Node of the data point j is Node 4_4_1 The method comprises the steps of carrying out a first treatment on the surface of the At time T2, data points i and j are migrated, data point i is migrated to subspace 3_2_1, data point j is migrated to subspace 3_3_1, and the storage Node of data point i is changed to Node 3_2_1 The storage Node of data point j is changed to Node 3_3_1 . If there is a preset calculation task (data point i-data point j), its corresponding calculation Node is Node 1_1_1 The computing nodes of the preset computing task remain unchanged even if the data points i and j undergo position migration.
After the storage position of the data point changes, the storage position in the storage information of the data point is updated, namely the current storage node of the data point is used as the storage position of the updated data point. Still taking fig. 6 as an example, the stored information of data point i records dataPoint storage locations (Node) 4_1_1 ) Change to data point storage location (Node 3_2_1 )。
And after the storage position of the data point changes, reporting the changed storage position of the data point to a calculation node pointed by a calculation position pointer of the data point in time.
Step 205: after the computing node receives the storage location of the data point, the storage location pointer of the data point is updated.
Still taking fig. 6 as an example, the Node is calculated 1_1_1 The storage location of the received data point i (Node 3_2_1 ) Thereafter, the storage location pointer p (Node 4_1_1 ) Updated to p (Node) 3_2_1 ) And a computing Node 1_1_1 The storage location of the received data point j (Node 3_3_1 ) Thereafter, the storage location pointer p (Node 4_4_1 ) Updated to p (Node) 3_3_1 )。
In summary, according to the processing method for the scientific computing task provided by the embodiment of the invention, the time for acquiring the attribute data of the data point can be shortened by storing the position pointer and calculating the position pointer, so that the execution efficiency of the scientific computing task is improved; the problem space of the scientific calculation task is segmented to obtain a plurality of subspaces, a mapping relation between the subspaces and the storage nodes is established, and the storage nodes store the storage information of the data points in the corresponding subspaces, so that the accurate calculation of the scientific calculation task can be facilitated; and the distribution of the computing nodes is performed through multi-dimensional data such as resource overhead, node number, node load, node processing capacity and the like, so that the load balancing of the computing nodes is realized, and the processing efficiency of scientific computing tasks is improved.
Example III
The embodiment of the invention provides a flow diagram of a processing method of a scientific computing task, referring to fig. 7, the method flow comprises the following steps:
Step 701: the computing node determines a target preset computing task to be currently executed, and determines data points associated with the target preset computing task.
For any computing node, the computing node is pre-designed in the processingAnd when calculating the task, determining a preset calculation task to be executed currently according to a calculation task list in the calculation node, wherein the preset calculation task to be executed currently is a target preset calculation task. And further determining each data point associated with the target preset computing task. Taking fig. 3 as an example, if the target preset computing task to be currently executed is { T Elei_1 -T Elei_2 Then the data point associated with the target preset computing task is T Elei_1 T and T Elei_2 。
Step 702: the computing node extracts attribute data for the associated data point from the storage node pointed to by the storage location pointer for the associated data point.
Since the storage position pointer of the data point associated with the target preset computing task is also stored in the computing node, the storage position pointer points to the storage node currently storing the attribute data of the data point, and therefore, for any associated data point, the computing node can extract the attribute data of the data point from the storage node pointed by the storage position pointer of the data point. In the specific implementation process, a part of data points associated with each preset computing task in a task list of the computing node are stored in other nodes, and in this case, the computing node can firstly acquire attribute data of the data points stored in other nodes according to the storage position pointer, so that the subsequent rapid execution of the preset computing task is facilitated.
In step 703, the computing node performs processing of the target preset computing task based on the extracted attribute data.
The attribute data of the data points required for processing the target preset computing task can be rapidly extracted through steps 701 and 702, and the target preset computing task can be processed based on the attribute data after the attribute data is acquired.
In step 704, the computing node obtains the processing result of the target preset computing task, and returns the processing result to the storage node pointed by the storage location pointer of the data point associated with the target preset computing task.
The computing node obtains the target pre-set after processing the target pre-set computing taskAnd setting a processing result of the calculation task, and further feeding back the processing result to a storage node pointed by a storage position pointer of a data point associated with the target preset calculation task by the task processing calculation node in order to ensure the precision of the subsequent task processing. The computing task { T } is still preset with the target Elei_1 -T Elei_2 For example, if the computing task is T Elei_1 And T Elei_2 Internal force calculation of (2) is performed during processing of the target preset calculation task { T } Elei_1 -T Elei_2 After } a T is obtained Elei_1 And T Elei_2 The internal force value of (2) is fed back to T Elei_1 To update T Elei_1 Attribute data of (2) and feeding back the internal force value to T Elei_2 To update T Elei_2 Attribute data of (a). And when other calculation task processing is carried out subsequently, processing is carried out according to the new attribute data.
In summary, according to the processing method for a scientific computing task provided by the embodiment of the invention, when a computing node processes a preset computing task allocated to the computing node, attribute data of data points can be rapidly extracted according to storage position pointers of the data points associated with the preset computing task, so that the execution efficiency of the scientific computing task is improved; and after the computing node processes the preset computing task, the processing result is written back to the storage node of the data point, so that the processing precision of the subsequent task is ensured.
Example IV
Referring to fig. 8, an embodiment of the present invention provides a processing apparatus for a scientific computing task, where the apparatus includes:
a computing node allocation module 801, configured to determine, for any preset computing task included in a scientific computing task, a computing node of the preset computing task; the preset calculation task is a data point-data point calculation task;
a storage node allocation module 802, configured to store, for any data point included in a scientific computing task, attribute data of the data point and a computing location pointer of the data point to a storage node of the data point; the computing position pointer of the data point is generated according to the node position of the computing node of the preset computing task participated by the data point;
The location reporting module 803 is configured to report, each time a storage location of a data point changes, the storage location of the data point after the change to a computing node pointed by a computing location pointer of the data point; and after the computing node receives the storage position of the data point, updating the storage position pointer of the data point.
In an alternative embodiment, the apparatus further comprises: a subspace dividing module (not shown in the figure) for dividing the problem space of the scientific computing task into a plurality of subspaces; establishing a mapping relation between each subspace and each storage node;
the storage node allocation module 802 is configured to: for any data point contained in a scientific calculation task, determining a target subspace in which the data point is located, and taking a storage node with a mapping relation with the target subspace as a storage node of the data point.
In an alternative embodiment, the subspace partitioning module is configured to: for any space dimension in the problem space, determining the number of storage nodes corresponding to the space dimension;
and uniformly segmenting the space dimension of the corresponding problem space according to the number of storage nodes corresponding to each space dimension, so that the segmentation segment number of the problem space in each space dimension is consistent with the number of storage nodes corresponding to the corresponding space dimension.
In an alternative embodiment, the apparatus further comprises: a migration module (not shown) for determining migration data points at which location migration occurs at the beginning of a new time step;
determining a target subspace where the migration data point is located after migration, and taking a storage node with a mapping relation with the target subspace as a storage node of the migration data point;
and migrating the attribute data of the migration data point and the calculation position pointer to a storage node of the migration data point.
In an alternative embodiment, the storage node allocation module 802 is configured to: generating storage information of the data point according to the attribute data of the data point, the calculated position pointer of the data point and the storage position of the data point; the stored information for the data point is stored in a storage node for the data point.
In an alternative embodiment, the apparatus further comprises: an updating module (not shown in the figure) is used for updating the storage position in the storage information of the data point every time the storage position of the data point changes.
In an alternative embodiment, the computing node allocation module 801 is configured to: and determining the computing nodes of each preset computing task contained in the scientific computing task according to the resource overhead of the preset computing task, the number of the computing nodes, the load of the computing nodes and/or the processing capacity of the computing nodes.
In an alternative embodiment, the computing node allocation module 801 is configured to: in the static allocation stage, allocating preset calculation tasks with the same quantity to each calculation node;
in the dynamic allocation stage, different numbers of preset computing tasks are allocated to each computing node according to the resource overhead of the preset computing tasks which are not allocated, the load of each computing node at present and/or the residual processing capacity of each computing node at present.
In an alternative embodiment, a computing node determines a target preset computing task to be currently executed, and determines data points associated with the target preset computing task; the computing node extracts attribute data of the associated data point from the storage node pointed by the storage position pointer of the associated data point; and the computing node processes a target preset computing task based on the extracted attribute data.
In an alternative embodiment, a computing node obtains a processing result of a target preset computing task, and returns the processing result to a storage node pointed by a storage location pointer of a data point associated with the target preset computing task.
In summary, according to the processing device for a scientific computing task provided by the embodiment of the invention, by recording the computing position pointer of a data point in the storage node of the data point, the storage position of the data point with position change can be timely reported to the computing node pointed by the computing position pointer, and the computing node generates the storage position pointer of the data point according to the reported storage position, so that the storage position of the attribute data currently storing the data point required for computing can be determined when the preset computing task is processed, the time for acquiring the attribute data of the data point is greatly reduced, and the execution efficiency of the scientific computing task is improved.
It should be noted that: in the processing device for a scientific computing task according to the above embodiment, only the division of the above functional modules is used for illustration when the scientific computing task is processed, and in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the processing device for the scientific computing task provided in the above embodiment and the processing method embodiment for the scientific computing task belong to the same concept, and specific implementation processes thereof are detailed in the method embodiment and are not described herein again.
Example five
Referring to FIG. 9, an embodiment of the present invention provides a computing system comprising: a processing device 901 for scientific computing tasks and a plurality of processing nodes 902.
The specific structure of the processing device 901 for scientific computing task may refer to the description in the fourth embodiment, and will not be described herein.
Each processing node 902 is used as a computing node of a preset computing task in the processing device 901 serving as a scientific computing task; and/or the processing node is used as a storage node of data points in the processing device 901 which is a scientific computing task.
Example six
An embodiment of the present invention provides a computing device including: a processor; the processor is used for executing the operation corresponding to the processing method of the scientific calculation task.
Example seven
The embodiment of the invention provides a computer readable storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute operations corresponding to the processing method of the scientific computing task.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims (14)

1. A method for processing a scientific computing task, the method comprising:
determining a computing node of a preset computing task aiming at any preset computing task contained in a scientific computing task; the preset calculation task is a data point-data point calculation task;
storing attribute data of any data point contained in a scientific calculation task and a calculation position pointer of the data point to a storage node of the data point; the computing position pointer of the data point is generated according to the node position of the computing node of the preset computing task participated by the data point;
whenever the storage position of a data point changes, reporting the storage position of the changed data point to a calculation node pointed by a calculation position pointer of the data point;
after receiving the storage position of the data point, the computing node updates the storage position pointer of the data point.
2. The method of claim 1, wherein prior to storing the attribute data of the data point and the calculated position pointer of the data point to the storage node of the data point for any data point included in the scientific computing task, the method further comprises:
Dividing the problem space of the scientific computing task into a plurality of subspaces;
establishing a mapping relation between each subspace and each storage node;
for any data point contained in a scientific calculation task, determining a target subspace in which the data point is located, and taking a storage node with a mapping relation with the target subspace as a storage node of the data point.
3. The method of claim 2, wherein the dividing the problem space of the scientific computing task into a plurality of subspaces further comprises:
for any space dimension in the problem space, determining the number of storage nodes corresponding to the space dimension;
and uniformly segmenting the space dimension of the corresponding problem space according to the number of storage nodes corresponding to each space dimension, so that the segmentation segment number of the problem space in each space dimension is consistent with the number of storage nodes corresponding to the corresponding space dimension.
4. The method according to claim 2, wherein the method further comprises:
determining a migration data point at which the location migration occurs at the beginning of the new time step;
determining a target subspace where the migration data point is located after migration, and taking a storage node with a mapping relation with the target subspace as a storage node of the migration data point;
And migrating the attribute data of the migration data point and the calculation position pointer to a storage node of the migration data point.
5. The method of claim 1, wherein storing the attribute data for the data point and the calculated location pointer for the data point to the storage node for the data point further comprises:
generating storage information of the data point according to the attribute data of the data point, the calculated position pointer of the data point and the storage position of the data point;
the stored information for the data point is stored in a storage node for the data point.
6. The method of claim 5, wherein the method further comprises:
the storage location in the stored information for a data point is updated each time the storage location for the data point changes.
7. The method of claim 1, wherein determining the computing node of the preset computing task for any preset computing task included in the scientific computing task further comprises:
and determining the computing nodes of each preset computing task contained in the scientific computing task according to the resource overhead of the preset computing task, the number of the computing nodes, the load of the computing nodes and/or the processing capacity of the computing nodes.
8. The method of claim 7, wherein determining the computing node of the preset computing task for any preset computing task included in the scientific computing task further comprises:
in the static allocation stage, allocating preset calculation tasks with the same quantity to each calculation node;
in the dynamic allocation stage, different numbers of preset computing tasks are allocated to each computing node according to the resource overhead of the preset computing tasks which are not allocated, the load of each computing node at present and/or the residual processing capacity of each computing node at present.
9. The method according to any one of claims 1-8, further comprising:
the method comprises the steps that a computing node determines a target preset computing task to be executed currently, and determines data points associated with the target preset computing task;
the computing node extracts attribute data of the associated data point from the storage node pointed by the storage position pointer of the associated data point;
and the computing node processes a target preset computing task based on the extracted attribute data.
10. The method of claim 9, wherein after the computing node performs the processing of the target preset computing task based on the extracted attribute data, the method further comprises:
The computing node obtains a processing result of a target preset computing task, and returns the processing result to a storage node pointed by a storage position pointer of a data point associated with the target preset computing task.
11. A processing device for scientific computing tasks, said device comprising:
the computing node distribution module is used for determining the computing node of a preset computing task aiming at any preset computing task contained in a scientific computing task; the preset calculation task is a data point-data point calculation task;
the storage node distribution module is used for storing attribute data of any data point contained in the scientific calculation task and a calculation position pointer of the data point to a storage node of the data point; the computing position pointer of the data point is generated according to the node position of the computing node of the preset computing task participated by the data point;
the position reporting module is used for reporting the changed storage position of the data point to a calculation node pointed by a calculation position pointer of the data point every time the storage position of the data point is changed; and after the computing node receives the storage position of the data point, updating the storage position pointer of the data point.
12. A computing system comprising the processing device of the scientific computing task of claim 11 and a plurality of processing nodes;
wherein the processing node is used as a computing node of a preset computing task in the processing device of the scientific computing task according to claim 11; and/or the processing node is used as a storage node of data points in the processing device of the scientific computing task according to claim 11.
13. A computing device, the computing device comprising: a processor;
the processor is configured to perform operations corresponding to the method for processing a scientific computing task according to any one of claims 1 to 10.
14. A computer-readable storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the method of processing a scientific computing task according to any one of claims 1-10.
CN202311791754.9A 2023-12-22 2023-12-22 Scientific computing task processing method, device and computing system Pending CN117873712A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311791754.9A CN117873712A (en) 2023-12-22 2023-12-22 Scientific computing task processing method, device and computing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311791754.9A CN117873712A (en) 2023-12-22 2023-12-22 Scientific computing task processing method, device and computing system

Publications (1)

Publication Number Publication Date
CN117873712A true CN117873712A (en) 2024-04-12

Family

ID=90582271

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311791754.9A Pending CN117873712A (en) 2023-12-22 2023-12-22 Scientific computing task processing method, device and computing system

Country Status (1)

Country Link
CN (1) CN117873712A (en)

Similar Documents

Publication Publication Date Title
CN109783237B (en) Resource allocation method and device
CN111488205B (en) Scheduling method and scheduling system for heterogeneous hardware architecture
CN102609303B (en) Slow-task dispatching method and slow-task dispatching device of Map Reduce system
CN109408590B (en) Method, device and equipment for expanding distributed database and storage medium
CN112465146B (en) Quantum and classical hybrid cloud platform and task execution method
US10664555B2 (en) Two-stage distributed estimation system
CN106569892B (en) Resource scheduling method and equipment
CN111782404A (en) Data processing method and related equipment
CN108833592A (en) Cloud host schedules device optimization method, device, equipment and storage medium
US20210390405A1 (en) Microservice-based training systems in heterogeneous graphic processor unit (gpu) cluster and operating method thereof
CN116701001B (en) Target task allocation method and device, electronic equipment and storage medium
Touheed et al. A comparison of some dynamic load-balancing algorithms for a parallel adaptive flow solver
CN116431315A (en) Batch processing task processing method and device, electronic equipment and storage medium
CN117873712A (en) Scientific computing task processing method, device and computing system
Pu et al. MPEFT: A novel task scheduling method for workflows
Yassir et al. Graph-based model and algorithm for minimising big data movement in a cloud environment
CN112948087A (en) Task scheduling method and system based on topological sorting
US20210149746A1 (en) Method, System, Computer Readable Medium, and Device for Scheduling Computational Operation Based on Graph Data
CN115033377A (en) Service resource prediction method and device based on cluster server and electronic equipment
CN114691873A (en) Semantic processing method and device for automatic driving log data and storage medium
CN109617954B (en) Method and device for creating cloud host
CN109145052B (en) Data partition storage method, device, system, storage medium and electronic device
CN113157427A (en) Task migration method and device, computer equipment and readable storage medium
CN110750330A (en) Virtual machine creating method, system, electronic equipment and storage medium
Hussain et al. A novel approach of fair scheduling to enhance performance of hadoop distributed file system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination