CN117608862B - Data distribution control method, device, equipment and medium - Google Patents

Data distribution control method, device, equipment and medium Download PDF

Info

Publication number
CN117608862B
CN117608862B CN202410085838.9A CN202410085838A CN117608862B CN 117608862 B CN117608862 B CN 117608862B CN 202410085838 A CN202410085838 A CN 202410085838A CN 117608862 B CN117608862 B CN 117608862B
Authority
CN
China
Prior art keywords
data
computer
load
task
subtask
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202410085838.9A
Other languages
Chinese (zh)
Other versions
CN117608862A (en
Inventor
董万根
周恩著
田冬雷
于建
陈霞玲
魏祖志
王金海
杨军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Gold Products Computer Technology Tianjin Co ltd
Original Assignee
Gold Products Computer Technology Tianjin Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Gold Products Computer Technology Tianjin Co ltd filed Critical Gold Products Computer Technology Tianjin Co ltd
Priority to CN202410085838.9A priority Critical patent/CN117608862B/en
Publication of CN117608862A publication Critical patent/CN117608862A/en
Application granted granted Critical
Publication of CN117608862B publication Critical patent/CN117608862B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multi Processors (AREA)

Abstract

The application relates to a data distribution control method, a device, equipment and a medium, which are applied to the technical field of computers, and the method comprises the following steps: acquiring task data to be processed; determining the data type of the task data to be processed; determining a computer group corresponding to the task data to be processed based on the data type of the task data to be processed; and determining an allocation computer based on the computer group and the task data to be processed, wherein the allocation computer is a computer with load smaller than a load threshold value. The method and the device have the effect of improving the efficiency and the accuracy of task data processing.

Description

Data distribution control method, device, equipment and medium
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for controlling data distribution.
Background
With the rapid development of the fields of big data, artificial intelligence and the like, large-scale data processing and task allocation become more common, and in a group of computer clusters, how to distribute a large amount of task data, so that it becomes important to reasonably allocate computer resources and efficiently perform data processing.
In the related art, when task data is distributed, the task data is often distributed according to the load rate of a computer, however, because the task data is very complex, the task data is distributed only according to the load rate of the computer, and the accuracy and the efficiency of task data processing cannot be ensured.
Disclosure of Invention
In order to improve the efficiency and accuracy of task data processing, the application provides a data distribution control method, a device, equipment and a medium.
In a first aspect, the present application provides a data distribution control method, which adopts the following technical scheme:
a data distribution control method, comprising:
acquiring task data to be processed;
determining the data type of the task data to be processed;
determining a computer group corresponding to the task data to be processed based on the data type of the task data to be processed;
and determining an allocation computer based on the computer group and the task data to be processed, wherein the allocation computer is a computer with load smaller than a load threshold value.
By adopting the technical scheme, after the task data to be processed is obtained, the data type of the task data to be processed is determined, the computer group which can process the data type more accurately and efficiently is selected according to the data type, and the distribution computer for processing the task data to be processed is determined according to the load of each computer in the computer group.
Optionally, before the determining, based on the data type of the task data to be processed, a computer group corresponding to the task data to be processed, the method further includes:
acquiring computer information;
determining a computer attribute based on the computer information, the computer attribute being used to characterize data characteristics processed by a computer;
and grouping the computers based on the computer attributes to obtain at least one computer group, wherein the computers in each computer group correspond to the same computer attribute.
By adopting the technical scheme, before the computer groups are determined, the computer attributes are determined according to the computer information, the computers are grouped according to the computer attributes, the computers in each computer group correspond to the same computer attributes, and when data distribution is carried out, the corresponding distribution computer can be determined more conveniently according to the computer groups, and the data distribution efficiency is improved without determining from a large number of computers in the computer cluster.
Optionally, the determining, based on the computer group and the task data to be processed, an allocation computer includes:
acquiring computer unit information, wherein the computer unit information comprises the current load of each computer in the computer unit;
Calculating the idle load of each computer based on the computer group information and a load threshold value, wherein the idle load is a load which can be born by each computer;
determining task load based on the task data to be processed;
the distribution computer is determined based on the task load and the idle load.
By adopting the technical scheme, after the computer group is determined, the distribution computer is determined according to the idle load of each computer and the task load of the task data to be processed, and the distribution computer can timely process the task data to be distributed, so that the efficiency of processing the task data to be processed is improved.
Optionally, the determining the allocation computer based on the task load and the idle load includes:
if the idle load is smaller than the task load, dividing the task data to be processed to obtain a plurality of subtask data;
calculating a sum of idle loads, wherein the sum of idle loads is the sum of the idle loads of all computers in the computer group;
if the idle load sum is greater than or equal to the task load, determining the subtask load of each subtask data;
The distribution computer is determined based on the subtask load and the idle load.
By adopting the technical scheme, when task data to be processed is complex and the task load is large, the idle load of each computer can not bear the processing process of the task data to be processed, the task data to be processed is divided into a plurality of subtask data, if the total of the idle loads is greater than or equal to the task load, the distribution computer is determined according to the subtask load and the idle load, the task data to be processed is divided into a plurality of subtask data to be processed respectively, and the processing efficiency of the task data to be processed is improved.
Optionally, after the calculating the sum of idle loads, the method further includes:
if the idle load sum is smaller than the task load, determining the logic relation of each subtask data;
determining task processing grades of all the subtask data based on the logic relation, wherein the task processing grades comprise a first grade, and the subtask data corresponding to the first grade is the subtask data needing to be processed preferentially;
determining the subtask data with the task processing grade of the first grade as priority subtask data;
Determining the number of the priority subtask data as a first number;
determining the number of computers with the idle load larger than a preset load in the computer group as a second number;
the allocation computer is determined based on a magnitude relation of the first number and the second number.
By adopting the technical scheme, if the total idle load is smaller than the task load, determining the task processing grade of each subtask data according to the logic relationship, determining the distribution computer according to the number of the prior subtask data and the second number, and considering the logic relationship among the subtask data when the subtask data is processed, thereby improving the processing efficiency of the task data to be processed.
Optionally, the determining the allocation computer based on the magnitude relation of the first number and the second number includes:
if the first quantity is larger than the second quantity, determining a second computer group based on the data type of the priority subtask data;
acquiring an idle load corresponding to the computer in the second computer group;
the distribution computer is determined based on the subtask load and the idle load.
By adopting the technical scheme, if the first quantity is larger than the second quantity, the current computer set cannot timely bear the processing requirement, the second computer set is determined again, the two computer sets cooperatively process the task data to be processed, and the efficiency of processing the task data to be processed is improved.
Optionally, after the determining of the allocation computer based on the set of computers and the task data to be processed, the method further comprises:
acquiring state data of the distribution computer;
judging whether the distribution computer is abnormal or not based on the state data;
if the distribution computer is abnormal, determining the task data to be processed corresponding to the abnormal distribution computer;
and determining a distribution computer based on the task load of the task data to be processed.
By adopting the technical scheme, whether the distribution computer is abnormal or not is judged through the state data of the distribution computer, if the distribution computer is abnormal, the corresponding task data to be processed is redistributed, the distribution computer is monitored in real time, and the possibility that the distribution computer is abnormal and cannot process the task data to be distributed in time is reduced.
In a second aspect, the present application provides a data distribution control device, which adopts the following technical scheme:
A data distribution control apparatus comprising:
the data acquisition module is used for acquiring task data to be processed;
the type determining module is used for determining the data type of the task to be processed;
the combination determining module is used for determining a computer group corresponding to the task to be processed based on the data type of the task to be processed;
and the distribution determining module is used for determining a distribution computer based on the computer group and the task to be processed, wherein the distribution computer is a computer with a load smaller than a load threshold value.
By adopting the technical scheme, after the task data to be processed is obtained, the data type of the task data to be processed is determined, the computer group which can process the data type more accurately and efficiently is selected according to the data type, and the distribution computer for processing the task data to be processed is determined according to the load of each computer in the computer group.
In a third aspect, the present application provides an electronic device, which adopts the following technical scheme:
an electronic device comprising a processor coupled with a memory;
The memory has stored thereon a computer program that can be loaded by a processor and that executes the data distribution control method according to any one of the first aspects.
In a fourth aspect, the present application provides a computer readable storage medium, which adopts the following technical scheme:
a computer-readable storage medium storing a computer program capable of being loaded by a processor and executing the data distribution control method according to any one of the first aspects.
Drawings
Fig. 1 is a flow chart of a data distribution control method according to an embodiment of the present application.
Fig. 2 is a block diagram of a data distribution control device according to an embodiment of the present application.
Fig. 3 is a block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The present application is described in further detail below with reference to the accompanying drawings.
The embodiment of the application provides a data distribution control method, which can be executed by electronic equipment, wherein the electronic equipment can be a server or terminal equipment, the server can be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, and a cloud server for providing cloud computing service. The terminal device may be, but is not limited to, a smart phone, a tablet computer, a desktop computer, etc.
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
In addition, the term "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In this context, unless otherwise specified, the term "/" generally indicates that the associated object is an "or" relationship.
As shown in fig. 1, a data distribution control method is described as follows (steps S101 to S104):
step S101, task data to be processed are acquired.
The task data to be processed is related data of a task to be processed, namely, related data of the task to be processed, and since a large amount of task data to be processed exists in many application scenarios, there are many sources of data, for example: the task data to be processed can be obtained from the user request, the task data to be processed can be obtained from the sensor, and the task data to be processed can be obtained from the log file.
Step S102, determining the data type of the task data to be processed.
The data types of the task data to be processed, such as an image type, an audio type, a numerical value type, etc., cover multiple fields and types, and if the task to be processed is to identify or process a video or an image, the data type of the task data to be processed is an image type, if the task to be processed is to identify or process an audio, the data type of the task data to be processed is an audio type, and if the task to be processed is to analyze or calculate a numerical value, the data type of the task data to be processed is a numerical value type.
Step S103, determining a computer group corresponding to the task data to be processed based on the data type of the task data to be processed.
In order to improve the efficiency and accuracy of data processing of tasks to be processed, when data distribution is performed, a more adaptive computer needs to be selected for data processing, namely a computer group capable of more efficiently and accurately processing the data of the data type is selected.
Specifically, before determining the computer group corresponding to the task data to be processed based on the data type of the task data to be processed, the method further includes: acquiring computer information; determining computer attributes based on the computer information, the computer attributes being used to characterize data characteristics processed by the computer; grouping the computers based on the computer attributes to obtain at least one computer group, wherein the computers in each computer group correspond to the same computer attribute.
In this embodiment, a large number of computers for processing task data to be processed are included in the computer cluster, relevant information, i.e., computer information, of all computers in the computer cluster is obtained from a database, and because of differences in software or hardware configuration of the computers, each computer can provide different computer resources, i.e., data that is good for processing will be different, the computer information includes data features that are good for processing of the computers, the data features that are good for processing are determined as computer attributes, the computer attributes include image attributes, numerical attributes and audio attributes, and the computers with the same computer attributes are divided into one computer group, so that at least one computer group is obtained, and the computers in each computer group have the same computer attribute.
Step S104, determining an allocation computer based on the computer group and the task data to be processed.
The computer is allocated to be a computer with a load smaller than a load threshold, the load threshold is preset by a worker according to the configuration condition of the computer, and when the load of the computer exceeds the load threshold, the computer may fail, for example: computer crash, etc.
When the data of the task to be processed is distributed, the data is distributed according to the loads of all computers in the corresponding computer group, so that the task to be processed can be processed in time.
Specifically, determining an allocation computer based on the computer group and the task data to be processed includes: acquiring computer unit information, wherein the computer unit information comprises the current load of each computer in a computer unit; calculating the idle load of each computer based on the computer group information and the load threshold value, wherein the idle load is the load born by each computer; determining task load based on task data to be processed; the allocation computer is determined based on the task load and the idle load.
In this embodiment, corresponding computer group information is obtained from a database, the current load of each computer in the corresponding computer group is known through the computer group information, the load threshold value minus the current load of each computer can obtain the idle load of each computer, that is, the current load that each computer can bear, the corresponding relation between each type of task data to be processed and the task load is stored in the database, the database is constructed according to the situation that the historical task data to be processed occupies the load, the corresponding task load is obtained from the database according to the task data to be processed, the computer with the idle load greater than or equal to the task load is determined as the first distribution computer, if a plurality of first distribution computers exist, the first distribution computer with the largest idle load is selected as the distribution computer corresponding to the task data to be processed, and if only one first distribution computer exists, the first distribution computer is determined as the distribution computer corresponding to the task data to be processed.
Further, determining a distribution computer based on the task load and the idle load, comprising: if the idle load is smaller than the task load, dividing task data to be processed to obtain a plurality of subtask data; calculating the sum of idle loads, wherein the sum of idle loads is the sum of idle loads of all computers in the computer group; if the idle load sum is greater than or equal to the task load, determining the subtask load of each subtask data; the allocation computer is determined based on the subtask load and the idle load.
In this embodiment, if the idle load of all the computers in the corresponding computer group is smaller than the task load, the task to be processed is divided to obtain a plurality of subtasks, the corresponding task data to be processed is also divided into a plurality of subtask data, the idle load sum of all the computers in the current computer group (i.e., the computer group corresponding to the task data to be processed) is calculated, if the idle load sum is greater than or equal to the task load, i.e., the current computer group can bear the processing work of the task data to be processed, the corresponding subtask load is obtained from the database according to the subtask data, and the allocation computer corresponding to each subtask data is determined according to the subtask load and the idle load.
When the corresponding relation between the allocation computer and the subtask data is determined, the allocation computer is sequentially determined according to the subtask load of the subtask data from top to bottom, and when the allocation computer corresponding to each subtask data is determined, the allocation computer with the smallest difference value between the idle load and the subtask load is determined as the allocation computer corresponding to the subtask data.
Specifically, after calculating the sum of idle loads, the method further comprises: if the sum of the idle loads is smaller than the task load, determining the logic relation of each subtask data; determining task processing grades of all the subtask data based on the logic relation, wherein the task processing grades comprise a first grade, and the subtask data corresponding to the first grade is the subtask data needing to be processed preferentially; determining subtask data with a task processing level of a first level as priority subtask data; determining a number of priority subtask data as a first number; determining the number of computers with idle loads larger than a preset load in the computer group as a second number; the allocation computer is determined based on a size relationship of the first number and the second number.
In this embodiment, if the sum of idle loads of all computers in the current computer group is smaller than the task load, there may be a case that the current computer group cannot completely process the task data to be processed, at this time, it is necessary to determine to allocate the computers through the following judgment, and determine the logic relationship between each subtask through the division manner of the task to be processed, where the logic relationship includes a logic relationship between each subtask, that is, the logic relationship between each subtask data, which has a sequence and is not related to each other.
Determining task processing grades of all the subtask data according to the logic relation among the subtask data, wherein the task processing grades comprise a first grade, a second grade and a third grade, if the subtask data has follow-up subtask data and does not have lead subtask data, namely the subtask data has influence on other subtask data, the corresponding follow-up subtask data can be processed only after the subtask data is processed, and the lead subtask data which has influence on the subtask data does not exist, the task processing grade corresponding to the subtask data is the first grade; if the subtask data has both the preamble subtask data and the follow-up subtask data, namely, the subtask data can be processed only after the preamble subtask data of the subtask data is processed, and the corresponding follow-up subtask data can be processed after the subtask data is processed, the task processing grade corresponding to the subtask data is a second grade; if the subtask data has the preamble subtask data and the follow-up subtask data does not exist, namely the subtask data does not influence other subtask data, and only after the preamble subtask data corresponding to the subtask data is processed, the subtask data can be processed, and then the task processing grade corresponding to the subtask data is a third grade.
The subtask data of the first level is required to be processed preferentially, the subtask data of the first level of task processing level is determined to be the priority subtask data, the number of the priority subtask data is counted, the number of computers with idle loads larger than the preset load in the computer group is counted to be the first number, the number is determined to be the second number, the preset load is the minimum load in the subtask loads corresponding to the priority subtask data, the first number and the second number are compared, and the allocation computer is determined according to the size relation of the first number and the second number.
Further, determining an allocation computer based on the magnitude relation of the first number and the second number, comprising: if the first number is greater than the second number, determining a second computer group based on the data type of the priority subtask data; acquiring an idle load corresponding to a computer in the second computer group; the allocation computer is determined based on the subtask load and the idle load.
In this embodiment, if the first number is greater than the second number, it is indicated that the computers in the current computer group cannot meet the requirement of timely processing the priority subtask data, so that the task data to be processed cannot be timely processed, at this time, another computer group needs to be selected to cooperate for processing, the selected computer group is determined to be the second computer group, and when the second computer group is selected, the selection is also performed according to the data type, and the data type of the task data to be processed is not identical to the data type of each subtask data, for example: the data type of the task data to be processed is an image type, namely the task to be processed mainly processes an image, but the task to be processed only processes the image, and a task for numerical value and/or audio processing exists, so that the corresponding data type also needs to be determined for the sub-task data after division.
Determining a second computer group according to the data types of the priority subtask data, counting the number of the priority subtask data corresponding to each data type, determining the data type corresponding to the task data to be processed as a first data type, determining the data type with the highest number of the priority subtask data corresponding to other data types (the data types except the first data type are other data types) as the priority data type, determining the corresponding computer attribute according to the priority data type, namely the second computer attribute, determining the computer group with the computer attribute being the second computer attribute as the second computer group, acquiring the current load of each computer in the second computer group from a database, and subtracting the current load from the idle load corresponding to each computer.
And sequencing the computers in the second computer group from top to bottom according to the idle load, and selecting a third number of computers before the computers are selected as distribution computers, wherein the third number is equal to the first number minus the second number.
The method comprises the steps that a distribution computer exists in a current computer set and a second computer set, all the distribution computers are determined to be a task computer set at the moment, namely, when data distribution is carried out, the computer set is not fixed and can be adjusted at any time according to a task to be processed, when the corresponding relation between the distribution computer and priority subtask data is determined, the distribution computers are sequentially determined from top to bottom according to the subtask load of the priority subtask data, and when the distribution computer corresponding to each priority subtask data is determined, the distribution computer with the smallest difference value between idle load and subtask load is determined to be the distribution computer corresponding to the priority subtask data.
When determining the allocation computer corresponding to the subtask data of the second level or the third level, the allocation method of the priority subtask data of the first level is determined in the same manner, and is not repeated here.
Specifically, after determining the allocation computer based on the computer group and the task data to be processed, the method further includes: acquiring state data of an allocation computer; judging whether the distribution computer is abnormal or not based on the state data; if the distribution computer is abnormal, determining task data to be processed corresponding to the abnormal distribution computer; the allocation computer is determined based on the task load of the task data to be processed.
When the distribution computer processes task data and/or subtask data to be processed, state data, such as: the electric quantity, the load, the heat and the like, judging whether the distribution computer has an abnormality according to the state data, and if the abnormality exists, affecting the data processing, so that the task to be processed cannot be processed in time, and the abnormality exists, for example: if the distribution computer is abnormal, determining to-be-processed task data and/or subtask data corresponding to the abnormal distribution computer according to the corresponding relation, and re-determining the distribution computer according to the task load corresponding to the to-be-processed task data and/or the subtask load corresponding to the subtask data, wherein the determination mode is the same as that of the method and is not repeated here.
Fig. 2 is a block diagram of a data distribution control device 200 according to an embodiment of the present application.
As shown in fig. 2, the data distribution control apparatus 200 mainly includes:
a data acquisition module 201, configured to acquire task data to be processed;
a type determining module 202, configured to determine a data type of a task to be processed;
the combination determining module 203 is configured to determine a computer group corresponding to a task to be processed based on a data type of the task to be processed;
the allocation determining module 204 is configured to determine an allocation computer based on the computer group and the task to be processed, where the allocation computer is a computer with a load less than a load threshold.
As an optional implementation manner of this embodiment, the combination determining module 203 is further specifically configured to, before determining, based on the data type of the task data to be processed, a computer group corresponding to the task data to be processed, include: acquiring computer information; determining computer attributes based on the computer information, the computer attributes being used to characterize data characteristics processed by the computer; grouping the computers based on the computer attributes to obtain at least one computer group, wherein the computers in each computer group correspond to the same computer attribute.
As an optional implementation manner of this embodiment, the allocation determining module 204 is further specifically configured to determine, based on the computer group and the task data to be processed, an allocation computer, including: acquiring computer unit information, wherein the computer unit information comprises the current load of each computer in a computer unit; calculating the idle load of each computer based on the computer group information and the load threshold value, wherein the idle load is the load born by each computer; determining task load based on task data to be processed; the allocation computer is determined based on the task load and the idle load.
As an optional implementation manner of this embodiment, the allocation determining module 204 is further specifically configured to determine, based on the task load and the idle load, an allocation computer, including: if the idle load is smaller than the task load, dividing task data to be processed to obtain a plurality of subtask data; calculating the sum of idle loads, wherein the sum of idle loads is the sum of idle loads of all computers in the computer group; if the idle load sum is greater than or equal to the task load, determining the subtask load of each subtask data; the allocation computer is determined based on the subtask load and the idle load.
As an alternative implementation manner of this embodiment, the allocation determining module 204 is further specifically configured to, after calculating the sum of idle loads, further include: if the sum of the idle loads is smaller than the task load, determining the logic relation of each subtask data; determining task processing grades of all the subtask data based on the logic relation, wherein the task processing grades comprise a first grade, and the subtask data corresponding to the first grade is the subtask data needing to be processed preferentially; determining subtask data with a task processing level of a first level as priority subtask data; determining a number of priority subtask data as a first number; determining the number of computers with idle loads larger than a preset load in the computer group as a second number; the allocation computer is determined based on a size relationship of the first number and the second number.
As an alternative implementation of the present embodiment, the allocation determining module 204 is further specifically configured to determine an allocation computer based on the size relationship of the first number and the second number, including: if the first number is greater than the second number, determining a second computer group based on the data type of the priority subtask data; acquiring an idle load corresponding to a computer in the second computer group; the allocation computer is determined based on the subtask load and the idle load.
As an alternative implementation manner of the present embodiment, the data distribution control apparatus 200 is further specifically configured to, after determining the distribution computer based on the computer group and the task data to be processed, further include: acquiring state data of an allocation computer; judging whether the distribution computer is abnormal or not based on the state data; if the distribution computer is abnormal, determining task data to be processed corresponding to the abnormal distribution computer; the allocation computer is determined based on the task load of the task data to be processed.
In one example, a module in any of the above apparatuses may be one or more integrated circuits configured to implement the above methods, for example: one or more application specific integrated circuits (application specific integratedcircuit, ASIC), or one or more digital signal processors (digital signal processor, DSP), or one or more field programmable gate arrays (field programmable gate array, FPGA), or a combination of at least two of these integrated circuit forms.
For another example, when a module in an apparatus may be implemented in the form of a scheduler of processing elements, the processing elements may be general-purpose processors, such as a central processing unit (central processing unit, CPU) or other processor that may invoke a program. For another example, the modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the apparatus and modules described above may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.
Fig. 3 is a block diagram of an electronic device 300 according to an embodiment of the present application.
As shown in FIG. 3, electronic device 300 includes a processor 301 and memory 302, and may further include an information input/information output (I/O) interface 303, one or more of a communication component 304, and a communication bus 305.
Wherein the processor 301 is configured to control the overall operation of the electronic device 300 to perform all or part of the steps of the data distribution control method described above; the memory 302 is used to store various types of data to support operation at the electronic device 300, which may include, for example, instructions for any application or method operating on the electronic device 300, as well as application-related data. The Memory 302 may be implemented by any type or combination of volatile or non-volatile Memory devices, such as one or more of static random access Memory (Static Random Access Memory, SRAM), electrically erasable programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), erasable programmable Read-Only Memory (Erasable Programmable Read-Only Memory, EPROM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk, or optical disk.
The I/O interface 303 provides an interface between the processor 301 and other interface modules, which may be a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 304 is used for wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as Wi-Fi, bluetooth, near field communication (Near Field Communication, NFC for short), 2G, 3G, or 4G, or a combination of one or more thereof, and accordingly the communication component 304 can include: wi-Fi part, bluetooth part, NFC part.
The electronic device 300 may be implemented by one or more application specific integrated circuits (Application Specific Integrated Circuit, abbreviated as ASIC), digital signal processors (Digital Signal Processor, abbreviated as DSP), digital signal processing devices (Digital Signal Processing Device, abbreviated as DSPD), programmable logic devices (Programmable Logic Device, abbreviated as PLD), field programmable gate arrays (Field Programmable Gate Array, abbreviated as FPGA), controllers, microcontrollers, microprocessors, or other electronic components for performing the data distribution control methods given by the above embodiments.
Communication bus 305 may include a pathway to transfer information between the aforementioned components. The communication bus 305 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture ) bus, or the like. The communication bus 305 may be divided into an address bus, a data bus, a control bus, and the like.
The electronic device 300 may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), car terminals (e.g., car navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like, and may also be a server, and the like.
The present application also provides a computer-readable storage medium having a computer program stored thereon, which when executed by a processor, implements the steps of the data distribution control method described above.
The computer readable storage medium may include: a U-disk, a removable hard disk, a read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
The foregoing description is only of the preferred embodiments of the present application and is presented as a description of the principles of the technology being utilized. It will be appreciated by persons skilled in the art that the scope of the application referred to in this application is not limited to the specific combinations of features described above, but it is intended to cover other embodiments in which any combination of features described above or their equivalents is possible without departing from the spirit of the application. Such as the above-mentioned features and the technical features having similar functions (but not limited to) applied for in this application are replaced with each other.

Claims (6)

1. A data distribution control method, characterized by comprising:
acquiring task data to be processed;
determining the data type of the task data to be processed;
determining a computer group corresponding to the task data to be processed based on the data type of the task data to be processed;
Determining an allocation computer based on the computer group and the task data to be processed, wherein the allocation computer is a computer with load smaller than a load threshold value;
the determining an allocation computer based on the computer group and the task data to be processed comprises the following steps:
acquiring computer unit information, wherein the computer unit information comprises the current load of each computer in the computer unit;
calculating the idle load of each computer based on the computer group information and a load threshold value, wherein the idle load is a load which can be born by each computer;
acquiring task load from a database based on the task data to be processed, wherein the database is constructed according to the condition that the historical task data to be processed occupies the load;
determining the distribution computer based on the task load and the idle load;
the determining the distribution computer based on the task load and the idle load includes:
if the idle load is smaller than the task load, dividing the task data to be processed to obtain a plurality of subtask data;
calculating a sum of idle loads, wherein the sum of idle loads is the sum of the idle loads of all computers in the computer group;
If the idle load sum is greater than or equal to the task load, determining the subtask load of each subtask data; sequencing the subtask data from top to bottom according to the corresponding subtask load to obtain a sequencing result; determining an allocation computer for the subtask data in turn based on the sequencing result, wherein the allocation computer is a computer with the minimum difference value between the idle load and the subtask load; after said calculating the sum of idle loads, the method further comprises:
if the idle load sum is smaller than the task load, determining a logic relation of each subtask data, wherein the logic relation comprises a presence sequence and mutual independence;
determining task processing grades of all the subtask data based on the logic relation, wherein the task processing grades comprise a first grade, a second grade and a third grade, and the subtask data corresponding to the first grade is the subtask data needing to be processed preferentially;
determining the subtask data with the task processing grade of the first grade as priority subtask data;
determining the number of the priority subtask data as a first number;
Determining the number of computers with the idle load larger than a preset load in the computer group as a second number;
determining the allocation computer based on a magnitude relation of the first number and the second number;
the determining task processing level of each subtask data based on the logical relationship includes:
if the subtask data has the follow-up subtask data and the subtask data does not have the lead subtask data, the task processing grade corresponding to the subtask data is a first grade;
if the subtask data has the front subtask data and the rear subtask data, the task processing grade corresponding to the subtask data is a second grade;
if the subtask data have the preamble subtask data and the follow-up subtask data do not exist, the task processing grade corresponding to the subtask data is a third grade;
the determining the allocation computer based on the magnitude relation of the first number and the second number includes:
if the first quantity is larger than the second quantity, determining a second computer group based on the data type of the priority subtask data;
acquiring an idle load corresponding to the computer in the second computer group;
The distribution computer is determined based on the subtask load and the idle load.
2. The method of claim 1, wherein prior to the determining the set of computers to which the task data to be processed corresponds based on the data type of the task data to be processed, the method further comprises:
acquiring computer information;
determining a computer attribute based on the computer information, the computer attribute being used to characterize data characteristics processed by a computer;
and grouping the computers based on the computer attributes to obtain at least one computer group, wherein the computers in each computer group correspond to the same computer attribute.
3. The method of claim 1, wherein after said determining an allocation computer based on said set of computers and said task data to be processed, said method further comprises:
acquiring state data of the distribution computer;
judging whether the distribution computer is abnormal or not based on the state data;
if the distribution computer is abnormal, determining the task data to be processed corresponding to the abnormal distribution computer;
and determining a distribution computer based on the task load of the task data to be processed.
4. A data distribution control apparatus, comprising:
the data acquisition module is used for acquiring task data to be processed;
the type determining module is used for determining the data type of the task to be processed;
the combination determining module is used for determining a computer group corresponding to the task to be processed based on the data type of the task to be processed;
the distribution determining module is used for determining a distribution computer based on the computer group and the task to be processed, wherein the distribution computer is a computer with load smaller than a load threshold value;
the allocation determining module is further specifically configured to determine an allocation computer based on the computer group and the task data to be processed, and includes:
acquiring computer unit information, wherein the computer unit information comprises the current load of each computer in the computer unit;
calculating the idle load of each computer based on the computer group information and a load threshold value, wherein the idle load is a load which can be born by each computer;
acquiring task load from a database based on the task data to be processed, wherein the database is constructed according to the condition that the historical task data to be processed occupies the load;
Determining the distribution computer based on the task load and the idle load;
the allocation determination module is further specifically configured to determine the allocation computer based on the task load and the idle load, and includes:
if the idle load is smaller than the task load, dividing the task data to be processed to obtain a plurality of subtask data;
calculating a sum of idle loads, wherein the sum of idle loads is the sum of the idle loads of all computers in the computer group;
if the idle load sum is greater than or equal to the task load, determining the subtask load of each subtask data; sequencing the subtask data from top to bottom according to the corresponding subtask load to obtain a sequencing result; determining an allocation computer for the subtask data in turn based on the sequencing result, wherein the allocation computer is a computer with the minimum difference value between the idle load and the subtask load; the allocation determining module is further specifically configured to, after the calculating the sum of idle loads, further include:
if the idle load sum is smaller than the task load, determining a logic relation of each subtask data, wherein the logic relation comprises a presence sequence and mutual independence;
Determining task processing grades of all the subtask data based on the logic relation, wherein the task processing grades comprise a first grade, a second grade and a third grade, and the subtask data corresponding to the first grade is the subtask data needing to be processed preferentially;
determining the subtask data with the task processing grade of the first grade as priority subtask data;
determining the number of the priority subtask data as a first number;
determining the number of computers with the idle load larger than a preset load in the computer group as a second number;
determining the allocation computer based on a magnitude relation of the first number and the second number;
the allocation determining module is further specifically configured to determine a task processing level of each subtask data based on the logical relationship, and includes:
if the subtask data has the follow-up subtask data and the subtask data does not have the lead subtask data, the task processing grade corresponding to the subtask data is a first grade;
if the subtask data has the front subtask data and the rear subtask data, the task processing grade corresponding to the subtask data is a second grade;
If the subtask data have the preamble subtask data and the follow-up subtask data do not exist, the task processing grade corresponding to the subtask data is a third grade;
the allocation determination module is further specifically configured to determine the allocation computer based on the magnitude relation of the first number and the second number, and includes:
if the first quantity is larger than the second quantity, determining a second computer group based on the data type of the priority subtask data;
acquiring an idle load corresponding to the computer in the second computer group;
the distribution computer is determined based on the subtask load and the idle load.
5. An electronic device comprising a processor coupled to a memory;
the processor is configured to execute a computer program stored in the memory to cause the electronic device to perform the method of any one of claims 1 to 3.
6. A computer readable storage medium comprising a computer program or instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 3.
CN202410085838.9A 2024-01-22 2024-01-22 Data distribution control method, device, equipment and medium Active CN117608862B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410085838.9A CN117608862B (en) 2024-01-22 2024-01-22 Data distribution control method, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410085838.9A CN117608862B (en) 2024-01-22 2024-01-22 Data distribution control method, device, equipment and medium

Publications (2)

Publication Number Publication Date
CN117608862A CN117608862A (en) 2024-02-27
CN117608862B true CN117608862B (en) 2024-04-12

Family

ID=89944707

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410085838.9A Active CN117608862B (en) 2024-01-22 2024-01-22 Data distribution control method, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN117608862B (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105729491A (en) * 2016-04-01 2016-07-06 纳恩博（北京）科技有限公司 Executing method, device and system for robot task
CN109343962A (en) * 2018-10-26 2019-02-15 北京知道创宇信息技术有限公司 Data processing method, device and distribution service
CN110673944A (en) * 2018-07-03 2020-01-10 杭州海康威视数字技术股份有限公司 Method and device for executing task
CN111459678A (en) * 2020-04-02 2020-07-28 上海极链网络科技有限公司 Resource scheduling method and device, storage medium and electronic equipment
CN116700955A (en) * 2023-05-22 2023-09-05 曙光信息产业(北京)有限公司 Job processing method, apparatus, computer device, and readable storage medium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150236974A1 (en) * 2013-04-26 2015-08-20 Hitachi, Ltd. Computer system and load balancing method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105729491A (en) * 2016-04-01 2016-07-06 纳恩博（北京）科技有限公司 Executing method, device and system for robot task
CN110673944A (en) * 2018-07-03 2020-01-10 杭州海康威视数字技术股份有限公司 Method and device for executing task
CN109343962A (en) * 2018-10-26 2019-02-15 北京知道创宇信息技术有限公司 Data processing method, device and distribution service
CN111459678A (en) * 2020-04-02 2020-07-28 上海极链网络科技有限公司 Resource scheduling method and device, storage medium and electronic equipment
CN116700955A (en) * 2023-05-22 2023-09-05 曙光信息产业(北京)有限公司 Job processing method, apparatus, computer device, and readable storage medium

Also Published As

Publication number Publication date
CN117608862A (en) 2024-02-27

Similar Documents

Publication Publication Date Title
US10558498B2 (en) Method for scheduling data flow task and apparatus
US8726290B2 (en) System and/or method for balancing allocation of data among reduce processes by reallocation
CN107124472A (en) Load-balancing method and device, computer-readable recording medium
CN108205469B (en) MapReduce-based resource allocation method and server
CN106874100B (en) Computing resource allocation method and device
CN112256417B (en) Data request processing method and device and computer readable storage medium
CN112000467A (en) Data tilt processing method and device, terminal equipment and storage medium
CN111813535A (en) Resource configuration determining method and device and electronic equipment
CN113703975A (en) Model distribution method and device, electronic equipment and computer readable storage medium
CN111382141B (en) Master-slave architecture configuration method, device, equipment and computer readable storage medium
CN117608862B (en) Data distribution control method, device, equipment and medium
CN107391262B (en) Job scheduling method and device
US20140047454A1 (en) Load balancing in an sap system
CN114143322A (en) Method for evaluating configuration action of computing resources in mixed cloud scene
CN114546652A (en) Parameter estimation method and device and electronic equipment
CN114168439A (en) Pressure measurement control method and device for service in cluster, storage medium and terminal
CN113127327A (en) Test method and device for performance test
CN111258729A (en) Redis-based task allocation method and device, computer equipment and storage medium
CN117591382B (en) Intelligent monitoring method, device, equipment and medium for server faults
CN114598705B (en) Message load balancing method, device, equipment and medium
CN116566992B (en) Dynamic collaboration method, device, computer equipment and storage medium for edge calculation
EP2698713A2 (en) Load balancing in data processing system
CN113900800B (en) Distribution method of edge computing system
CN113778657B (en) Data processing method and device
CN112631743B (en) Task scheduling method, device and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant