CN117707727A - Distributed task distribution system, method and device and electronic equipment - Google Patents

Distributed task distribution system, method and device and electronic equipment Download PDF

Info

Publication number
CN117707727A
CN117707727A CN202311660701.3A CN202311660701A CN117707727A CN 117707727 A CN117707727 A CN 117707727A CN 202311660701 A CN202311660701 A CN 202311660701A CN 117707727 A CN117707727 A CN 117707727A
Authority
CN
China
Prior art keywords
task
management server
target
server
database
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311660701.3A
Other languages
Chinese (zh)
Inventor
刘姿阳
崔义芳
王志海
喻波
韩振国
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Wondersoft Technology Co Ltd
Original Assignee
Beijing Wondersoft Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Wondersoft Technology Co Ltd filed Critical Beijing Wondersoft Technology Co Ltd
Priority to CN202311660701.3A priority Critical patent/CN117707727A/en
Publication of CN117707727A publication Critical patent/CN117707727A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/50—Indexing scheme relating to G06F9/50
    • G06F2209/5017—Task decomposition

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the application discloses a distributed task distribution system, a distributed task distribution method, a distributed task distribution device and electronic equipment. The system comprises: the scheduling server is in communication connection with the first management server and the second management server and is used for selecting a target database task from the database tasks to be processed according to a preset scheduling rule, selecting a target management server from the first management server and the second management server according to a preset issuing rule and issuing the target database task to the target management server, wherein the first management server and the second management server are in a simultaneous operation state; the target management server is in communication connection with the plurality of work servers and is used for dividing the target database task into a plurality of subtasks and distributing the plurality of subtasks to the plurality of work servers; and the work server is used for processing the received subtasks and returning the subtask processing results to the target management server. This embodiment improves task processing efficiency.

Description

Distributed task distribution system, method and device and electronic equipment
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to a distributed task distribution system, a distributed task distribution method, a distributed task distribution device and electronic equipment.
Background
In the big data age, the data task scheduling distribution is a concern of system performance, and the efficient and automatic task scheduling distribution system can efficiently and concurrently process continuous tasks in the system, so that the system performance and the task execution efficiency are greatly improved.
In the prior art, tasks are typically distributed by one management server to a plurality of work servers, which execute the tasks. This approach is inefficient in task processing.
Disclosure of Invention
The embodiment of the application provides a distributed task distribution system, a distributed task distribution method, a distributed task distribution device and electronic equipment, so as to solve the technical problem of low task processing efficiency in the prior art.
In a first aspect, an embodiment of the present application provides a distributed task distribution system, including: the scheduling server is in communication connection with the first management server and the second management server and is used for selecting a target database task from database tasks to be processed according to a preset scheduling rule, selecting a target management server from the first management server and the second management server according to a preset issuing rule and issuing the target database task to the target management server, wherein the first management server and the second management server are in a simultaneous operation state; the target management server is in communication connection with a plurality of working servers and is used for dividing the target database task into a plurality of subtasks and distributing the subtasks to the plurality of working servers; the work server is used for processing the received subtasks and returning subtask processing results to the target management server.
In a second aspect, an embodiment of the present application provides a distributed task distribution method, which is applied to a target management server, where the target management server is a first management server or a second management server that is communicatively connected to a scheduling server, and the first management server and the second management server are in a state of running simultaneously, and the method includes: receiving a target database task issued by the scheduling server, wherein the target database task is a database task selected from database tasks to be processed according to a preset scheduling rule; dividing the target database task into a plurality of subtasks, and distributing the subtasks to a plurality of working servers; and receiving a subtask processing result returned by each working server in the plurality of working servers.
In a third aspect, an embodiment of the present application provides a distributed task distribution device, which is applied to a target management server, where the target management server is a first management server or a second management server that is communicatively connected to a scheduling server, and the first management server and the second management server are in a state of running simultaneously, where the device includes: the scheduling unit is used for receiving a target database task issued by the scheduling server, wherein the target database task is a database task selected from database tasks to be processed according to a preset scheduling rule; the splitting unit is used for splitting the target database task into a plurality of subtasks and distributing the subtasks to a plurality of working servers; and the receiving unit is used for receiving the subtask processing results returned by each working server in the plurality of working servers.
In a fourth aspect, an embodiment of the present application provides an electronic device, including: one or more processors; a storage device having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement the method as described in any embodiment of the second aspect.
In a fifth aspect, embodiments of the present application provide a computer readable medium having stored thereon a computer program which, when executed by a processor, implements a method as described in any of the embodiments of the second aspect.
According to the distributed task distribution system, the distributed task distribution method, the distributed task distribution device and the distributed task distribution electronic equipment, the scheduling server can select a target database task from database tasks to be processed according to a preset scheduling rule, select a target management server from a first management server and a second management server which are in communication connection according to a preset issuing rule, and issue the target database task to the target management server; the target management server can divide the target database task into a plurality of subtasks and distribute the subtasks to a plurality of working servers; the work server may process the received subtasks and return subtask processing results to the target management server. Because the first management server and the second management server are in a simultaneous operation state, and the target database task is split into a plurality of subtasks to be executed concurrently, the task processing efficiency can be greatly improved, and the concurrent and efficient execution of a large data volume task is ensured.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the following drawings, in which:
FIG. 1 is a block diagram of an embodiment of a distributed task distribution system according to the present application;
FIG. 2 is a flow chart of an embodiment of a distributed task distribution method according to the present application;
FIG. 3 is a schematic structural view of an embodiment of a distributed task distribution device according to the present application;
fig. 4 is a schematic structural diagram of an electronic device for implementing an embodiment of the present application.
Detailed Description
All actions taken in this application to obtain signals, information or data are performed in compliance with the corresponding data protection legislation policies of the country of location and to obtain authorization granted by the owner of the corresponding device.
The present application is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
FIG. 1 illustrates a block diagram of an embodiment of a distributed task distribution system to which the present application may be applied.
As shown in fig. 1, the distributed task distribution system may include a dispatch server, a first management server, a second management server, and a work server cluster. A plurality of working servers may be included in a working server cluster.
In this embodiment, the scheduling server may be communicatively connected to the first management server and the second management server. Some of the working servers in the working server cluster may be communicatively coupled to a first management server and another of the working servers may be communicatively coupled to a second management server.
In this embodiment, each of the work servers may be communicatively coupled to a data source. The data sources may include one or more databases. The database may include, but is not limited to, at least one of: (Hadoop Distributed File System, distributed file system), hive (a data warehouse tool based on Hadoop), mySQL (a relational database management system), OB (OceanBase, oxstar base), OBs (Object Storage Service ), clickHouse (a columnar database management system for online analysis), hbase (a distributed, nematic open source database), elastsearch (a distributed, highly extended, highly real-time search and data analysis engine).
In this embodiment, the first management server and the second management server may be in a state of running simultaneously. Thus, the task processing efficiency can be improved as compared with the use of a single management server.
In this embodiment, the number of working servers in the working server cluster may be extended as needed, so that the service requirement may be better met.
In this embodiment, the scheduling server may store database tasks to be processed. The scheduling server may be configured to select a target database task from the database tasks to be processed according to a preset scheduling rule, select a target management server from the first management server and the second management server according to a preset issuing rule, and issue the target database task to the target management server.
The preset scheduling rules can be set according to requirements. As an example, it may be set to select a target database task from among database tasks to be processed in order of priority from high to low. It may also be provided that the target database task is selected from the database tasks to be processed here from the large to the small waiting time. The present invention is not particularly limited herein. Similarly, the preset issuing rule can be set as required. As an example, it may be set to poll the first management server and the second management server, and when which management server is currently polled, the management server is set as the target management server.
In this embodiment, the target management server may be configured to split the target database task into a plurality of subtasks, and distribute the plurality of subtasks to a plurality of working servers. Here, the target database task may be split by using a preset splitting rule. The preset segmentation rules can be set according to the needs. As an example, a target database task may involve multiple files, and may be split by the number of files. As yet another example, the target database task may involve multiple task types, and may be split by task type. The present invention is not particularly limited herein.
In this embodiment, the work server may be configured to process the received subtasks, and return the subtask processing result to the target management server. Wherein the work servers can process the subtasks in parallel.
Because the first management server and the second management server are in a simultaneous operation state, and the target database task is split into a plurality of subtasks to be executed concurrently, the task processing efficiency can be greatly improved, and the concurrent and efficient execution of a large data volume task is ensured. The system can efficiently and concurrently process continuous tasks in the system, and greatly provides system performance and task execution efficiency.
In some alternative embodiments, different ones of the plurality of work servers communicatively coupled to the target management server may be used to process sub-tasks of different task types. The number of task types of subtasks that each work server can handle is not limited. For example, a certain work server may be used to process HDFS tasks and hive tasks. Another work server may be used to process MySQL tasks and OBS tasks.
The above-mentioned target management server may further be configured to: dividing the target database task into a plurality of subtasks according to the task type related to the target database task; and distributing each subtask to a corresponding work server according to the task type of each subtask. Therefore, different types of tasks are distributed for each work server reasonably, a plurality of subtasks can be executed in parallel, resource isolation can be carried out on the work server, the processing speed of the tasks is greatly improved, and concurrent and efficient execution of large-data-volume tasks is guaranteed.
In some alternative embodiments, the target management server may be further configured to: and dividing the target database task into a plurality of subtasks according to the type of the source database or the type of the destination database related to the target database task. The source database may refer to a database from which data is to be acquired. The destination database may refer to a database to which data is to be written.
In some alternative embodiments, multiple work servers communicatively coupled to the target management server may handle the same type of task. The above-mentioned target management server may further be configured to: and dividing the target database task into a plurality of subtasks according to the number of files related to the target database task. For example, the number of files involved in a target database task is ten thousand. It can be split into 10 subtasks. The number of files involved per subtask is one thousand. The 10 subtasks may be distributed to a plurality of work servers communicatively coupled to the target management server, e.g., to 10 work servers, each of which handles one subtask. For another example, 5 work servers may be distributed, each processing two subtasks. Therefore, a plurality of subtasks can be executed in parallel, and the task processing efficiency of the work server is improved.
In some alternative embodiments, the target management server may be further configured to: and updating the task state of the target database task based on the subtask processing results returned by each working server in the plurality of working servers.
In some alternative embodiments, the scheduling server may be further configured to obtain database tasks to be processed. And adding the acquired database tasks to different task queues according to the task scheduling parameters. And determining a target task queue according to a preset scheduling rule, and selecting a target database task from the target task queue. The task queues described above may be partitioned as desired. For example, the partitioning may be based on the priorities of the tasks. Database tasks of different priorities may be added to different task queues.
In some alternative embodiments, the system may further include a third management server (not shown). The third management server can be used for creating database connection information so that each working server can be connected with the database through the database connection information. In addition, the third management server may be further configured to create task information, where the task information may include the task scheduling parameters. Correspondingly, the scheduling server is further used for acquiring the database task to be processed from the third management server.
The scheduling server, the first management server, the second management server, the working server, and the third management server may be hardware or software. When the server is hardware, the server may be implemented as a distributed server cluster formed by a plurality of servers, or may be implemented as a single server. When the server is software, it may be implemented as a plurality of software or software modules (e.g., to provide distributed services), or as a single software or software module. For example, the first management server and the second management server may be regarded as Master nodes (Master nodes or management nodes). The working servers in the working server cluster may be considered executor nodes (working nodes or executing nodes). The present invention is not particularly limited herein.
It should be understood that the number of scheduling servers, first management server, second management server and working server clusters and working servers therein in fig. 1 are merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring to FIG. 2, a flow 200 of one embodiment of a distributed task distribution method according to the present application is shown. The distributed task distribution method can be applied to a target management server. The target management server is a first management server or a second management server which is in communication connection with the scheduling server. The first management server and the second management server are in a simultaneous operation state. Thus, the task processing efficiency can be improved as compared with the use of a single management server.
The distributed task distribution method comprises the following steps:
step 201, receiving a target database task issued by a scheduling server, wherein the target database task is a database task selected from database tasks to be processed according to a preset scheduling rule.
In this embodiment, the execution body (for example, the above-mentioned target management server) of the distributed task distribution method may receive the target database task issued by the scheduling server. The target database task is a database task selected from the database tasks to be processed according to a preset scheduling rule.
In this embodiment, the scheduling server may store database tasks to be processed. The scheduling server may be configured to select a target database task from the database tasks to be processed according to a preset scheduling rule, select a target management server from the first management server and the second management server according to a preset issuing rule, and issue the target database task to the target management server. The preset scheduling rules can be set according to requirements. As an example, it may be set to select a target database task from among database tasks to be processed in order of priority from high to low. It may also be provided that the target database task is selected from the database tasks to be processed here from the large to the small waiting time. The present invention is not particularly limited herein. Similarly, the preset issuing rule can be set as required. As an example, it may be set to poll the first management server and the second management server, and when which management server is currently polled, the management server is set as the target management server.
Step 202, dividing the target database task into a plurality of subtasks, and distributing the plurality of subtasks to a plurality of working servers.
In this embodiment, the execution body may divide the target database task into a plurality of subtasks, and distribute the plurality of subtasks to a plurality of work servers. Here, the target database task may be split by using a preset splitting rule. The preset segmentation rules can be set according to the needs. As an example, a target database task may involve multiple files, and may be split by the number of files. As yet another example, the target database task may involve multiple task types, and may be split by task type. The present invention is not particularly limited herein.
In some alternative implementations, among a plurality of work servers communicatively coupled to the execution body described above, different work servers may be used to process sub-tasks of different task types. The number of task types of subtasks that each work server can handle is not limited. For example, a certain work server may be used to process HDFS tasks and hive tasks. Another work server may be used to process MySQL tasks and OBS tasks. The execution body may first segment the target database task into a plurality of subtasks according to a task type related to the target database task. And then distributing each subtask to a corresponding work server according to the task type of each subtask. Therefore, not only can a plurality of subtasks be executed in parallel, but also the resource isolation can be carried out on the working server, the task processing efficiency of the working server is improved, and the high availability of the system is ensured.
In some optional implementations, the execution body may segment the target database task into a plurality of subtasks according to a type of a source database or a type of a destination database related to the target database task. The source database may refer to a database from which data is to be acquired. The destination database may refer to a database to which data is to be written.
In some alternative implementations, multiple work servers communicatively connected to the target management server may handle the same type of task. The execution body may divide the target database task into a plurality of subtasks according to the number of files related to the target database task. For example, the number of files involved in a target database task is ten thousand. It can be split into 10 subtasks. The number of files involved per subtask is one thousand. The 10 subtasks may be distributed to a plurality of work servers communicatively coupled to the target management server, e.g., to 10 work servers, each of which handles one subtask. For another example, 5 work servers may be distributed, each processing two subtasks. Therefore, a plurality of subtasks can be executed in parallel, and the task processing efficiency of the work server is improved.
And 203, receiving a subtask processing result returned by each working server in the plurality of working servers.
In this embodiment, the work server may be configured to process the received subtasks, and return the subtask processing result to the target management server. Wherein the work servers can process the subtasks in parallel.
Because the first management server and the second management server are in a simultaneous operation state, and the target database task is split into a plurality of subtasks to be executed concurrently, the task processing efficiency can be greatly improved, and the concurrent and efficient execution of a large data volume task is ensured.
In some alternative embodiments, the execution body may update the task state of the target database task based on a subtask processing result returned by each of the plurality of work servers.
According to the method provided by the embodiment of the application, the scheduling server can select a target database task from the database tasks to be processed according to the preset scheduling rule, select a target management server from the first management server and the second management server which are in communication connection according to the preset issuing rule, and issue the target database task to the target management server; the target management server can divide the target database task into a plurality of subtasks and distribute the subtasks to a plurality of working servers; the work server may process the received subtasks and return subtask processing results to the target management server. Because the first management server and the second management server are in a simultaneous operation state, and the target database task is split into a plurality of subtasks to be executed concurrently, the task processing efficiency can be greatly improved, and the concurrent and efficient execution of a large data volume task is ensured.
With further reference to fig. 3, as an implementation of the method shown in the foregoing figures, the present application provides an embodiment of a distributed task distribution device, where the embodiment of the device corresponds to the embodiment of the method shown in fig. 1, and the device may be specifically applied to a target management server, where the target management server is a first management server or a second management server that is communicatively connected to a scheduling server, and the first management server and the second management server are in a state of running simultaneously.
As shown in fig. 3, the distributed task distribution apparatus 300 of the present embodiment includes: a scheduling unit 301, configured to receive a target database task issued by the scheduling server, where the target database task is a database task selected from database tasks to be processed according to a preset scheduling rule; a splitting unit 302, configured to split the target database task into a plurality of subtasks, and distribute the plurality of subtasks to a plurality of working servers; and a receiving unit 303, configured to receive a subtask processing result returned by each of the plurality of working servers.
In some optional implementations of this embodiment, different work servers in the plurality of work servers are used to process subtasks of different task types; the splitting unit 302 is further configured to split the target database task into a plurality of subtasks according to a task type related to the target database task; and distributing each subtask to a corresponding work server according to the task type of each subtask.
In some optional implementations of this embodiment, the splitting unit 302 is further configured to split the target database task into a plurality of subtasks according to a type of a source database or a type of a destination database related to the target database task.
In some optional implementations of this embodiment, the splitting unit 302 is further configured to split the target database task into a plurality of subtasks according to the number of files related to the target database task.
In some optional implementations of this embodiment, the apparatus further includes an updating unit, configured to update a task state of the target database task based on a subtask processing result returned by each of the plurality of work servers.
According to the device provided by the embodiment of the application, the scheduling server can select the target database task from the database tasks to be processed according to the preset scheduling rule, select the target management server from the first management server and the second management server which are in communication connection according to the preset issuing rule, and issue the target database task to the target management server; the target management server can divide the target database task into a plurality of subtasks and distribute the subtasks to a plurality of working servers; the work server may process the received subtasks and return subtask processing results to the target management server. Because the first management server and the second management server are in a simultaneous operation state, and the target database task is split into a plurality of subtasks to be executed concurrently, the task processing efficiency can be greatly improved, and the concurrent and efficient execution of a large data volume task is ensured.
Referring now to fig. 4, a schematic diagram of an electronic device for implementing some embodiments of the present application is shown. The electronic device shown in fig. 4 is only an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present application.
As shown in fig. 4, the electronic device 400 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 401, which may perform various suitable actions and processes according to a program stored in a Read Only Memory (ROM) 402 or a program loaded from a storage means 408 into a Random Access Memory (RAM) 403. In the RAM403, various programs and data necessary for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM403 are connected to each other by a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
In general, the following devices may be connected to the I/O interface 405: input devices 406 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 407 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 408 including, for example, magnetic disks, hard disks, etc.; and a communication device 409. The communication means 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. While fig. 4 shows an electronic device 400 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead. Each block shown in fig. 4 may represent one device or a plurality of devices as needed.
In particular, according to some embodiments of the present application, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, some embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowchart. In such embodiments, the computer program may be downloaded and installed from a network via communications device 409, or from storage 408, or from ROM 402. The above-described functions defined in the methods of some embodiments of the present application are performed when the computer program is executed by the processing means 401.
It should be noted that, in some embodiments of the present application, the above-mentioned computer readable medium may be a computer readable signal medium or a computer readable storage medium, or any combination of the above-mentioned two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present application, however, the computer-readable signal medium may comprise a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText TransferProtocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving a target database task issued by the scheduling server, wherein the target database task is a database task selected from database tasks to be processed according to a preset scheduling rule; dividing the target database task into a plurality of subtasks, and distributing the subtasks to a plurality of working servers; and receiving a subtask processing result returned by each working server in the plurality of working servers.
Computer program code for carrying out operations for some embodiments of the present application may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++; conventional procedural programming languages, such as the "C" language or similar programming languages, are also included. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., connected via the internet using an internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present application may be implemented in software or in hardware. The described units may also be provided in a processor, for example, described as: a processor includes a first determination unit, a second determination unit, a selection unit, and a third determination unit. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
The foregoing description is only illustrative of the principles of the technology being employed and of some of the preferred embodiments of the present application. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present application is not limited to the specific combination of the above technical features, but encompasses other technical features formed by any combination of the above technical features or their equivalents without departing from the spirit of the invention. Such as the above-described features, are mutually replaced with (but not limited to) features having similar functions as disclosed in the embodiments of the present application.

Claims (15)

1. A distributed task distribution system, comprising:
the scheduling server is in communication connection with the first management server and the second management server and is used for selecting a target database task from database tasks to be processed according to a preset scheduling rule, selecting a target management server from the first management server and the second management server according to a preset issuing rule and issuing the target database task to the target management server, wherein the first management server and the second management server are in a simultaneous operation state;
the target management server is in communication connection with a plurality of working servers and is used for dividing the target database task into a plurality of subtasks and distributing the subtasks to the plurality of working servers;
the work server is used for processing the received subtasks and returning subtask processing results to the target management server.
2. The system of claim 1, wherein different ones of the plurality of work servers are configured to process subtasks of different task types;
the target management server is further configured to:
dividing the target database task into a plurality of subtasks according to the task type related to the target database task;
and distributing each subtask to a corresponding work server according to the task type of each subtask.
3. The system of claim 2, wherein the target management server is further configured to:
and dividing the target database task into a plurality of subtasks according to the type of the source database or the type of the destination database related to the target database task.
4. The system of claim 1, wherein the target management server is further configured to:
and dividing the target database task into a plurality of subtasks according to the number of files related to the target database task.
5. The system of claim 1, wherein the target management server is further configured to:
and updating the task state of the target database task based on the subtask processing result returned by each working server in the plurality of working servers.
6. The system of claim 1, wherein the scheduling server is further configured to:
acquiring a database task to be processed;
according to the task scheduling parameters, adding the acquired database tasks to different task queues;
and determining a target task queue according to a preset scheduling rule, and selecting a target database task from the target task queue.
7. The system of claim 6, further comprising a third management server for creating database connection information to enable each work server to establish a connection with a database through the database connection information; creating task information, wherein the task information comprises the task scheduling parameters;
the scheduling server is further configured to obtain a database task to be processed from the third management server.
8. A distributed task distribution method, applied to a target management server, where the target management server is a first management server or a second management server that is communicatively connected to a scheduling server, and the first management server and the second management server are in a state of running simultaneously, the method includes:
receiving a target database task issued by the scheduling server, wherein the target database task is a database task selected from database tasks to be processed according to a preset scheduling rule;
dividing the target database task into a plurality of subtasks, and distributing the subtasks to a plurality of working servers;
and receiving a subtask processing result returned by each working server in the plurality of working servers.
9. The method of claim 8, wherein different ones of the plurality of work servers are used to process subtasks of different task types;
the splitting the target database task into a plurality of subtasks, and distributing the plurality of subtasks to a plurality of working servers, including:
dividing the target database task into a plurality of subtasks according to the task type related to the target database task;
and distributing each subtask to a corresponding work server according to the task type of each subtask.
10. The method of claim 9, wherein the splitting the target database task into a plurality of sub-tasks according to the task type involved in the target database task comprises:
and dividing the target database task into a plurality of subtasks according to the type of the source database or the type of the destination database related to the target database task.
11. The method of claim 8, wherein the splitting the target database task into a plurality of sub-tasks comprises:
and dividing the target database task into a plurality of subtasks according to the number of files related to the target database task.
12. The method of claim 8, wherein the method further comprises:
and updating the task state of the target database task based on the subtask processing result returned by each working server in the plurality of working servers.
13. A distributed task distribution device, applied to a target management server, where the target management server is a first management server or a second management server that is communicatively connected to a scheduling server, and the first management server and the second management server are in a state of running simultaneously, the device comprising:
the scheduling unit is used for receiving a target database task issued by the scheduling server, wherein the target database task is a database task selected from database tasks to be processed according to a preset scheduling rule;
the splitting unit is used for splitting the target database task into a plurality of subtasks and distributing the subtasks to a plurality of working servers;
and the receiving unit is used for receiving the subtask processing results returned by each working server in the plurality of working servers.
14. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 8-12.
15. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 8-12.
CN202311660701.3A 2023-12-05 2023-12-05 Distributed task distribution system, method and device and electronic equipment Pending CN117707727A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311660701.3A CN117707727A (en) 2023-12-05 2023-12-05 Distributed task distribution system, method and device and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311660701.3A CN117707727A (en) 2023-12-05 2023-12-05 Distributed task distribution system, method and device and electronic equipment

Publications (1)

Publication Number Publication Date
CN117707727A true CN117707727A (en) 2024-03-15

Family

ID=90154499

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311660701.3A Pending CN117707727A (en) 2023-12-05 2023-12-05 Distributed task distribution system, method and device and electronic equipment

Country Status (1)

Country Link
CN (1) CN117707727A (en)

Similar Documents

Publication Publication Date Title
CN109523187B (en) Task scheduling method, device and equipment
US20180373540A1 (en) Cluster graphical processing unit (gpu) resource sharing efficiency by directed acyclic graph (dag) generation
CN109408205B (en) Task scheduling method and device based on hadoop cluster
US8819683B2 (en) Scalable distributed compute based on business rules
CN109508326B (en) Method, device and system for processing data
CN111338797B (en) Task processing method, device, electronic equipment and computer readable storage medium
US20180248934A1 (en) Method and System for a Scheduled Map Executor
US10423442B2 (en) Processing jobs using task dependencies
CN111352711A (en) Multi-computing engine scheduling method, device, equipment and storage medium
US11968248B2 (en) Content-based distribution and execution of analytics applications on distributed datasets
CN109298948A (en) Distributed computing method and system
CN113722056A (en) Task scheduling method and device, electronic equipment and computer readable medium
CN113760488A (en) Method, device, equipment and computer readable medium for scheduling task
CN109582445A (en) Message treatment method, device, electronic equipment and computer readable storage medium
CN115357350A (en) Task configuration method and device, electronic equipment and computer readable medium
CN109729110B (en) Method, apparatus and computer readable medium for managing dedicated processing resources
CN116821187A (en) Database-based data processing method and device, medium and electronic equipment
CN110716809B (en) Method and device for scheduling cloud resources
CN116302271A (en) Page display method and device and electronic equipment
CN117707727A (en) Distributed task distribution system, method and device and electronic equipment
US20230266997A1 (en) Distributed scheduling in container orchestration engines
CN115629853A (en) Task scheduling method and device
CN110825461A (en) Data processing method and device
CN111625692B (en) Feature extraction method, device, electronic equipment and computer readable medium
CN115170026A (en) Task processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination