CN109165088B - Elastic expansion task segmentation method based on elastic-jobframe and capable of achieving single machine - Google Patents
Elastic expansion task segmentation method based on elastic-jobframe and capable of achieving single machine Download PDFInfo
- Publication number
- CN109165088B CN109165088B CN201811038389.3A CN201811038389A CN109165088B CN 109165088 B CN109165088 B CN 109165088B CN 201811038389 A CN201811038389 A CN 201811038389A CN 109165088 B CN109165088 B CN 109165088B
- Authority
- CN
- China
- Prior art keywords
- task
- elastic
- framework
- joba
- instances
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a task fragmentation method capable of single-machine elastic expansion based on an elastic-jobb framework, which mainly comprises 6 steps, wherein the main process is established for a task instance, and task codes are compiled based on the elastic-jobb framework; defining a task declaration (. xml); defining task descriptions (. properties); the task analyzes the fragment items and the self-defined parameters of the task at the running time; according to the constructed example matrix, a service slicing strategy written by a user automatically determines how to process different data concurrently or execute a concurrent program; service expansion or contraction in a distributed environment. The invention provides a task segmentation method capable of flexibly expanding the capacity of a single machine, so that elastic-jobs can adapt to the scene of executing concurrent tasks by multiple threads of the single machine, and the service concurrency efficiency and the resource utilization rate are greatly improved. The fragmentation method does not need to increase excessive extra development cost and supports dynamic expansion and contraction.
Description
Technical Field
The invention relates to the field of computers, in particular to a task fragmentation method capable of achieving single-machine elastic expansion based on an elastic-jobframe.
Background
In computer software systems, it is often necessary to count, summarize, and store the results of a large amount of data. The process of preparing data ahead of time may be performed automatically by the system at any time specified. The tasks automatically executed by the system are called background tasks; all background tasks which are executed once or periodically at a given time are called background timing tasks, and are called timing tasks for short.
The timing task scheduling framework is responsible for providing flexible and various timing strategies for scheduling and executing the timing tasks and monitoring the execution of the timing tasks in real time. Some scheduling frameworks also provide certain extended functions for task deployment or task reliability, such as a distributed flexible deployment scheme, a service heartbeat mechanism, and the like.
The existing timing task scheduling framework realized by JAVA is realized by Timer, Quartz, elastic-joba and the like. Timer is an API provided after JDK 1.4. The method of Timer can be used for specifying the scheduling rule, and has higher integration level with a Spring framework, and the timed task code can be registered through configuration. However, the task scheduling of Timer lacks a flexible configuration such as crontab. Quartz is a container for scheduling, supports the crontab syntax, and can be integrated with spring to realize thread pool management. Quartz, however, focuses on timing tasks rather than data and does not have a customized flow based on data processing. While Quartz can achieve high availability of jobs based on databases, it lacks the functionality to execute jobs in distributed parallel.
elastic-jobis a zookeeper-based open source distributed timed task framework. The method supports distributed deployment, task fragmentation, elastic capacity expansion and contraction, failover and state monitoring. It also has higher integration with Spring frame. Thus, elastic-jobis a mature choice of the existing enterprise-level JAVA timed task framework. elastic-jobs support distributed sharded execution of the same task, i.e., machines assigned to task sharded items execute tasks concurrently. Because the elastic-jobs can only deploy one service under one IP, the same task in the native service only has one instance on one machine, the elastic capacity expansion of the native service cannot reach the granularity of single machine multithreading, and the native-jobs have certain limitation.
Therefore, the invention provides a task fragmentation method capable of single machine elastic expansion, so that the elastic-jobs can adapt to the scene of single machine multithread execution concurrent tasks, and the service concurrent efficiency and the resource utilization rate are greatly improved. The fragmentation method does not need to increase excessive extra development cost and supports dynamic expansion and contraction.
Disclosure of Invention
The invention aims to solve the defects and problems of the prior art, so that a developer can obtain a more flexible architecture with higher concurrency granularity when using an elastic-jobframe, and the cost of a deployed server is saved.
The invention comprises the following steps:
step 1: establishing a task instance, and compiling a task code based on an elastic-joba framework; the callback entry function parameters of the elastic-jobframe comprise the fragment items allocated by the task scheduling, and the fragment items are read and recorded;
step 2: defining a task declaration (. xml); loading a job statement xml in a starting program, generating an elastic-jobtask and registering the elastic-jobtask in a zookeeper service; a plurality of task instances are required to be declared by using the task class written in the step 1, and a plurality of task instances are registered;
and step 3: defining task descriptions (. properties); the number of the slicing items of the task instance is determined by referring to the number of deployable machines; the user-defined parameters of the tasks are used as configuration information during running;
and 4, step 4: the task analyzes the fragment items and the user-defined parameters of the task at the running time; through the acquired information in the step 3, the number of machines in the distributed environment and the number of instances of the same task under a single machine can be dynamically analyzed, and an instance matrix combining the distributed environment and the single machine environment is constructed;
and 5: according to the example matrix constructed in the step 4, a service slicing strategy written by a user automatically determines how to process different data concurrently or execute a concurrent program;
step 6: service capacity expansion or capacity reduction in a distributed environment; the fragmentation method can dynamically sense the distributed environment change and automatically adjust the data range to be processed of each instance; under a single machine environment, task instances need to be increased or decreased, namely task processing threads need to be increased or decreased, and only self-defined parameters need to be modified in elastic-joba configuration to restart.
Preferably, the main parameters of the callback entry function of the framework in step 1 are provided by the framework, and the self-definition of the task fragment items and the parameter information allocated this time is dynamically obtained by implementing this function.
Preferably, the plurality of task instances spring bean stated in the step 2 further specifies that the operation of loading the xml file is used for completing the timing task based on the elastic-jobb framework.
Preferably, since the elastic-joba supports single-machine execution of multi-sharded items, the number of sharded items in the step 3 is greater than the number of running machines.
Preferably, the instance matrix in the distributed environment is calculated according to the obtained instance sharding item and the self-defined parameters during task execution.
Preferably, the information transferred in step 3 includes the number of instances of the same task in step 2 and the identifiable sequence ID of the instance.
Preferably, in the step 4, a remainder value obtained by dividing the account number by the total number is used as a basis for the account number fragment induction processing.
Preferably, each instance can calculate a remainder set borne by a single server ordering instance through the instance serial numbers in the sharding item and the custom parameter, and further process a corresponding account set.
Preferably, machine changes to the distributed environment do not affect the automatic regeneration of the sharding method.
Preferably, the number of instances in the single machine is changed by modifying the self-defined parameters and rewriting bean statements in the task xml, so that the total number and the sequence number of the instances are changed.
Compared with the prior art, the invention provides the task segmentation method capable of flexibly expanding the capacity of the single machine, so that the elastic-jobs can adapt to the scene of executing concurrent tasks by the single machine in a multi-thread manner, and the service concurrency efficiency and the resource utilization rate are greatly improved. The fragmentation method does not need to increase excessive extra development cost and supports dynamic expansion and contraction. When a developer uses the elastic-jobframework, a more flexible framework with higher concurrency granularity can be obtained, and the cost of the deployed server is saved.
Drawings
FIG. 1 is a flow chart of a case implementation of the present invention;
FIG. 2 is a schematic diagram of streaming task authoring of the present invention;
FIG. 3 is a diagram illustrating an elastic-joba framework entry function and parameters;
FIG. 4 is a task declaration template xml of the stand-alone extensible fragmentation method;
FIG. 5 is a diagram illustrating the definition of custom parameters in a task description.
Detailed Description
In order to further understand the objects, structures, features and functions of the present invention, the following embodiments are described in detail.
Referring to fig. 1, fig. 2, fig. 3, fig. 4 and fig. 5, a flowchart of an embodiment of a task slicing method for single-machine flexible capacity expansion based on elastic-jobs according to the present invention is shown in fig. 1. In the embodiment, the account number associated information is processed according to the account number in the database. Due to the large number of accounts, the service is required to support concurrent fragmentation processing. The invention comprises the following steps:
step 1: establishing a task instance, and compiling a task code based on an elastic-joba framework; the callback entry function parameters of the elastic-jobframe comprise the fragment items allocated by the task scheduling, and the fragment items are read and recorded;
step 2: defining a task declaration (. xml); loading a job statement xml in a starting program, generating an elastic-jobtask and registering the elastic-jobtask in a zookeeper service; a plurality of task instances are required to be declared by using the task class written in the step 1, and a plurality of task instances are registered;
and step 3: defining task descriptions (. properties); the number of the slicing items of the task instance is determined by referring to the number of deployable machines; the user-defined parameters of the tasks are used as configuration information during running; the transmitted information includes the number of the instances of the same task in the step 2 and the identifiable sequence ID of the instance.
And 4, step 4: the task analyzes the fragment items and the user-defined parameters of the task at the running time; through the acquired information in the step 3, the number of machines in the distributed environment and the number of instances of the same task under a single machine can be dynamically analyzed, and an instance matrix combining the distributed environment and the single machine environment is constructed;
and 5: according to the example matrix constructed in the step 4, a service slicing strategy written by a user automatically determines how to process different data concurrently or execute a concurrent program;
step 6: service capacity expansion or capacity reduction in a distributed environment; the fragmentation method can dynamically sense the distributed environment change and automatically adjust the data range to be processed of each instance; under a single machine environment, task instances need to be increased or decreased, namely task processing threads need to be increased or decreased, and only self-defined parameters need to be modified in elastic-joba configuration to restart.
Referring to fig. 2, firstly, a streaming task class is written based on an elastic-joba framework, and the main parameters of the callback entry function of the framework in the step 1 are provided by the framework, referring to fig. 3; and dynamically acquiring the self-definition of the task slicing items and the parameter information distributed at this time by realizing the function.
And the plurality of task instances spring beans stated in the step 2 specify that the operation of loading the xml file is used for completing the timing task based on the elastic-jobframe. In one embodiment, we set two instances of the same task under a single server, and write xml with reference to fig. 4, and specify the implementation class of beans as the task class in fig. 1.
We need to define the number of pieces of a task in a distributed environment. In one embodiment, we have two servers to deploy the timed task service, defining the number of shards as 2. Due to the fact that
The elastic-jobs supports a single machine to execute multi-sharded items, and the number of the sharded items in the step 3 is larger than that of the running machines. In addition, a custom parameter of the task needs to be defined, which is used for identifying the number of task instances and the instance serial number under a single machine, and the specific operation is combined with fig. 5.
After the work is finished, the instance matrix in the distributed environment is calculated according to the obtained instance sharding item and the user-defined parameters when the task is executed. In one embodiment, the example matrix calculation process in the case of dual-server dual-standalone example is combined with table 1, and it is assumed that we have two servers deployed timing tasks, and the task sharding item is 0/1/2. While we have two instance beans specified in xml, with custom parameters sequence 0 and sequence 1, we can get the total number of instances in the distributed environment to be 2 x 3-6. The remainder value of the division between the account number and the total number can be used as the basis for the fragmentation induction processing of the account number.
TABLE 1 example matrix under Dual Server and Dual Stand-alone example
Each instance can calculate a remainder set born by the single server ordering instance through the sharding item and the instance serial number in the user-defined parameter, and further process the corresponding account set.
When the server expands and contracts, the processing and calculation conditions of the example matrix are different. When the server expands, the processing conditions of the example matrix are shown in table 2; when the server is dynamically scaled, the example matrix is calculated as shown in table 3.
Table 2 example matrix under three-server dual-standalone example
TABLE 3 example matrix under Single Server Dual Single machine example
From tables 2 and 3, we can see that the machine change of the distributed environment has no influence on the fragmentation method of the present invention, and can be automatically regenerated.
Preferably, the number of instances in the single machine is changed by modifying the self-defined parameters and rewriting bean statements in the task xml, so that the total number and the sequence number of the instances are changed. In one embodiment, the number of instances of a single machine is changed to 3, the matrix of instances always changes to 3 × 3 — 9, and the calculation results are shown in table 4:
table 4 example matrix under two-server three-stand-alone example
Compared with the prior art, the invention provides the task segmentation method capable of single machine flexible capacity expansion, so that the elastic-jobs can adapt to the scene of single machine multi-thread execution concurrent tasks, and the service concurrent efficiency and the resource utilization rate are greatly improved. The fragmentation method does not need to increase excessive extra development cost and supports dynamic expansion and contraction. When a developer uses the elastic-jobframework, a more flexible framework with higher concurrency granularity can be obtained, and the cost of the deployed server is saved.
The present invention has been described in relation to the above embodiments, which are only exemplary of the implementation of the present invention. It should be noted that the disclosed embodiments do not limit the scope of the invention. Rather, it is intended that all such modifications and variations be included within the spirit and scope of this invention.
Claims (10)
1. A task fragmentation method based on elastic-joba framework and capable of single machine elastic expansion is characterized by comprising the following steps:
step 1: establishing a task instance, compiling a task code based on an elastic-joba framework, and taking the task code compiled based on the elastic-joba framework as a task class; the callback entry function parameters of the elastic-jobframe comprise fragment items distributed by task scheduling, and the fragment items are read and recorded;
step 2: define task declaration, xml; loading a job statement xml in a starting program, generating an elastic-jobtask and registering the elastic-jobtask in a zookeeper service; utilizing the task class written in the step 1 to declare a plurality of task instances and register the plurality of task instances;
and step 3: defining task descriptions, properties; the number of the slicing items of the task instance is determined by the number of deployed machines; the user-defined parameters of the tasks are used as configuration information during running; the self-defining parameters comprise the number of instances and the serial number of the instances;
and 4, step 4: the task analyzes the fragment items and the user-defined parameters of the task at the running time; dynamically analyzing the number of machines in the distributed environment and the number of instances of the same task under a single machine through the number of instances and the number of instance sequences obtained in the step 3, and constructing an instance matrix combining the distributed environment and the single machine environment;
and 5: according to the example matrix constructed in the step 4, a service slicing strategy written by a user determines to concurrently process different data or execute a concurrent program;
step 6: service capacity expansion or capacity reduction in a distributed environment; dynamically sensing the distributed environment change, and automatically adjusting the data range to be processed of each instance; and under the single-machine environment, increasing and decreasing task processing threads, and restarting after modifying custom parameters in elastic-joba configuration.
2. The elastic-joba framework-based task sharding method capable of standalone elastic capacity expansion according to claim 1, wherein parameters of the callback entry function of the framework in the step 1 are provided by the framework, and the dynamically-acquired assigned task sharding items and the customization of parameter information are realized through the callback entry function.
3. The method for task sharding based on elastic-joba framework capable of standalone elastic expansion of claim 1, wherein the plurality of task instances spring bean declared in step 2 specify the operation of loading the xml file for completing the timing task based on elastic-joba framework.
4. The elastic-jobframe-based task sharding method capable of stand-alone elastic capacity expansion, according to claim 1, wherein since elastic-jobs supports stand-alone execution of multiple sharded items, the number of sharded items in step 3 is greater than the number of running machines.
5. The elastic-joba framework based task sharding method capable of standalone elastic capacity expansion according to claim 1, wherein during task execution, the instance matrix in the distributed environment is calculated according to the obtained instance sharding item and custom parameters.
6. The method for task segmentation based on elastic-joba framework capable of single-machine flexible capacity expansion of claim 1, wherein the information passed in step 3 includes the number of instances of the same task in step 2 and an identifiable sequence ID of the instances.
7. The elastic-joba-frame-based task fragmentation method capable of single-machine flexible expansion according to claim 1, wherein a remainder value obtained by dividing an account number by the number of machines is used as a basis for account number fragmentation induction processing in step 4.
8. The elastic capacity-expansion task sharding method based on the elastic-joba framework is characterized in that each instance calculates a remainder set born by a singleton instance of a single server through the sharding item and the instance serial number in the custom parameter, and further processes a corresponding account set.
9. The elastic-joba framework based task sharding method capable of stand-alone elastic capacity expansion is characterized in that machine change of a distributed environment does not affect automatic regeneration of the sharding method.
10. The elastic-joba-framework-based task fragmentation method capable of single-machine flexible expansion is characterized in that the number of instances in a single machine is changed, and the total number and the sequence number of the instances are changed by modifying the custom parameters, rewriting bean statements in the task xml.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811038389.3A CN109165088B (en) | 2018-09-06 | 2018-09-06 | Elastic expansion task segmentation method based on elastic-jobframe and capable of achieving single machine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811038389.3A CN109165088B (en) | 2018-09-06 | 2018-09-06 | Elastic expansion task segmentation method based on elastic-jobframe and capable of achieving single machine |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109165088A CN109165088A (en) | 2019-01-08 |
| CN109165088B true CN109165088B (en) | 2022-04-19 |
Family
ID=64894337
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811038389.3A Active CN109165088B (en) | 2018-09-06 | 2018-09-06 | Elastic expansion task segmentation method based on elastic-jobframe and capable of achieving single machine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109165088B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110262848B (en) * | 2019-06-05 | 2022-03-04 | 创新先进技术有限公司 | Method and device for loading application program instance in application development framework |
| CN112422683B (en) * | 2020-11-19 | 2023-02-03 | 浪潮云信息技术股份公司 | API gateway service high-availability implementation method in K8S environment |
| CN114840268A (en) * | 2022-04-25 | 2022-08-02 | 平安普惠企业管理有限公司 | Service code execution method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106371918A (en) * | 2016-08-23 | 2017-02-01 | 北京云纵信息技术有限公司 | Task cluster scheduling management method and apparatus |
| CN106528297A (en) * | 2016-11-14 | 2017-03-22 | 山东浪潮云服务信息科技有限公司 | System for managing tasks |
| CN106874109A (en) * | 2016-12-29 | 2017-06-20 | 朗新科技股份有限公司 | A kind of distributed job distribution processing method and system |
| CN107370773A (en) * | 2016-05-11 | 2017-11-21 | 北京京东尚科信息技术有限公司 | The method and system of the telescopic distributed data filing of High Availabitity |
| US20180143856A1 (en) * | 2016-11-18 | 2018-05-24 | Sap Se | Flexible job management for distributed container cloud platform |
-
2018
- 2018-09-06 CN CN201811038389.3A patent/CN109165088B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107370773A (en) * | 2016-05-11 | 2017-11-21 | 北京京东尚科信息技术有限公司 | The method and system of the telescopic distributed data filing of High Availabitity |
| CN106371918A (en) * | 2016-08-23 | 2017-02-01 | 北京云纵信息技术有限公司 | Task cluster scheduling management method and apparatus |
| CN106528297A (en) * | 2016-11-14 | 2017-03-22 | 山东浪潮云服务信息科技有限公司 | System for managing tasks |
| US20180143856A1 (en) * | 2016-11-18 | 2018-05-24 | Sap Se | Flexible job management for distributed container cloud platform |
| CN106874109A (en) * | 2016-12-29 | 2017-06-20 | 朗新科技股份有限公司 | A kind of distributed job distribution processing method and system |
Non-Patent Citations (2)
| Title |
|---|
| Elastic-Job - 分布式定时任务框架;张亮_Apache;《https://my.oschina.net/u/719192/blog/506062》;20150914;全文 * |
| 分布式任务elasticjob框架原理了解及使用;秋石_cc61;《https://www.jianshu.com/p/80d162f789c0》;20180813;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109165088A (en) | 2019-01-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109165088B (en) | Elastic expansion task segmentation method based on elastic-jobframe and capable of achieving single machine | |
| US10951491B2 (en) | Automatic microservice problem detection in enterprise applications | |
| CN107957903B (en) | Asynchronous task scheduling method, server and storage medium | |
| CN106970921B (en) | Data migration method and device | |
| US10365964B1 (en) | Data processing platform monitoring | |
| US9405630B2 (en) | Methods and apparatus to perform site recovery of a virtual data center | |
| JP2019523462A (en) | Multitask scheduling method, system, application server, and computer-readable storage medium | |
| CN110401700B (en) | Model loading method and system, control node and execution node | |
| CN107016029B (en) | Method, device and system for processing service data | |
| US8296769B2 (en) | Order-relation analyzing apparatus, method, and computer program product thereof | |
| CN106940699B (en) | Synchronous processing method, device, server and system for memory data | |
| Karcher et al. | Run-time automatic performance tuning for multicore applications | |
| US10310877B2 (en) | Category based execution scheduling | |
| CN110599341A (en) | Transaction calling method and system | |
| CN110609749A (en) | Distributed task operation method, system and equipment | |
| CN112181522A (en) | Data processing method and device and electronic equipment | |
| US9442818B1 (en) | System and method for dynamic data collection | |
| CN110413300A (en) | Rolling upgrade control method, device, equipment and storage medium | |
| CN106547861A (en) | A kind of method and device of the data base of intelligent management machine node | |
| CN112766470B (en) | Feature data processing method, instruction sequence generating method, device and equipment | |
| CN116107728A (en) | Task execution method and device, storage medium and electronic equipment | |
| CN110109986B (en) | Task processing method, system, server and task scheduling system | |
| CN112965798A (en) | Big data processing method and system based on distributed multithreading | |
| CN111984294A (en) | Service system updating method and device based on Groovy dynamic loading technology | |
| US20160239372A1 (en) | Undoing changes made by threads |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |