CN114238137A - Batch processing task testing method and device, storage medium and program product - Google Patents

Abstract

The present specification provides a batch task testing method, apparatus, storage medium, and program product, which can be applied to the field of big data. The method comprises the following steps: acquiring at least one group of tasks to be tested; screening key tasks from the at least one group of tasks to be tested based on the task properties of the tasks to be tested; respectively obtaining single batch execution results when each key task is executed independently; under a batch execution scene, acquiring a full batch execution result of each key task; determining a test result based on a comparison of the single batch execution result and the full batch execution result. According to the method, the workload of test analysis is reduced by screening key tasks, the influence between different tasks in full-batch execution is effectively determined by comparing a single-batch execution result with a full-batch execution result, and the effect of batch execution is effectively reflected by the test result, so that the test on the batch processing tasks in a batch processing environment is efficiently and accurately finished, and the production research is facilitated.

Description

Batch processing task testing method and device, storage medium and program product

Technical Field

The embodiment of the specification relates to the technical field of big data, in particular to a batch processing task testing method, a batch processing task testing device, a storage medium and a program product.

Background

With the development of technology and business level, the development of supporting related businesses by using a distributed architecture is widely applied. In the distributed architecture, a certain transaction can be decomposed, so that a plurality of nodes can respectively process the transaction, thereby improving the expansibility and the usability as a whole. Based on the characteristics of the distributed architecture, the advantages of the distributed architecture can be furthest exerted when batch processing operation is executed, and the efficiency of service processing is improved.

At present, when a distributed system is applied to execute batch processing operation, due to the fact that concurrent processing among batch processing tasks is involved, the distributed system is different from a traditional centralized architecture, interference among various services exists based on influence conditions among the services during service execution, and therefore the difference between the traditional service execution flow and the traditional service execution flow is highlighted. However, when testing is performed on a batch processing task at present, it is difficult to comprehensively and effectively realize the test on the batch processing task under the condition of combining the characteristics of distributed batch processing, and it is difficult to analyze the test result based on the current application context, so that a proper test conclusion cannot be obtained. Therefore, there is a need for a method for intuitively and effectively obtaining the test results of batch processing tasks under a distributed architecture.

Disclosure of Invention

The invention provides a batch processing task testing method, a batch processing task testing device, a storage medium and a program product, and aims to solve the problem of how to effectively obtain a batch processing task testing result under a distributed architecture.

In order to solve the above technical problem, the present invention provides a batch task testing method, including: acquiring at least one group of tasks to be tested; screening key tasks from the at least one group of tasks to be tested based on the task properties of the tasks to be tested; respectively obtaining single batch execution results when each key task is executed independently; under a batch execution scene, acquiring a full batch execution result of each key task; determining a test result based on a comparison of the single batch execution result and the full batch execution result.

The invention also provides a batch processing task testing device, which comprises: the task acquisition module to be tested is used for acquiring at least one group of tasks to be tested; the key task screening module is used for screening key tasks from the at least one group of tasks to be tested based on the task properties of the tasks to be tested; the single-batch execution result acquisition module is used for respectively acquiring single-batch execution results when each key task is independently executed; the full-batch execution result acquisition module is used for acquiring full-batch execution results of each key task in a batch execution scene; and the test result determining module is used for determining a test result based on the comparison result of the single batch execution result and the full batch execution result.

The present invention also proposes a computer readable storage medium having stored thereon a computer program/instructions which, when executed by a processor, implement the above batch task testing method.

The present invention also proposes a computer program product having stored thereon a computer program/instructions which, when executed by a processor, implement the above batch task testing method.

According to the technical scheme provided by the invention, when the batch processing task is tested, key tasks are firstly screened from at least one group of tasks to be tested according to the task properties of the tasks to be tested, then single batch execution results when each key task is executed independently are respectively obtained, full batch execution results corresponding to each key task under a batch execution scene are obtained, and the test of the batch processing task is realized by comparing the single batch execution results with the full batch execution results. In the process, the workload of test analysis is reduced by screening key tasks, the influence between different tasks in full-batch execution is effectively determined by comparing a single-batch execution result with a full-batch execution result, and then the effect of batch execution is effectively reflected by a test result, so that the test on the batch processing tasks in a batch processing environment is efficiently and accurately finished, and the production research is facilitated.

Drawings

In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the specification, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a flow chart of a batch task testing method according to an embodiment of the present disclosure;

FIG. 2 is a block diagram of a batch task testing device according to an embodiment of the present disclosure;

FIG. 3 is a block diagram of a batch task test equipment according to an embodiment of the present disclosure.

Detailed Description

The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it is obvious that the described embodiments are only some embodiments, not all embodiments, of the present specification. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present specification without any creative effort shall fall within the protection scope of the present specification.

In order to solve the above technical problem, a batch task testing method provided by the present invention is introduced. The execution main body of the batch processing task testing method is batch processing task testing equipment, and the batch processing task testing equipment comprises a server, an industrial personal computer, a Personal Computer (PC) and the like. As shown in fig. 1, the batch task testing method may include the following implementation steps.

S110: and acquiring at least one group of tasks to be tested.

The task to be tested may be a task to be tested, specifically, the test for the task to be tested may be to execute the task to be tested according to a normal business execution process, and take a process execution process and an execution result as corresponding test results; the test for the task to be tested may also be to formulate a specific execution environment, and obtain a task processing result obtained under the execution environment as a test result. In practical application, the test task can be set according to the requirement, and the method is not limited to this.

In the batch processing task testing method provided by the invention, the task to be tested can be a task executed based on a batch processing execution mode, the batch processing is to perform batch processing on the task, and corresponding operations are synchronously executed on the large batch of tasks generally through a preset program script. When the batch processing task is executed, the system calls the batch processing program, so that the batch processing task can be automatically processed line by line, and a corresponding execution result is obtained.

Specifically, in the application scenario to which the present invention is directed, the task to be tested may be a task processed based on a batch execution mode in a distributed architecture. The distributed architecture is a tool and an application for realizing distributed computation, and a plurality of computers are converged through a computer network, so that corresponding computation tasks are distributed based on load balance of nodes corresponding to different computers when the tasks are executed, the computation pressure is effectively dispersed under the condition that the nodes of the computers are highly autonomous, and the batch processing function of the system can be better realized.

In some embodiments, the important point of testing the task to be tested is to consider the influence of a batch execution scenario on the execution effect of the task to be tested, and compared with the case of executing the task to be tested separately, when the task to be tested is executed in batch, due to the influence among tasks in the execution process, the occupied resources of the task and the execution time of the task may overlap, so that the execution effect of the task to be tested in the batch processing process is influenced. Therefore, an effective test scheme needs to be formulated to obtain the test result of the task to be tested.

S120: and screening key tasks from the at least one group of tasks to be tested based on the task properties of the tasks to be tested.

The distributed architecture is more focused on the concurrency of task execution compared with the traditional centralized architecture. The concurrent execution process is difficult to avoid the interference or the overlapping of the time period required by the execution and the execution resources based on the execution relation among the tasks, and further has a difference with the effect when the service is executed independently. In order to analyze the execution effect of the batch processing task under the distributed architecture, when the task to be tested is tested, the effect of the task to be tested when being executed independently can be obtained, and the effect is compared with the effect of the task to be tested when being executed concurrently, so that the quality of the concurrent processing process is analyzed according to the comparison result, and further the business execution process is optimized.

However, if each task to be tested is separately executed, a test result of each task to be tested in the case of being executed separately is obtained, and the test result is compared with a batch execution result, it obviously needs to consume much time and resources. In addition, under the condition that the execution process of a part of tasks to be tested has particularity, the test result of the part of tasks to be tested may be different from the normal test result, but the test result is interfered, so that the obtained service test result is not representative.

Therefore, in order to deal with the above situation, in the present invention, when obtaining the execution effect of the task to be tested in the case of being executed alone, a key task may be first screened from the at least one group of tasks to be tested.

The key tasks can be tasks with higher importance or priority and can effectively embody the overall execution effect of the tasks. The task to be tested may be a task with a greater difficulty in execution, such as occupying more computing resources or having a longer processing period, so as to cover the processing flow of the task more typically.

In the invention, key tasks can be screened according to the task properties of the tasks to be tested. The task properties of the tasks to be tested are used for reflecting the characteristics of the tasks, and the relevance exists between the task properties and the execution process and the execution effect of the tasks, so that the key tasks need to be screened from the tasks to be tested by considering the task properties when the key tasks are determined.

In some embodiments, the task properties include at least one of task resource consumption, task priority, task complexity, task correspondence time window, task background pressure, and task operation type.

The task resource consumption amount is used to indicate the amount of resources consumed to execute the task. The resource amount may be a computing resource or a time resource, and for example, the resource amount may be used to indicate the number of computing nodes required for executing a task, or the time consumed for executing the task according to a preset flow.

In addition, the task resource consumption can also be used for representing the consumption condition of the task to be tested on the public resource. During the concurrent processing of tasks, a plurality of different tasks may involve occupying the same resource, which may be, for example, a computing resource of a computing node or a shared resource that needs to be locked. When the occupation ratio of the tasks to be tested to the public resources is higher, the execution results of the special tasks can be reflected better, and the execution results are further used for feeding back the execution flow of the tasks.

The task priority may be a preset priority level, for example, the manager may set corresponding priority levels for different types of tasks in advance according to different importance degrees of the tasks, and the preset task priority level may be directly called to perform the determination when screening the important tasks. The task priority can also be associated with the attribute of the task to be tested, and when the task to be tested corresponds to different categories, the priority of the task to be tested can be determined according to the corresponding relation between the categories and the priorities.

In some specific embodiments, in determining the task priority, the determination may be made based on an association between the task under test and the impact factor. The influence factor may be a factor that directly reflects the execution effect of the task. For example, in the processing of banking, the impact factor may include at least one of business operation, annual settlement, accounting, and customer experience. In practical application, the corresponding impact factor may be determined according to the application environment of the service and the characteristics of the service itself, which is not limited to the above example and is not described herein again.

The task complexity is directly related to the data size and scale of the task itself, e.g., a task with a larger data size has a higher task complexity. The task complexity can also be hooked with the hierarchical structure of the task, when the task is sequentially executed according to a fixed flow, the task can be directly executed according to a corresponding script program, and the corresponding complexity is also low; when the tasks have different subtasks and influence factors such as incidence relation or execution sequence exist among the subtasks, the automation degree is low when the task to be tested is executed, and the task complexity degree is obviously high.

The time window corresponding to the task may correspond to a time window for executing the task, the length of the time window may represent time consumed for executing the task, the position of the time window may be used to represent an order or a moment for executing the task, and analyzing the time window corresponding to the task may comprehensively consider influence relationships between the time windows.

The task background pressure can represent the relevance between the task and the execution environment, and the task background pressure can represent the relationship between the task and the online transaction, and when the online transaction process exists, the more complex interaction relationship exists between the task and the execution environment; the task background pressure may also refer to a batch processing background, and different tasks to be tested may have different batch processing backgrounds, which are used to refer to influences of the tasks in a batch processing execution environment.

The task operation types may reflect operations related to the task, and may be divided by the administrator according to different execution modes or execution characteristics of the task, where the task operation types may specifically include different operations such as large file operation, cross-library operation, database full-table processing, and the like. According to the operation type of the task, the execution flow of the task can be effectively combined for analysis.

When screening key tasks, one or more of the listed task properties may be analyzed according to actual application requirements, which is not limited.

After the task properties of the tasks are obtained, the key tasks can be screened according to the task properties. By combining the specific introduction of the task properties, the corresponding screening standards can be preset, so that the corresponding key tasks can be intuitively and preferentially acquired. For example, a task to be measured whose execution consumption resource exceeds a certain amount may be regarded as an important task, or a task to be measured whose task priority satisfies a certain condition may be regarded as an important task. The specific screening criteria can be set according to the requirements of the actual application, and are not described herein again.

S130: and respectively obtaining single batch execution results when each key task is executed independently.

After each key task is obtained, each key task can be executed independently in a single batch scene to obtain a single batch execution result of each key task. In a single batch scene, only one task may be executed independently to obtain a corresponding test result, or only one key task may be arranged to be executed in an original batch execution environment, thereby achieving the effect of single batch execution.

The single batch execution result may include a task execution result and task execution data. The task execution result is used for directly reflecting whether the task is successfully executed or not. The task execution data can reflect the execution effect of the task, such as the execution time of the task, the number of computing nodes processing the task, the occupation condition of the memory/server/database and other parameters, and the effect of the test can be more finely obtained through the task execution data.

In some embodiments, in a single batch scenario, a non-functional test mode may be adopted to obtain a single batch execution result of each key task. Non-functional tests are generally tests performed in challenging environments, for example, to obtain corresponding parameters during task execution in cases of low or no amount of resources. Accordingly, the test results of non-functional tests are generally not directly accessible through testing, but rather need to be inferred indirectly through test data, such as by testing failure rates corresponding to different loads to determine load capacity. Specific items of the non-functional test can be set according to the requirements of practical application, and are not described herein again.

When each key task is executed independently, the occupation of resources and the robbery of common resources are not involved between the key tasks, and certain difference exists between the key tasks and the task batch processing execution process. Therefore, in the subsequent process, the final test result can be determined by comparing the single batch execution result with the result obtained under the condition that all tasks to be tested are executed in batch.

In some embodiments, regarding the single batch test result, whether the execution of each key task meets the task execution index may also be considered, so that the test for the key tasks is completed in advance according to the single batch scenario.

The task execution index may be an index set in advance for judging the task execution effect, for example, the task execution index may be judged by using processing capacity, time window, and resource consumption as judgment criteria. The processing capability may indicate whether each key task was successfully executed, the number of exceptions in the execution process, and the like. The time window may represent the time taken by the emphasis task, as well as the specific moment of execution. Resource consumption may represent a degree of utilization of computing resources, such as memory, server, and database occupancy, among others. The task execution index may be set based on the above specific content, for example, whether the task execution is successful or not, or a certain resource consumption and time consumption are set for evaluating the task execution effect. The specific setting of the task execution index may be set based on the requirements of the actual application, which is not limited to this.

S140: and under a batch execution scene, acquiring a full batch execution result of each key task.

After the execution results of each key task are obtained in a single batch execution scene, for comparison, the full batch execution results of each key task can be obtained in a batch execution scene.

And the batch execution scene is to perform batch processing on all tasks to be tested according to the batch processing execution flow. Under the batch execution scene, each task to be tested is executed in batch according to a preset program script.

And the full-batch execution result is used for reflecting the execution result of each task to be tested in the batch execution scene. The full-batch execution result may include whether the task to be tested is successfully executed or not, or may include related parameters reflected in the process of being executed by the task to be tested, such as usage of resources, time consumed by execution, and the like. In some embodiments, the content involved in the full batch execution result may correspond to the content involved in a single batch execution result, thereby facilitating comparison in subsequent steps to obtain a final test result.

In some embodiments, the batch execution scenario is an execution scenario in a distributed architecture, that is, each task to be tested is processed by a distributed cluster. The distributed cluster comprises a plurality of distributed nodes, and each distributed node can finish the processing of each task in an autonomous mode. When all tasks to be tested are processed in batch, the tasks to be tested can be distributed to all distributed nodes based on the requirements of the processing flow based on the load balance among the distributed nodes, and the distributed nodes are respectively used for processing to obtain corresponding full-batch execution results.

Based on the above embodiment, the data in the database corresponding to the distributed cluster may be stored based on a narrow table structure. Compared with the concept of a wide table, the narrow table has the advantages that more fields are involved in a database table in the wide table, and the structure of the narrow table is strictly designed according to the database, so that the redundancy degree of data is greatly reduced, the simplification of the structure of the database is ensured, the storage capacity of the database is improved, and the narrow table is more suitable for a batch processing task execution environment.

Accordingly, the Batch execution environment may also be an environment built based on a Spring Batch framework. The Spring Batch framework is a lightweight, comprehensive Batch framework for developing powerful enterprise-level Batch applications for daily operations. The Batch execution environment is built based on the Spring Batch framework, and the business can be processed based on each block, so that the execution effect of Batch processing is effectively guaranteed.

Because the subsequent steps mainly involve comparison between key tasks, although the execution of all tasks to be tested is involved in a full-batch execution scene, in order to reduce the time consumed by analysis, only the full-batch execution result corresponding to the key tasks can be extracted, so that the test efficiency is improved while the test result is effectively obtained.

The full-batch execution result is used for reflecting the execution performance of each key task, and specifically can be embodied in the characteristics of task execution integrity, batch processing efficiency, task execution time window state, server resource use condition, public resource use condition and the like.

The task execution integrity is used for reflecting the integrity of the execution process corresponding to the key task. The integrity may be embodied during normal execution of the task, i.e. whether the task is executed completely in case of normal execution. In some other examples, corresponding abnormal conditions and fault conditions may also be constructed, for example, situations such as computing resources are unavailable or a database fails, and whether the task can be completely executed is determined, so as to further determine the robustness of the task execution.

Batch processing efficiency is used to analyze the execution efficiency of a task, which may be associated with the execution time and complexity of the task itself. The less time is spent on tasks with the same complexity, the higher the execution efficiency is, and the more reasonable the distribution and processing of the tasks are.

The task execution time window state is used for describing a time window corresponding to the task execution, and comprises a time length and the time when the task starts and ends, so that the time windows of different tasks can be comprehensively compared, and the influence conditions among different tasks can be analyzed.

The server resource usage may be a usage of a computing resource of the server during task execution, for example, the number of occupied computing nodes or the number of occupied processes is refined, so that a relevant situation of the resource used during task execution can be determined more finely.

Common resource usage is used to represent the occupation of common resources by different tasks in a batch environment. Where there are multiple nodes processing tasks in parallel, the common resource may be a resource shared between the nodes, such as shared storage, shared compute nodes, and the like. In order to further analyze the influence relationship among different tasks, the specific execution condition of the tasks can be effectively determined by acquiring the use condition of the public resources for analysis.

The above is merely an exemplary description of the full batch execution result, and other full batch execution results may be set according to requirements in practical applications, which is not limited to this.

In some embodiments, obtaining the full batch execution result includes a resource scheduling status of each key task. And when a full-batch execution result is obtained, obtaining the resource scheduling condition of each key task in a batch execution scene. The resource scheduling condition comprises at least one of a resource load balancing condition, a resource scheduling fault switching condition and a job automatic scheduling condition.

And in the resource load balancing case, description is made on the load conditions of different distributed nodes in the distributed architecture. Different distributed nodes have different computing capacities, and the task amount allocated to the different distributed nodes is different, so that different distributed nodes have different load conditions, for example, some nodes have an idle state under the condition that some nodes are overloaded. In order to fully utilize the computing power of each node, the resource load balancing situation needs to be analyzed to achieve load balancing.

The resource scheduling fault switching is used for judging whether the resource node can transfer other nodes in time under the condition that sudden faults occur so as to ensure that the task is continuously and smoothly executed. The sudden failure of the resource node can be artificially set to test the processing effect of the task to be tested.

The job automatic scheduling case may indicate an automatic execution case of the task. Because the automatic execution of the tasks to be tested is mainly performed by using the program scripts under the batch processing environment, the automatic execution effect of the tasks to be tested can be effectively determined by analyzing the automatic scheduling condition of the jobs, and the automatic execution effect is also used for reflecting the full-batch execution result of the tasks.

After the at least one resource scheduling condition is obtained, the resource scheduling condition may be used as a full-batch execution result, or after the resource scheduling condition is analyzed, a corresponding analysis result may be used as a full-batch execution result. In practical application, other resource scheduling conditions may also be set according to the test requirements and the needs of the practical application, which are not limited to the above examples and are not described herein again.

In some embodiments, the full batch execution result includes a database call status. And when the full-batch execution result is obtained, obtaining the database calling condition under a batch execution scene is involved. Since the task to be tested may also involve calling data in the database during execution, and there may be situations such as data occupation, etc., the corresponding database calling status also needs to be analyzed.

Specifically, the database calling condition may include at least one of an RAC switching availability condition, an influence condition of a database exception on task processing, and a database shared storage availability condition.

Rac (Oracle Real Application cluster) is an Oracle grid computing technology, and is generally a technology for supporting a grid computing environment under the support of two or more homogeneous computers and storage devices. The RAC has the advantages of high availability, high performance, capability of being replaced according to needs and the like. When RAC is utilized, switching among databases related to RAC may be involved in the process of executing tasks to be tested, and verifying the availability of RAC switching can effectively judge the availability of the databases and can effectively support the application of RAC.

The influence of the database exception condition on the task processing may be an execution effect of the task to be tested in the case of the database exception. The database exception may be an exception condition artificially simulated for obtaining a strain capability for an emergency condition of the database exception during task execution. In the case of an abnormal database, whether the callable backup data exists or not and whether the task can be executed without depending on the data in the database or not can be the test content for the task to be tested, so as to further analyze the high availability of the database.

The database shared storage available condition is used for describing the calling condition of shared contents among different databases under the condition that the shared storage contents exist among the databases. In practical applications, in order to save storage space, shared storage may exist among databases, that is, corresponding contents exist in a shared manner, so as to save storage space and improve the callability of data. When the tasks are executed in batches, the database common storage available condition can be analyzed because the tasks executed concurrently can involve the calling of the databases in different databases. In a specific test process, corresponding data can be placed in the shared storage unit in advance according to the data calling condition in the execution process of the task to be tested, so that the shared storage condition of the database is verified.

The RAC switching availability, the influence of the database exception on the task processing, and the database shared storage availability are mainly used for analyzing and testing the influence on the database in the batch task execution process, and because the calling status of the database also has a large influence on the test result, the influence of the database on the execution result needs to be considered during the test, so as to ensure the effective calling of the data in the database.

After the at least one database calling condition is obtained, the database calling condition may be used as a full-batch execution result, or after the database calling condition is analyzed, a corresponding analysis result may be used as a full-batch execution result. In the practical application, other database calling conditions may also be set according to the test requirements and the needs of the practical application, which are not limited to the above examples and are not described herein again.

In some embodiments, the building of the batch execution scenario may be done in advance before the full batch execution process is executed. When the batch execution scene is constructed, the execution overlapping condition of each task to be tested can be determined firstly. The execution overlap condition is used to reflect the impact of the important tasks on each other during execution. The execution overlapping conditions comprise time period overlapping and/or data resource overlapping conditions, because batch processing relates to batch processing of a large number of tasks, different tasks can be executed in the same time period or different tasks need to call the same computing resource for execution, the influence relationship among the key tasks can be determined by integrating the execution overlapping conditions, the construction of batch execution scenes is further completed, and the batch execution effect of the key tasks is ensured.

S150: determining a test result based on a comparison of the single batch execution result and the full batch execution result.

After the single batch execution result and the full batch execution result are obtained, the single batch execution result and the full batch execution result can be compared. The single-batch execution result and the full-batch execution result are respectively used for reflecting the execution results of executing a certain task independently and executing all tasks in batches, and the difference between the single-batch execution result and the full-batch execution result is analyzed according to the comparison result of the single-batch execution result and the full-batch execution result, so that the interference of the batch processing execution process on the execution process of the tasks to be detected can be intuitively and effectively determined.

Specifically, a corresponding analysis index may be preset, and when the difference determined according to the comparison result meets the definition of the analysis index, it may be determined that the batch execution status interferes with the effective execution of the task.

After determining the test results, the task execution process may be adjusted based on the test results, such as changing the allocation of resources or staggering the order between certain ones to ensure efficient execution of tasks in the batch process. The specific adjustment process may be set based on the requirements of the actual application, and is not described herein again.

In some embodiments, in an exception test scenario, exception handling conditions corresponding to each key task may also be obtained. The abnormal test scenario may be a special scenario set manually, for example, the resource is unavailable, and the execution process is reported to be wrong. Accordingly, the exception handling condition includes at least one of an emergency handling condition, an associated job impact condition, and an exception recovery condition. The emergency treatment situation may be that the emergency treatment situation can be automatically and effectively handled against the abnormal situation in the task execution process. The associated job impact condition may be an execution condition of other associated tasks of the tasks involved in the abnormal situation. The exception recovery condition may be whether the task is effectively recovered from the exception, resumed or re-executed to complete the execution process normally.

And determining a corresponding test result according to the exception handling condition, and further effectively analyzing the corresponding capability aiming at the exception in the task execution process.

Through the introduction of the batch processing task testing method in the above embodiment, it can be seen that, when testing is performed on batch processing tasks, the method firstly screens key tasks from at least one group of tasks to be tested according to task properties of the tasks to be tested, then respectively obtains a single batch execution result when each key task is executed independently, obtains a full batch execution result corresponding to each key task in a batch execution scene, and realizes testing on the batch processing tasks by comparing the single batch execution result with the full batch execution result. In the process, the workload of test analysis is reduced by screening key tasks, the influence between different tasks in full-batch execution is effectively determined by comparing a single-batch execution result with a full-batch execution result, and then the effect of batch execution is effectively reflected by a test result, so that the test on the batch processing tasks in a batch processing environment is efficiently and accurately finished, and the production research is facilitated.

A batch task testing apparatus according to the present invention is described based on the batch task testing method corresponding to fig. 1. The batch processing task testing device is arranged on the batch processing task testing equipment. As shown in fig. 2, the batch job testing apparatus includes the following modules.

The task-to-be-tested obtaining module 210 is configured to obtain at least one group of tasks to be tested.

And the key task screening module 220 is configured to screen a key task from the at least one group of tasks to be tested based on the task property of the task to be tested.

The single batch execution result obtaining module 230 is configured to obtain single batch execution results when each key task is executed independently.

And a full-batch execution result obtaining module 240, configured to obtain a full-batch execution result of each key task in a batch execution scenario.

And a test result determining module 250 for determining a test result based on the comparison result of the single batch execution result and the full batch execution result.

In some embodiments, the task properties include at least one of task resource consumption, task priority, task complexity, task correspondence time window, task background pressure, and task operation type.

In some embodiments, the task priority is determined based on an association between the task under test and the impact factor; the impact factor includes at least one of business operation, end-of-year settlement, accounting, and customer experience.

In some embodiments, the batch task testing device further comprises: the execution overlapping condition determining module is used for determining the execution overlapping condition of each task to be tested; the performing an overlap condition comprises performing a time period overlap and/or a data resource utilization overlap; and the batch execution scene construction module is used for constructing the batch execution scene based on the execution overlapping condition.

In some embodiments, the full batch execution result obtaining module 240 includes: the resource scheduling state acquiring unit is used for acquiring the resource scheduling state of each key task in a batch execution scene; the resource scheduling condition comprises at least one of a resource load balancing condition, a resource scheduling fault switching condition and an automatic job scheduling condition; and the full-batch execution result determining unit is used for determining a full-batch execution result according to the resource scheduling condition.

In some embodiments, the full batch execution result obtaining module 240 includes: the database calling condition acquisition unit is used for acquiring database calling conditions in a batch execution scene; the database calling condition comprises at least one of an RAC switching available condition, an influence condition of the database exception on the task processing and a database shared storage available condition.

In some embodiments, the task to be tested corresponds to at least one execution system; the execution systems have an association relation; the single batch execution result and/or the full batch execution result comprises: data provision between execution systems with associative relationships affects effects and/or data dependent effects.

In some embodiments, in the batch execution scenario, each task to be tested is processed by a distributed cluster.

In some embodiments, the data in the database corresponding to the distributed cluster is stored based on a narrow table structure.

In some embodiments, the Batch execution scenario is built based on a Spring Batch framework.

In some embodiments, the full batch execution result includes task execution integrity, batch processing efficiency, task execution time window state, server resource usage, and common resource usage.

In some embodiments, the batch task testing device further comprises: the exception handling state acquisition module is used for acquiring exception handling states of all key tasks in an exception test scene; the exception handling condition comprises at least one of an emergency handling condition, an associated operation influence condition and an exception recovery condition; and the test result acquisition module is used for determining a test result according to the abnormal processing condition.

Based on the batch task testing method corresponding to fig. 1, the present invention provides a computer readable storage medium having stored thereon a computer program/instructions. As shown in fig. 3, the computer-readable storage medium can be read by a processor based on an internal bus of a device, and program instructions in the computer-readable storage medium are implemented by the processor.

In this embodiment, the computer-readable storage medium may be implemented in any suitable manner. The computer-readable storage medium includes, but is not limited to, a Random Access Memory (RAM), a Read-Only Memory (ROM), a Cache (Cache), a Hard Disk Drive (HDD), a Memory Card (Memory Card), and the like. The computer storage medium stores computer program instructions. The computer program instructions, when executed, implement the program instructions or modules of the embodiments corresponding to fig. 1 of the present specification.

In this embodiment, the processor may be implemented in any suitable manner. For example, the processor may take the form of, for example, a microprocessor or processor and a computer-readable medium that stores computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, an embedded microcontroller, and so forth. Specifically, the processor may execute the computer program instructions to implement the following steps: acquiring at least one group of tasks to be tested; screening key tasks from the at least one group of tasks to be tested based on the task properties of the tasks to be tested; respectively obtaining single batch execution results when each key task is executed independently; under a batch execution scene, acquiring a full batch execution result of each key task; determining a test result based on a comparison of the single batch execution result and the full batch execution result.

Based on the batch task testing method corresponding to fig. 1, the present invention provides a computer program product having a computer program/instructions stored thereon. The computer program product may be a program written in a corresponding computer program language, stored in a corresponding storage device in a programmed manner, and transmittable via a computer network. The computer program product may be executed by a processor. In the present invention, the computer program product, when executed, implements the program instructions or modules of the corresponding embodiments of fig. 1.

It should be noted that, the batch processing task testing method, the batch processing task testing device, the storage medium and the program product may be applied to the technical field of big data, and may also be applied to other technical fields except the technical field of big data, which is not limited to this.

While the process flows described above include operations that occur in a particular order, it should be appreciated that the processes may include more or less operations that are performed sequentially or in parallel (e.g., using parallel processors or a multi-threaded environment).

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, tape storage, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that may be used to store information that may be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

The embodiments of this specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The described embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment. In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of an embodiment of the specification. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims (15)

1. A batch task testing method is characterized by comprising the following steps:

acquiring at least one group of tasks to be tested;

screening key tasks from the at least one group of tasks to be tested based on the task properties of the tasks to be tested;

respectively obtaining single batch execution results when each key task is executed independently;

under a batch execution scene, acquiring a full batch execution result of each key task;

determining a test result based on a comparison of the single batch execution result and the full batch execution result.

2. The method of claim 1, wherein the task properties include at least one of task resource consumption, task priority, task complexity, task correspondence time window, task background pressure, and task operation type.

3. The method of claim 2, wherein the task priority is determined based on an association between a task under test and an impact factor; the impact factor includes at least one of business operation, end-of-year settlement, accounting, and customer experience.

4. The method of claim 1, wherein before obtaining the full-batch execution result of each key task in the batch execution scenario, the method further comprises:

determining the execution overlapping condition of each task to be tested; the performing an overlap condition comprises performing a time period overlap and/or a data resource utilization overlap;

and constructing the batch execution scene based on the execution overlapping condition.

5. The method of claim 1, wherein the obtaining of the full-batch execution result of each key task in the batch execution scenario includes:

acquiring a resource scheduling condition of each key task in a batch execution scene; the resource scheduling condition comprises at least one of a resource load balancing condition, a resource scheduling fault switching condition and an automatic job scheduling condition;

and determining a full-batch execution result according to the resource scheduling condition.

6. The method of claim 1, wherein the full batch execution result comprises a database call status; under the batch execution scene, acquiring a full batch execution result of each key task, including:

acquiring a database calling condition in a batch execution scene; the database calling condition comprises at least one of an RAC switching available condition, an influence condition of the database exception on the task processing and a database shared storage available condition.

7. The method of claim 1, wherein the task under test corresponds to at least one execution system; the execution systems have an association relation; the single batch execution result and/or the full batch execution result comprises: data provision between execution systems with associative relationships affects effects and/or data dependent effects.

8. The method of claim 1, wherein in the batch execution scenario, each task under test is processed by a distributed cluster.

9. The method of claim 8, wherein data in the database corresponding to the distributed cluster is stored based on a narrow table structure.

10. The method of claim 8, wherein the Batch execution scenario is built based on a Spring Batch framework.

11. The method of claim 1, wherein the full batch execution results include task execution integrity, batch processing efficiency, task execution time window status, server resource usage, common resource usage.

12. The method of claim 1, wherein after obtaining a full batch execution result of each key task in a batch execution scenario, the method further comprises:

acquiring exception handling conditions corresponding to each key task in an exception test scene; the exception handling condition comprises at least one of an emergency handling condition, an associated operation influence condition and an exception recovery condition;

and determining a test result according to the abnormal processing condition.

13. A batch job testing apparatus, comprising:

the task acquisition module to be tested is used for acquiring at least one group of tasks to be tested;

the key task screening module is used for screening key tasks from the at least one group of tasks to be tested based on the task properties of the tasks to be tested;

the single-batch execution result acquisition module is used for respectively acquiring single-batch execution results when each key task is independently executed;

the full-batch execution result acquisition module is used for acquiring full-batch execution results of each key task in a batch execution scene;

and the test result determining module is used for determining a test result based on the comparison result of the single batch execution result and the full batch execution result.

14. A computer-readable storage medium, on which a computer program/instructions is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1-12.

15. A computer program product having stored thereon computer programs/instructions, characterized in that the computer programs/instructions, when executed by a processor, implement the steps of the method according to any of claims 1-12.

Images