CN113094185A - Method and device for detecting parallel data processing abnormity of database batch program - Google Patents
Method and device for detecting parallel data processing abnormity of database batch program Download PDFInfo
- Publication number
- CN113094185A CN113094185A CN202110394104.5A CN202110394104A CN113094185A CN 113094185 A CN113094185 A CN 113094185A CN 202110394104 A CN202110394104 A CN 202110394104A CN 113094185 A CN113094185 A CN 113094185A
- Authority
- CN
- China
- Prior art keywords
- batch
- partition
- information
- program
- data table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000005192 partition Methods 0.000 claims abstract description 314
- 230000002159 abnormal effect Effects 0.000 claims abstract description 26
- 238000001514 detection method Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 17
- 238000004458 analytical method Methods 0.000 claims description 10
- 230000005856 abnormality Effects 0.000 claims description 8
- 238000013473 artificial intelligence Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 18
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000012937 correction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000717 retained effect Effects 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
- G06F9/524—Deadlock detection or avoidance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and a device for detecting the processing abnormity of parallel data of a database batch program, which can be used in the technical field of artificial intelligence, wherein the method comprises the following steps: determining partition information of the data table according to the database partition rule table; determining batch information according to a batch number control table of a batch program; according to the partition information and the batch information, whether the data table is abnormal or not during the parallel data processing of the batch program is determined.
Description
Technical Field
The invention relates to the technical field of databases, in particular to the technical field of artificial intelligence, and particularly relates to a method and a device for detecting abnormal processing of batch program parallel data of a database.
Background
In order to improve the efficiency of data processing such as query, a host data table in a database is usually divided into a plurality of partitions. When a batch program processes parallel data of different batches, if different batches process data of the same data table partition at the same time, a phenomenon that a plurality of programs lock one data table partition at the same time occurs, and further, the batch program may be deadlocked and interrupted. At present, due to the fact that the data volume of a test environment is insufficient, the difference between the host resource and the production environment resource is large, and the test environment is difficult to test abnormal conditions which may occur when batch programs are processed in parallel due to the fact that data table partitions conflict with batch setting.
Disclosure of Invention
The invention aims to provide a method for detecting the exception of parallel data processing of a database batch program, which is used for checking data table partitions and batch program batch settings and avoiding the exception of the data table during the parallel data processing of the batch program. The invention also aims to provide a database batch program parallel data processing abnormity detection device. It is a further object of this invention to provide such a computer apparatus. It is a further object of this invention to provide such a readable medium.
In order to achieve the above object, the present invention discloses a method for detecting parallel data processing exception of a database batch program, which comprises:
determining partition information of the data table according to the database partition rule table;
determining batch information according to a batch number control table of a batch program;
and determining whether the data table is abnormal or not during the parallel data processing of the batch program according to the partition information and the batch information.
Preferably, the determining the partition information of the data table according to the database partition rule table specifically includes:
scanning a database partition rule table;
determining the number of partitions, partition conditions and partition rules of partition ranges of each data table in the database according to the partition rule table;
and determining all the partitions of the data table and partition information corresponding to the partitions according to the partition rules.
Preferably, the determining the batch information according to the batch number control table of the batch program specifically includes:
scanning a batch number control table of a batch program;
and determining the batch number and the corresponding area number range of each batch of the batch program according to the number control table to obtain batch information.
Preferably, the determining whether an exception exists in the parallel data processing of the batch program according to the partition information and the batch information specifically includes:
and determining whether partitions corresponding to a plurality of batches exist in one data table according to the partition information and the batch information of the one data table, and if so, determining that the parallel data processing of the batch program has an exception.
Preferably, the determining whether partitions corresponding to a plurality of batches exist in one data table according to the partition information and the batch information of the one data table specifically includes:
determining the area number range corresponding to each partition of the data table according to the partition information;
and determining whether the area number range of one partition of the data table is positioned in the area number ranges of a plurality of batches or not according to the area number range corresponding to each partition of the data table and the area number range corresponding to each batch.
Preferably, the determining, according to the partition information, the area number range corresponding to each partition of the data table specifically includes:
determining account number ranges corresponding to all partitions of the data table according to the partition information;
and converting the account number range into a region number range to obtain a region number range corresponding to each partition.
Preferably, the method further comprises the following steps:
if the abnormality exists;
and modifying the partition information of the data table in the database partition rule table so that each partition of the data table corresponds to one batch of the batch program.
The invention also discloses a device for detecting the parallel data processing abnormity of the database batch program, which comprises the following components:
the partition analysis module is used for determining partition information of the data table according to the database partition rule table;
the batch analysis module is used for determining batch information according to a batch number control table of the batch program;
and the abnormality detection module is used for determining whether the data table has abnormality during parallel data processing of the batch program according to the partition information and the batch information.
The invention also discloses a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor,
the processor, when executing the program, implements the method as described above.
The invention also discloses a computer-readable medium, having stored thereon a computer program,
which when executed by a processor implements the method as described above.
The invention determines the partition information of the data table in the database according to the database partition rule table, and determines the batch information of the batch corresponding to the parallel program according to the batch number control table of the batch program. And determining whether the abnormal condition caused by the fact that the plurality of programs simultaneously process the data of one partition of the data table when the plurality of programs in the batch program process the data table in parallel exists according to the batch information of the batch program and the partition information of the data table. And further, the partition information of the data table or the batch information of the batch program can be adjusted according to the abnormal detection result so as to avoid the abnormal condition of the data table during the parallel data processing of the batch program.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow diagram illustrating one embodiment of a method for detecting an exception in parallel data processing of a database batch program according to the present invention;
FIG. 2 is a flowchart of an embodiment S100 of the method for detecting the exception of parallel data processing of a database batch program according to the present invention;
FIG. 3 is a flowchart of a method for detecting exception in parallel data processing of a database batch program according to an embodiment S200 of the present invention;
FIG. 4 is a flowchart illustrating a database batch program parallel data processing exception detection method S310 according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a method for detecting exception of parallel data processing of a database batch program according to an embodiment of the present invention S311;
FIG. 6 is a flowchart of a method for detecting exception in parallel data processing of a database batch program according to an embodiment S400 of the present invention;
FIG. 7 is a block diagram illustrating an embodiment of an apparatus for detecting exception in parallel data processing of a database batch program according to the present invention;
FIG. 8 is a diagram illustrating an exception detection apparatus for parallel data processing of a database batch program according to an embodiment of the present invention, including an exception correction module;
FIG. 9 shows a schematic block diagram of a computer device suitable for use in implementing embodiments of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the method and the device for detecting the abnormal processing of the parallel data of the database batch program disclosed by the application can be used in the technical field of artificial intelligence and can also be used in any field except the technical field of artificial intelligence.
The method aims at solving the problem that at present, due to the fact that partition setting of a host database data table conflicts with batch setting of each program of a batch program, deadlock and interruption possibly exist during parallel data processing of the batch program are caused. The invention checks the partition setting of the data table and the batch setting of the batch program in the database, avoids the phenomenon of batch program running interruption caused by partition error of the data table or batch program batch error, and perfects the coverage range of program test.
According to one aspect of the invention, the embodiment discloses a method for detecting the exception of parallel data processing of a database batch program. As shown in fig. 1, in this embodiment, the method includes:
s100: and determining the partition information of the data table according to the database partition rule table.
S200: the batch information is determined from a batch number control table of the batch program.
S300: and determining whether the data table is abnormal or not during the parallel data processing of the batch program according to the partition information and the batch information.
The invention determines the partition information of the data table in the database according to the database partition rule table, and determines the batch information of the batch corresponding to the parallel program according to the batch number control table of the batch program. And determining whether the abnormal condition caused by the fact that the plurality of programs simultaneously process the data of one partition of the data table when the plurality of programs in the batch program process the data table in parallel exists according to the batch information of the batch program and the partition information of the data table. And further, the partition information of the data table or the batch information of the batch program can be adjusted according to the abnormal detection result so as to avoid the abnormal condition of the data table during the parallel data processing of the batch program.
In a preferred embodiment, as shown in fig. 2, the determining, by the S100, the partition information of the data table according to the database partition rule table specifically includes:
s110: the database partition rule table is scanned.
S120: and determining the partition number, the partition condition and the partition rule of the partition range of each data table in the database according to the partition rule table.
S130: and determining all the partitions of the data table and partition information corresponding to the partitions according to the partition rules.
Specifically, the partition rule table of the database stores the partition rule of each data table in the database. Therefore, the partition rule table of the database can determine that all the data tables with the partition number larger than 0 exist, that is, the data tables of a plurality of partitions exist. The partition rule may include partition rules such as the number of partitions of the data table, partition conditions, and partition ranges, and partition information of each partition of the data table in which a plurality of partitions exist may be determined according to the partition rules. For example, in a specific example, for relational database DB2, partition rule table mtbttprt in DB2 may be obtained and analyzed to obtain a list of data tables with Partitions (PART) greater than 0 in the mtbttprt table. And determining the partition number and the data range of each partition in the data table according to the corresponding partition rule for each data table in the data table list. Wherein the range of area numbers is characterized by a start area number and an end area number. It is understood that the data range may be a range of area numbers, or a range of fields such as a range of account numbers. Table 1 shows the partition information of the different data tables in this specific example.
TABLE 1
Preferably, data filtering may be performed on the data table according to the partition rule of the data table, that is, data in the data table is screened, and data in a field corresponding to the partition rule in the data table is retained, so as to facilitate subsequent anomaly detection.
In a preferred embodiment, as shown in fig. 3, the determining, by the S200, the batch information according to the batch number control table of the batch program specifically includes:
s210: a batch number control table of a batch program is scanned.
S220: and determining the batch number and the corresponding area number range of each batch of the batch program according to the number control table to obtain batch information.
Specifically, scanning a preset batch number control table in the batch program can determine the batch accessed by each program in the batch program, so as to obtain the batch information of the batch program. Specifically, the batch number and the corresponding area number range of each batch in the batch program can be screened from the batch number control table to obtain the batch information. For example, in one specific example, the batch number control table NTHPABZN of the batch program is scanned, and the NTHPABZN table is analyzed to obtain the batch number of each batch and the corresponding area number range. Wherein the range of area numbers is characterized by a start area number and an end area number. Table 2 shows the batch information for the batch program in this particular example.
TABLE 2
In a preferred embodiment, the step S300 of determining whether there is an exception in the parallel data processing of the batch program according to the partition information and the batch information specifically includes:
s310: and determining whether partitions corresponding to a plurality of batches exist in one data table according to the partition information and the batch information of the one data table, and if so, determining that the parallel data processing of the batch program has an exception.
Specifically, there is a corresponding relationship between the partition information of the data table and the batch information of the batch programs, and if one partition in the data table corresponds to multiple batches, this indicates that there may be a case where multiple programs access one partition of one data table at the same time when the batch program is running. This may result in multiple programs simultaneously locking a data table partition whose data processing may generate exceptions that result in batch program interrupts. Thus, in the preferred embodiment, it can be determined whether the data table partition runs in a batch program and is at risk of being subjected to data processing by a plurality of programs to generate an exception by determining the number of batches to which the data table partition corresponds.
In a preferred embodiment, as shown in fig. 4, the determining, by the S310, whether partitions corresponding to multiple batches exist in one data table according to the partition information and the batch information of the one data table specifically includes:
s311: and determining the area number range corresponding to each partition of the data table according to the partition information.
S312: and determining whether the area number range of one partition of the data table is positioned in the area number ranges of a plurality of batches or not according to the area number range corresponding to each partition of the data table and the area number range corresponding to each batch.
Specifically, the partition information of the data table may include a partition number and a corresponding area number range of each partition of the data table. The batch information for a batch program may include a batch number and a corresponding range of area numbers. Thus, the area number range of each partition may be compared with the area number range of each batch, and if the start area number or the end area number of the area number range of one partition falls within the area number ranges of at least two batches, it indicates that one partition of the data table corresponds to at least two batches of the batch program. For example, with respect to the partition information and the batch information of tables 1 and 2, it can be determined whether there is a case where one partition corresponds to a plurality of batches by comparing the area number range of each partition of tables 1 and the area number range of each batch of tables 2. Table 3 shows the results of the region number range alignments of tables 1 and 2.
TABLE 3
As can be seen from table 3, the area numbers in the area number range of the partition with partition number 9 of the BTBRSFXH table fall into 02 lots and 04 lots. The abnormal situation of occupation conflict can be caused when the data of the BTBRSFXH table partition 9 is processed by the 02 batch and the 04 batch in parallel. Similarly, the partition with the BTHCRDTP table partition number 9, the partition with the BTHRSKFQ table partition number 9, the partition with the BTHRSMUP table partition number 1, and the partition with the CTHRSCFM table partition number 50 all have a range of area numbers falling into a plurality of lots.
In a preferred embodiment, as shown in fig. 5, the determining, by the S311, an area number range corresponding to each partition of the data table according to the partition information specifically includes:
s3111: and determining the account number range corresponding to each partition of the data table according to the partition information.
S3112: and converting the account number range into a region number range to obtain a region number range corresponding to each partition.
Specifically, it is understood that the partition information of the data table may also include a partition number and a corresponding account number range. The account number range needs to be converted into the area number range through a data conversion mode. Therefore, the area number range corresponding to each partition is obtained, the partition information is the same as the partition and batch division standards of the batch information, and whether the range conflict exists between each partition and each batch can be analyzed. In a specific example, when the partitions are preset, the account number ranges corresponding to the partitions are set to form partition information, and meanwhile, the corresponding relation between the account number ranges and the area number ranges is preset. After the account number range corresponding to each partition is determined according to the partition information, the area number range corresponding to each partition can be further determined according to the corresponding relationship between the account number range and the area number range.
In a preferred embodiment, as shown in fig. 6, the method further includes S400:
s410: if there is an anomaly.
S420: and modifying the partition rule table of the database or the batch number control table of the batch program so that each partition of the data table corresponds to one batch of the batch program.
Specifically, if there is a data table due to a data partitioning or batch error of the batch program, there is an exception in the parallel data processing of the batch program. The partition information of the data table or the batch information of the batch program needs to be adjusted, so that one partition in the data table only corresponds to one batch, and the conflict phenomenon caused when one partition is called by a plurality of batches for data processing is avoided. Specifically, in actual operation, the partition rule table of the database needs to be modified to adjust the partition information of the data table, or the batch number control table of the batch program needs to be modified to adjust the batch information of the batch program. For example, for the partition with the partition number 9 in the BTBRSFXH table in table 3, the partition 9 may be split into two partitions, where the region number range of one partition is 201-299, corresponding to 02 batches; the region number range of the other partition is 300-302, corresponding to 04 batches. The two divided partitions of the partition 9 correspond to one batch respectively, so that the abnormal phenomenon of parallel data processing of batch programs is avoided.
The invention determines the partition information of the data table in the database according to the database partition rule table, and determines the batch information of the batch corresponding to the parallel program according to the batch number control table of the batch program. And determining whether the abnormal condition caused by the fact that the plurality of programs simultaneously process the data of one partition of the data table when the plurality of programs in the batch program process the data table in parallel exists according to the batch information of the batch program and the partition information of the data table. And further, the partition information of the data table or the batch information of the batch program can be adjusted according to the abnormal detection result so as to avoid the abnormal condition of the data table during the parallel data processing of the batch program.
Based on the same principle, the embodiment also discloses a device for detecting the parallel data processing exception of the database batch program. As shown in fig. 7, in the present embodiment, the apparatus includes a partition analysis module 11, a batch analysis module 12, and an abnormality detection module 13.
The partition analysis module 11 is configured to determine partition information of the data table according to the database partition rule table.
The batch analysis module 12 is configured to determine batch information according to a batch number control table of a batch program.
And the anomaly detection module 13 is used for determining whether the data table has an anomaly during parallel data processing of the batch program according to the partition information and the batch information.
The invention determines the partition information of the data table in the database according to the database partition rule table, and determines the batch information of the batch corresponding to the parallel program according to the batch number control table of the batch program. And determining whether the abnormal condition caused by the fact that the plurality of programs simultaneously process the data of one partition of the data table when the plurality of programs in the batch program process the data table in parallel exists according to the batch information of the batch program and the partition information of the data table. And further, the partition information of the data table or the batch information of the batch program can be adjusted according to the abnormal detection result so as to avoid the abnormal condition of the data table during the parallel data processing of the batch program.
In a preferred embodiment, the partition analysis module 11 is specifically configured to scan a database partition rule table. And determining the partition number, the partition condition and the partition rule of the partition range of each data table in the database according to the partition rule table. And determining all the partitions of the data table and partition information corresponding to the partitions according to the partition rules.
Specifically, the partition rule table of the database stores the partition rule of each data table in the database. Therefore, the partition rule table of the database can determine that all the data tables with the partition number larger than 0 exist, that is, the data tables of a plurality of partitions exist. The partition rule may include partition rules such as the number of partitions of the data table, partition conditions, and partition ranges, and partition information of each partition of the data table in which a plurality of partitions exist may be determined according to the partition rules. For example, in a specific example, for relational database DB2, partition rule table mtbttprt in DB2 may be obtained and analyzed to obtain a list of data tables with Partitions (PART) greater than 0 in the mtbttprt table. And determining the partition number and the data range of each partition in the data table according to the corresponding partition rule for each data table in the data table list. Wherein the range of area numbers is characterized by a start area number and an end area number. It is understood that the data range may be a range of area numbers, or a range of fields such as a range of account numbers. Table 1 shows the partition information of the different data tables in this specific example.
Preferably, data filtering may be performed on the data table according to the partition rule of the data table, that is, data in the data table is screened, and data in a field corresponding to the partition rule in the data table is retained, so as to facilitate subsequent anomaly detection.
In a preferred embodiment, the batch analysis module is specifically configured to scan a batch number control table of a batch program. And determining the batch number and the corresponding area number range of each batch of the batch program according to the number control table to obtain batch information.
Specifically, scanning a preset batch number control table in the batch program can determine the batch accessed by each program in the batch program, so as to obtain the batch information of the batch program. Specifically, the batch number and the corresponding area number range of each batch in the batch program can be screened from the batch number control table to obtain the batch information. For example, in one specific example, the batch number control table NTHPABZN of the batch program is scanned, and the NTHPABZN table is analyzed to obtain the batch number of each batch and the corresponding area number range. Wherein the range of area numbers is characterized by a start area number and an end area number. Table 2 shows the batch information for the batch program in this particular example.
In a preferred embodiment, the anomaly detection module 13 is specifically configured to determine whether partitions corresponding to multiple batches exist in one data table according to partition information and batch information of the one data table, and if so, determine that an anomaly exists in parallel data processing of a batch program.
Specifically, there is a corresponding relationship between the partition information of the data table and the batch information of the batch programs, and if one partition in the data table corresponds to multiple batches, this indicates that there may be a case where multiple programs access one partition of one data table at the same time when the batch program is running. This may result in multiple programs simultaneously locking a data table partition whose data processing may generate exceptions that result in batch program interrupts. Thus, in the preferred embodiment, it can be determined whether the data table partition runs in a batch program and is at risk of being subjected to data processing by a plurality of programs to generate an exception by determining the number of batches to which the data table partition corresponds.
In a preferred embodiment, the anomaly detection module 13 is further configured to determine an area number range corresponding to each partition of the data table according to the partition information. And determining whether the area number range of one partition of the data table is positioned in the area number ranges of a plurality of batches or not according to the area number range corresponding to each partition of the data table and the area number range corresponding to each batch.
Specifically, the partition information of the data table may include a partition number and a corresponding area number range of each partition of the data table. The batch information for a batch program may include a batch number and a corresponding range of area numbers. Thus, the area number range of each partition may be compared with the area number range of each batch, and if the start area number or the end area number of the area number range of one partition falls within the area number ranges of at least two batches, it indicates that one partition of the data table corresponds to at least two batches of the batch program. For example, with respect to the partition information and the batch information of tables 1 and 2, it can be determined whether there is a case where one partition corresponds to a plurality of batches by comparing the area number range of each partition of tables 1 and the area number range of each batch of tables 2. Table 3 shows the results of the region number range alignments of tables 1 and 2.
As can be seen from table 3, the area numbers in the area number range of the partition with partition number 9 of the BTBRSFXH table fall into 02 lots and 04 lots. The abnormal situation of occupation conflict can be caused when the data of the BTBRSFXH table partition 9 is processed by the 02 batch and the 04 batch in parallel. Similarly, the partition with the BTHCRDTP table partition number 9, the partition with the BTHRSKFQ table partition number 9, the partition with the BTHRSMUP table partition number 1, and the partition with the CTHRSCFM table partition number 50 all have a range of area numbers falling into a plurality of lots.
In a preferred embodiment, the anomaly detection module 13 is further configured to determine an account range corresponding to each partition of the data table according to the partition information. And converting the account number range into a region number range to obtain a region number range corresponding to each partition.
Specifically, it is understood that the partition information of the data table may also include a partition number and a corresponding account number range. The account number range needs to be converted into the area number range through a data conversion mode. Therefore, the area number range corresponding to each partition is obtained, the partition information is the same as the partition and batch division standards of the batch information, and whether the range conflict exists between each partition and each batch can be analyzed. In a specific example, when the partitions are preset, the account number ranges corresponding to the partitions are set to form partition information, and meanwhile, the corresponding relation between the account number ranges and the area number ranges is preset. After the account number range corresponding to each partition is determined according to the partition information, the area number range corresponding to each partition can be further determined according to the corresponding relationship between the account number range and the area number range.
In a preferred embodiment, as shown in fig. 8, the apparatus further comprises an anomaly correction module 14. The anomaly correction module 14 is configured to correct an anomaly, if any. And modifying the partition rule table of the database or the batch number control table of the batch program so that each partition of the data table corresponds to one batch of the batch program.
Specifically, if there is a data table due to a data partitioning or batch error of the batch program, there is an exception in the parallel data processing of the batch program. The partition information of the data table or the batch information of the batch program needs to be adjusted, so that one partition in the data table only corresponds to one batch, and the conflict phenomenon caused when one partition is called by a plurality of batches for data processing is avoided. Specifically, in actual operation, the partition rule table of the database needs to be modified to adjust the partition information of the data table, or the batch number control table of the batch program needs to be modified to adjust the batch information of the batch program. For example, for the partition with the partition number 9 in the BTBRSFXH table in table 3, the partition 9 may be split into two partitions, where the region number range of one partition is 201-299, corresponding to 02 batches; the region number range of the other partition is 300-302, corresponding to 04 batches. The two divided partitions of the partition 9 correspond to one batch respectively, so that the abnormal phenomenon of parallel data processing of batch programs is avoided.
The invention determines the partition information of the data table in the database according to the database partition rule table, and determines the batch information of the batch corresponding to the parallel program according to the batch number control table of the batch program. And determining whether the abnormal condition caused by the fact that the plurality of programs simultaneously process the data of one partition of the data table when the plurality of programs in the batch program process the data table in parallel exists according to the batch information of the batch program and the partition information of the data table. And further, the partition information of the data table or the batch information of the batch program can be adjusted according to the abnormal detection result so as to avoid the abnormal condition of the data table during the parallel data processing of the batch program.
Since the principle of the device for solving the problems is similar to the method, the implementation of the device can refer to the implementation of the method, and the detailed description is omitted here.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer device, which may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
In a typical example, the computer device specifically comprises a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method performed by the client as described above when executing the program, or the processor implementing the method performed by the server as described above when executing the program.
Referring now to FIG. 9, shown is a schematic diagram of a computer device 600 suitable for use in implementing embodiments of the present application.
As shown in fig. 9, the computer apparatus 600 includes a Central Processing Unit (CPU)601 which can perform various appropriate works and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM)) 603. In the RAM603, various programs and data necessary for the operation of the system 600 are also stored. The CPU601, ROM602, and RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output section 607 including a Cathode Ray Tube (CRT), a liquid crystal feedback (LCD), and the like, and a speaker and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted as necessary on the storage section 608.
In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A parallel data processing exception detection method for a database batch program is characterized by comprising the following steps:
determining partition information of the data table according to the database partition rule table;
determining batch information according to a batch number control table of a batch program;
and determining whether the data table is abnormal or not during the parallel data processing of the batch program according to the partition information and the batch information.
2. The method for detecting the processing exception of the parallel data of the database batch program according to claim 1, wherein the determining the partition information of the data table according to the database partition rule table specifically comprises:
scanning a database partition rule table;
determining the number of partitions, partition conditions and partition rules of partition ranges of each data table in the database according to the partition rule table;
and determining all the partitions of the data table and partition information corresponding to the partitions according to the partition rules.
3. The method for detecting the processing exception of the parallel data of the database batch program according to claim 1, wherein the determining the batch information according to the batch number control table of the batch program specifically comprises:
scanning a batch number control table of a batch program;
and determining the batch number and the corresponding area number range of each batch of the batch program according to the number control table to obtain batch information.
4. The method for detecting the processing exception of the parallel data of the database batch program according to claim 1, wherein the determining whether the processing exception of the parallel data of the batch program exists according to the partition information and the batch information specifically comprises:
and determining whether partitions corresponding to a plurality of batches exist in one data table according to the partition information and the batch information of the one data table, and if so, determining that the parallel data processing of the batch program has an exception.
5. The method of claim 4, wherein the determining whether partitions corresponding to the plurality of batches exist in the one data table according to the partition information and the batch information of the one data table specifically comprises:
determining the area number range corresponding to each partition of the data table according to the partition information;
and determining whether the area number range of one partition of the data table is positioned in the area number ranges of a plurality of batches or not according to the area number range corresponding to each partition of the data table and the area number range corresponding to each batch.
6. The method for detecting the processing abnormality of the parallel data of the database batch program according to claim 5, wherein the determining the area number range corresponding to each partition of the data table according to the partition information specifically comprises:
determining account number ranges corresponding to all partitions of the data table according to the partition information;
and converting the account number range into a region number range to obtain a region number range corresponding to each partition.
7. The method of claim 1, further comprising:
if the abnormality exists;
and modifying the partition information of the data table in the database partition rule table so that each partition of the data table corresponds to one batch of the batch program.
8. The utility model provides a database batch procedure parallel data processing anomaly detection device which characterized in that includes:
the partition analysis module is used for determining partition information of the data table according to the database partition rule table;
the batch analysis module is used for determining batch information according to a batch number control table of the batch program;
and the abnormality detection module is used for determining whether the data table has abnormality during parallel data processing of the batch program according to the partition information and the batch information.
9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor,
the processor, when executing the program, implements the method of any of claims 1-7.
10. A computer-readable medium, having stored thereon a computer program,
the program when executed by a processor implementing the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110394104.5A CN113094185A (en) | 2021-04-13 | 2021-04-13 | Method and device for detecting parallel data processing abnormity of database batch program |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110394104.5A CN113094185A (en) | 2021-04-13 | 2021-04-13 | Method and device for detecting parallel data processing abnormity of database batch program |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113094185A true CN113094185A (en) | 2021-07-09 |
Family
ID=76676624
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110394104.5A Pending CN113094185A (en) | 2021-04-13 | 2021-04-13 | Method and device for detecting parallel data processing abnormity of database batch program |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113094185A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1794285A (en) * | 2006-01-05 | 2006-06-28 | 中国工商银行股份有限公司 | Parallel batch processing method |
| CN107193543A (en) * | 2017-03-31 | 2017-09-22 | 北京奇艺世纪科技有限公司 | A kind of batch operation performs method and device |
| CN107273195A (en) * | 2017-05-24 | 2017-10-20 | 上海艾融软件股份有限公司 | A kind of batch processing method of big data, device and computer system |
| CN110263048A (en) * | 2019-05-05 | 2019-09-20 | 平安科技（深圳）有限公司 | High-volume data processing method, device, computer equipment and storage medium |
| CN111966687A (en) * | 2020-08-18 | 2020-11-20 | 中国银行股份有限公司 | Method and device for partitioning database table of mainframe DB2 |
| CN112363860A (en) * | 2020-11-11 | 2021-02-12 | 中国建设银行股份有限公司 | Batch processing operation abnormal interruption detection method and device |
-
2021
- 2021-04-13 CN CN202110394104.5A patent/CN113094185A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1794285A (en) * | 2006-01-05 | 2006-06-28 | 中国工商银行股份有限公司 | Parallel batch processing method |
| CN107193543A (en) * | 2017-03-31 | 2017-09-22 | 北京奇艺世纪科技有限公司 | A kind of batch operation performs method and device |
| CN107273195A (en) * | 2017-05-24 | 2017-10-20 | 上海艾融软件股份有限公司 | A kind of batch processing method of big data, device and computer system |
| CN110263048A (en) * | 2019-05-05 | 2019-09-20 | 平安科技（深圳）有限公司 | High-volume data processing method, device, computer equipment and storage medium |
| CN111966687A (en) * | 2020-08-18 | 2020-11-20 | 中国银行股份有限公司 | Method and device for partitioning database table of mainframe DB2 |
| CN112363860A (en) * | 2020-11-11 | 2021-02-12 | 中国建设银行股份有限公司 | Batch processing operation abnormal interruption detection method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020042804A1 (en) | Database query optimization method and apparatus and computer device | |
| US20130144893A1 (en) | Systems and Methods for Extraction of Concepts for Reuse-based Schema Matching | |
| CN110019298B (en) | Data processing method and device | |
| CN111931183A (en) | Open source software security vulnerability processing method and device | |
| CN116303286A (en) | Batch file processing method, device and system | |
| CN113287096A (en) | Test result classification for failed code verification | |
| CN111105307A (en) | Post-loan management method, system, equipment and readable medium | |
| CN113220666B (en) | Batch processing log playback method, device and system after database migration | |
| CN114090113B (en) | Method, device, equipment and storage medium for dynamically loading data source processing plug-in | |
| CN110990412A (en) | Data structure design method and device | |
| CN113094185A (en) | Method and device for detecting parallel data processing abnormity of database batch program | |
| CN116957602A (en) | Product process performance traceability method and device and electronic equipment | |
| CN113204541A (en) | Heterogeneous database data migration automatic correction method and device | |
| CN110795308A (en) | Server inspection method, device, equipment and storage medium | |
| CN113094243B (en) | Node performance detection method and device | |
| CN112199401B (en) | Data request processing method, device, server, system and storage medium | |
| EP4207006A1 (en) | Model generation program, model generation method, and model generation device | |
| CN115114612A (en) | Access processing method, device, electronic equipment and storage medium | |
| CN114357072A (en) | Method for detecting positioning accuracy of positioning control equipment for numerical control machining | |
| CN111159311B (en) | Method, device, equipment and storage medium for integrating data mapping | |
| CN113641702A (en) | Method and device for interactive processing with database client after statement audit | |
| CN115017518B (en) | Vulnerability correction information detection method and device | |
| CN110908976A (en) | Accumulation fund data acquisition method and system | |
| CN115202967A (en) | Batch program monitoring method and device | |
| CN113891378B (en) | Method and device for calculating signal coverage of base station and calculating equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |