CN115858372A - OLAP system-based batch data construction and automatic verification method and system - Google Patents
OLAP system-based batch data construction and automatic verification method and system Download PDFInfo
- Publication number
- CN115858372A CN115858372A CN202211633956.6A CN202211633956A CN115858372A CN 115858372 A CN115858372 A CN 115858372A CN 202211633956 A CN202211633956 A CN 202211633956A CN 115858372 A CN115858372 A CN 115858372A
- Authority
- CN
- China
- Prior art keywords
- data
- batch
- business
- olap
- data source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000012795 verification Methods 0.000 title claims abstract description 49
- 238000010276 construction Methods 0.000 title claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 96
- 238000012360 testing method Methods 0.000 claims abstract description 44
- 230000008569 process Effects 0.000 claims abstract description 19
- 230000005856 abnormality Effects 0.000 claims abstract description 15
- 238000012544 monitoring process Methods 0.000 claims description 17
- 238000009826 distribution Methods 0.000 claims description 8
- 230000001419 dependent effect Effects 0.000 claims description 2
- 238000004891 communication Methods 0.000 abstract description 5
- 230000008901 benefit Effects 0.000 abstract description 2
- 238000004904 shortening Methods 0.000 abstract 1
- 238000004458 analytical method Methods 0.000 description 15
- 238000002360 preparation method Methods 0.000 description 8
- 238000007405 data analysis Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012384 transportation and delivery Methods 0.000 description 5
- 230000006399 behavior Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000011835 investigation Methods 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 210000001503 joint Anatomy 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000012502 risk assessment Methods 0.000 description 2
- 238000012954 risk control Methods 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 230000000391 smoking effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000013024 troubleshooting Methods 0.000 description 2
- 238000013475 authorization Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000009960 carding Methods 0.000 description 1
- 238000005352 clarification Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013501 data transformation Methods 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000011068 loading method Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000006116 polymerization reaction Methods 0.000 description 1
- 238000000275 quality assurance Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000013522 software testing Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a batch data construction and automatic verification method and a system based on an OLAP system, comprising the following steps: calling a corresponding data source template file and a business data association relation according to the business type to generate a business association data source file, wherein the business association data source file is generated based on a predefined data type; judging whether the current environment meets a precondition, if so, sending a batch processing instruction to the OLAP system; in the process of executing batch processing test tasks, acquiring processing state information in real time, judging whether abnormality exists or not, generating alarm push information when the abnormality occurs, and automatically completing result user touch through a communication tool used by an enterprise; and acquiring batch processing results, and performing automatic verification on accuracy based on the comparison indexes. Compared with the prior art, the method has the advantages of shortening data construction time, configuring calibration rule parameters, automatically asserting result accuracy, automatically pushing results, improving efficiency and the like.
Description
Technical Field
The invention relates to the technical field of background data analysis type system testing, in particular to a batch data construction and automatic verification method and system based on an OLAP system.
Background
With the development of internet technology and modern financial industry, big data goes deep into various industries and daily production of residents, big data analysis type technology and products and the like are continuously developed, and the latitude of the big data is expanded and the quantity of the big data is continuously increased, so that great challenge is brought to the verification work of data processing correctness. Particularly, internet data analysis type companies or systems, banks, insurance, electronic commerce and government agencies are the main representatives.
The system for analyzing and predicting the account risk in the credit card loan of the bank needs to perform online analysis according to the comprehensive data such as customer consumption habits, income, living habits, repayment behaviors, credit investigation data, industry occupation, external data changes and the like, and the adopted data has the following characteristics: the data source path is multiple; the variety is various: such as DUMP files of oracle databases, XML format message data, TXT data, excel data and other non-typed text data; counting time latitude inconsistency: for example, annual data, monthly data, daily data, aperiodic data and the like, a certain cycle is required to be performed on the time latitude on the client latitude; different data sources and different service data have different associated logic conditions, such as customer numbers, credit card numbers, identity card numbers, service serial numbers and the like; there are many fields logically associated with test data volumes ranging from hundreds to thousands, tens of thousands to millions, tens of millions, etc. The characteristics lead to high difficulty in constructing the test data conforming to the characteristics of the wind control business rule in the loan and large workload for preparing the test data and verifying the correctness of the full-scale batch operation of the system.
The existing testing method is to compile a testing case and construct data according to static requirement specification documents, development program design documents, communication confirmation with a business party in the verification process and subjective understanding of testing and verifying personnel in combination with a software testing method and theory, so that the processing correctness of big data is verified, and data preparation mainly depends on manual testing or some simple code scripts to construct data. And verifying the correctness of the manually constructed and contrasted case one by a verifier. Due to the state change of the service data and the service mutual exclusion rule, the manually constructed data can not be reused generally, and the data needs to be repeatedly constructed along with the iterative modification of the version. The verification data can not be mapped with the verification scene, and can not be deposited. The existing data structure mainly meets the local functions of a certain service module or certain service points, and cannot be used in the whole service link of the loan wind control system through one-time data transformation, so that the test efficiency is low, and automatic data structure and automatic assertion cannot be achieved.
With the gradual agility of the software industry, the original data construction and verification method cannot meet the requirement of agile rapid iterative delivery of MVP (minimum Viable Product).
The existing OLAP type system test method comprises the following steps:
and through analyzing the business requirement document and confirming the business rule by the user, combining different data types and data volumes, and manually or by adopting a simple script or a small self-research tool to complete the data construction of the current version.
When the project construction period is short, the data verification work is completed in the project milestones through face-to-face guidance and experience teaching of personnel, and testing manpower and driving are increased, so that the delivery quality is guaranteed.
And increasing testing environment resources, decoupling verification work of the analysis link with the data dependency relationship, and increasing the parallelism of the verification work.
The existing verification method for the business data preparation depends on the degree of familiarity of a verifier with the business, the reusability of data is low, long-term automatic preparation cannot be carried out, and under the condition of large data processing batch, a large amount of manpower is needed for verifying the correctness of batch operation, data extraction, conversion and loading of reports and files. The associated disadvantages are as follows:
1. the data construction time is long, the manpower waste of rework or problem troubleshooting caused by data errors can occur, and when the historical data are associated after the system is iteratively developed, the preparation of linked regression test data is time-consuming.
2. Test data assets with real business association logic cannot be precipitated for business products or system levels.
3. The data scene is not rich in structure, which may cause partial distortion of the data processing program and affect the business decision.
4. The state inspection before the batch processing of the system and the state inspection in the system execution process mainly depend on manual work, and an effective unattended automatic solution is lost.
5. The flow regression test of the system full-batch operation introduced by the containerization transformation, the security vulnerability repair, the spring cloud technical component upgrading and other bottom layer transformations in the software engineering technology is short in online delivery cycle and needs a large amount of manpower to cover.
Disclosure of Invention
The present invention is directed to overcome the above-mentioned drawbacks of the prior art, and provides a batch data constructing and automatic verification method and system based on an OLAP (on-line transaction processing) system, which can automatically shorten the data constructing time and improve the overall verification efficiency.
The purpose of the invention can be realized by the following technical scheme:
a batch data construction and automatic verification method based on an OLAP system comprises the following steps:
1) Calling a corresponding data source template file and a business data association relation according to the business type to generate a business association data source file, wherein the business association data source file is generated based on a predefined data type;
2) Judging whether the current environment meets a precondition, if so, sending a batch processing instruction to an OLAP system, executing a batch processing test task by the OLAP system based on the service relevance data source file, executing the step 3), and if not, executing the step 2 again after waiting for a set time;
3) Acquiring processing state information in real time in the process of executing a batch processing test task by the OLAP system, judging whether an abnormality exists or not, and generating alarm push information when the abnormality occurs;
4) Configuring a comparison index of a system processing state and a batch processing result based on business requirement processing logic and system expectation;
5) And acquiring a batch processing result of the OLAP system executing a batch processing test task, and automatically verifying the accuracy of the batch processing result based on the comparison index.
Furthermore, each data source template file corresponds to a service table or a file, and the data source template file can be mapped and maintained.
Further, each of the data source template files includes field data values of configurable generation rules.
Further, the generation rule includes one or more of a data type, a length, an enumerated value.
Further, when the service relevance data source file is generated, the generation is performed based on the pre-configured data generation frequency and number.
Further, the precondition comprises that batch operation in operation is not stored, the last test task is completely finished, the redis information is cleaned and initialized, the data of the associated configuration table is cleaned, and the service dependent on the batch program is normally initialized.
Further, the processing state information includes one or more of a specific task currently running to, a core basic data processing result, a derivative variable processing job, a model variable processing job, a data handling conclusion, and a batch task execution result.
Further, the determining whether there is an abnormality specifically includes:
an exception is determined to exist when the system program task makes an error report or exceeds an expected time.
Further, the comparison index includes one or more of a basic data quantity comparison index, a disposal type distribution comparison index, a disposal type probability value distribution comparison index, a business decision reference value, and a business report sequence index.
The invention also provides a batch data construction and automatic verification system based on the OLAP system, which comprises:
the data construction module is used for calling the corresponding data source template file and the business data association relation according to the business type to generate a business association data source file, and the business association data source file is generated based on the predefined data type;
the batch running instruction generating module is used for judging whether the current environment meets a precondition or not, if so, a batch processing instruction is sent to an OLAP (on-line analytical processing) system, the OLAP system executes a batch processing test task based on the service relevance data source file, and if not, the batch running instruction generating module is restarted after waiting for a set time;
the monitoring module is used for acquiring processing state information in real time in the process of executing batch processing test tasks by the OLAP system, judging whether an abnormality exists or not and generating alarm push information when the abnormality occurs;
and the automatic verification module is used for acquiring a batch processing result of the OLAP system executing a batch processing test task, verifying the accuracy of the batch processing result based on a preset comparison index, and processing logic and system expected configuration based on the service requirement.
Compared with the prior art, the invention has the following beneficial effects:
1. the invention can realize the automatic verification of the business scene data construction, program operation monitoring, system batch operation state and business data processing correctness assertion of the target system, save the time consumption of iterative version repeated data construction, dependence manual assertion, data processing state manual monitoring and other work, shorten the data construction time, can be applied to a smoking test stage in the software research and development process, a system test and a regression test stage for modifying correlation influence verification, can realize local iterative adjustment and optimization along with the change of a data source and program operation, and improve the efficiency and delivery quality of OLAP data analysis type system verification.
2. The invention can finish the automatic assertion of the processing correctness of the OLAP characteristic business system and the monitoring of the system running state on the premise of not invading business codes, and forms quick butt joint by combining with the API interface of the existing communication tool to finish the timely notification of the verification result.
3. On the basis of the file type, the field type, the length and the like of a data source on which the original manual carding big data analysis depends, the method can be conveniently adopted to automatically generate the data file, and the data can be generated randomly, by a specified rule or fixedly.
4. The method has the advantages that the large-data batch programs are automatically monitored, time consumed by long-term manual waiting and observation is avoided, the running condition of the system program can be monitored under the condition that service codes are not invaded, the program processing result and the log information of key processing steps are automatically notified, and the error reporting program and reasons are assisted to be positioned.
5. Through the code which keeps consistent with the continuous updating and the business system iteration, the dynamic organization process assets can be formed, project members can be familiar with the business modules and the verification points of different modules, and the quality assurance risks of incomplete system business verification methods and test ranges introduced by the deputy of core personnel can be reduced to a certain extent.
6. The method can be applied to a wind control business system in credit card credit, has stronger pertinence and professiveness to automatic construction of business data and business logic mapping relation of wind control in credit card industry credit, can be carried and used based on the existing environmental management scheme components, such as a continuous delivery platform, a log platform and the like, and assists software developers to eliminate relevant environmental interference factors when troubleshooting problems, thereby improving software research and development efficiency.
Drawings
FIG. 1 is a schematic diagram of an implementation of the present invention;
FIG. 2 is a principal flow diagram of the present invention;
FIG. 3 is a schematic diagram of a process for applying the present invention to a system for analyzing the wind control in credit card credit;
fig. 4 is a schematic diagram of the implementation procedure of the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation manner and a specific operation process are given, but the scope of the present invention is not limited to the following embodiments.
The invention provides a batch data construction and automatic verification method based on an OLAP system, which comprises the following steps as shown in figure 1:
s1, establishing a data source file structure dictionary, and maintaining a corresponding data source template file in the dictionary according to the service type for the generation and calling of a subsequent specific service data instance;
s2, establishing basic information logic association data to obtain a business data association relation;
s3, generating a service relevance data source file according to the step S1 and the step S2, wherein the service relevance data source file is generated based on a predefined data type;
s4, transmitting the service relevance data source file generated by the device to a corresponding server path;
s5, checking the precondition of the batch running environment: judging whether the current environment meets the precondition, if so, executing the step S6, otherwise, executing the step S5 again after waiting for the set time;
s6, sending a batch processing instruction to an OLAP system, wherein the OLAP system executes a batch processing test task based on the service relevance data source file, and a system program runs service data analysis and processing;
s7, monitoring task nodes: acquiring processing state information in real time in the process of executing a batch processing test task by the OLAP system, judging whether an abnormality exists or not, and generating alarm push information when the abnormality occurs;
s8, automatically judging the correctness based on the data result: obtaining a batch processing result of the OLAP system executing a batch processing test task, and verifying the accuracy of the batch processing result based on a preset comparison index, wherein the comparison index is based on business requirement processing logic and system expected configuration, namely the expectation of automatic assertion.
In step S2, the service association field attribute association relationship needs to be configured, and complete simulation data of the service chain dependency relationship can be established by parameterizing the production frequency and number of configuration files based on the data association between the real service and the preamble. Such as the maintenance of the corresponding relationship of consumption behavior, bill, overdue and client by the card number.
In step S3, the data types may be defined as txt, dump, xls, xml, etc. The data warehouse in the existing big data technology can process data of different data types and different formats, and the invention predefines the data types and can conveniently and automatically generate the data according to the specified types.
Each data source template file corresponds to a service table or a file. Each of the data source template files includes field data for configurable generation rules, which may be predefined, including definitions for one or more of data type, length, enumerated values. For example, identification card type 01, length 18 bits, gender: m or F.
The data of different industries has typical industry label characteristics, such as an order number, a logistics order number, payment time and the like of electronic commerce, five elements of a client of a bank credit card, a real name system, a cardholder relation, an account type, a person credit investigation report and the like. The invention can simulate and construct the high data quality standard of the service scene and the real service meaning in the credit card through the configuration of the generation rule.
In step S3, when the service relevance data source file is generated, the generation is performed based on the pre-configured data generation frequency and number. Data generation frequency such as year/month/day/fixed date, etc.
In step S5, the preconditions include that there is no batch job in operation, the last test task is complete, cleaning and initialization are related, redis information and associated table data are cleaned or initialized, and the services on which the batch program depends are normal.
In step S7, the task node monitoring mainly includes monitoring of the task list, whether to run, and the running state result, and the specific processing state information includes one or more of the currently running specific task, the core basic data processing result, the derivative variable processing operation, the model variable processing operation, the data processing conclusion, and the batch task execution result.
In step S7, when an abnormal error is reported or the expected time is exceeded, it is determined that there is an abnormality, and then an alarm push message is generated, where the alarm push message can implement timely notification of batch procedures and verification results by interfacing with an open API of an enterprise mailbox or an enterprise WeChat or nailer communication tool.
In step S8, the comparison index includes one or more of a basic data amount comparison index, a treatment type distribution comparison index, a treatment type probability value distribution comparison index, a business decision reference value, and a business report order index. The step can be used for finishing the correctness automatic assertion of the business data processing result according to the constructed data, the data processing result table of the OLAP and the customized rule.
The main flow is shown in fig. 2.
The method can realize the automatic verification of the construction of the business scene data, the program operation monitoring, the system batch running state and the business data processing correctness assertion of the target system by realizing the one-time clarification of the architecture design of the business system and the system of the OLAP online analysis type system, saves the time consumption of repeated data construction of an iterative version, dependence on manual assertion, manual data processing state monitoring and other work, can be applied to a smoking test stage in a software research and development process, a system test and a regression test stage for modifying correlation influence verification, and can realize local iterative adjustment optimization along with the change of a data source and program operation. The invention can complete the automatic assertion of the processing correctness of the OLAP characteristic business system and the monitoring of the running state of the system on the premise of not invading business codes, and combines the API interface of the existing communication tool to form rapid butt joint, and complete the complete automatic solution method and system such as the timely notification of the verification result.
The above functions, if implemented in the form of software functional units and sold or used as a separate product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The invention also provides a batch data construction and automatic verification system based on the OLAP system, which comprises a data construction module, a batch run instruction generation module, a monitoring module and an automatic verification module, wherein the data construction module is used for calling a corresponding data source template file and a business data association relation according to the business type to generate a business association data source file, and the business association data source file is generated based on a predefined data type; the batch running instruction generating module is used for judging whether the current environment meets a precondition or not, if so, a batch processing instruction is sent to an OLAP system, the OLAP system executes a batch processing test task based on the service relevance data source file, and if not, the batch running instruction generating module is restarted after waiting for a set time; the monitoring module is used for acquiring processing state information in real time in the process of executing batch processing test tasks by the OLAP system, judging whether an abnormality exists or not and generating alarm push information when the abnormality occurs; the automatic verification module is used for acquiring a batch processing result of the OLAP system executing a batch processing test task, and performing automatic verification on the accuracy of the batch processing result based on a preset comparison index.
The working process of the batch data construction and automatic verification system based on the OLAP system is shown in fig. 1, wherein step S6 is executed by the OLAP system, and the rest steps are executed by the data construction and automatic verification system.
Examples
In this embodiment, the method is applied to a wind control analysis system for credit card credit, so as to implement batch data construction and automatic verification of the wind control analysis system for credit card credit, and further implement actual service testing work.
As shown in fig. 3, the implementation steps and details of this embodiment are as follows:
s1: and establishing necessary data source template files in the bank credit based on the logical relation of the credit-related control business characteristics, wherein each data source template corresponds to a business table or file. The number of the field data under each data source template can be increased or decreased based on actual business requirements, and the generation rules can be configured. Such as: the customer billing date can be fixed for a certain day or can be configured with the number of days ahead of the current system date. And establishing a data source full quantity list which is depended by the loan risk control service and comprises basic information of a client, a card layer, an account layer, bills, credit investigation, consumption, repayment behavior, earnings, public security network information, industrial and commercial information, intermediate variables and derivative variables in the client loan processed by a business department strategy model and the like.
S2: the associated fields of various service data in the loan insurance system are mainly as follows: card number, mobile phone number, customer number, account number and the like, all data source tables can be connected in series through the associated condition fields of the scheme, so that the associated relation of various credit authorization risks in reasonable credit judgment latitudes is established among the data sources, and the system can be used for subsequent processing, conversion, analysis and decision-making result output.
S3: and combining the data source template list of the S1 and the relation of the business core associated fields in the credit of the S2, wherein the core in the step is used for generating data of different business logic layers.
And S4, generating a corresponding predefined data type file based on the content of the S3, and generating an entity data file in a GBK or UTF-8 coding format. For example, a special list csv, a new member list txt, a credit card information table dump, a client bill issue list dump, a bill overdue client list dump, client personal credit data detail, xml, text files without suffix names and the like form various image basic data of clients in credit.
S5: the proposal calls a preposed condition checking module before the main program of the loan wind control system. Checking whether the main program is called up: the primary checkpoints are: whether the current OLAP system has batch operation in operation, whether the previous account date is completely finished, relevant redis information is cleared and initialized, associated table data is cleared, whether the service depended by the batch program is normal and the like.
S6: and calling a batch program of the air control system in credit to enable the system to carry out the whole normal processing operation process until the system analysis task is finished or the system execution task is abnormally interrupted and the like, wherein the system is abnormal in anticipation.
S7: according to the scheme, the task list monitoring module of the loan wind control system mainly monitors the batch operation state, the time consumption and the specific task list. The main tasks are as follows: specific tasks currently running, core basic data processing results, derivative variable processing operations, model variable processing operations, risk disposition conclusions, batch task execution results and the like, and necessary batch tasks of the wind control service system in credit card credit. The monitoring device can be adjusted or increased or decreased according to actual needs. If abnormal error reporting is found in the monitoring process or the expected time is exceeded, triggering an alarm mechanism of the scheme to carry out enterprise WeChat and mail notification, and adopting manual intervention measures according to specific error reporting.
S8: the automatic comparison module of the scheme presets rules according to a processing module, a business analysis result and a report statistic item of a pneumatic control online analysis type system in credit card credit, after the execution of a main program of the pneumatic control system in credit is finished, the final result of the data processed by the system is used, the data of S1 and S2 are collected, automatic assertion of the results is carried out on preset comparison indexes according to bill days by using categories such as a polymerization analysis method, a business expert experience numerical value and a system standard check item, and main indexes are basic data quantity comparison, disposal type distribution comparison, disposal type probability value distribution comparison, a business decision reference value and a business report sequence index.
The main data source main body of the risk analysis system in the loan is the basic data which is filled by the client of the card issuer, the card used by the client or the authorized and collected by the client, such as the behavior data of card, account, consumption, bill, repayment, installments and the like, and the related data of external third parties such as personal credit, industrial and commercial network, student and credit network data and the like are combined to carry out online analysis on the bill or other time latitudes, and various comprehensive system judgments of the risk client in the loan are carried out.
The main decision output results of the bank credit card are as follows: the credit line value, the line adjustment, the risk level and the processing suggestion of the risk level, the independent treatment of important clients in the client hierarchy, the early overdue risk and the retrieval treatment of the initial overdue clients, the identification and treatment suggestion of external high common debt clients and the like.
After complete data preparation, processing and result correctness verification of one account book day in the system are finished, batch execution and automatic preparation of multiple account book days can be realized, and continuous batch execution and verification of the risk control system in credit card loan, such as monthly continuous data generation, data processing and program execution result analysis, are realized. When the OLAP system architecture is not greatly changed or reconstructed, the device and the method can continuously serve the full-flow quality guarantee work of the credit card online analysis loan electronic control system, the full-amount and subsequent version associated data preparation work of the analysis type system is prepared from manual reduction to automatic generation, all task work of the system is monitored in a listing manner, and the task result and report processing correctness of the important core of the system are automatically declared along with the iterative development of a service system. An automated method for constructing and verifying batch data based on an OLAP system is a complete solution method and a complete solution device.
The method is based on Python technology and an open source third-party library, is innovatively developed according to the characteristics of credit card business, can complete automatic construction of various data sources for credit wind control, automatic monitoring of data processing operation programs, processing of structural data quantity, automatic judgment of correctness and mail pushing of final structures to appointed persons of the trunk.
Fig. 4 is a schematic diagram of a program for implementing the method, where a, B, C, and D are initial preparation programs, including database operations, file generation file writing, auxiliary data file generation, and data source generation result monitoring programs. After the Main program of Main is called, the operation steps of S1-S7 are controlled to generate data, trigger and execute a credit card risk analysis type system in credit, monitor the state and service of the system execution process and finally automatically assert the correctness of the generated batch result. After the billing day is completed, relevant data checking and system subtask checking are performed to check whether the next polling is performed, i.e., the next billing day.
The foregoing detailed description of the preferred embodiments of the invention has been presented. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions available to those skilled in the art through logic analysis, reasoning and limited experiments based on the prior art according to the concept of the present invention should be within the scope of protection defined by the claims.
Claims (10)
1. A batch data construction and automatic verification method based on an OLAP system is characterized by comprising the following steps:
1) Calling a corresponding data source template file and a business data association relation according to the business type to generate a business association data source file, wherein the business association data source file is generated based on a predefined data type;
2) Judging whether the current environment meets a precondition, if so, sending a batch processing instruction to an OLAP system, executing a batch processing test task by the OLAP system based on the service relevance data source file, executing the step 3), and if not, executing the step 2 again after waiting for a set time;
3) Acquiring processing state information in real time in the process of executing a batch processing test task by the OLAP system, judging whether an abnormality exists or not, and generating alarm push information when the abnormality occurs;
4) Configuring a comparison index of a system processing state and a batch processing result based on business requirement processing logic and system expectation;
5) And acquiring a batch processing result of the OLAP system executing a batch processing test task, and automatically verifying the accuracy of the batch processing result based on the comparison index.
2. The OLAP system based batch data construction and automatic verification method according to claim 1, wherein each data source template file corresponds to a business table or a file, and the data source template file can be mapped and maintained.
3. The OLAP system-based batch data construction and automated verification method of claim 1, wherein each of the data source template files includes field data values of configurable generation rules.
4. The OLAP system based batch data construction and automated verification method of claim 3, wherein the generation rules include one or more of data type, length, enumerated values.
5. The OLAP system-based batch data construction and automated verification method of claim 1, wherein generating the business association data source file is performed based on a preconfigured data generation frequency and quantity.
6. The OLAP system-based batch data construction and automated verification method according to claim 1, wherein the preconditions include absence of a running batch job, complete completion of a last test task, cleanup and initialization redis information, associated configuration table data cleanup, and normal initialization of batch program dependent services.
7. The OLAP system-based batch data construction and automated verification method of claim 1, wherein the process state information includes one or more of a specific task currently running to, a core base data processing result, a derived variable processing job, a model variable processing job, a data handling conclusion, a batch task execution result.
8. The OLAP system-based batch data construction and automatic verification method according to claim 1, wherein the determining whether there is an abnormality specifically comprises:
an exception is determined to exist when a system program task makes an error or exceeds an expected time.
9. The OLAP system-based batch data construction and automatic verification method according to claim 1, wherein the comparison index comprises one or more of a basic data amount comparison index, a treatment type distribution comparison index, a treatment type probability value distribution comparison index, a business decision reference value, and a business report order index.
10. An OLAP system based batch data construction and automated verification system, comprising:
the data construction module is used for calling the corresponding data source template file and the business data association relation according to the business type to generate a business association data source file, and the business association data source file is generated based on the predefined data type;
the batch running instruction generating module is used for judging whether the current environment meets a precondition or not, if so, a batch processing instruction is sent to an OLAP (on-line analytical processing) system, the OLAP system executes a batch processing test task based on the service relevance data source file, and if not, the batch running instruction generating module is restarted after waiting for a set time;
the monitoring module is used for acquiring processing state information in real time in the process of executing batch processing test tasks by the OLAP system, judging whether abnormity exists or not and generating alarm push information when the abnormity appears;
and the automatic verification module is used for acquiring a batch processing result of the OLAP system executing a batch processing test task, verifying the accuracy of the batch processing result based on a preset comparison index, and processing logic and system expected configuration based on the service requirement.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211633956.6A CN115858372B (en) | 2022-12-19 | 2022-12-19 | Batch data construction and automatic verification method and system based on OLAP system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211633956.6A CN115858372B (en) | 2022-12-19 | 2022-12-19 | Batch data construction and automatic verification method and system based on OLAP system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115858372A true CN115858372A (en) | 2023-03-28 |
| CN115858372B CN115858372B (en) | 2024-04-26 |
Family
ID=85674142
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211633956.6A Active CN115858372B (en) | 2022-12-19 | 2022-12-19 | Batch data construction and automatic verification method and system based on OLAP system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115858372B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090248698A1 (en) * | 2008-03-31 | 2009-10-01 | Stephan Rehmann | Managing Consistent Interfaces for Internal Service Request Business Objects Across Heterogeneous Systems |
| CN105786695A (en) * | 2014-12-26 | 2016-07-20 | 北大医疗信息技术有限公司 | Data test method and system |
| CN107169862A (en) * | 2017-05-25 | 2017-09-15 | 中国建设银行股份有限公司辽宁省分行 | A kind of bad customer deposit automatic tracking system of bank |
| US20180165180A1 (en) * | 2016-12-14 | 2018-06-14 | Bank Of America Corporation | Batch File Creation Service |
| CN110413483A (en) * | 2019-07-30 | 2019-11-05 | 中国工商银行股份有限公司 | Monitoring method, device, electronic equipment and the storage medium of batch jobs data |
| CN111858065A (en) * | 2020-07-28 | 2020-10-30 | 中国平安财产保险股份有限公司 | Data processing method, device, storage medium and device |
| WO2020233330A1 (en) * | 2019-05-20 | 2020-11-26 | 深圳壹账通智能科技有限公司 | Batch testing method, apparatus, and computer-readable storage medium |
| CN112256978A (en) * | 2020-12-22 | 2021-01-22 | 畅捷通信息技术股份有限公司 | Data processing method, device and medium based on data model |
| CN112835932A (en) * | 2021-01-11 | 2021-05-25 | 中国邮政储蓄银行股份有限公司 | Batch processing method and device of service table and nonvolatile storage medium |
| CN112948265A (en) * | 2021-03-30 | 2021-06-11 | 中信银行股份有限公司 | Batch processing automatic verification method and device |
| CN113064866A (en) * | 2021-03-02 | 2021-07-02 | 深圳供电局有限公司 | Power business data integration system |
| CN113360393A (en) * | 2021-06-25 | 2021-09-07 | 武汉众邦银行股份有限公司 | Continuous verification method and device based on production environment flow monitoring |
-
2022
- 2022-12-19 CN CN202211633956.6A patent/CN115858372B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090248698A1 (en) * | 2008-03-31 | 2009-10-01 | Stephan Rehmann | Managing Consistent Interfaces for Internal Service Request Business Objects Across Heterogeneous Systems |
| CN105786695A (en) * | 2014-12-26 | 2016-07-20 | 北大医疗信息技术有限公司 | Data test method and system |
| US20180165180A1 (en) * | 2016-12-14 | 2018-06-14 | Bank Of America Corporation | Batch File Creation Service |
| CN107169862A (en) * | 2017-05-25 | 2017-09-15 | 中国建设银行股份有限公司辽宁省分行 | A kind of bad customer deposit automatic tracking system of bank |
| WO2020233330A1 (en) * | 2019-05-20 | 2020-11-26 | 深圳壹账通智能科技有限公司 | Batch testing method, apparatus, and computer-readable storage medium |
| CN110413483A (en) * | 2019-07-30 | 2019-11-05 | 中国工商银行股份有限公司 | Monitoring method, device, electronic equipment and the storage medium of batch jobs data |
| CN111858065A (en) * | 2020-07-28 | 2020-10-30 | 中国平安财产保险股份有限公司 | Data processing method, device, storage medium and device |
| CN112256978A (en) * | 2020-12-22 | 2021-01-22 | 畅捷通信息技术股份有限公司 | Data processing method, device and medium based on data model |
| CN112835932A (en) * | 2021-01-11 | 2021-05-25 | 中国邮政储蓄银行股份有限公司 | Batch processing method and device of service table and nonvolatile storage medium |
| CN113064866A (en) * | 2021-03-02 | 2021-07-02 | 深圳供电局有限公司 | Power business data integration system |
| CN112948265A (en) * | 2021-03-30 | 2021-06-11 | 中信银行股份有限公司 | Batch processing automatic verification method and device |
| CN113360393A (en) * | 2021-06-25 | 2021-09-07 | 武汉众邦银行股份有限公司 | Continuous verification method and device based on production environment flow monitoring |
Non-Patent Citations (3)
| Title |
|---|
| ZHIFANG LI等: "XeFlow: Streamlining Inter-Processor Pipeline Execution for the Discrete CPU-GPU Platform", IEEE TRANSACTIONS ON COMPUTERS, 31 January 2020 (2020-01-31) * |
| 田圳;李丛林;王伟力;徐元孚;曹旌;李赛峰;: "基于多源监控数据的设备状态感知方法", 电力系统及其自动化学报, no. 07, 31 December 2020 (2020-12-31) * |
| 赵曦;: "云计算架构在银行批处理流程优化中的应用研究", 软件导刊, no. 10, 25 October 2013 (2013-10-25) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115858372B (en) | 2024-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110020660B (en) | Integrity assessment of unstructured processes using Artificial Intelligence (AI) techniques | |
| US11580475B2 (en) | Utilizing artificial intelligence to predict risk and compliance actionable insights, predict remediation incidents, and accelerate a remediation process | |
| US8346683B2 (en) | System, program, and method for representation, utilization, and maintenance of regulatory knowledge | |
| US20050080821A1 (en) | System and method for managing collections accounts | |
| US20120150820A1 (en) | System and method for testing data at a data warehouse | |
| US11250513B2 (en) | Computer implemented system for generating assurance related planning process and documents for an entity and method thereof | |
| US8533109B2 (en) | Performance of control processes and management of risk information | |
| WO2022155740A1 (en) | Systems and methods for outlier detection of transactions | |
| CN112000656A (en) | Intelligent data cleaning method and device based on metadata | |
| Milani et al. | Decomposition driven consolidation of process models | |
| US11030565B1 (en) | System and method for audit report generation from structured data | |
| CN115858372B (en) | Batch data construction and automatic verification method and system based on OLAP system | |
| Offutt et al. | An industrial study of applying input space partitioning to test financial calculation engines | |
| English | Total quality data management (TQdM) | |
| Schreck et al. | Augmenting software project managers with predictions from machine learning | |
| Chu et al. | Best practices for managing predictive models in a production environment | |
| Yao | RPA Technology Enables Highly Automated Development of Corporate Financial Accounting Processes | |
| Rao et al. | Data Validation for Eligibility Criteria and Death Benefit Claims for Insurance Policies | |
| Nayak et al. | Benefits of Robotic Process Automation (RPA): Today and Tomorrow of the Manufacturing Industries | |
| Nikiforova et al. | Data quality model-based testing of information systems: Two-level testing of the insurance system | |
| CN117314665A (en) | Financial risk prediction method based on accounting | |
| da Rocha Seabra | Issue Creator | |
| Seabra | Issue Creator | |
| Kneuper et al. | Selected Current Trends in Software Processes | |
| Sooryanarayana Shetty et al. | Relational database system design for FMECA program creation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |