TWI459225B - Data Exploration Model Automated Maintenance System and Method - Google Patents

Description

Data exploration model automatic transportation system and method

The invention is a data exploration model system and method, in particular to a data exploration model automatic transportation system and method.

In the conventional data exploration model system, after the user sets the relevant accuracy judgment parameters, the system confirms whether the data to be used is ready, and starts the process, and performs the accuracy verification calculation and determination on the model after the required data is generated. Whether to start the model reconstruction mechanism, or use the data of different time points to calculate the model accuracy according to the user's setting, compare the model itself or compare with other models, and select the appropriate model to replace according to the setting. Update, although the above two conventional systems can effectively perform accuracy verification calculation, module reconstruction, replacement update and visualization, but the implementation lacks a data exploration classification, group model establishment and application in its system process steps. Important processes such as data prediction and scoring, data clustering, model accuracy monitoring, and data calculation of important variables are cascaded and integrated.

It can be seen that the above-mentioned data exploration model system still has complicated processes and many inconveniences in actual implementation. It is not a good designer and needs to be improved.

To this end, the Applicant has in view of the shortcomings of the above-mentioned conventional data exploration model system, and upholds the spirit of research and innovation, and strives for excellence, using its professional vision and professional knowledge to research the automatic data transmission system of the data exploration model of this creation and method.

The invention provides a data exploration model automatic transportation and transportation system, which comprises a user operation interface, a data exploration automatic maintenance engine, a data access module, a model metadata, a result set, and a model. Type monitoring module, model reconstruction module, data scoring module and individualized important variable module.

The user operation interface receives the parameter data input by the user number; wherein the data exploration automatic maintenance engine stores the parameter data received by the user operation interface, and retrieves, transmits and receives the parameter data for job assignment and control. The process execution status, error status determination and error handling; wherein, the data access module accesses the parameter data of different storage properties, performs the required parameter data acquisition and calculation results, and the data exploration automation maintenance engine The metadata is written; wherein, the model metadata stores the metadata of the data mining automation engine; wherein, the result set stores the data calculation result obtained by the data mining automatic transportation engine; wherein, the plurality of data sources are used The information provided by the data can be retrieved through the data access module; wherein the model monitoring module will calculate the accuracy of the model when the parameter data assigned by the data mining automation engine is calculated, and the model is accurate. Whether the degree is below the set threshold; where the model reconstruction module will be composed of data The parameter data assigned by the survey automation maintenance engine is trained by different algorithms. The several models obtained are compared with the test data to select the model with the best accuracy. Among them, the data scoring module will be automated by data mining. The parameter data assigned by the engine is calculated by the model rules, and the parameter data to be scored is calculated to obtain the score; and the individual variable module is selected, and the parameter data assigned by the data mining automation engine is transmitted through the internal information of the model, and the ordering and proportion are utilized. Formula, stratification, and importance calculation for the field attributes of each parameter data.

The above-mentioned data exploration and automation maintenance engine further includes: a process parameter data area, a work parameter data area, a scheduler, a process control module, and a job assignment module.

The process parameter data area stores the parameter data obtained by the user operation interface, wherein the working parameter data area stores the parameter data obtained by the user operation interface. Position, model accuracy threshold, data date; wherein, the scheduler is connected to the schedule of the parameter data obtained by the user operation interface, and the process is started when the time point is reached; wherein the process control module is composed of the scheduler After starting, and the process parameter data area, the working parameter data area and the process status record area, dynamic process scheduling, parameter preparation and marking work step coordinating; wherein the job assignment module accepts the work entered by the process control module Obtain the information required for the work of the model metadata, result set or data source through the data access module, and assign the job to the model monitoring module, model reconstruction module, data scoring module or individualized important variable module And the process status of the model monitoring module, the model reconstruction module, the data scoring module or the individualized important variable module is recorded in the process status record area; wherein, the process status record area, the status of the record process execution is taken as The judgment basis of the process; and the retrospective reorganization module, accepting the error status record of the job assignment module, and performing the retrospective reorganization plan After the arrangement, will enter the process parameter data area.

The above process control module further comprises: a process decision unit, a dynamic process worksheet, a corresponding work parameter, a message notification unit, and a work item coordinating unit. The process decision unit receives the data from the process parameter data, the work parameter data, the process status record and the process rule base, and performs workflow scheduling, parameter sorting and work step marking; wherein the process rule base includes the correct process. Logic rules provide the basis for the process sequence arrangement; wherein, the dynamic process worksheet is a worksheet generated by the process decision unit for workflow scheduling, parameter sorting, and work step marking; wherein, the corresponding working parameter is a process decision unit The parameters required for the corresponding work items generated by the workflow arrangement, parameter sorting, and work step marking; wherein, the message notification unit is a notification message generated by the process decision unit after the workflow arrangement, parameter sorting, and work step marking, Inform the user of the process status; and the work item coordinating unit, coordinating according to the dynamic process worksheet and corresponding working parameters The work of the next step and its required parameters are assigned to the job assignment module.

The above job assignment module further comprises: a data preparation and checking unit, a flow state processing unit, a system load detecting unit and a job starting unit.

The data preparation and checking unit checks the work and parameters to be executed accepted by the process control module; wherein, the process state processing unit receives the inspection result of the data preparation and inspection unit, and writes the inspection result into the process. In the status record area, wherein the system load detection unit receives the check result of the data preparation and check unit, and checks the load status of the back end host according to the check result, determines the work host; and the job start unit, according to the system load detection The decision of the unit is to transfer the work and required information to the module to be executed.

The backtracking reforming module further includes: a data storage architecture association table, a parameter parsing unit, a data backtracking planning unit, a backtracking instruction generating unit, an archive data processing module, a breakpoint position resolution unit, a flow reply instruction generating unit, and Process status backhaul processing module. The data storage architecture association table records the records and data locations and corresponding relationships generated by different users' operating systems; wherein the parameter parsing unit receives the parameters of the job assignment module for parsing, and determines whether to execute the error. Processing the plan and determining the error handling method; wherein, the data backtracking planning unit receives the data backtracking operation determined by the parameter analyzing unit, and obtains the user operating system record file and the data position and the corresponding relationship thereof by the data storage architecture association table To define the scope of the data backtracking and the associated file data, the scope of the data to be replenished, generate the data replenishment plan and transmit the plan to the parameter process data area; wherein the backtracking instruction generating unit is based on the data backtracking The range defined by the drawing unit generates a backtracking instruction, and the instruction is issued through the data access module; wherein the file data processing module executes the file backtracking specified by the backtracking instruction generating unit; wherein the interrupt point location analyzing unit receives the parameter Parsing list The data judged by the yuan is returned to the operation, and the scope and content of the reply data processing are defined; wherein the process reply instruction generating unit generates a reply command according to the range and content defined by the break point location analyzing unit, and performs a reply command through the data access module. The instruction is released; and the process state return processing module receives the process state processed by the data access module and returns to the job assignment module.

The invention provides a method for data mining model automatic transportation and transportation system, the steps thereof comprise: determining, by each time, the operation time point of the operation time, if the operation is performed in a scheduling manner, the operation is started according to the scheduling time, in time When the time is up, start the job to enter the process decision; if it is to execute the job immediately, go directly to the process decision; in the process decision, arrange the scope, sequence and preparation parameters of the job, and mark the work steps, then submit to the work project coordinator Carry out the integration of parameters and steps; judge whether all the arranged work items are completed, and if so, the operation ends; if otherwise, the data preparation and inspection will be carried out, and the data obtained from the data source will be initially checked and returned, and then the process will be carried out. State judgment; if the data check finds an error, the secondary job will determine that it cannot be automatically run, then enter the notification message and end the job; if the data check does not find an error, enter the job in the process running state. According to the load status of the backend host, after the assignment is made, according to the secondary work item Differently, perform model monitoring, model reconstruction, data scoring, or individualized important variable calculations, respectively, after receiving the job, receive the process status in the process status of the process return, and then return to the process decision to continue the next step; if the data is checked The core does not find an error, and when the status is backtracking and reorganization, it enters the execution of the retrospective reorganization process, processes the backend data, and then returns to the process decision to continue the next step.

The above process decision includes the following steps: when the operation time point is reached, after the scheduling time is completed, after the execution of the retrospective reorganization process is completed, the process status of the process is returned, the process status is received, and the data is written to After the process status is recorded, it is controlled by a driving process control module. After the completion of the arrangement, the process proceeds to the process decision; after reading the process status record, the process status record is taken out, and the process flow is determined as the basis for determining whether the process should be modified; if the process status record marked as incorrect is read When the process is modified, the process parameters are read, and the work item to be executed in the process is obtained. After the work process worksheet is generated, after reading the work parameters, the parameters of the work item to be executed are obtained, and the corresponding parameters are generated to generate corresponding The working parameters, then read the process rule base, obtain the logic rules of the process operation, to correctly arrange the execution steps of the work item, and modify the work order of the existing work table in the work flow design and update basis. Update, enter the dynamic process worksheet after completion, output the new dynamic process worksheet according to the modification of the process parameters, and then enter the corresponding parameters of the production work, and output the new work corresponding parameters according to the modification of the process parameters and the working parameters; Get the status record marked as normal and enter the existing work Corresponding parameters, using the existing work corresponding parameters, without modifying them, then using the existing dynamic flow table, using the existing dynamic process worksheet, and not modifying it; after the dynamic process worksheet and corresponding parameters are determined, enter the process Work step mark, mark the next step in the dynamic process worksheet, and enter the work item coordinator.

The above-mentioned execution of the retrospective reorganization process further includes the following steps: after the workflow is determined to be retroactively reworked, the process error state record is first taken, and the error state is taken out and judged. If it is determined that data backtracking is required, Enter the data state analysis; if it is determined that the process needs to be replied, enter the process breakpoint location analysis; analyze the parameter data obtained by the data state analysis, and after determining the backtracking range and the data backfilling content, enter the system backtracking command, according to The obtained parameters and the information obtained from the data storage architecture association table are generated, and the data table and the file system are cleaned up, and the contents of the file and the data table are cleaned up, and the data to be cleaned back is cleaned up, and the data recovery process is completed after completion. , according to the parameters obtained, the data recovery operation process Calculate the process plan into the process parameters, and finally return to the process decision; at the process break point location analysis analysis process due to the error caused by the state of the interrupt, and determine the data processing scope and content, enter the data processing instruction According to the obtained parameters, the data processing instruction is used to restore the back-end data to the state before the execution of the interrupted operation, and execute the data processing instruction in the data processing, and the data to be cleaned up before the process is restored is completed, and the self-interruption point recovery process is executed. Maintain the arrangement and sequence of the original work items, continue the unfinished work, achieve the purpose of the process response, and finally return to the process decision to continue the next step.

When the data mining model automatic transportation system of the present invention is compared with other conventional technologies, the advantage is that the data exploration classification, the group model establishment, the application data prediction score, the data grouping, and the model accurate in the past rely on human resources. Important but complex processes such as degree monitoring and data calculation of important variables are serially integrated, and a control mechanism is designed for all possible errors in the process, so that all processes are smoothly driven according to the logic flow that should be in place, and when an error occurs. According to different error conditions, data cleaning, breakpoint records, backup responses, etc., can be smoothly run when the process is restarted, and the effect of process driving and condition control automation is achieved.

The technical features, contents, and advantages of the present invention, as well as the advantages thereof, can be understood by the reviewing committee, and the present invention will be described in detail with reference to the accompanying drawings. The subject matter is only for the purpose of illustration and description. It is not intended to be a true proportion and precise configuration after the implementation of the present invention. Therefore, the scope and configuration relationship of the attached drawings should not be interpreted or limited. First described.

Please refer to FIG. 1 , which is an architectural diagram of the data exploration model automatic transportation and transportation system of the present invention, which includes: a user operation interface 1 , and a parameter data for the user to input the operation; Survey automation maintenance engine 2, store parameters and capture, transmit and receive relevant data according to requirements for job assignment, control process execution status, error status determination and error handling; model metadata 3 for storing descriptive data exploration Model metadata; result set 4 is used to store data calculation results obtained in the data exploration and transportation process; the data access module 5 can perform access operations on data of different storage properties, and perform required data acquisition and calculation The result, the data mining automatic transmission engine metadata is written; the plurality of data sources 10, the information provided by the user, used in model monitoring, establishment, testing and data scoring; the model monitoring module 6, the data After the calculation, the accuracy of the model is established, and the accuracy of the model is determined to be lower than the set threshold. The model reconstruction module 7 uses a plurality of different algorithms to perform model training, and the obtained models are tested with test data. Compare and select the model with the best accuracy; the data scoring module 8 uses the model rules to calculate the data to be scored. To score; individualized important variable module 9, through internal model information, the use of sequencing, ratio-type, class and other technology, the field properties of each piece of data were important calculation. In the first figure, the relationship between the components is as follows: the data mining automation engine 2 stores the parameters obtained by the user operation interface 1, and retrieves the metadata from the model through the data access module 5 according to the content of the operation content. The result set 4 or the data source 10 data is individually assigned to the model monitoring module 6, the model reconstruction module 7, the data scoring module 8, the individualized important variable module 9, and receives the returned modules. The calculation result and the process status record, if the process is in error, different error processing methods are implemented according to the status, and if the process status is correct, the operation result is written into the model metadata 3 and the result set 4 through the data access module 5.

Please refer to FIG. 2, which is a structural diagram of the data exploration and transportation engine 2 of the present invention, which further includes a process parameter data area 21 for storing the operation to be performed by the user; the working parameter data area 22, Used to store data location, model accuracy threshold, data date and other parameters; scheduling The device 23 receives the schedule set by the user, and starts the process when the time is up; the retroactive reforming module 24 performs the error state determination when the process is in error, and adopts the best error handling method according to the determination; The pipe module 25 performs dynamic process scheduling, parameter preparation, marking work steps and coordinating the work items of the next step according to the process state; the process status record area 26 is used to record the state of the process execution as the judgment basis of the process; the job assignment mode Group 27, launching the module to be executed, transmitting the information required for the work, receiving the return result, and transmitting the data to the process status record area 26 for recording. In Fig. 2, the relationship of each component is as follows: the process parameter data area 21, the work parameter data area 22 and the scheduler 23 receive and store the parameters obtained by the user operation interface 1, and when the time is up, the schedule is The controller 23 activates the process control module 25. The process control module 25 reads the data of the process parameter data area 21, the work parameter data area 22 and the process status record area 26, and performs dynamic process scheduling, parameter preparation, marking work steps, and work item coordination, and the work is handed over. A module 27 is assigned to the job. The job assignment module 27 retrieves the data required for the work from the model metadata 3, the result set 4, or the data source 10 through the data access module 5 according to the current work item, and assigns the job according to the load status of the backend host. After the job assignment, it is responsible for receiving the process status returned by the model monitoring module 6, the model reconstruction module 7, the data scoring module 8 or the individualized important variable module 9, and transmitting the process data to the process status record area. After 26, the process control module 25 is driven to continue the work arrangement of the next step. If the job assignment module 27 receives the status record indicated as an error, the backtracking reforming module 24 is driven, and after the module performs the backtracking reorganization plan, the parameters are written into the process parameter data area 21, and then The process control module 25 performs the work sequence adjustment, and the job assignment module 27 drives the retrograde reforming module 24 to perform retrospective reorganization of the model metadata 3 or the result set 4 through the data access module 5, and then continues Follow-up steps.

Please refer to FIG. 3, which is a flow control module 25 of the data exploration and transportation engine of the present invention. The composition, further comprising: a process decision unit 251, configured to determine the job status and parameter data, and determine the arrangement of the work process steps, the corresponding parameter arrangement and the work step mark; the process rule library 252, which contains the logic rules of the correct process, provides The basis of the process sequence arrangement; the dynamic process work table 253, the objects generated after the process arrangement is completed, will be modified according to the different working conditions; corresponding work parameters 254, corresponding to the parameters required for each work item, will The information notification unit 255 is configured to generate a notification message to inform the user of the process status; the work item coordinating unit 256 is responsible for coordinating the work of the next step and its required parameters, and delivering the work. The job assignment module 27 performs the assignment. In Fig. 3, the relationship of the components is as follows: the scheduler 23 starts the process control module 25 at the time point, and the process decision unit 251 reads the process parameter data 21, the work parameter data 22, the process state record 26 and The data of the process rule base 252 performs workflow scheduling, parameter sorting and work step marking, generates a dynamic process work table 253 and corresponding work parameters 254, and performs message notification through the message notifying unit 255 as necessary. The work coordinating unit 256 then coordinates the work items and parameters of the next step according to the mark of the dynamic process work table 253 and the corresponding work parameter 254, and assigns the work delivery job assignment module 27, after the job assignment module 27 is assigned. After receiving the process status message and writing is completed, the process control module 25 is started again for subsequent steps.

Please refer to FIG. 4 , which is a structural diagram of the job assignment module 27 of the data exploration and transportation engine of the present invention, which further includes: a data preparation and checking unit 271, which is responsible for preparing the materials required for the work project, and obtaining After the data is initially checked and the result is returned; the process status return unit 272 is responsible for transmitting the obtained process status to the process status record 26; the system load detecting unit 273 is configured to detect the load of the back end host. The condition, in order to provide the basis of the job assignment, the job initiation unit 274, according to the current work, starts the corresponding operation module, and transmits the required materials together. After receiving the return data. In Fig. 4, the relationship of the components is as follows: the process control module 25 transmits the work and parameters to be executed to the data preparation and checking unit 271, and the unit obtains the work from the working parameter data 22 and the data access module 5. The required information is checked and the inspection result is passed to the process status processing unit 272. If the data check is correct, the system load detection unit 273 starts to check the load status of the back end host, and after determining the work host, the work start unit 274 transmits the work and required data to the module to be executed; the model monitoring module 6. The model reconstruction module 7, the data scoring module 8 or the individualized important variable module 9 or the retroactive reforming module 24, and receives the returned data, and then passes the received process status to the process state return processing unit. 272, finally written into the process status record area 26, and drive the process control module 25 to read the newly written process status record for subsequent steps.

Please refer to FIG. 5 , which is a structural diagram of the retrospective reorganization module 24 of the data exploration and maintenance engine of the present invention, which further includes: a data storage architecture association table 241 for recording the operating systems of different users. The generated record file and the data location and their mutual correspondence relationship; the data retrospective planning unit 242 establishes a data replenishment plan according to the process state when the user arrangement or error condition occurs and determines that the data backtracking is required; the parameter parsing unit 243 And parsing the received parameter, determining whether to execute the error processing plan, and determining the error processing mode; the interruption point position analyzing unit 244 is configured to determine the state of the process due to the error and to define the state when the process returns to the operation, and defines The scope and content of the data processing; the backtracking instruction generating unit 245, when performing data backtracking, is used to generate an instruction to trace the recorded data back; the flow reply instruction generating unit 246, when performing the process reply, is used to generate the command to generate the backend data. Reverting to the state before the execution of the interrupt job; the process state return processing unit 247 is responsible for obtaining Process transfer state after processing a job assigned to the module 27; 248, when the data back, the system file related data processing module Backtracking of archives. In Fig. 5, the relationship between the components is as follows: the parameter parsing unit 243 receives the data obtained by the job assignment module 27 for analysis, and determines the processing mode thereof, and judges whether the data backtracking or the process reply should be performed according to the situation, if the data is executed. Backtracking, the job is delivered to the data traceback reorganization unit 242; if the execution process is replied, the job is delivered to the breakpoint location resolution unit 244. After the data retrospective plan unit 242 obtains the analysis result by the parameter parsing unit, the data storage architecture association table 241 is read to define the scope of the data backtracking and the associated file data, the data range to be replenished, and the data replenishment plan. The plan is transmitted to the parameter flow data area 21. The backtracking instruction generating unit 245 generates a backtracking instruction according to the range defined by the data backtracking unit, and issues an instruction through the data access module 5, and the file data processing module 248 performs backtracking of the related file; After the process interruption state is judged, after defining the scope and content of the data processing, the process reply command generating unit 246 generates a reply command, and sends the required command through the data access module 5 to release the process. The data backtracking or the process reply command release result is processed by the process state return processing module 247, and then the process state is transmitted back to the job assignment module 27 for subsequent processing.

Please refer to FIG. 6 , which is a flow chart of a method for data mining model automatic transportation system of the present invention, which comprises the following steps: First, the system judges the working time point every time the operation is received, and if the operation is performed in a scheduled manner. The entry starts 100 according to the schedule time, and when the time is up, the start operation enters the process decision 200; if the job is executed immediately, the process decision 200 is directly entered. In the process decision 200, the scope, sequence and preparation parameters of the operation are arranged, and the work steps are marked, and the work item coordinator 300 is used to integrate the parameters and steps. Next, it is judged whether all the arranged work items are completed, and if so, the current work is finished; if otherwise, the execution data preparation and check 400 is entered, and the data obtained from the source of the data is initially checked and returned, and then the flow is processed. State judgment. If the data check finds an error, the job will be judged as not being able to run automatically, and then enter the generate notification message 500 to end the job; if the data check does not find an error, in the process running state, enter the job start 700 After determining the assignment according to the load status of the back-end host, perform model monitoring 800, model reconstruction 900, data score 1000, or individualized important variable calculation 1100 according to the different work items, and process each after processing the job. The process state 1200 receives the process status, and then returns to the process decision 200 to continue the operation of the next step; if the data check does not find an error and the status is backtracking and reorganization, then the execution of the retrospective re-routing process 600 is performed. After the end data is processed, it returns to the process decision 200 to continue the next step.

Please refer to FIG. 7 , which is a flowchart of a method for an automated data transmission system of a data exploration model according to the present invention. The process decision 200 further includes the following steps: when the time point of the operation is reached, after the completion of the scheduling time 100 is completed, After the execution of the backtracking and re-routing process 600 is completed, the process status 1200 of the process back is processed, the process status is received, and the data is written to the process status record, and the process control module 25 is driven to perform subsequent operations, and the process decision is made after completion. 200. First, after reading the process status record 201, the process status record is taken out, and as a basis for determining the workflow arrangement, it is determined whether the process should be modified. If the process status record marked as an error is read, the process modification is performed, and the read process parameter 202 is entered to obtain the work item to be executed in the current process, and after the process flow work table is generated, the work parameter 203 is read. Obtaining parameters of the work item to be executed to generate corresponding work parameters, and then reading the process rule base 204 to obtain logic rules of the process work, to correctly arrange the execution steps of the work item, and based on the design and update of the work flow 205 The above-mentioned data will be used to modify the work order and update the content of the existing worksheet. Upon completion, the dynamic workflow worksheet 206 is generated, and the new dynamic process worksheet is output according to the modification of the process parameters, and then the corresponding work corresponding parameter 207 is entered, according to Modification of process parameters and working parameters will The new work corresponds to the parameter output; if the status record marked as normal is read, the existing work corresponding parameter 209 is entered, the existing work corresponding parameter is used, and the existing dynamic flow table 210 is used, and the existing one is used. The dynamic process worksheet is also not modified. After the dynamic process worksheet and corresponding parameters are determined, the process work step marker 208 is entered, and the next step is marked in the dynamic process worksheet to enter the work project coordination 300.

Please refer to FIG. 8 , which is a flowchart of a method for the automated data transmission system of the data exploration model of the present invention. The execution of the retrospective re-routing process 600 further includes the following steps: when the workflow is determined to be retroactively reorganized, first The process error status record 601 is retrieved, and the error status is taken out and judged. If it is determined that the data backtracking is required, the data status analysis 602 is entered; if it is determined that the process response is required, the process interruption point position analysis 606 is entered. The data state analysis 602 analyzes the obtained parameter data, and after determining the backtracking range and the data backfilling content, enters the generating system backtracking instruction 603, and generates an instruction to the data table according to the obtained parameter and the data storage structure association table. The file system is cleaned up, and the contents of the file and data sheet 604 are cleaned up, and the data to be cleaned back is cleaned up. After completion, the data recovery process 605 is entered, and the data replenishment process is designed according to the obtained parameters. The plan is written into the process parameters, and finally returned to the process decision 200; the process break point location analysis 606 analyzes the state of the interrupt caused by the error, and determines the data processing scope and content, then enters the generate data processing command 607, according to The obtained parameter generating data processing instruction returns the back-end data to the state before the execution of the interrupting operation, and executes the data processing instruction in the data processing 608, and the data to be cleaned up before the process is restored is completed, and the self-interruption point recovery process is executed 609. Maintain the arrangement and sequence of the original work items and continue unfinished Work to achieve the purpose of the return process, the decision-making process and finally back to 200 to continue the work of the next step.

The following is an embodiment of an automated transportation system and method for using the data exploration model of the present invention. Suppose a company regularly improves the customer churn rate of a specific business service by means of data mining. By using the present invention, a customer with a high loss tendency is automatically calculated before the prediction accuracy is ensured, and is used for subsequent marketing operations.

Please refer to FIG. 2 and FIG. 6 again. After inputting the data of Table 1 in the user operation interface 1, the user stores the work execution item from the process parameter data 21; the work parameter data 22 stores the data source "North District Database" and the topic. The name "customer churn calculation"; the scheduler 23 stores the execution frequency "1/month: 12:00:00". The user operation interface 1 performs data analysis on the model uploaded by the user, and inputs the model data into the model metadata 3. After the completion of the determination of the operation time point, this embodiment is operated in a scheduled manner, so the entry is initiated by the scheduled time 100, and the project is executed at 12 o'clock every month. In this example, the date 2012-05-01 is taken as an example. When the time is up, the process decision 200 is entered.

Please refer to Figure 3 and Figure 7 again to enter the read process status record 201. Since this issue is newly established and does not contain any previous process status, it will be judged at the decision whether the process should be modified. "Yes". The process decision unit 251 reads the process parameter 202, and then reads the work parameter 203, and then after reading the process rule base 204, according to the obtained information: 1. The process is started for the first time, 2. The work item to be executed is Monitoring, reconstruction, scoring and individualization of important variables, 3. Source of data for the North District Database, 4. Applicable process rules, planning. Generating dynamics The flow work table 206 and the work corresponding parameter 207 are obtained. Table 2 is a dynamic flow work table, and the work items to be executed are arranged in order, and Table 3 is a parameter correspondence table for corresponding to the work table of Table 2. Correspond to the parameters required for the work to be performed under each sequence. Since this embodiment is a newly created scheduling project, there is no scoring data available for model monitoring. Therefore, in the process working step mark 208, the sequence 1 and the sequence 2 of Table 4 are skipped, and the current stage will be skipped. The steps are marked in the third order. As shown in Table 4, the "*" mark is marked on the work item to be executed at the current stage, and after completion, the work item is co-ordinated 300.

Please refer to the 3rd, 4th and 6th drawings again. After the work item coordinator 300 reads the results of Table 3 and Table 4, it will judge the sequenced three work items of the marked Table 4 and their corresponding parameters. Whether all the work items are completed, since the work items of the third and fourth order of Table 4 in this embodiment have not been executed, the determination is "NO", and the execution data preparation and check 400 is entered. The data preparation and checking unit 271 performs preliminary check on the data preparation and the contents of the assigned work items. In this example, the assigned work is "data rating". Since the known data source is the North District Database, it is transparent. The data access module 5 obtains the data to be scored and checks the score Is the date of the material incorrect? If the data is found to be incorrect, the data preparation and checking unit 271 passes the error message "The data scoring process is incorrect, and the reason: the date of the scoring data content is incorrect." is passed to the process state return processing unit 272, as shown in Table 5. A record with an error code of 201 is written, and the process control module 25 is started after the process status record 26 is written. At this time, the process decision unit 251 reads the error message of the data check, the process status is determined not to be automatically run, and the notification message 500 is entered, and the message notification unit 255 sends a message to notify the user to correct the data.

Assuming that the data is correct, the process status is determined to be "running" or "backtracking reorganization" according to the work item. The work item of this embodiment is scored for the data, so it is determined to be "running" and the job start 700 is entered. The system load detection unit 273 will detect the CPU load and the remaining space of the RAM in the system environment, and obtain the result as shown in Table 6. It is known from Table 6 that the current load of the host 3 is low and the resources are sufficient. Therefore, it is decided that the host 3 performs the work of scoring the data, and the job initiation unit 274 starts the data scoring module 8 and enters the data score of 1000. The data required for the job is transferred to the host computer 3 for calculation, and the completed operation result is transmitted back to the result set 4 and then enters the process state 1200 of the process return. In the process status 12000 for processing the return, the job activation unit 274 will receive the process status "data scoring process has been successfully completed." returned by the data scoring module 8, and the process returns to the status by the process return processing unit 272. Into the process status record 26, a process status record as shown in Table 7 is obtained, and the process control module 25 is initiated, returning to the step flow decision 200.

Please refer to Figure 7 again. After entering the process decision 200, since the process status of the process is normal, after the process status record 201 is read, the decision process does not need to be modified, and after entering the existing work corresponding parameter 209, the entry proceeds to the existing The dynamic flow table 210 is as shown in Table 4. In the process work step mark 208, the "*" sign is marked in the fourth order, as shown in Table 8, into the work item coordinator 300.

Please refer to the 5th to 8th drawings again, enter the status of the subsequent operation after the work item is coordinated, and assign the job to the specific host, and enter the individualized important variable calculation 1100. At this time, if the host has an error, it will enter the return processing. After the process status 1200 writes the resulting error message to the process status record 26, the process control module 25 is launched again and returns to the process decision 200. After reading the process status record 201, the error message returned will be read, and the process modification will be performed, and the backtracking and re-routing process will be inserted into the work table and the work steps will be marked to the corresponding work. The numbers are shown in Table 9 and Table 10.

Since the retrospective reorganization operation does not require data preparation and verification, the execution data preparation and verification 400 will not perform the operation, and directly enters the execution retrospective reorganization process 600. After the data is acquired by the process error status record 601, the parameter analysis unit 243 analyzes the error code and the operation parameter to determine the operation to perform data traceback or process reply. There may be two situations at this time: Condition 1, as shown in the last record in Table 11, the reason for the error in the process is that the network problem is not caused by the back-end data (model metadata 3, result set 4), then The process reply operation, enters the process break point location analysis 606, analyzes the process interruption and the current working parameters to determine the data processing scope and content, and generates the data processing instruction 607.

Since the cause of the error is not caused by the back-end data, the data written before the interruption should be cleaned up before the process is resumed to avoid the problem of repeated data writing. Therefore, the data processing instructions as shown in Table 12 are generated. The access module 5 performs the data processing 608. After the execution of the instruction is completed, the execution result is written to the process status record 26, as shown in Table 13, and the process decision 200 is returned to the unfinished operation.

In the second case, as shown in the last record in Table 14, because the content of the scoring data (result set 4) is abnormal, resulting in an error in the calculation of the individualized important variables, the data backtracking process is adopted, and the data state analysis 602 is entered, and the data backtracking is performed. The drawing unit 242 reads the contents of the data storage architecture association table 241 and generates a backtracking instruction 603. Due to the incorrect content of the previously calculated scoring data, the calculation process of the individualized important variables is wrong. For this reason, the data in the back-end data must be deleted and replenished, and the individualized important variable data to be calculated must be And clean up to avoid repeated writing of data. As shown in Table 15, the data backtracking instructions are divided into SQL and DOS. The database and system files are processed separately. The SQL command numbered 1 will be written before the interrupt. After the individualized important variable data is cleaned up, the error scoring data is cleaned up by the SQL instruction number 2, and the file parsed file is cleared by the DOS command number 1.

After completion, the process proceeds to the establishment data replenishment process 605, and the replenishment plan shown in Table 16 is generated, sent to the process parameter data 21, and returned to the process decision 200 to perform the replenishment operation. Please refer to Figure 6 again. After returning to process decision 200, due to the data back-tracking process, the state is determined to be the process to be modified, and the data is replenished after the process parameters are read 202, and the dynamic process work is generated at 7. Table 206 shows the data replenishment plan, as shown in Table 17 and Table 18.

After the status is completed, the retrofit or status 2 is completed, and then the last job is continued: the important variable calculation is individualized. Please refer to Figure 6 again. After the process decision 200 completes the worksheet and its marking and parameter setting, it will enter the work project coordination 300, and the work items to be carried out at this stage: the individualized important variable calculation and its parameters are extracted, This work item is still required to be executed, so the implementation data preparation and inspection 400 is entered. After the data preparation and verification are correct, the operation start 700 is entered, and the individualized important variable calculation is delivered to the host with the lower load, and then the individualized important variable calculation 1100 is entered, and after the calculation is completed, the process state of the process return is entered. 1200, after writing the record in the process state, return to process decision 200. At this time, since the calculation of the individualized important variables of the genius has been successfully completed, there are no unexecuted work items in the worksheet. After entering the work project coordination 300, there is no need to process it, and after the determination, it is determined that all work items have been completed. End the workflow of this example.

The above-mentioned embodiments are intended to be illustrative only and not to limit the scope of the present invention. Any equivalent modifications and changes thereto are not included in the spirit and scope of the present invention. In the scope.

1‧‧‧User interface

2‧‧‧Automatic maintenance engine

3‧‧‧Model metadata

4‧‧‧ result set

5‧‧‧ Data Access Module

6‧‧‧Model Monitoring Module

7‧‧‧Model Reconstruction Module

8‧‧‧Information scoring module

9‧‧‧Individualized important variable modules

10‧‧‧ Multiple sources

21‧‧‧Process parameter data area

22‧‧‧Working parameter data area

23‧‧‧ Scheduler

24‧‧‧Retrospective reforming module

25‧‧‧Process Control Module

26‧‧‧Process Status Record Area

27‧‧‧ Job Assignment Module

101‧‧‧Source 1

102‧‧‧Source 2

103‧‧‧Source 3

251‧‧‧Process decision unit

252‧‧‧Process rule base

253‧‧‧Dynamic Process Worksheet

254‧‧‧ corresponding working parameters

255‧‧‧Message Notification Unit

256‧‧‧Working Project Coordination Unit

271‧‧‧Information Preparation and Inspection Unit

272‧‧‧Process Status Return Unit

273‧‧‧System load detection unit

274‧‧‧Operation starter unit

241‧‧‧Data Storage Architecture Association Table

242‧‧‧Material retrospective unit

243‧‧‧Parameter resolution unit

244‧‧‧ Breakpoint position resolution unit

245‧‧‧ Backtracking instruction generation unit

246‧‧‧Process reply instruction generation unit

247‧‧‧Process Status Return Processing Unit

248‧‧‧Archive Data Processing Module

100‧‧‧Start by schedule time

200‧‧‧ Process decision

300‧‧‧Working project coordination

400‧‧‧ Implementation of data preparation and verification

500‧‧‧ Generate notification message

600‧‧‧ Perform retrospective reorganization process

700‧‧‧Starting the work

800‧‧‧Model Monitoring

900‧‧‧Model reconstruction

1000‧‧‧ data rating

1100‧‧‧Individualization of important variable calculations

1200‧‧‧Processing the status of the return process

201‧‧‧Read process status record

202‧‧‧Read process parameters

203‧‧‧Read working parameters

204‧‧‧Read process rule base

205‧‧‧Workflow design and update

206‧‧‧ Generate dynamic process worksheet

207‧‧‧ Generate work correspondence parameters

208‧‧‧Process work step marking

209‧‧‧Use existing work corresponding parameters

210‧‧‧Use existing dynamic process tables

601‧‧‧Acquisition process error status record

602‧‧‧Data status analysis

603‧‧‧ Generate system backtracking instructions

604‧‧‧Clean up files and data sheets

605‧‧‧ Establish data recovery process

606‧‧‧Process breakpoint location resolution

607‧‧‧Generation of data processing instructions

608‧‧‧Data processing

609‧‧‧ resume process execution from breakpoint

Figure 1 is an architectural diagram of the present invention.

Figure 2 is an architectural diagram of the present invention.

Figure 3 is an architectural diagram of the present invention.

Figure 4 is an architectural diagram of the present invention.

Figure 5 is a block diagram of the present invention.

Figure 6 is a flow chart of the present invention.

Figure 7 is a flow chart of the present invention.

Figure 8 is a flow chart of the present invention.

1‧‧‧User interface

2‧‧‧Automatic maintenance engine

3‧‧‧Model metadata

4‧‧‧ result set

5‧‧‧ Data Access Module

6‧‧‧Model Monitoring Module

7‧‧‧Model Reconstruction Module

8‧‧‧Information scoring module

9‧‧‧Individualized important variable modules

10‧‧‧ Multiple sources

101‧‧‧Source 1

102‧‧‧Source 2

106‧‧‧Source 3

Claims (8)

A data exploration model automatic transportation and transportation system comprises: a user operation interface, receiving parameter data input by a user number; and a data exploration and automation operation engine, storing parameter data received by the user operation interface, and extracting, Transfer and receive parameter data for job assignment, control process execution status, error status determination and error handling; a data access module, accessing parameter data of different storage properties, performing required parameter data capture and The calculation result, the writing of the metadata of the data mining automatic transportation engine; a model metadata storing the metadata of the data mining automation engine; and a result set for storing the data calculation result obtained by the data mining automatic transportation engine; A plurality of data sources are provided by the user and can be retrieved through the data access module; a model monitoring module calculates the parameter data assigned by the data mining automation engine and compares the models Accuracy at the time of establishment, determine whether the accuracy of the model is lower than the set threshold; The module will use the different algorithms to carry out model training on the parameter data assigned by the data mining automation engine. The several models will be compared with the test data to select the model with the best accuracy; a data scoring module, The parameter data assigned by the data mining automation engine is calculated by the model rule, and the parameter data to be scored is calculated to obtain a score; and an important variable module is to be assigned by the data mining automatic maintenance engine. The data is used to calculate the importance of the field attributes of each parameter data through the internal information of the model, using sorting, proportionaling, and stratification. For example, the data exploration model automatic transportation and transportation system described in claim 1 of the patent application, wherein the data exploration automation maintenance engine further comprises: a process parameter data area, and the parameter data obtained by storing the user operation interface is to be executed. Job location; a working parameter data area, a location of the parameter data obtained by the user operation interface, a model accuracy threshold, a data date; a scheduler, receiving a schedule of the parameter data obtained by the user operation interface, waiting time The process is started at a point in time; a process control module is started by the scheduler, and the process parameter data area, the work parameter data area and a process status record area are arranged, and dynamic process scheduling, parameter preparation and marking are performed. Work-step co-ordination; a job assignment module that accepts work entered by the process control module, and obtains information required for work from the model metadata, the result set, or the source of the data through the data access module, And assigning an operation to the model monitoring module, the model reconstruction module, the data scoring module or the individualized important variable module, and The process status of the model monitoring module, the model reconstruction module, the data scoring module or the individualized important variable module is recorded in the process status record area; the process status record area, the status of the record process execution As a basis for judging the process; and a backtracking reorganization module, accepting the status record of the error of the job assignment module, and performing the backtracking reorganization plan, will enter the process parameter data area. For example, the data exploration model automatic transportation and transportation system described in claim 2, wherein the process control module further comprises: a process decision unit, which receives data from the process parameter, the working parameter data, and the process status. Recording and data of a process rule base, marking workflow, parameter sorting and work step marking; the process rule base includes logic rules for correct processes, providing basis for process sequence arrangement; a dynamic process worksheet is for The process decision unit performs workflow scheduling, parameter sorting, and worksheets generated after the work steps are marked; a corresponding work parameter is required for the workflow decision unit to perform work flow scheduling, parameter sorting, and corresponding work items generated after the work step is marked. The parameter notification unit is a notification message generated by the process decision unit for the workflow decision arrangement, the parameter arrangement and the work step mark, and the user is informed of the process status; and a work item coordinating unit according to the dynamic process work form And the corresponding working parameters, coordinating the next step Work and its required parameters are assigned to the job assignment module. For example, the data exploration model automatic transportation and transportation system described in claim 2, wherein the operation assignment module further comprises: a data preparation and inspection unit to check the work to be performed accepted by the process control module. And a parameter; a process state processing unit, receiving the inspection result of the data preparation and checking unit, and writing the inspection result into the process status recording area; a system load detecting unit receiving the data preparation and checking unit Check the results, and check the load status of the back-end host based on the check results to determine the job host; A job activation unit transmits the work and required data to the module to be executed according to the decision of the system load detection unit. For example, the data recovery model automatic maintenance system described in claim 2, wherein the retrospective reforming module further comprises: a data storage architecture association table, which records the record files generated by different users' operating systems and Data location and its mutual correspondence; a parameter parsing unit receives the parameters of the job assignment module for parsing, determines whether to execute an error processing plan, determines an error processing mode, and a data backtracking unit receives the parameter parsing unit Judging the data back-tracking operation, the data storage architecture association table obtains the location and data position of the user's operating system records and their mutual correspondence, to define the scope of the data backtracking and the associated file data, the scope of the data to be replenished, Generating a data replenishment plan and transmitting the plan to the parameter process data area; a backtracking instruction generating unit generates a backtracking instruction according to the range defined by the data backtracking unit, and issues an instruction through the data access module An archive data processing module executing the file assigned by the backtracking instruction generating unit Backtracking; a breakpoint location parsing unit receives the data reply operation determined by the parameter parsing unit and defines the scope and content of the reply data processing; a process reply instruction generating unit, according to the range defined by the interrupt point location parsing unit Content, generating a reply command, and issuing the command through the data access module; and a process state return processing module, receiving the process state processed by the data access module, and transmitting back to the job assignment module. A method for data mining model automatic transportation system, the steps thereof include: a. The system judges the time of the work every time the operation is received, and if the operation is performed in a scheduled manner, the operation starts at the scheduling time, at the time point. When it is time, start the job to enter the process decision; b. If the job is executed immediately, go directly to the process decision; c. In the process decision, arrange the scope, sequence and preparation parameters of the job, and mark the work steps, then Coordinating the integration of parameters and steps by the work project; d. Judging whether all the arranged work items are completed, and if so, the operation is completed; e. If the data preparation or verification is entered, the data obtained from the source will be initially checked. After verifying the status, the process status is judged; f. If the data check finds an error, the operation will determine that the operation cannot be automatically performed, and then enter the notification message to complete the operation; g. The core did not find an error. In the process running state, the job is started, and after the job assignment is determined according to the load status of the back-end host, Different work items, respectively perform model monitoring, model reconstruction, data scoring or individualized important variable calculation, respectively receive the process status after processing the return process, and then return to step c to continue the next step; h. If no error is found in the data check and the status is backtracking and reorganization, go to the execution of the retrospective reorganization process, and then process the backend data, and then return to step c to continue the next step. For example, the method for applying the data exploration model automatic transportation and transportation system described in claim 6 wherein the process decision further comprises the following steps: when the operation time point is reached, the execution time is completed after the scheduling time is completed, and the execution backtracking is performed. Heavy rectification After the process is completed, the process status of the process is returned, the process status is received, and the data is written to the process status record, and the subsequent operation is arranged through a driving process control module, and the process decision is made after completion; Take the process status record, take out the process status record, and determine whether the process should be modified as the judgment basis of the workflow arrangement; if the process status record marked as wrong is read, the process modification is performed, and the read process parameters are entered. After obtaining the work item to be executed in the current process, after the working process worksheet is generated, after reading the working parameters, the parameters of the work item to be executed are obtained, and the corresponding working parameters are generated, and then the process rule library is read and obtained. The logic rules of the process operation are used to correctly arrange the execution steps of the work items, and the existing worksheets are modified in sequence and updated in the content according to the data obtained from the design and update of the operation process. Upon completion, the dynamic process worksheet is generated, according to The modification of the process parameters will output the new dynamic process worksheet, and then enter the production Corresponding parameters, according to the flow parameter and the modification of the working parameter, the new work corresponding parameter is output; if the status record marked as normal is read, the corresponding work corresponding parameter is entered, and the existing work corresponding parameter is used, and the corresponding parameter is not used. Modify, continue to use the existing dynamic process table, use the existing dynamic process worksheet, and do not modify it; after the dynamic process worksheet and corresponding parameters are determined, enter the process work step mark, mark the next step in the dynamic process work In the table, enter the work project coordination. For example, the method for applying the data exploration model automatic transportation and transportation system described in claim 6 wherein the performing the retrospective reorganization process further comprises the following steps: after the workflow is determined to be retroactively reorganized, the extraction process is first performed. Error status record, take the error status out If it is judged that data backtracking is required, the data state analysis is entered; if it is determined that the process needs to be replied, the flow break point position analysis is entered; the parameter data obtained by the data state analysis is analyzed, and the backtracking range and data are determined. After replenishing the content, enter the system backtracking instruction, according to the information obtained from the parameter and the data storage architecture association table, generate instructions to clean up the data table and the file system, and clean up the contents of the file and data table, which will be cleaned up. The data is cleaned up, and after completion, the data replenishment process is entered, the data replenishment operation process is designed according to the obtained parameters, the process plan is written into the process parameters, and finally the process decision is returned; the process interruption point location analysis and analysis When the process is interrupted due to an error, and the data processing scope and content are determined, the data processing instruction is entered, and the data processing instruction is generated according to the obtained parameter to return the back-end data to the state before the execution of the interrupt job, and the data processing is performed. Execute data processing instructions to clear the process before returning The data processing is complete, enter the recovery process from the point of interruption implement and maintain arrangements with the order of the original work item, continue unfinished work to achieve the purpose of the return process, and finally back to step c to continue the work of the next step.