JP5251718B2 - Data conversion program, data conversion method, and data conversion apparatus - Google Patents

Description

  The present invention relates to a recovery technique when a conversion error occurs in a data conversion system.

  A data conversion system that collects data from a data supply source system, performs conversion / processing, and distributes the data to the data supply destination system is becoming popular. As shown in FIG. 23, the data conversion system collects data from the data supply source systems (A, B), performs a plurality of data conversions (conversions 1, 2, 3), and processes the processed data. Are distributed to the data supply destination systems (C, D).

  For data collection from the data supplier system, depending on the type of business system, a general-purpose machine program transfers files to the data conversion system side, and data stored in an open relational database is SQL (Structured Query). (Language: structured query language).

  Conventionally, when an error due to data fraud or invalid data occurs in any of a plurality of conversion processes (code conversion of characters etc., cleansing, etc.), (1) the intermediate data in the conversion process in the middle is corrected to the user (2) Have the user correct the abnormal record from all the collected data, re-enter all the data, and convert all the data from the first conversion process. (3) When a conversion error occurs, an error recovery method has been adopted, such as skipping an abnormal record processing and continuing normal record processing (see, for example, Patent Document 1).

  In the above-described conventional method, when correcting and recovering data at the time of error occurrence, the user confirms / corrects intermediate data being converted, not data at the time of collection. Since the intermediate data has undergone a change of order of records from the collected data and a plurality of conversions, it is difficult for the user to determine the cause of the error and correct the data.

  In addition, in the method of skipping the abnormal record conversion process and continuing the normal record, since the number and types of errors are large, the determination of the error location and error content becomes more complicated. Further, the point at which recovery is started for each conversion error is different, and the user's recovery operation becomes complicated.

  Furthermore, when conversion abnormal data is included in the data conversion flow, there are cases where you want to skip that data (record) and continue the conversion process of other data. There are problems such as incorrect conversion results due to lack of data. Recovery in this case requires re-execution from the beginning of the data conversion flow with all data including correction data, and the amount of data is large and recovery takes time.

JP 2006-3978 A

  In order to solve the above-described problems, in the present invention, in the data conversion system, a collection data tracking mechanism and an accumulation file mechanism for accumulating and managing collection data and intermediate data for a certain period (data retention / reuse / management) Efficient data conversion error recovery apparatus, method, and program combining the recovery mechanism using

  One aspect of the present invention relates to a data conversion program in a data conversion apparatus that converts data collected from a supply source system and supplies the data to a supply destination system, and stores the collected data in a storage file and manages the collected data in a computer A management step, a record information adding step for extracting target data from the accumulated file and adding a record identification number to the data, and a plurality of conversion processes for each record with respect to the data to which the record identification number is added. A data conversion step to be performed; a conversion continuation step in which, if an error occurs in the conversion process, except for an error record, other records are continuously processed to a predefined meeting point; and the error record is corrected by the user Get a correction record and change the correction record A conversion reprocessing step for performing reprocessing, and a distribution data output step for cutting out the record identification number from the data and outputting it as distribution data when conversion of all records in the data is normally completed; By doing so, when an abnormality occurs in data conversion, it becomes possible to perform recovery without being affected by the error location, the error content, and the change of the record order from the collected data.

  As described above, according to the present invention, when a conversion abnormality occurs in the data conversion flow, only the abnormal record at the time of collection can be extracted, so that it is affected by the conversion error location, the error content, and the order of the records from the collected data. In addition, it is not necessary to specify recovery target data when a large number of errors occur.

  Also, in the recovery execution of the data conversion flow, it is necessary to prepare all the data depending on the conversion processing (sorting and aggregation), but the processing is performed only with the correction data (record) immediately before the conversion processing to prepare all the data. Recovery time can be shortened.

  In addition, the conversion process to collect all data is determined and held as a data conversion flow interruption point, and by waiting control until abnormal data (record) is removed, the output data of the data conversion flow is not fixed. The recovery operation of only the correction data can be repeatedly performed while keeping the data.

1 shows a basic configuration of a data conversion system according to an embodiment of the present invention. The data structure example of the flow execution management table of the data conversion which becomes embodiment of this invention is shown. The example of a data structure of the storage file management table which becomes embodiment of this invention is shown. The record management information of the accumulation | storage file which becomes embodiment of this invention is shown. It is a figure (Example 1) explaining the flow of the data conversion at the time of the occurrence of the error which becomes embodiment of this invention. It is a figure (Example 1) explaining the correction procedure of the conversion error data by the user at the time of the data conversion error which becomes embodiment of this invention. The structure (Example 1) of the recovery process by the recovery execution operation of the data conversion flow which becomes embodiment of this invention and resource reuse is shown. 1 shows an overall flow of data conversion (Example 1) in a data conversion system according to an embodiment of the present invention. 7 shows a normal data holding point determination processing flow (embodiment 1) according to the embodiment of the present invention. The processing flow of record identification information addition which becomes embodiment of this invention is shown. The processing flow (Example 1) of the data conversion flow execution which becomes embodiment of this invention is shown. The processing flow of the abnormal record information output which becomes embodiment of this invention is shown. 6 shows a processing flow of abnormal data extraction / storage according to an embodiment of the present invention. The recovery conversion processing flow of the data conversion which becomes embodiment of this invention is shown. The processing flow of interruption resumption which becomes embodiment of this invention is shown. The figure (Example 2) explaining the flow of the data conversion at the time of the error generation which becomes embodiment of this invention is shown. The figure (Example 2) explaining the correction procedure of the conversion error data by the user at the time of the data conversion error which becomes embodiment of this invention is shown. The structure (Example 2) of the recovery process by the recovery execution operation and resource reuse of the data conversion flow which becomes embodiment of this invention is shown. The whole flow (Example 2) of the data conversion in the data conversion system which becomes embodiment of this invention is shown. The normal data retention point determination processing flow (Example 2) according to the embodiment of the present invention will be described. The processing flow (Example 2) of the data conversion flow execution which becomes embodiment of this invention is shown. The processing flow continuation (Example 2) of the data conversion flow execution which becomes embodiment of this invention is shown. The structure of the conventional data conversion system is shown.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Regarding a data conversion system that combines a collection data tracking mechanism and a recovery mechanism using a storage file mechanism (data retention / reuse / management) that accumulates and manages collection data and intermediate data for a certain period of time, Two examples will be described below.

  In the first embodiment, when a conversion error occurs in the data conversion flow, except for the error record, the other data (records) continues the conversion process of all records up to the data holding point, and only the corrected record is processed at the time of recovery. And handle the method of converting all data when the data retention point is reached.

In the second embodiment, in the data holding point waiting process according to the first embodiment, the influence of the subsequent conversion process is determined, and if there is no influence, the process is continued immediately before the end of the flow to shorten the recovery time. Handle.
Example 1
FIG. 1 shows a basic configuration of a data conversion system according to an embodiment of the present invention. This data conversion system includes a data conversion apparatus 1, a data supply source system 2 (systems 1 and 2), and a data supply destination system 3 (systems 3 and 4) that are connected to a network such as a LAN (Local Area Network). Is done.

  The data conversion apparatus 1 collects data from the data supply source system 2, uses the storage file mechanism, retains / manages data collected by the data collection / storage unit 10 and the storage file 100, and performs the next data conversion flow processing unit 4, the record data adding unit 11 takes out the collected data and adds a record identification number that is unique within the apparatus to the data.

  Thereafter, the conversion processing is performed with the record identification numbers added by the plurality of conversion units 12a, 12b. When all the conversions are normally completed, the record identification information cutting unit 13 separates the record identification number from the data and stores it as distribution data in the accumulation file.

  When a conversion error occurs in the data conversion flow, except for the error record, the other data (record) continues to be processed until the data holding point (waiting point) managed by the flow execution management table 200 is reached. Continue the conversion process for the current record. At the data holding point, the flow definition contents are referred to at the start of the data conversion flow, and the presence / absence and arrangement of conversion processing (sorting and counting) that require batch processing of all records are determined.

  When only conversion processing that does not include batch processing of all records is performed, the data retention point is managed immediately before the completion of the data conversion flow, and when batch processing of all records is included, the data immediately before is managed as a data retention point. .

  In the recovery operation, in response to the user's recovery data retrieval operation, the storage / extraction processing unit 15 extracts only the abnormal record on the collected data while referring to the abnormal record information, and the user receives a correction record (including an uncorrected record). ) Is stored again.

  In the recovery execution, regardless of the error location and the error content, the data conversion flow is reprocessed only for the correction record in accordance with the user's data conversion flow re-execution operation. When the process reaches the data holding point, the recovery processing unit 14 aggregates data with normal records in the middle of the conversion process, determines that the recovery operation is completed based on whether or not abnormal record information is lost, and triggers it. All data conversion processing is performed.

  Here, although not shown, the data conversion device 1 is a computer including a CPU (Central Processing Unit) and a memory, and a program for performing the above-described data conversion processing is an auxiliary storage device (see FIG. Are loaded into the memory when the power is turned on and executed by the CPU. The program is stored in a storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disk), a flexible disk, or an MO (Magneto Optical Disk), and is loaded into the data conversion apparatus 1 by a drive device of each medium. You may comprise as follows.

FIG. 2 shows an example of the data structure of the data conversion flow execution management table according to the embodiment of the present invention. The flow execution management table 200 manages the data conversion flow, and includes items of “execution ID (Identification)”, “processing state”, “normal data holding point”, and “processing continuation”.
“Execution ID” is the serial number for each execution of the data conversion flow, “Processing state” is the processing state of the data conversion flow (during execution, suspension, or restarting), and “Normal data retention point” is the normal conversion data for recovery The “holding point” and “continue processing” indicate a flag indicating whether or not the processing is continued until the final conversion processing even if the data holding point exists.

  FIG. 3 shows an example of the data structure of the accumulated file management table according to the embodiment of the present invention. The accumulated file management table 101 manages the status of the accumulated file 100. The “accumulated file ID”, “data box name”, “collection source system No.”, “conversion flow ID”, “data status”, and It consists of items of “accumulated file record management information”.

  “Storage file ID” is the management record ID of the storage file (unique serial number for each collection file / storage file), “Data box name” is an arbitrary name for each data type of the storage file, “Collection source system No. "Collection source system number", "conversion flow ID" is a serial number for each data conversion flow execution, and "data state" is a data processing state in units of accumulated file IDs (stored: data storage completed, extracting: data extraction Medium (extraction process in progress / interrupted state), extracted: data extraction complete), and “accumulated file record management information” indicate an abnormal record management / error information output file linked to the conversion flow ID.

  FIG. 4 shows record management information of the accumulation file according to the embodiment of the present invention. The accumulated file record management information 101a includes: a: accumulated file ID−serial number for each accumulated file ID / b: invalid record flag (ING: invalid record ON) / c: collection source system number / d: conversion process number of conversion error occurrence / E: Error item number / f: Abnormal record collection data record length / g: Conversion error type (code, cleansing, etc.) / h: Syslog (system log) error message (conversion content, error item, error content), etc. It is composed of a plurality of CSV (Comma Separated Value) format information records.

  FIG. 5 is a diagram (Example 1) for explaining the flow of data conversion when an error occurs according to the embodiment of the present invention. In this example, the procedure of the preparation process for improving the recovery efficiency when a conversion abnormality occurs in the execution of the data conversion flow is described.

  When data is collected from the data supply source system, the data is stored using the storage file 100 in preparation for data reuse at the time of data collection. Information management of the accumulation file 100 is performed by the accumulation file management table 101.

  When the data conversion flow execution is started by the user's data conversion flow execution (ID = 42) instruction, in the sort / aggregate conversion processing, it is necessary to process all the records together, and normal for recovery The data holding point discriminating unit 20 discriminates the waiting point (normal data holding point) from the conversion flow definition and holds it in the data conversion flow execution management table 200.

  The normal data holding point determination unit 20 detects all data batch conversion processing (conversion type such as sorting / aggregation) that appears first, and holds the normal data holding point in the flow execution management table 200.

  When data processing of the data conversion flow is started, the data extraction unit 21 extracts target data from the accumulation file 100, and the record information addition unit 11 adds record information to the extracted data. More specifically, a record number is added to the storage file ID managed for each collected file, and the ID for each record is added to the data. Thereafter, management for each record is performed only for an abnormal record when an abnormality occurs.

  By embedding the storage file ID and the record number for each storage file ID as an ID (identifier) for each record of data, it is possible to track the conversion error record in the conversion flow.

  However, in this mechanism, data management information is held in units of collected files, and performance is improved by reducing the number of accesses and capacity is reduced by reducing the number of management information.

  Next, conversions 12a and 12b are executed. At this time, if a data conversion error occurs (record 2 is a code conversion error and record 4 is a cleansing error), the abnormal record information is written in the accumulated file record management information 101a, and only the abnormal record is managed for each record. . On the other hand, the normal records in the conversions 12a and 12b continue processing up to the normal data holding point that controls the waiting 16 and are held in the accumulated file 100 to set the “data state” in the accumulated file management table 101 to “stored”. Data conversion flow processing is interrupted (waiting for recovery).

  FIG. 6 is a diagram (Example 1) for explaining a procedure for correcting conversion error data by a user at the time of a data conversion error according to an embodiment of the present invention.

  FIG. 6 shows a procedure of processing from data extraction to storage accompanying user data correction following FIG. When the user specifies the data conversion flow execution ID = 42 from the accumulated file 100 and performs the data extraction operation 01, the recovery data storage / extraction unit 15 executes the flow of the accumulated file management table 101 in the extraction process (recovery) 15a. The accumulated file record management information 101a linked to the ID is acquired. An abnormal record ID is acquired from the accumulation file 100, an offset (position) of the abnormal record is specified from the accumulation file 100 based on the ID (accumulation file ID + record number), and a record ID is added to the extracted data file 100 ′. (The accumulated file record management information is also output so that the user can know the extracted data information). With this extraction process, it is possible to collect and extract only abnormal records from records at the time of data collection without being affected by the location of the conversion error and the error contents.

  Next, when the user corrects data 03 (only record 2 is corrected and record 4 is discarded as an invalid record) and the data conversion flow execution ID is specified and the storage data file is stored, the added record ID is added. The record is replaced by specifying the offset (position) in the accumulation file 100 of the corrected record.

  Since records that are not included in the stored data are treated as invalid records in the subsequent recovery, an invalid record flag is set in the accumulated file record management information 101a (file A). In data extraction / storage operations, it is possible to specify and reuse collected records, so that repeated recovery operations can be performed, and only some abnormal records can be corrected and recovery can be executed. Error recovery is possible.

  FIG. 7 shows the configuration (Example 1) of the recovery process by the recovery execution operation and resource reuse of the data conversion flow according to the embodiment of the present invention. This example shows the procedure of the recovery execution process that continues from FIG.

  In the recovery execution operation of the user, even if the location of the conversion error is scattered or the cause of the error is different (data error / invalid data), it is possible to identify the error record at the time of collection and reuse / replace it. The recovery operation can be performed in the same procedure as the data conversion flow re-execution.

  In the recovery execution process (execution ID = 42), the data extraction unit 21 from the storage file 100 refers to the storage file record management information 101a and extracts all correction records (including uncorrected records) corresponding to the abnormal records. (Invalid records are suppressed from being extracted).

  The record information adding unit 11 for the extracted data adds the record ID of the abnormal record corresponding to the correction record with reference to the accumulated file record management information 101a.

  The conversion units 12a and 12b perform the conversion process only on the correction record, and the conversion process up to the normal data holding point managed by the data conversion flow execution management table 200. In the waiting process of the interruption / resumption process 22, when it is determined that all the abnormal records in the accumulated file record management information 101a have disappeared, the corresponding record is deleted and the subsequent process is performed.

  In the merge process of the recovery process (suspend / resume process 22), the corrected record and the normally processed data held in the accumulated file 100 are merged, and the sort process is performed in the order of the record ID (accumulated file + serial number) at the start of the flow. Return data in the order of records at the beginning of the flow. This sort processing has very little processing overhead because normal records are equivalent to already sorted records.

  Thereafter, after performing conversion 3 (aggregation) with all data, the record information cut processing unit 13 cuts the record ID and completes the data conversion flow.

  Hereinafter, the flow of each conversion process in the data conversion system will be described with reference to FIGS.

FIG. 8 shows an overall flow of data conversion (Example 1) in the data conversion system according to the embodiment of the present invention. Processing of each step (S11 to S23) is as follows.
S11: The data conversion apparatus 1 collects data from the data supply source system 2, holds the collected data in the accumulation file 100, and manages the data by the accumulation file management table 101.
S12: The data conversion process flow is started by an input operation by the user.
S13: A normal data holding point in the data conversion flow is determined (details are shown in FIG. 9).
S14: Data is extracted from the accumulation file 100, and record identification information addition processing for assigning an ID that can be identified in the system for each record is performed (details are shown in FIG. 10).
S15: A data conversion flow execution process is performed (details are shown in FIG. 11).
S16: It is determined whether or not there is an abnormality in each data conversion process.
S17: If there is an abnormality, extraction / storage processing of abnormality data including data correction by the user is performed (details are shown in FIG. 13).
S18: A data conversion flow process for recovery is performed (details are shown in FIG. 14).
S19: The record identification information cutout unit 13 cuts out the record unit identification information added in step S14 after performing conversion 3 (aggregation) with all data.
S20: The above processing result is output as the distribution file 300, and the data conversion flow is completed.
S21: Distribute the distribution data to the supply destination system 3.

FIG. 9 shows a normal data holding point determination processing flow (Example 1) according to the embodiment of the present invention.
S31: Read each conversion definition content and conversion processing order of the data conversion flow.
S32: It is determined whether or not there is a conversion (sort / aggregate) that requires waiting for all records in the conversion flow.
S33: Search for a conversion process that requires waiting for all the records appearing first in the data conversion flow.
S34: Immediately after finishing the conversion process immediately before the searched conversion process, it is determined as a normal data holding point and set in the data conversion flow execution management table 200.
S35: If there is no conversion requiring waiting for all records in step S32, a normal data holding point is set immediately after the final conversion process of the data conversion flow (set in the management information for each execution of the data conversion flow). .

FIG. 10 shows a processing flow for adding record identification information according to the embodiment of the present invention.
S41: Data to be processed in the first data conversion flow is extracted from the accumulation file 100 to a temporary file.
S42: Cut out from the temporary file for each record unit.
S43: Write the data conversion flow ID (numbered every execution) to the management record in units of storage file ID (ID for each collected file) in the storage file management information 101a.
S44: One item is added to the head of the record, storage file ID (upper 8 bits) + record number (lower 8 bits) in units of storage file ID is set, and written to the input file for subsequent conversion processing. Thereby, each record of the accumulated file information for each collected file is managed by the record number.
S45: It is determined whether the identification information adding process to the final record for the extracted data is completed. If not completed, the process returns to step S42 and the subsequent processes are repeated.

FIG. 11 shows a processing flow (Example 1) for executing the data conversion flow according to the embodiment of the present invention.
S51: It is determined whether the normal data holding point is immediately after the conversion process.
S52: If the normal data holding point is immediately after the conversion process, the storage file is assigned to the output destination of the conversion process.
S53: A new data file management record is added to the accumulated file management information.
S54: If it is not a normal data holding point in step S51, the output destination of each conversion process is a memory or a temporary work file.
S55: The conversion process is executed for each record.
S56: It is determined whether or not the processing has been normally performed.
S57: If there is an abnormality in the conversion in step S56, the abnormal record information output process is performed without converting the abnormal record (details are shown in FIG. 12).
S58: It is determined whether it is the last record in the conversion process. If it is not in the last record, the process returns to step S55 and the subsequent processing is repeated.
S59: If it is the final record, it is determined whether it is after the normal data holding point or in the final conversion process of the data conversion flow. If it is not final, the process returns to step S51 and the subsequent processing is repeated.
S60: If it is final, it is determined whether or not an abnormal record remains in the accumulated file record management information.
S61: If an abnormal record remains, the storage file of the storage file management information is stored, and the data conversion flow process is interrupted.
S62: If no abnormal record remains, it is determined whether or not the recovery flow is being executed. If the recovery flow is not being executed, this flow ends.
S63: When the recovery flow is being executed, an interruption / resumption process is performed (for details, see FIG. 15).
S64: It is determined whether the data conversion process is final. If the data conversion process is not final, the process returns to step S51 to repeat the subsequent processes.

At the time of recovery, the flow of (A) is run following step S56.
S65: It is determined whether recovery is in progress. If recovery is not in progress, the flow (A) ends.
S66: If recovery is in progress, if there is corresponding abnormal record information in the stored file management information and the conversion process is after the occurrence of an error, the abnormal record information is deleted.

FIG. 12 shows a processing flow of output of abnormal record information according to the embodiment of the present invention.
S71: The record ID (accumulation file ID + serial number), error conversion number, error content, etc. are embedded in the syslog (system log) and an error message is output.
S72: It is determined whether there is accumulated file record management information linked to the corresponding flow ID in the accumulated file management information.
S73: If there is accumulated file record management information, the following CSV format information record is added to the accumulated file record management information (one record is added).

Record ID (accumulated file ID + serial number), invalid record flag, collection source system, record length, error conversion number, error item number, record length,.
S74: When there is no accumulated file record management information, the accumulated file record management information is generated, and the generated file name is set in the management record of the corresponding flow ID in the accumulated file management information.

FIG. 13 shows a processing flow of abnormal data extraction / storage according to the embodiment of the present invention.
S81: The user specifies the data conversion flow ID (ID for each execution) in which an error has occurred and performs data extraction.
S82: The storage file record management information is specified from the management record of the corresponding flow execution ID of the storage file management information.
S83: One record information (CSV format) is read from the accumulated file record management information, and an abnormal record ID is specified.
S84: An arrangement offset is specified from the collected data managed in the storage file from the storage file ID corresponding to the record ID and the serial number (record number).
S85: The corresponding record with the record ID added is added to the extracted data file.
S86: It is determined whether it is the last record of the accumulated file record management information. If it is not the last record, the process returns to step S83 and the subsequent processing is repeated.
S87: If it is the last record, the accumulated file record management information is copied and output.
S88: The user is made to correct the abnormal record (leave the added record ID).
S89: The user stores the stored data file (corrected record file) by specifying the data conversion flow ID (ID for each execution) in which an error has occurred.
S90: The record ID is read from the stored data file.
S91: An arrangement offset is specified from the collected data managed in the storage file based on the storage file ID corresponding to the record ID and the serial number (record number).
S92: The record ID is cut out, and the collected data of management is overwritten with the correction record in the storage file with the specified offset.
S93: The comparison result between the record ID in the stored file record management information and the record ID in the stored data file is held.
S94: It is determined whether or not the comparison of all the records in the stored file record management information is completed. If the comparison is not completed, the process returns to step S90 and the subsequent processing is repeated.
S94: If the comparison is completed, an invalid record flag (IGN) is set when the record ID in the stored file record management information is more than the record ID in the stored data file.

FIG. 14 shows a recovery execution processing flow of data conversion according to the embodiment of the present invention.
S101: The user designates the data conversion flow ID (ID for each execution) in which an error has occurred and re-executes the data conversion flow.
S102: An abnormal record ID is specified from the accumulated file record management information (invalid records are ignored).
S103: The arrangement offset is specified from the collected data managed in the accumulation file from the accumulation file ID corresponding to the record ID and the serial number (record number).
S104: Record identification information addition processing is performed (details are shown in FIG. 10).
S105: Data conversion flow execution processing (details are shown in FIG. 11) and interruption / resumption processing are performed (details are shown in FIG. 15).

FIG. 15 shows a processing flow for resuming interruption according to the embodiment of the present invention.
S112: The saved data is specified from the accumulated file allocation ID at the time of saving the record, and copied to the data output destination file (the output record of the subsequent recovery conversion process is additionally written).
S113: Sort record IDs as keys and return to the order of the flow start time.
(Example 2)
Hereinafter, with reference to FIGS. 16 to 22, a method for determining the influence of the subsequent conversion process at the time of a conversion error and continuing the process until immediately before the end of the flow will be described with a focus on differences from the first embodiment.

  FIG. 16 is a diagram (Example 2) for explaining the flow of data conversion when an error occurs according to the embodiment of the present invention.

  First, the normal data retention point determination unit 20 refers to the flow execution management table 200 to determine whether or not there is an instruction to continue the error up to the normal data retention point. If error continuation is specified, in the waiting process 16 when a conversion error occurs before the sort / aggregate conversion process, an error occurrence item and a sort / aggregate process key item (for example, record data) to be processed thereafter If there is no relation, the subsequent processing is continued without waiting.

  At this time, the abnormal record continues to the final conversion process of the data conversion flow while processing the item data other than the error item.

  In the example of FIG. 16, the item relation information of the data conversion flow is searched to determine whether or not the subsequent conversion process can be continued. Since an error occurs in an item that does not affect the key item 3 of the subsequent sort process, the waiting process is performed. Is ignored. Then, in conversion 3, the item 3 is sorted, and the processing result is output as data B to the storage file 100. Further, the abnormal record is stored in another area of the accumulation file 100.

  Further, as a procedure for performing the recovery, the user corrects the abnormal record and performs a recovery operation in the same manner as in the first embodiment. In the recovery execution process, the storage file 100 is not subjected to the sort / aggregate process again. The conversion process is completed by replacing the retained abnormal record with the correction record.

  According to the present invention described above, even if conversion processing (sorting / aggregation) for processing all records after the conversion error occurs, it is possible to determine whether or not to continue the processing (waiting / not affecting) in the waiting processing. In this case, by adopting a configuration in which the conversion process is continued, in the recovery process, all records are not processed from the middle of the data conversion flow, and the recovery time is further shortened compared to the first embodiment. Is possible.

  FIG. 17 is a diagram (Example 2) for explaining the conversion error data correction procedure by the user at the time of a data conversion error according to the embodiment of the present invention.

  In this embodiment in which the influence of the subsequent conversion process is determined at the time of the conversion error and the process is continued until just before the end of the flow, the configuration of the correction process of the conversion error data by the user is the same as that in FIG. In the example, it is shown that the error record 4 with the accumulation file ID = 852 is extracted by the extraction process 15a, and is replaced and re-stored by the storage process 15b by the record 4x that is corrected and input by the user.

  FIG. 18 shows a configuration (Example 2) of the recovery process by the recovery execution operation and resource reuse of the data conversion flow according to the embodiment of the present invention.

  An abnormal record in the accumulation file 100 is replaced with a correction record (record 4x) by the user, and correction is performed even when there is a conversion process that originally needs to wait for all records, such as sorting and aggregation. The recovery time can be further shortened by performing the recovery process for only the record.

  FIG. 19 shows an overall flow (Example 2) of data conversion in the data conversion system according to the embodiment of the present invention. In the overall flow, the normal data retention point determination process (detailed in FIG. 20) in step S203 and the data conversion flow execution process (detailed in FIG. 21) in step S205 are different, but the subsequent process is the above-described FIG. This is the same as Example 1. The processing of steps S203 and S205 will be described in detail below with reference to FIGS.

FIG. 20 shows a normal data holding point determination processing flow (embodiment 2) according to the embodiment of the present invention. Steps S221 to S225 are the same processing as that of the first embodiment of FIG. 9 described above, and thus the following description is omitted.
S226: It is determined whether there is an error continuation designation as the definition content of the data conversion flow.
S227: If there is an error continuation designation, a flag “present” is set in the “process continuation” item of the flow execution management table 200 as management information for each execution of the data conversion flow. As a result, the data conversion flow is continued until immediately after the final conversion process.

  FIG. 21 shows a processing flow (Example 2) for executing the data conversion flow according to the embodiment of the present invention. In the following, a part of the data conversion flow execution process that is different from the first embodiment of FIG. 11 will be described.

After executing the conversion process for each record, it is determined in step S236 whether or not the process has been normally performed.
S243: When there is an abnormality in the process, the error record number and the item number are output with the abnormal record information as the record management information 101a according to the flow of FIG.
S244: Referring to the flow execution table 200, it is determined whether or not processing continuation is designated.
S245: If there is no processing continuation designation, conversion processing for abnormal records is not performed.
S246: If processing continuation is specified, data other than the abnormal item is output as a conversion processing result record, while no data (blank or 0) is set for the abnormal item, and then the next record is recorded until the final record is reached. Process.

In step S247 for determining whether or not an abnormal record remains in the accumulated file record management information, the following processing is performed if it remains.
S247: It is determined whether (the process continuation designation is specified) and (other than the final conversion process of the data conversion flow). If it is not final conversion processing, the following processing is performed.
S248: The definition information of each conversion item is searched, and key item information whose subsequent conversion type is “sort” or “total” is acquired.
S249: An item list linked to the key item is extracted from the mapping information (item link information) between the item definition information of each conversion (item corresponding to each conversion).
S250: It is determined whether the key item is an item linked to the conversion error item. If the item is linked to the conversion error item, in step S251, the storage file of the storage file management information is stored, the data conversion flow process is interrupted, and this flow is ended.

  If the key item is not an item linked to the conversion error item, the process returns to step S231 and the subsequent processing is repeated.

  Note that the determination processing in step S250 is repeated for the error records.

  On the other hand, if the data conversion flow is the final process in step S247, the stored file of the stored file management information is stored in step S251, and the data conversion flow process is interrupted.

FIG. 22 shows a processing flow continuation (Example 2) of the data conversion flow execution according to the embodiment of the present invention. In this example, in step S236 of FIG. 21, when there is no abnormality in the conversion process, a process flow (B) that is executed during recovery between S236 and S247 is shown.
S261: It is determined whether or not this process is being executed for recovery. If recovery is not in progress, the process (B) is not applied. If recovery is in progress, the following process is performed.
S262: If the storage file management information includes the corresponding abnormal record information and the conversion process is performed after the error occurs, the abnormal record information is deleted.
S263: It is determined whether (the processing continuation is specified) and (the final conversion processing). If it is not final processing, the flow of (B) is ended, and if it is final processing,
S264: The record ID is searched from the saved accumulated file data, and the offset of the corresponding abnormal record is specified.
S265: Replace the abnormal record with the normal conversion record.

  Data utilization field for collecting / processing / distributing data between multiple systems

DESCRIPTION OF SYMBOLS 1 Data converter 2 Supply source system 3 Supply destination system 4 Data conversion flow process part 10 Data collection storage part 11 Record identification information addition part 12 Conversion part 13 Record identification information clipping part 14 Recovery process part 15 Recovery data storage extraction part 20 Normal Data retention point 21 Data extraction unit 50 Recovery execution 100 Accumulated file 101 Accumulated file management table 101a Record management information 102 Accumulated data (normal processing data)
103 Abnormal record 200 Flow execution management table 300 Output file

Claims (5)

A data conversion program for converting data collected from a supply source system into a computer accessible to a file storage means for holding data and causing a supply destination system to execute supply processing,
In the computer,
A record information adding step of extracting target data from the file storage means and adding a record identification number to the data;
A data conversion step of performing a plurality of conversion processes for each record on the data to which the record identification number is added;
When an error occurs in the conversion process, a conversion continuation step of continuing the other records up to a predefined meeting point, excluding the error record,
A conversion reprocessing step of acquiring a correction record corrected by the user with respect to the error record, and performing a reprocessing of conversion on the correction record;
A data conversion program for executing a distribution data output step of cutting out the record identification number from the data and outputting it as distribution data when conversion of all records in the data is normally completed.   2. The data conversion program according to claim 1, wherein the waiting point is set in a conversion processing unit that needs to perform batch processing when all of the plurality of conversion processes are completed.   The conversion continuation step when an error occurs determines whether or not there is an influence on a subsequent conversion process, and if there is no influence, ignores the waiting point and continues the process until immediately before the end of the flow. 2. The data conversion program according to 2. A data conversion method in which a computer accessible to a file storage means for holding data converts data collected from a supply source system and executes supply processing to the supply destination system.
The computer is
A record information adding step of extracting target data from the file storage means and adding a record identification number to the data;
A data conversion step of performing a plurality of conversion processes for each record on the data to which the record identification number is added;
When an error occurs in the conversion process, a conversion continuation step of continuing the other records up to a predefined meeting point, excluding the error record,
A conversion reprocessing step of acquiring a correction record corrected by the user with respect to the error record, and performing a reprocessing of conversion on the correction record;
A data conversion method characterized by executing a distribution data output step of cutting out the record identification number from the data and outputting it as distribution data when conversion of all records in the data is normally completed. A data conversion device that converts data collected from a supply source system and supplies the converted data to a supply destination system,
File storage means for holding the collected data;
Record information adding means for extracting target data from the file storage means and adding a record identification number to the data;
Data conversion means for performing a plurality of conversion processes for each record on the data to which the record identification number is added;
If an error occurs in the conversion process, conversion continuation means for continuously processing other records up to a predefined meeting point, excluding error records;
A conversion reprocessing means for acquiring a correction record corrected by the user with respect to the error record, and performing a reprocessing of conversion on the correction record;
A data conversion apparatus comprising distribution data output means for cutting out the record identification number from the data and outputting it as distribution data when conversion of all the records in the data is normally completed.

Images