JP2013196404A - Data processing apparatus, data processing method and data processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the developing efficiency of processing definition for performing the extraction/working/writing of data by making source definition selectable.SOLUTION: The data processing apparatus includes: a source definition information storage section 131 for defining a source table 351, and storing source definition 133 including source location information on a data source 300 in which the source table 351 is present and source definition ID132 in association; a display section 120 for displaying a list of the source definition ID132 on a display screen 121; an input section 110 for receiving input of at least one source definition ID132 selected from the list of the source definition ID132; and a table access section 140 for extracting the source definition 133 corresponding to the at least one input source definition ID132 from the source definition information storage section 131, and performing access to the source table 351 defined by the extracted source definition 133 for the data source 300 shown by the source location information included in the extracted source definition 133.

Description

  The present invention relates to a data processing technique that extracts data from a source data source, processes the extracted data, and writes the extracted data in an output destination database or the like. More specifically, for example, the present invention relates to a data processing technique used for an ETL (Extract, Transform, Load) tool.

  The ETL tool uses the data reading definition (source) stored in advance to define data processing for extraction, processing, and writing in the data processing editing definition (mapping). Normally, at the start of processing, the ETL tool opens a database session or opens a flat file for all objects (database tables and flat files) corresponding to pre-stored data reading definitions (sources). Perform access.

  In the prior art, there is a technology that enables a user to input an extraction condition for a data reading definition (source) (see Patent Document 1). In addition, there is a technique for automatically generating an extraction condition program in accordance with an extraction condition input by a user (see Patent Document 2).

JP 2011-008811 A JP 2000-315151 A

  In the conventional ETL tool, as described above, all objects (database tables and flat files) corresponding to the data reading definition (source) are accessed. When there is something that does not exist, such as being processed, there is a problem that processing is interrupted or processing cannot be started.

  Even if there is a definition (source) of data reading that is not used during execution of data processing editing definition (mapping) processing, access is also made to the object corresponding to the definition (source) of unused data reading. Therefore, there is a problem that wasteful resources are consumed.

  In addition, when extracting data with different extraction conditions from the same object (database table or flat file), the definition of data reading (source) must be rewritten and executed, resulting in inferior processing efficiency. is there.

  In addition, when the extraction condition can be input from the user, there is a problem that the processing may cause an error, for example, an object that does not exist is specified due to an input error of the user.

  One of the main objects of the present invention is to solve the above-described problems. By making it possible to select a definition (source) of data reading stored in advance, data extraction / processing / writing is possible. An object of the present invention is to provide a data processing apparatus capable of efficiently executing the process of performing the above.

A data processing apparatus according to the present invention is a data processing apparatus that accesses a plurality of source tables in an acquisition source database that is an acquisition source of data,
Associating a source definition program that defines a plurality of source tables, acquisition source location information that indicates the location of an acquisition source database in which each source table of the plurality of source tables exists, and a source identifier that identifies the source definition program A definition program storage unit stored in the storage device;
A display unit that extracts a source identifier according to a predetermined condition from the source identifiers stored in the definition program storage unit, and displays a list of the extracted source identifiers on a display device;
An input unit for inputting at least one source identifier selected from the source identifier list displayed by the display unit;
At least one source identifier input by the input unit is input, and the source definition program and the source location information corresponding to the input at least one source identifier are extracted from the definition program storage unit and extracted. A table access unit for accessing a source table defined by the extracted source definition program with respect to an acquisition source database indicated by the acquisition source location information;

The data processing device further includes:
A database determination unit that determines whether or not the acquisition source database is deleted from the location of the acquisition source database indicated by the acquisition source location information stored in the definition program storage unit;
The display unit
Among the source identifiers stored in the definition program storage unit, the source identifier corresponding to the acquisition source location information indicating the acquisition source database determined not to be deleted by the database determination unit is extracted, and the extracted A source identifier list is displayed on a display device.

The display unit further displays on the display device a selection screen for selecting at least one of the source identifiers from the displayed source identifier list,
The input unit is
At least one of the source identifiers selected on the selection screen displayed by the display unit is input.

The definition program storage unit
The source definition program describing extraction conditions for extracting each source table of the plurality of source tables from the acquisition source database indicated by the acquisition source location information, wherein the plurality of source definitions describe mutually exclusive extraction conditions A program is memorized.

The data processing device further includes:
A data writing unit that processes data in the source table accessed by the table access unit and writes the processed data to at least one output table in an output table in a predetermined output destination database. And

The definition program storage unit further includes:
An output definition program that defines the output table, output location information indicating the location of an output destination database where the output table is located, and an output identifier that identifies the output definition program are stored in a storage device in association with each other,
The display unit further extracts an output identifier according to a predetermined condition from the output identifiers stored in the definition program storage unit, and displays the extracted output identifier list on a display device,
The input unit further inputs at least one output identifier selected from the output identifier list displayed by the display unit,
The table access unit inputs at least one output identifier input from the input unit, and outputs the output definition program and the output location information corresponding to the input at least one output identifier from the definition program storage unit. Extracting, accessing the output destination database indicated by the extracted output location information, accessing the output table defined by the extracted output definition program;
The data writing unit writes the processed data into the output table accessed by the table access unit.

The data processing method according to the present invention identifies a definition program that defines a plurality of source tables, source location information that indicates the location of a source database in which each source table of the plurality of source tables is located, and the definition program A data processing method of a data processing apparatus including a definition program storage unit that stores a source identifier in association with each other,
A list display step of extracting a source identifier according to a predetermined condition from the source identifiers stored in the definition program storage unit, and displaying the extracted source identifier list on a display device;
An input step for inputting at least one source identifier selected from the source identifier list displayed by the list display step;
The table access unit inputs at least one of the source identifiers input in the input step, and the definition program corresponding to the input at least one of the source identifiers and the acquisition source location information from the definition program storage unit And a table access step of accessing a source table defined by the extracted definition program with respect to an acquisition source database indicated by the extracted acquisition source location information.

The data processing method further includes:
A database determination step of determining whether or not the acquisition source database is deleted from the location of the acquisition source database indicated by the acquisition source location information stored in the definition program storage unit,
The list display step includes:
Among the source identifiers stored in the definition program storage unit, the source identifier corresponding to the source location information indicating the source database determined not to be deleted by the database determination step is extracted, and the extracted A source identifier list is displayed on a display device.

A data processing program according to the present invention identifies a definition program that defines a plurality of source tables, source location information that indicates the location of a source database in which each source table of the plurality of source tables is located, and the definition program A data processing program to be executed by a data processing apparatus that is a computer including a definition program storage unit that stores a source identifier in association with each other,
A list display step of extracting the source identifier according to a predetermined condition from the source identifiers stored in the definition program storage unit, and displaying the extracted source identifier list on a display device;
An input step of inputting at least one source identifier selected from the source identifier list displayed by the list display step;
Input at least one source identifier input in the input step, extract the definition program corresponding to the input at least one source identifier and the acquisition source location information from the definition program storage unit, and extract And a table access step of accessing a source table defined by the extracted definition program with respect to an acquisition source database indicated by the acquisition source location information.

The data processing program further includes:
A database determination step of determining by a processing device whether or not the acquisition source database is deleted from the location of the acquisition source database indicated by the acquisition source location information stored in the definition program storage unit;
The list display step includes:
Among the source identifiers stored in the definition program storage unit, the source identifier corresponding to the source location information indicating the source database determined not to be deleted by the database determination step is extracted, and the extracted A source identifier list is displayed on a display device.

  The data processing apparatus according to the present invention, an input unit for inputting at least one source identifier selected from the list of source identifiers displayed by the display unit, and at least one source identifier input by the input unit, The source definition program corresponding to at least one input source identifier and the source location information are extracted from the source definition information storage unit, and the source database indicated by the extracted source location information is extracted by the extracted source definition program. Since there is a table access unit that accesses the source table to be defined, even if there is a source definition program that is not used, access is not performed to the source database corresponding to the source definition program that is not used. Processing without consuming resources An effect that rate is improved.

  In addition, since source definition programs that are not used can be stored in the source definition program storage unit, even when extracting data with different extraction conditions from the same source database, the source definition program is newly rewritten and executed. There is no need, and the working efficiency is improved.

  Further, the data processing apparatus according to the present invention includes a database determination unit that determines whether or not the acquisition source database is deleted from the location of the acquisition source database indicated by the acquisition source location information stored in the definition program storage unit. The display unit extracts a source identifier corresponding to the source location information indicating the source database determined not to be deleted by the database determination unit from the source identifiers stored in the definition program storage unit, The list of extracted source identifiers is displayed on the display device, so there is no risk of accessing a non-existing source database that has already been deleted, etc., and processing efficiency is improved without interruption. The effect of doing.

  The display unit displays a selection screen for selecting at least one source identifier from the displayed list of source identifiers, and the input unit inputs at least one source identifier selected on the selection screen. Thus, it is possible to reduce input errors by the user, such as designating a non-existing source database, and to improve the work efficiency.

2 is a functional block diagram of the data processing apparatus 100 according to Embodiment 1. FIG. 6 is a diagram showing a configuration of a source definition information storage unit 131 according to Embodiment 1. FIG. 3 is a diagram illustrating an example of hardware resources of the data processing apparatus 100 according to Embodiment 1. FIG. FIG. 4 is a flowchart showing an operation of the data processing apparatus 100 according to the first embodiment. 6 is a diagram illustrating an example of a mapping screen 125 for data processing editing definition in the data processing apparatus 100 according to Embodiment 1. FIG. FIG. 6 is a diagram illustrating an example of a start condition input screen in the data processing apparatus 100 according to the first embodiment, where (a) is a selection screen 123 for inputting a process start condition by a GUI operation, and (b) is a command line parameter. It is a command input screen 124 for inputting processing start conditions. 6 is a functional block diagram of a data processing apparatus 101 according to Embodiment 2. FIG. 10 is a flowchart showing operations of a database determination unit 180 and a display unit 120 according to Embodiment 2. FIG. 6 is a functional block diagram of a data processing apparatus 102 according to Embodiment 3. FIG. FIG. 10 is a diagram showing a configuration of a target definition information storage unit 138 according to Embodiment 3. FIG. 10 is a flowchart showing a source and target input process in the data processing apparatus 102 according to the third embodiment. FIG. 10 is a flowchart showing a table access process in the data processing apparatus 102 according to the third embodiment. 14 is an example of a command input screen displayed on a display screen 121 in the data processing apparatus 102 according to the third embodiment. 14 is an example of a mapping screen 125 displayed on the display screen 121 in the data processing apparatus 102 according to the third embodiment.

Embodiment 1 FIG.
In this embodiment, a data processing device that extracts data from various data sources such as various databases and various files, performs processing, and outputs data suitable for an output destination data warehouse, data mart, etc. 100 will be described. In the present embodiment, processing for extracting data from various data sources will be described in particular.

  FIG. 1 is a functional block diagram of a data processing apparatus 100 according to the present embodiment. Functional blocks of the data processing apparatus 100 will be described with reference to FIG.

  The data processing apparatus 100 is an apparatus that executes processing for realizing, for example, an ETL (Extract, Transform, Loading) tool. The ETL tool has an access method that can access various data sources 300 (for example, Oracle database, flat file, Excel, XML, etc.), and the user can use the data source 300 without being aware of the access method. It is a tool that can extract, process, and write data. The data source 300 is an example of an acquisition source database that is an acquisition source of data.

  In the data processing apparatus 100, a user defines a program (hereinafter referred to as a source definition 133) described using a database language such as SQL in order to extract data from the data source 300 (hereinafter referred to as a source table 351). Is defined in advance. Then, the user creates a data processing edit definition (mapping) using the source definition 133 or the like stored in advance, and executes data extraction / processing / writing processing.

  The data processing apparatus 100 extracts data from the data source 300 (for example, the database 310 and the file 320) according to the source definition 133 stored in advance. Then, the data processing apparatus 100 processes the extracted data and writes it in a table (hereinafter referred to as a target table 451) in the data target 400 (output destination database). There are a plurality of files 320 and files 420 corresponding to the source definition.

  As shown in FIG. 1, the data processing apparatus 100 includes a display unit 120, an input unit 110, a repository unit 130, a table access unit 140, a data extraction unit 150, a data processing unit 160, and a data writing unit 170.

  The repository unit 130 includes a source definition information storage unit 131 that stores a source definition 133 (source definition program) in which a source table 351 extracted from the data source 300 is defined in advance. The source definition information storage unit 131 includes a source definition 133 that defines a plurality of source tables 351, and source location information 134 (source location information) that indicates the location of the data source 300 (an example of the source database) in which each source table 351 exists. An example) (see FIG. 2) and a source definition ID 132 (source identifier) for identifying the source definition 133 are associated with each other and stored in the storage device.

  FIG. 2 is a diagram showing a configuration of the source definition information storage unit 131 according to the present embodiment. The source definition information storage unit 131 is information in which a source definition 133 is associated with a source definition ID 132 that uniquely identifies the source definition 133. The source definition ID 132 may be the name of the source definition 133 or a symbol attached to the source definition 133. The source definition ID 132 only needs to be able to uniquely identify the source definition 133.

  The source definition 133 is a program written in a database language or the like, and defines a source table 351. The definition of the source definition #A is to describe in the source definition #A, for example, as “source definition #A is a table having item 1, item 2, and item 3” in the database language. That is, in the source definition 133, information defining the contents of the source table 351 (for example, information that there is an item 1, item 2, and item 3) and source location information indicating the location of the data source 300 in which the source table 351 is located. 134 (for example, information that the data source 300 is the database 310 source table #A) is included.

  Similarly, the definition of the source definition #B is, for example, to describe in the source definition #B that “the source file having the item X and the item Y is set as the source definition #B”. That is, the source definition 133 includes information defining the contents of the file 320 (for example, information that there is an item X and an item Y) and source location information 134 indicating the location of the data source 300 in which the file 320 is located (for example, The data source 300 is a file 320).

  As shown in FIG. 2, an extraction condition for extracting the source table 351 can be described in the source definition 133. For example, as a source definition 133, “source definition #A” (ID (source definition ID 132): #A) and “source definition #A with condition 1” (ID with ID 1 added to the source definition #A) (ID : # A1), "Source definition with condition 2 #A" with the extraction condition 2 (ID: # A2), and "Source definition with condition 3 #A" with the extraction condition 3 (ID: # A3) Can also be memorized.

  Here, “source definition with condition 1 #A”, “source definition with condition 2 #A”, and “source definition with condition 3 #A” may be extraction conditions having an exclusive relationship. . For example, source definition # A1, which is “source definition #A with condition 1”, is a source table #A for only today's data among source tables #A defined by source definition #A, and “with condition 2” The source definition # A2 that is “source definition #A” is the source table #A of only the data of one month before in the source table #A, and the source definition # A3 that is “source definition #A with condition 3” is The source table #A may be only the source table #A one year before. The source definition information storage unit 131 stores all source definitions 133 that the user is likely to use for data extraction processing, and also stores source definitions 133 in which any extraction condition is added to each source definition 133. it can.

  Returning to FIG. The display unit 120 extracts the source definition ID 132 from the source definition IDs 132 stored in the source definition information storage unit 131 of the repository unit 130 according to a predetermined condition, and a list of the extracted source definition IDs 132 (hereinafter referred to as a source ID list). 122 (see FIG. 6A)) is displayed on the display screen 121 of the display device. For example, the display unit 120 may extract all of the stored source definition IDs 132 as the predetermined condition, or the extraction condition attached to the source definition 133 in order to limit to the items after the predetermined date. It is not necessary to extract the thing more than three years ago.

  Further, the display unit 120 displays a selection screen 123 (see FIG. 6A) for selecting at least one source definition ID 132 from the source ID list 122 of the displayed source definition ID 132 on the display screen 121. The method of selecting the source definition ID 132 used by the user for data processing editing definition (mapping) from the source ID list 122 includes a method of displaying a selection screen 123 using a GUI screen, a command input screen 124 (FIG. 6B). ))) To display as a command line parameter.

  The input unit 110 inputs at least one source definition ID 132 selected from the source ID list 122 (an example of a source identifier list) displayed by the display unit 120. The input unit 110 inputs at least one source definition ID 132 selected on the selection screen 123 or the command input screen 124 displayed on the display screen 121 by the display unit 120.

  Further, the input unit 110 may control the input so that only one of the source definitions in an exclusive relationship such as # A1, # A1, and # A3 can be selected. Also, the input may be controlled so that it can be specified at the same time even in an exclusive relationship.

  The table access unit 140 inputs at least one source definition ID 132 input by the input unit 110, and sets the source definition 133 including the source location information 134 corresponding to the input at least one source definition ID 132 to the source definition of the repository unit 130. The source table 351 defined by the extracted source definition 133 is accessed to the data source 300 (an example of the source database) extracted from the information storage unit 131 and indicated by the extracted source location information 134.

  The above-mentioned “accessing the source table 351 defined by the source definition 133 to the data source 300” means that, for example, when the source definition ID 132 “#A” shown in FIG. Is to execute the source definition #A corresponding to “#A”, establish a session in the database 310, and connect to the source table #A in an accessible manner.

  Therefore, the table access unit 140 accesses only the source table 351 selected by the user, and does not access the source table 351 not selected by the user.

  The data extraction unit 150 extracts data from the source table 351 in accordance with the data processing editing definition defined by the user. The data processing unit 160 processes and edits the data extracted by the data extraction unit 150 according to the data processing editing definition defined by the user. The data writing unit 170 converts the data processed and edited by the data processing unit 160 according to the data processing editing definition defined by the user into a target table 451 (an example of an output table) in the data target 400 (an example of an output destination database). ).

  Next, a hardware configuration example of the data processing apparatus 100 according to the present embodiment will be described. FIG. 3 is a diagram illustrating an example of hardware resources of the data processing apparatus 100 according to the present embodiment. Note that the configuration in FIG. 3 is merely an example of the hardware configuration of the data processing device 100, and the hardware configuration of the data processing device 100 is not limited to the configuration described in FIG. Also good.

  In FIG. 3, the data processing apparatus 100 includes a CPU 911 (also referred to as a central processing unit, a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, or a processor) that executes a program. The CPU 911 receives, for example, a ROM (Read / Only / Memory) 913, a RAM (Random / Access / Memory) 914, a communication board 915, a display device 901, a keyboard 902, a mouse 903, a magnetic disk device 920, and the like via the bus 912. Connected and controls these hardware devices. The display device 901 includes a display screen 121.

  Further, the CPU 911 may be connected to an FDD 904 (Flexible / Disk / Drive), a compact disk device 905 (CDD), a printer device 906, and a scanner device 907. Further, instead of the magnetic disk device 920, a storage device such as an optical disk device or a memory card (registered trademark) read / write device may be used.

  The RAM 914 is an example of a volatile memory. The storage media of the ROM 913, the FDD 904, the CDD 905, and the magnetic disk device 920 are an example of a nonvolatile memory. These are examples of the storage device.

  The “repository unit” and “source definition information storage unit” described in the present embodiment are realized by the RAM 914, the ROM 913, the magnetic disk device 920, and the like.

  A communication board 915, a keyboard 902, a mouse 903, a scanner device 907, an FDD 904, and the like are examples of input devices. Further, the communication board 915, the display device 901, the printer device 906, and the like are examples of an output unit and an output device. The display device 901 includes a display screen 121.

  The communication board 915 is connected to the network. For example, the communication board 915 is connected to a LAN (local area network), the Internet, a WAN (wide area network), a SAN (storage area network), and the like.

  The magnetic disk device 920 stores an operating system 921 (OS), a window system 922, a program group 923, and a file group 924. The programs in the program group 923 are executed by the CPU 911 using the operating system 921 and the window system 922.

  The RAM 914 temporarily stores at least part of the operating system 921 program and application programs to be executed by the CPU 911. The RAM 914 stores various data necessary for processing by the CPU 911.

  The ROM 913 stores a BIOS (Basic Input / Output / System) program, and the magnetic disk device 920 stores a boot program. When the data processing apparatus 100 is activated, the BIOS program in the ROM 913 and the boot program in the magnetic disk device 920 are executed, and the operating system 921 is activated by the BIOS program and the boot program.

  The program group 923 stores programs that execute the functions described as “˜unit” (other than “repository unit 130” in the following) in the description of the present embodiment. The program is read and executed by the CPU 911.

  In the description of the present embodiment, the file group 924 includes “determination of”, “determination of”, “comparison of”, “reference of”, “extraction of”, “designation of”, Information, data, signal values, variable values, and parameters indicating the results of the processing described as “specification”, “selection”, etc. are stored as items of “˜file” and “˜database”. ing. The “˜file” and “˜database” are stored in a recording medium such as a disk or a memory. Information, data, signal values, variable values, and parameters stored in a storage medium such as a disk or memory are read out to the main memory or cache memory by the CPU 911 via a read / write circuit, and extracted, searched, referenced, compared, and calculated. Used for CPU operations such as calculation, processing, editing, output, printing, and display. Information, data, signal values, variable values, and parameters are stored in the main memory, registers, cache memory, and buffers during the CPU operations of extraction, search, reference, comparison, calculation, processing, editing, output, printing, and display. It is temporarily stored in a memory or the like.

  Also, the arrows in the flowchart described in this embodiment mainly indicate input / output of data and signals, and the data and signal values are the RAM 914 memory, the FDD904 flexible disk, the CDD905 compact disk, and the magnetic disk device. It is recorded on a recording medium such as a 920 magnetic disk, other optical disks, minidisks, and DVDs. Data and signals are transmitted online via a bus 912, signal lines, cables, or other transmission media.

  In addition, what is described as “˜unit” in the description of the present embodiment may be “˜circuit”, “˜device”, “˜device”, and “˜step”, “˜”. “Procedure” and “˜Process” may be used.

That is, the operation of the data processing apparatus 100 can be understood as a data processing method by the steps, procedures, and processes shown in the flowchart described in this embodiment.
Further, what is described as “˜unit” may be realized by firmware stored in the ROM 913. Alternatively, it may be implemented only by software, or only by hardware such as elements, devices, substrates, and wirings, by a combination of software and hardware, or by a combination of firmware. Firmware and software are stored as programs in a recording medium such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, and a DVD. The program is read by the CPU 911 and executed by the CPU 911. In other words, the program causes the computer to function as “to part” of the present embodiment. Alternatively, the procedure or method of “˜unit” in the present embodiment is executed by a computer.

  As described above, the data processing device 100 described in this embodiment includes a CPU as a processing device, a memory as a storage device, a magnetic disk, etc., a keyboard as an input device, a mouse, a communication board, etc., a display device as an output device, a communication board, etc. As described above, the function indicated as “to part” is realized by using these processing device, storage device, input device, and output device.

  FIG. 4 is a flowchart showing the operation of the data processing apparatus 100 according to the present embodiment. The operation of the data processing apparatus 100 will be described with reference to FIG.

<Source designation process (list display process, input process): S100 to S101>
As described above, the source definition information storage unit 131 of the repository unit 130 stores a source definition 133 that may be used in advance. For example, it is assumed that the source definition information storage unit 131 stores “source definition #A”, “source definition # A1”, “source definition #B”, and “source definition #C”.

  FIG. 5 is a diagram showing an example of the mapping screen 125 for data processing editing definition in the data processing apparatus 100 according to the present embodiment.

  In S100, when the data processing apparatus 100 is activated, the display unit 120 displays a mapping screen 125 for data processing editing definition on the display screen 121. The mapping screen 125 is a screen used by the user, and the user uses the mapping screen 125 to perform data processing editing definition.

  As shown in FIG. 5, all source tables 351 corresponding to the source definition 133 stored in the source definition information storage unit 131 of the repository unit 130 are displayed in the source table list 126 of the mapping screen 125. Therefore, “source definition #A”, “source definition # A1”, “source definition #B”, and “source definition #C” are displayed as the source table list 126. However, at this time, the data processing apparatus 100 has not yet accessed the data source 300 corresponding to the source table 351 displayed in the source table list 126.

  Next, the display unit 120 displays a screen (selection screen 123 or command input screen 124) for selecting the source definition 133 used by the user from the source definitions 133 stored in the source definition information storage unit 131. It is displayed on the screen 121. The selection screen 123 or the command input screen 124 is a start condition input screen for allowing the user to input a source definition 133 to be used as a data processing start condition by the data processing apparatus 100.

  The display unit 120 displays, for example, a start condition input screen (selection screen 123 or command input screen 124) superimposed on the mapping screen 125. Alternatively, when the data processing apparatus 100 is activated, the display unit 120 may first display a start condition input screen (screen selection screen 123 or command input screen 124) before displaying the mapping screen 125.

  6A and 6B are diagrams illustrating an example of a start condition input screen. FIG. 6A is a selection screen 123 for inputting a process start condition by a GUI operation, and FIG. 6B is a command for inputting a process start condition by a command line parameter. This is an input screen 124.

  As shown in FIG. 6A, the display unit 120 displays the source ID list 122 of the source definition ID 132 stored in the source definition information storage unit 131 on the display screen 121. The display unit 120 displays on the display screen 121 a selection screen 123 that can be selected from the source ID list 122 by a GUI operation. Here, since the source definition 133 stored in the source definition information storage unit 131 is “source definition #A”, “source definition # A1”, “source definition #B”, and “source definition #C”, In the ID list 122, “#A”, “# A1”, “#B”, and “#C” are displayed. The selection screen 123 in FIG. 6A shows a state where “# A1” and “#B” check boxes are selected in the source ID list 122. That is, “source definition # A1” and “source definition #B” are selected by the user, and “source definition #A” and “source definition #C” are not selected.

  Alternatively, as illustrated in FIG. 6B, the display unit 120 displays on the display screen 121 a command input screen 124 that allows the source definition ID 132 to be selected using command line parameters. At this time, the display unit 120 may display the source ID list 122 by right-clicking the mouse or the like so that the user can refer to it. Alternatively, the command input screen 124 may be displayed superimposed on the mapping screen 125. In this case, the user can input a command while referring to the source table list 126 on the mapping screen 125 as the source ID list 122.

  In the command input screen 124 of FIG. 6B, “> COMMAND01, # A1, #B” is input, and the source definitions 133 of “# A1” and “#B” are selected from the source ID list 122. It shows the state. That is, “source definition # A1” and “source definition #B” are selected, and “source definition #A” and “source definition #C” are not selected.

  In S101, the input unit 110 inputs the source definition ID 132 selected on the selection screen 123 or the command input screen 124. In the example of FIGS. 6A and 6B, the input unit 110 inputs “# A1” and “#B” of the source definition ID 132.

<Table access process: S102 to S106>
The table access unit 140 inputs the source definition ID 132 selected by the user from the input unit 110. The selected source definition ID 132 is set as a designated source definition ID 139.

  The table access unit 140 sequentially extracts the source definition ID 132 and the corresponding source definition 133 from the source definition information storage unit 131 and sets the source definition ID 132 and the source definition 133 to be processed. The table access unit 140 executes the following processing for the extracted source definition ID 132 and source definition 133 to be processed.

  In S102, the table access unit 140 compares the source definition ID 132 to be processed with the specified source definition ID 139 input from the input unit 110, and determines whether or not the source definition ID 132 to be processed is the specified source definition ID 139. Judge by device. If the target source definition ID 132 is not the designated source definition ID 139 (NO in S102), the table access unit 140 advances the process to S106.

  If the source definition ID 132 to be processed is the designated source definition ID 139 (YES in S102), the table access unit 140 has source location information 134 indicating the location of the data source 300 included in the corresponding source definition 133 to be processed. Is determined by the processing device. The table access unit 140 determines whether the processing target source definition 133 is the database 310 or the file 320 from the source location information 134 included in the processing target source definition 133.

  If it is determined that the data source 300 of the source definition 133 to be processed is the database 310 (the database 310 in S103), the process proceeds to S104. In S104, the table access unit 140 establishes a session in the database 310. That is, the table access unit 140 establishes a session for the database 310 and connects to the source table 351 defined by the source definition 133 to be processed so as to be accessible.

  If it is determined that the data source 300 of the source definition 133 to be processed is the file 320 (file 320 in S103), the process proceeds to S105. In S105, the table access unit 140 opens the file 320. That is, the table access unit 140 opens the file 320 and connects the file 320 so that the source table 351 defined in the source definition 133 to be processed can be accessed.

  Here, it is assumed that the database 310 or the file 320 is discriminated. As described above, as the data source 300, for example, various data sources 300 such as an Oracle database, a flat file, Excel, and XML can be considered.

  In S106, the table access unit 140 determines whether or not there is an unprocessed source definition 133 by the processing device. When it is determined that there is an unprocessed source definition 133 (YES in S106), the table access unit 140 returns the process to S102 and executes the process for the unprocessed source definition 133. If the table access unit 140 determines that there is no unprocessed source definition 133 (NO in S106), the table access unit 140 proceeds to S107.

<Data extraction process / data processing process / data writing process: S107 to S109>
In S107, the data extraction unit 150 extracts data from the source table 351 in accordance with the data processing edit definition defined on the mapping screen 125. Next, in S108, the data processing unit 160 processes and edits the data extracted by the data extraction unit 150 in accordance with the data processing editing definition defined on the mapping screen 125. Next, in S109, the data writing unit 170 converts the data processed and edited by the data processing unit 160 according to the data processing editing definition defined on the mapping screen 125 into the target table 451 (output table) in the data target 400. ). At the end of the process, the table access unit 140 disconnects the session with the database 310 to which the session is pasted in S104, and closes the file 320 opened in S105.

For example, a case where “# A1” and “#B” of the source definition ID 132 are selected in the source specifying step described above will be described below.
(1) As shown in FIG. 5, on the mapping screen 125, the user merges the data of the source table #A (ID: # A1) and the data of the source table #B as shown by the bold line, The process part # 1 is defined as a process of processing the data processed by the process part # 1 using the process part # 2 and writing the data processed by the process part # 2 in the target table #T.
(2) When the data processing apparatus 100 has a function of displaying the selection screen 123, the user selects “# A1” and “#B” on the selection screen 123 as shown in FIG. To do. Further, when the data processing apparatus 100 has a function of displaying the command input screen 124, the user displays “# A1” and “#B” on the command input screen 124, as shown in FIG. select.
(3) The table access unit 140 accesses the source table #A (ID: # A1) and the source table #B, but does not access the source table #A and the source table #C.

  As described above, according to the data processing apparatus 100 according to the present embodiment, for all the data sources 300 (objects (RDB tables and flat files)) corresponding to the data reading definition (source definition 133), Since access is not performed but only the data source 300 selected by the user is accessed, even if there are objects that have already been deleted, such as deleted objects, they are selected by the user. If not, access will not be executed, and processing will not be interrupted.

  Further, according to the data processing apparatus 100 according to the present embodiment, even when there is an unused data reading definition (source definition 133) at the time of data processing editing definition (mapping), the unused data reading definition (source) Since the data source 300 corresponding to the definition is not accessed (a session is not established), useless resources are not consumed.

  Further, according to the data processing device 100 according to the present embodiment, the source definition 133 for extracting data (source table 351) with different extraction conditions from the same data source 300 is stored in the source definition information storage unit 131. Therefore, it is not necessary to rewrite and execute the data reading definition (source definition 133), and work efficiency is improved.

  In addition, according to the data processing apparatus 100 according to the present embodiment, even when data of different extraction conditions (including extraction conditions having an exclusive relationship) is extracted from the same source table 351, different extractions are performed on the same source table 351. Since the source definition 133 with the condition (including the extraction condition having an exclusive relationship) can be stored in the source definition information storage unit 131, it is necessary to rewrite and execute the data reading definition (source definition 133) No work efficiency is improved.

  Further, according to the data processing apparatus 100 according to the present embodiment, the user can select the source definition 133 using the selection screen 123 or the command input screen 124. Therefore, when the extraction condition is input every time it is executed. In comparison, input errors of the user are reduced, and an apparatus that is easy for the user to use can be provided.

Embodiment 2. FIG.
FIG. 7 is a functional block configuration diagram of the data processing apparatus 101 according to the present embodiment. FIG. 7 is a diagram corresponding to FIG. 1, and functional blocks similar to those in FIG. 1 are denoted by the same reference numerals and description thereof is omitted.

  The data processing apparatus 101 in FIG. 7 is different from the data processing apparatus 100 in FIG. 1 in that a database determination unit 180 is provided. The database determination unit 180 determines whether or not the data source 300 is deleted from the location of the data source 300 indicated by the source location information 134 included in the source definition 133 stored in the source definition information storage unit 131 by the processing device. judge.

  FIG. 8 is a flowchart showing operations of the database determination unit 180 and the display unit 120 according to the present embodiment. The data processing apparatus 101 according to the present embodiment executes step 100a (S100a) and step 100b (S100b) in FIG. 8 instead of step 100 (S100) in FIG. The operations after step 101 (S101) are the same as those in FIG. The operations of the database determination unit 180 and the display unit 120 of the data processing apparatus 101 according to the present embodiment will be described with reference to FIG.

  In S100a, the database determination unit 180 processes whether or not the data source 300 is deleted from the location of the data source 300 indicated by the source location information 134 included in the source definition 133 stored in the source definition information storage unit 131. Judge by device.

  When the database determination unit 180 determines that the determination target data source 300 has been deleted, the database determination unit 180 adds information indicating deletion to the source definition 133 including the determination target data source 300.

  For example, the information indicating the deletion can be set by providing a deletion flag for each record of the source definition 133 in the source definition information storage unit 131. For example, if the database determination unit 180 determines that the determination target data source 300 has been deleted, the database determination unit 180 turns ON the deletion flag corresponding to the source definition 133 including the determination target data source 300.

  In S <b> 100 b, the display unit 120 displays the source corresponding to the source location information 134 indicating the data source 300 determined not to be deleted by the database determination unit 180 among the source definition IDs 132 stored in the source definition information storage unit 131. The definition ID 132 is extracted, and a list of the extracted source definition IDs 132 is displayed on the display screen 121 of the display device.

  The display unit 120 determines the deletion flag of the source definition ID 132 stored in the source definition information storage unit 131 by the processing device, extracts the processing flag that is not ON, and displays it as the source ID list 122.

  When displaying the mapping screen 125, the display unit 120 may display the source definition ID 132 whose processing flag is not ON as the source table list 126 displayed on the mapping screen 125.

  As described above, according to the data processing apparatus 101 according to the present embodiment, the display unit 120 displays the source ID list 122 or the source table list 126 without the source definition ID 132 corresponding to the nonexistent data source 300. be able to. Therefore, the user does not mistakenly select the source definition ID 132 corresponding to the nonexistent data source 300, and the workability can be further improved.

Embodiment 3 FIG.
The data processing apparatus 102 according to the present embodiment has a function that allows the user to select a target table 451 (output table) into which data is written by the data writing unit 170 as well as the source table 351. is there. That is, the table access unit 140 accesses only the target table 451 of the target location information 137 included in the target definition 136 selected by the user.

  FIG. 9 is a functional block configuration diagram of the data processing apparatus 102 according to the present embodiment. FIG. 9 is a diagram corresponding to FIG. 1, and functional blocks similar to those in FIG. 1 are denoted by the same reference numerals and description thereof is omitted.

  The data processing apparatus 102 in FIG. 9 is different from the data processing apparatus 100 in FIG. 1 described in the first embodiment in that the repository unit 130 includes a target definition information storage unit 138.

  FIG. 10 is a diagram showing a configuration of the target definition information storage unit 138 according to the present embodiment. The target definition information storage unit 138 is information in which a target definition 136 is associated with a target definition ID 135 that uniquely identifies the target definition 136. The target definition 136 includes target location information 137 (an example of output location information) indicating the location of the data target 400 (output destination database) having the target table 451 defined by the target definition 136.

  The target definition information storage unit 138, the target definition 136, the target definition ID 135, the target table 451, the data target 400, and the target location information 137 are respectively the source definition information storage unit 131, the source definition 133, the source definition ID 132, and the source table. 351, the data source 300, and the source location information 134 have the same functions and configurations.

  A target definition information storage unit 138 (an example of an output definition program storage unit) is a target definition 136 (output definition program) that defines a target table 451 (output table) and a target that indicates the location of the data target 400 where the target table 451 is located. The location information 137 (output location information) and the target definition ID 135 (output identifier) for identifying the target definition 136 are associated with each other and stored in the storage device. As shown in FIG. 10, the target location information 137 is included in the target definition 136.

  The display unit 120 extracts the target definition ID 135 from the target definition ID 135 stored in the target definition information storage unit 138 according to a predetermined condition, and displays a list of the extracted target definition ID 135 on the display screen 121.

  The input unit 110 inputs at least one target definition ID 135 selected from the list of target definition IDs 135 displayed by the display unit 120.

  The table access unit 140 inputs at least one target definition ID 135 input by the input unit 110, and sets a target definition ID 135 and target location information 137 corresponding to the input at least one target definition ID 135 as a target definition information storage unit 138. The target table 451 defined by the extracted target definition ID 135 is accessed for the data target 400 indicated by the extracted target location information 137.

  The data writing unit 170 writes the data processed and edited by the data processing unit 160 to the target table 451 accessed and connected by the table access unit 140.

  FIG. 11 is a flowchart showing a source and target input process in the data processing apparatus 102 according to the present embodiment. Steps 100c (S100c) to 101d (S101d) of the source and target input process of the data processing apparatus 102 are executed in place of steps 100 (S100) to 101 (S101) in FIG.

  FIG. 12 is a flowchart showing a table access process in the data processing apparatus 102 according to the present embodiment. The table access process (step 210 (S210) to step 215 (S215)) of the data processing apparatus 102 is a process added between step 106 (S106) and step 107 (S107) in FIG.

  As described above, in the data processing method of the data processing apparatus 102 according to the present embodiment, instead of step 100 (S100) to step 101 (S101) in FIG. 4, step 100c (S100c) to step 101d ( S101d) is executed, S210 to S215 in FIG. 12 are added between S106 and S107 in FIG. 4, and the other operations are the same as those in FIG.

  12 are the same as the operations of S101 to S106 of FIG. 4 and will be described using a specific example.

  FIG. 13 is an example of the command input screen 124 displayed on the display screen 121 according to the present embodiment. FIG. 14 is an example of the mapping screen 125 displayed on the display screen 121 according to the present embodiment. As shown in FIG. 13, two command lines 1271 and 1272 are input to the command input screen 124. Further, as shown in FIG. 14, the data processing edit definition is defined on the mapping screen 125 by the user.

  In S100c, the display unit 120 displays on the display screen 121 a command input screen 124 for selecting the source definition ID 132 and the target definition ID 135 by the command line parameter. At this time, the display unit 120 may display the source ID list 122 and the target ID list (not shown) on the display screen 121 by right-clicking the mouse.

  Alternatively, when the display unit 120 displays the command input screen 124 by superimposing it on the mapping screen 125, the user displays the source table list 126 and the target table list 129 (see FIG. 14) on the mapping screen 125 as the source ID list. 122 and the target ID list may be referred to.

  In the command input screen 124 of FIG. 13, “> COMMAND02, #T, # A1, #B” (command line 1271) and “> COMMAND02, #S, #C, #B” (command line 1272) are input. Yes. For example, the command line 1271 means that the source table #A (ID: # A1), #B is output to the target table #T. Further, the command line 1272 means that the data is output from the source tables #B and #C to the target table #S. By inputting the command lines 1271, 1272, the user selects “# A1”, “#B”, “#C” in the source ID list 122, and selects “#T”, “#S” in the target ID list. It means to choose.

  In S101d, the input unit 110 inputs “#T” and “#S” of the target definition ID 135 selected on the command input screen 127, and advances the processing to S102.

  As described in the first embodiment, the table access unit 140 accesses the source table 351 selected by the user through the processing of S102 to S106, and advances the processing to S210. Here, it is assumed that source tables #A (ID: # A1), #B, and #C are connected by command lines 1271 and 1272.

  In step S211, the table access unit 140 selects the target definition ID 135 to be processed in the target definition information storage unit 138 (referred to as “#S” in the first record in FIG. 10) and the selected designated target input from the input unit 110. The definition IDs “#T” and “#S” are compared. Since the processing target ID “#S” applies to the specified target definition ID (“#T”, “#S”) (YES in S211), the table access unit 140 sets the target definition of the processing target ID “#S”. The target location information 137 included in #S is determined by the processing device. The table access unit 140 determines that the target location information 137 included in the target definition #S of the processing target ID “#S” is the database 410 (see FIG. 10) (determined as a database in S212).

  Since it is determined that the data target 400 of the target definition #S of the processing target ID “#S” is the database 410, the table access unit 140 establishes a session in the database 410 in S213. That is, the table access unit 140 establishes a session for the database 410 and connects to the target table #S defined in the target definition #S to be processed so as to be accessible.

  In S215, the table access unit 140 determines whether or not there is an unprocessed target definition ID 135 by the processing device. The data writing unit 170 executes the processing of S210 to S215 for all target definition IDs 135 stored in the target definition information storage unit 138 shown in FIG. Here, target tables #S and #T are connected by command lines 1271 and 1272.

  In S107 of FIG. 4, the data extraction unit 150 extracts data from the source tables #A (ID: # A1), #B, and #C according to the data processing edit definition defined on the mapping screen 125. Next, in S108, the data processing unit 160 processes and edits the data extracted by the data extraction unit 150 in accordance with the data processing editing definition defined on the mapping screen 125. Next, in S109, the data writing unit 170 converts the data processed and edited by the data processing unit 160 according to the data processing editing definition defined on the mapping screen 125 into the target tables #S, # in the data target 400. Write to T.

  As described above, according to the data processing apparatus 102 according to the present embodiment, the target table 451 can also be selected by the user, and the processing efficiency and work efficiency can be improved.

  Further, according to the data processing apparatus 102 according to the present embodiment, even when the processing shown in the command line 1271 and the command line 1272 is performed, it is not necessary to generate two programs, and it is possible to cope with one program.

  Further, according to the data processing apparatus 102 according to the present embodiment, the source definition 133 and the target definition 136 of any condition can be stored, so even when the contents of the source definition 133 and the target definition 136 are changed. One change operation is sufficient, and an apparatus with high maintainability can be provided.

  Although the first to third embodiments have been described above, two or more of these embodiments may be combined. Alternatively, one of these embodiments may be partially implemented. Or you may implement combining two or more embodiment among these partially.

100, 101, 102 Data processing device, 110 input unit, 120 display unit, 121
Display screen, 122 Source ID list, 123 Selection screen, 124 Command input screen, 125 Mapping screen, 126 Source table list, 129 Target table list, 130 Repository section, 131 Source definition information storage section, 132 Source definition ID, 133 Source definition 134 Source location information, 135 Target definition ID, 136 Target definition, 137 Target location information, 138 Target definition information storage unit, 139 Designated source definition ID, 140 Table access unit, 150 Data extraction unit, 160 Data processing unit, 170 Data Write unit, 180 database determination unit, 300 data source, 310 database, 320 file, 351 source table, 400
Data target, 410 database, 420 file, 451 target table, 901 display device, 902 keyboard, 903 mouse, 904 FDD, 905 CDD, 906 printer device, 907 scanner device, 911 CPU, 912
Bus, 913 ROM, 914 RAM, 915 communication board, 920 magnetic disk unit, 921 OS, 922 window system, 924 file group, 1271, 1272 command line.

Claims (10)

In a data processing apparatus that accesses a plurality of source tables in an acquisition source database that is an acquisition source of data,
Associating a source definition program that defines a plurality of source tables, acquisition source location information that indicates the location of an acquisition source database in which each source table of the plurality of source tables exists, and a source identifier that identifies the source definition program A definition program storage unit stored in the storage device;
A display unit that extracts a source identifier according to a predetermined condition from the source identifiers stored in the definition program storage unit, and displays a list of the extracted source identifiers on a display device;
An input unit for inputting at least one source identifier selected from the source identifier list displayed by the display unit;
At least one source identifier input by the input unit is input, and the source definition program and the source location information corresponding to the input at least one source identifier are extracted from the definition program storage unit and extracted. A data processing apparatus comprising: a table access unit that accesses a source table defined by the extracted source definition program to an acquisition source database indicated by the acquisition source location information. The data processing device further includes:
A database determination unit that determines whether or not the acquisition source database is deleted from the location of the acquisition source database indicated by the acquisition source location information stored in the definition program storage unit;
The display unit
Among the source identifiers stored in the definition program storage unit, the source identifier corresponding to the acquisition source location information indicating the acquisition source database determined not to be deleted by the database determination unit is extracted, and the extracted The data processing apparatus according to claim 1, wherein a source identifier list is displayed on a display device. The display unit further displays on the display device a selection screen for selecting at least one of the source identifiers from the displayed source identifier list,
The input unit is
The data processing apparatus according to claim 1, wherein at least one of the source identifiers selected on the selection screen displayed by the display unit is input. The definition program storage unit
The source definition program describing extraction conditions for extracting each source table of the plurality of source tables from the acquisition source database indicated by the acquisition source location information, wherein the plurality of source definitions describe mutually exclusive extraction conditions The data processing apparatus according to claim 1, wherein a program is stored. The data processing device further includes:
A data writing unit that processes data in the source table accessed by the table access unit and writes the processed data to at least one output table in an output table in a predetermined output destination database. The data processing apparatus according to claim 1. The definition program storage unit further associates an output definition program that defines the output table, output location information that indicates a location of an output destination database in which the output table exists, and an output identifier that identifies the output definition program And store it in the storage device
The display unit further extracts an output identifier according to a predetermined condition from the output identifiers stored in the definition program storage unit, and displays the extracted output identifier list on a display device,
The input unit further inputs at least one output identifier selected from the output identifier list displayed by the display unit,
The table access unit inputs at least one output identifier input from the input unit, and outputs the output definition program and the output location information corresponding to the input at least one output identifier from the definition program storage unit. Extracting, accessing the output destination database indicated by the extracted output location information, accessing the output table defined by the extracted output definition program;
The data processing apparatus according to claim 5, wherein the data writing unit writes the processed data to the output table accessed by the table access unit. A definition program that defines a plurality of source tables, acquisition source location information indicating the location of an acquisition source database in which each source table of the plurality of source tables is located, and a source identifier that identifies the definition program are stored in association with each other. In a data processing method of a data processing apparatus including a definition program storage unit,
A list display step of extracting a source identifier according to a predetermined condition from the source identifiers stored in the definition program storage unit, and displaying the extracted source identifier list on a display device;
An input step for inputting at least one source identifier selected from the source identifier list displayed by the list display step;
The table access unit inputs at least one of the source identifiers input in the input step, and the definition program corresponding to the input at least one of the source identifiers and the acquisition source location information from the definition program storage unit And a table access step of accessing a source table defined by the extracted definition program with respect to an acquisition source database indicated by the extracted acquisition source location information. Method. The data processing method further includes:
A database determination step of determining whether or not the acquisition source database is deleted from the location of the acquisition source database indicated by the acquisition source location information stored in the definition program storage unit,
The list display step includes:
Among the source identifiers stored in the definition program storage unit, the source identifier corresponding to the source location information indicating the source database determined not to be deleted by the database determination step is extracted, and the extracted The data processing method of the data processing apparatus according to claim 7, wherein a source identifier list is displayed on a display device. A definition program that defines a plurality of source tables, acquisition source location information indicating the location of an acquisition source database in which each source table of the plurality of source tables is located, and a source identifier that identifies the definition program are stored in association with each other. In a data processing program to be executed by a data processing apparatus which is a computer having a definition program storage unit
A list display step of extracting the source identifier according to a predetermined condition from the source identifiers stored in the definition program storage unit, and displaying the extracted source identifier list on a display device;
An input step of inputting at least one source identifier selected from the source identifier list displayed by the list display step;
Input at least one source identifier input in the input step, extract the definition program corresponding to the input at least one source identifier and the acquisition source location information from the definition program storage unit, and extract A data processing program comprising: a table access step of accessing a source table defined by the extracted definition program with respect to an acquisition source database indicated by the acquisition source location information. The data processing program further includes:
A database determination step of determining by a processing device whether or not the acquisition source database is deleted from the location of the acquisition source database indicated by the acquisition source location information stored in the definition program storage unit;
The list display step includes:
Among the source identifiers stored in the definition program storage unit, the source identifier corresponding to the source location information indicating the source database determined not to be deleted by the database determination step is extracted, and the extracted The data processing program according to claim 9, wherein the source identifier list is displayed on a display device.

Images