JP2019032781A - Data integration support system and data integration support method - Google Patents

Abstract

To enable rule creation for solving incompatibility between pieces of data present on business or contrary to an analyst's expectation to be carried out without manual operation.SOLUTION: Provided is a data integration support system that integrates related pieces of data among a plurality of pieces of record data according to a rule, in which designation of a target table selected from among a plurality of tables that manage the plurality of pieces of record data and a specific key in the target table is accepted, and a specific rule is determined on the basis of the target table and the specific key.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a data integration support system and a data integration support method that integrate so-called data after name identification so as to be in an appropriate mode intended by an analyst, and in particular, when such data after name identification is integrated. The present invention is suitable for application to a data integration support system that supports the creation of rules used for.

  In recent years, there has been an increase in the movement to solve management problems by analyzing and utilizing various data in companies. During data analysis, data preprocessing often requires a lot of man-hours, so the analysis period becomes long or sufficient data cannot be prepared, which may lead to degradation of the quality of the analysis results. . For example, one of the problems is that the relationship between data entities (one-to-one, one-to-many) is different from the relationship that business or analysts expect, and it takes time to detect and resolve it. there were.

  As described above, when the relationship between the data entities is different from the expectation, if the analyst is unaware of this and performs the combination and aggregation of the data, an error occurs in the data, and the quality of the analysis result decreases. In order to solve such a problem, it is necessary for an analyst to manually set a rule for resolving inconsistencies in the data while checking the data one by one (see Patent Document 1). It takes special skills and time.

  In the conventional system, the first and second records are identified, and the common data is obtained by comparing the contents of the data elements of the first and second identified records by applying a record matching criterion. One of the first and second records is selected as a surviving record based on the earliest date of record generation or the first and second relative contents (see Patent Document 2).

JP 2013-137766 A JP 2005-502132 A

  However, even in the above-described conventional system, it is necessary to manually define a rule for selecting a record, and data preprocessing cannot be performed efficiently.

  The present invention has been made in consideration of the above points, and it is possible to efficiently create rules for eliminating contradictions between existing data against business or analyst expectations, without human intervention. A data integration support system and a data integration support method are proposed.

  In order to solve such a problem, in the present invention, in a data integration support system that integrates related data among a plurality of record data according to a rule, a plurality of tables for managing the plurality of record data, and the plurality of tables A rule that determines a specific rule based on the target table selected from among the target table, a specification receiving unit that receives a specification of a specific key in the target table, and the target table and the specific key whose specification has been received by the specification receiving unit A determination unit; a rule execution unit that selects one record data among the plurality of record data according to the specific rule determined by the rule determination unit; and the one record data selected by the rule execution unit A data analysis unit for analyzing the data and the data analysis Characterized in that by and a result display unit for displaying an analysis result for the one record data.

  In the present invention, in the data integration support method in a data integration support system that integrates related data among a plurality of record data according to a rule, the data integration support system manages a plurality of record data. A target reception table selected from among the tables, a specification reception step for receiving a specification of a specific key in the target table, and the data integration support system based on the target table and the specific key for which the specification was received in the specification reception step And a rule execution step in which the data integration support system selects one record data among the plurality of record data according to the specific rule determined in the rule determination step. And a data analysis unit step in which the data integration support system analyzes the one record data selected in the rule execution step, and the data integration support system displays an analysis result for the one record data. And a result display step.

  According to the present invention, it is possible to efficiently create a rule for eliminating a contradiction between data existing in business or against an analyst's expectation without human intervention.

It is a block diagram which shows an example of schematic structures, such as a data integration support system by this Embodiment. It is a figure which shows an example of the data stored in the customer table shown in FIG. It is a figure which shows an example of the data added to the customer table shown in FIG. It is a figure which shows an example of a rule creation assistance process. It is a figure which shows an example of the preparation which calculates the coverage of a selection rule. It is a figure which shows an example of the preparation which calculates the coverage of a selection rule. It is a figure which shows the example of calculation of the coverage of a selection rule. It is a figure which shows an example of the data stored in the application history table shown in FIG. It is a figure which shows an example of the one part record data referred from the application history table shown in FIG. It is a figure which shows an example which gave the score based on the record data shown in FIG. It is a figure which shows an example by which the use column was sorted by the coverage and the application history score. It is a figure which shows an example of selection conditions. It is a figure which shows an example of the selection result displayed according to condition selection.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

(1) Configuration Example of Data Integration Support System According to this Embodiment FIG. 1 shows an example of a schematic configuration of a data integration support system 101 according to this embodiment. This embodiment is characterized by support for creating a rule that should be used as a reference when summarizing the results after so-called name identification. This will be specifically described below.

  The data integration support system 101 has a storage device 102 having a target database (hereinafter referred to as “target data DB”) 103 that manages a plurality of tables including a customer table and a person-in-charge table, which will be described in detail later, via a network (not shown). It is connected to the. The data integration support system 101 may have a configuration in which the target data DB 103 is built in and the storage apparatus 102 is omitted.

  The target data DB 103 is a so-called relational database, and each of the plurality of tables manages data for each column item, and manages record data including a plurality of data corresponding to all column items. doing.

  The data integration support system 101 includes a designation reception unit 1, a data analysis unit 2, a DB connection unit 3, a rule determination unit 4, a rule execution unit 5, and a result display unit 7, and more preferably includes an application history table 6.

  When the DB connection unit 3 receives an inquiry command in a query language such as a so-called SQL (Structured Query Language) language, the DB connection unit 3 newly adds, deletes, or updates record data with the target data DB 103 according to the content of the inquiry. Then, the result of the inquiry is returned to the inquiry source.

  The designation receiving unit 1 receives designation of a target table selected from a plurality of tables and a specific column item (hereinafter also referred to as “specific key”) in the target table.

  The rule determining unit 4 determines a specific rule based on the target table and the specific key received by the designation receiving unit 1. In this embodiment, “determining” means not only creating a new specific rule, but also diverting a part of the rules applied in the past. Details of the rule determination method by the rule determination unit 4 will be described later.

  The rule execution unit 5 searches the record data from the table of the target data DB 103 by issuing an inquiry command via the DB connection unit 3 according to the specific rule determined by the rule determination unit 4. Thereby, one record data can be selected from among the record data.

  The data analysis unit 2 analyzes one record data selected by the rule execution unit 5. The result display unit 7 displays the analysis result of one record data by the data analysis unit 2. Details of this display content will be described later.

  The application history table 6 manages past rules (corresponding to selection rules Ri described later) applied to a plurality of data in association with application frequency scores Fi corresponding to the number of times of application of the past rules. .

(2) Data Integration Support Method by Data Integration Support System The data integration support system 101 is configured as described above. Next, a data integration support method as an operation example of the data integration support system 101 will be described. In the present embodiment, the rules may be determined from the target record data group, but here, a mode in which divertable rules can be adopted among the rules applied in the past will be described.

(2-1) Data Consistency Consistency FIGS. 2 and 3 each show an example of record data corresponding to each column item of the customer table 104 managed in the target data DB 103 described above. The customer table 104 has, as its column items, a customer ID, name, address, telephone number, gender, age, registration date, update date, update time, and invalid flag for identifying a plurality of customers. It is assumed that the unique key in the customer table 104 is a customer ID among all the column items.

  In the present embodiment, the record data shown in FIG. 2 is not different from the relation expected by the business or analyst because the customer ID that is a unique key is not duplicated in all the record data, and is particularly resolved in the data. There is nothing to be done (hereinafter referred to as “there is no data inconsistency”).

  In the present embodiment, the record data shown in FIG. 3 includes two record data that are duplicated in the customer IDs “A0001” and “A0003” that are unique keys among all the record data. Is necessary (hereinafter referred to as “there is a data inconsistency”).

  This is stored in the target data DB 103 in a one-to-many relationship depending on, for example, a system in which data having a one-to-one relationship in business or analyst's expectation handles the data or its operation or business flow. This is because there are cases.

For example, in a customer table for managing customer information, a customer and its address should have a one-to-one relationship in business, but in any of the following cases, a record having a different address for one customer There may be multiple data.
1) When address change processing is not performed at the time of updating record data, but is performed by invalidating new record data or old record data (for example, setting an invalid flag)
2) When name identification is performed probabilistically according to the similarity of records

(2-2) Resolving a state with data contradiction Therefore, in the present embodiment, a state having a contradiction on data is resolved using a rule determined as follows.

  FIG. 4 is a flowchart illustrating an example of the rule determination process. First, the designation receiving unit 1 receives designation of a target table and a specific key (step S1).

  The rule determination unit 4 determines whether or not the specific key is a unique key (step S2). If the specific key is not a unique key, the rule determining unit 4 ends the process, while the specific key is a unique key. The following step S3 is executed.

  In step S3, the rule determination unit 4 calculates the coverage Ci of the selection rule Ri using column items other than the specific key for each item.

  Specifically, the rule determination unit 4 uses a column item other than the specific key, for example, “telephone number” for a part of record data in which the unique key (customer ID) is duplicated in the customer table shown in FIGS. , Using the “update date”, “update time”, and “invalid flag”, the coverage Ci of the selection rule Ri is calculated for each item as shown in FIG.

  Next, the rule determination unit 4 acquires an application frequency score Fi (corresponding to the number of application times shown) of the selection rule Ri from the application history table 6 shown in FIG. 8 (step S4). Specifically, as shown in FIG. 9, the rule determination unit 4 selects a record whose table name is “customer table” and whose key column item is “customer ID” with respect to the application history table 6. Search is performed to obtain an inquiry result as shown in FIG.

  Further, in step S4, the rule determination unit 4 extracts, from these records, a rule that matches any of the table, the key column item, or the selection target from the application history table 6, and selects the selection rule (used column name, condition ) To calculate the application frequency score Fi as shown in FIG. In this case, the rule determination unit 4 gives a high score to a rule having a history in which all of the table, the key string item, and the selection target exist.

  Next, the rule determination unit 4 calculates the priority Si of the selection rule Ri according to a predetermined function f (Ci, Fi) (step S5).

  Next, the rule determination unit 4 sorts the selection rules Ri by the priority Si, and causes the result display unit 7 to display a selection rule candidate display screen including the selection rules Ri after the sorting (step S6).

  When sorting in this way, the rule determination unit 4 displays the column items to be used on the result display unit 7 according to the size of the coverage Ci and the application history score Fi as shown on the left side of FIG. Alternatively, it may be displayed so as to mask unused column items as shown on the right side of FIG.

  Further, the rule determination unit 4 superimposes a selection condition screen on which the selection conditions of “maximum” and “minimum” can be selected as shown in FIG. The analyzer may be made to select the selection condition. When selected in this way, as shown in FIG. 13, the result display unit 7 presents, for example, a selection rule candidate group having “update date” as a use column item and “maximum” as a selection condition. A selection rule candidate display screen is displayed.

  The analyst performs the operation of selecting the desired selection rule Ri from the selection rule group displayed on the selection rule candidate display screen in this way, thereby the customer ID that is record data in a state where there is a contradiction in the data Select one more appropriate record from the two record data with "A0001" and one more appropriate record from the two record data with the customer ID "A0003". Used for data analysis performed as a process.

  As described above, according to the present embodiment, when utilizing data that is expected to expand in the future, rule determination for resolving inconsistencies between data that is contrary to business or analyst expectations is manually performed. Therefore, it is possible to reduce the man-hours for data pre-processing that should be completed before performing the data analysis as the data post-processing described above. Further, since rule determination is not performed manually, the skill level of the creator is not affected, and the rules can be homogenized.

(3) Other Embodiments The above embodiment is an example for explaining the present invention, and is not intended to limit the present invention only to these embodiments. The present invention can be implemented in various forms without departing from the spirit of the present invention. For example, in the above-described embodiment, the processing of various programs is described sequentially, but this is not particularly concerned. Therefore, as long as there is no contradiction in the processing result, the processing order may be changed or the operation may be performed in parallel. Further, the program including each processing block in the above embodiment may be in a form stored in a non-transitory storage medium readable by a computer, for example.

  The present invention can be widely applied to a data integration support system and a data integration support method for integrating so-called name identification data so as to be in an appropriate mode intended by an analyst.

  DESCRIPTION OF SYMBOLS 1 ... Designation reception part, 2 ... Data analysis part, 3 ... DB connection part, 4 ... Rule determination part, 5 ... Rule execution part, 6 ... Application history table, 7 ... Result display part, 101 Data integration support system 102 Storage device 103 Data DB

Claims (12)

In a data integration support system that integrates related data among multiple record data according to rules,
A plurality of tables for managing the plurality of record data;
A target accepting unit selected from the plurality of tables and a designation receiving unit that accepts designation of a specific key in the target table;
A rule determining unit that determines a specific rule based on the target table and the specific key whose designation is received by the designation receiving unit;
A rule execution unit that selects one record data among the plurality of record data according to the specific rule determined by the rule determination unit;
A data analysis unit that analyzes the one record data selected by the rule execution unit;
A result display unit for displaying an analysis result of the one record data by the data analysis unit;
A data integration support system comprising: The rule determination unit
The data integration support system according to claim 1, wherein coverage of a selection rule using column items other than the specific key in the target table is calculated for each item. An application history table for managing past rules applied to the plurality of data;
The rule determination unit
The data integration support system according to claim 1, wherein a rule applicable to the one record data is searched from the past rules managed in the application history table. The result display section
3. A selection rule candidate display screen for presenting the rule candidates is displayed, and coverage and application history scores of the selection rules using column items in the target table are displayed together. And a data integration support system according to claim 3. The result display section
The data integration support system according to claim 4, wherein a selection condition screen for inputting a selection condition is displayed on the selection rule candidate display screen in an overlapping manner. The result display section
5. The data integration support system according to claim 4, wherein an unused column item among the column items of the target table is masked and displayed on the selection rule candidate display screen. In a data integration support method in a data integration support system that integrates related data among a plurality of record data according to a rule,
The data integration support system receives a designation of a target table selected from a plurality of tables managing the plurality of record data and a specific key in the target table; and
A rule determining step in which the data integration support system determines a specific rule based on the target table and the specific key whose designation is accepted in the designation accepting step;
A rule execution step in which the data integration support system selects one record data among the plurality of record data according to the specific rule determined in the rule determination step;
A data analysis step in which the data integration support system analyzes the one record data selected in the rule execution step;
A result display step in which the data integration support system displays an analysis result for the one record data; and
A data integration support method characterized by comprising: In the rule determining step,
The data integration support method according to claim 7, wherein the data integration support system calculates coverage of selection rules using column items other than the specific key in the target table for each item. In the rule determination step,
The data integration support system searches for a rule applicable to the one record data from the past rules managed in an application history table that manages past rules applied to the plurality of data. The data integration support method according to claim 7, wherein: The result display step includes:
The data integration support system displays a selection rule candidate display screen for presenting the rule candidates, and also displays coverage and application history scores of the selection rules using column items in the target table. The data integration support method according to claim 8 or 9, characterized by the above. The result display step includes:
The data integration support method according to claim 10, wherein the data integration support system displays a selection condition screen for inputting a selection condition on the selection rule candidate display screen in an overlapping manner. The result display step includes:
The data integration support method according to claim 10, wherein the data integration support system masks and displays unused column items among the column items of the target table on the selection rule candidate display screen.

Images