JP5640796B2 - Name identification support processing apparatus, method and program - Google Patents

Description

  The present technology relates to a technology for supporting name identification.

  Name identification refers to finding and integrating multiple records representing the same entity in a database. For example, in a customer database, if there are a plurality of records for a customer company or a customer individual, a process of integrating them. Typically, there are many columns in the database that is subject to name identification, and the amount of data is enormous. Therefore, if the name identification process is performed as it is, the calculation amount increases and a long processing time is required. Cause. Therefore, column selection is performed before name identification. This column selection is a process of determining which column in the database is to be processed for name identification. Normally, by selecting an appropriate column in the column selection, the processing speed is improved while maintaining the accuracy of the name identification process. If there is a column that adversely affects name identification in the original database, it is expected that the accuracy of name identification will be improved by selecting an appropriate column.

  However, in the past, column selection for name identification was performed based on human experience and intuition, column selection was performed after machine learning, or the number of different values for each column was calculated and a certain number of different values were calculated. Or a column having a number was selected. However, such a method may be time consuming and may select an inappropriate column. Examples of columns that are not appropriate include cases where both redundant columns representing almost the same information such as “company kanji name” and “company kana name” are selected, or customer company name identification such as trade product names. In some cases, an unrelated column is selected.

  Note that name identification is a technique for finding the identity between records, and the technique for finding the identity between columns is as follows. A technique for detecting an association between items in a plurality of tables already exists. For example, for each item in the table, calculate an item score that is a value based on the number of appearances for each value in the item, select different tables from the plurality of tables as the association source table and the association destination table, A relation source item that is an item in the table and a relation destination item that is an item in the relation destination table are selected, and a combination of the relation source item and the relation destination item is generated as a relation. Then, for each generated association, a related score that is a value based on the number of appearances of each value in the related source item in the related destination item, the item score of the related source item, and the item score of the related destination item is calculated. is there. However, in general, the association between items in a plurality of tables is not detected by name identification. As a technique for finding the identity of records in a plurality of tables, there is a reference name collation, but this technique is not targeted.

  In addition, data may be registered in all columns of each record, but there may be data loss. In such cases, there is a method to exclude the part containing missing values in the target data and perform normal analysis on the remaining data, but if the number of non-missing values is small, the reliability of the correlation coefficient value There is a problem that decreases. In addition, there is a method of substituting statistically meaningful numerical values (average value, regression estimation value, worst value, etc.) into the missing part of each column, but there is a high risk of deviating from the natural variance value. There is a problem that the correlation coefficient value becomes inaccurate. In addition, there are methods such as estimating multiple missing values, generating and analyzing multiple data sets, and finally integrating the results, but depending on the data size and number of missing data, the amount of calculation becomes enormous, It is not a realistic measure in the field of name identification.

JP 2008-27072 A

  Accordingly, an object of the present technology is, in one aspect, to provide a technology for automatically selecting a column that may help to perform name identification with high accuracy.

  A name identification support processing method according to an aspect of the present technology includes (A) calculating a first data similarity between records for each column in the database and storing the first data similarity in a data storage unit; and (B) a combination of columns. Calculating a correlation coefficient of the first data similarity stored in the data storage unit for each of the data, and storing the correlation coefficient in the data storage unit; and (C) an arbitrary column in the target column which is an arbitrary column of the columns The record having data similar to or similar to the data in the target record that is the record of the record is specified, and the second data similarity for each column is calculated in each of the range between the target record and the specified record Alternatively, the second data stored in the data storage unit for each combination of the column of interest and the other column is specified and stored in the data storage unit. A neighborhood correlation coefficient calculation step of calculating a correlation coefficient of the data similarity degree as a neighborhood correlation coefficient and storing it in the data storage unit; and (D) a positive correlation coefficient that is calculated and exceeds a predetermined significance level. A first extraction process is performed to extract a combination of columns from which a positive neighborhood correlation coefficient calculated by multiplying the percentage of records having no neighborhood correlation coefficient or data loss has been calculated from the data storage unit. An extraction step.

  You can automatically select columns that may be useful for accurate name identification.

FIG. 1 is a diagram for explaining the correlation and the neighborhood correlation. FIG. 2 is a diagram for explaining the correlation. FIG. 3 is a diagram for explaining the correlation. FIG. 4 is a diagram for explaining the correlation and the neighborhood correlation. FIG. 5 is a diagram for explaining the correlation and the neighborhood correlation. FIG. 6 is a diagram for explaining the correlation and the neighborhood correlation. FIG. 7 is a diagram for explaining the correlation and the neighborhood correlation. FIG. 8 is a diagram for explaining the relationship between the correlation and the neighborhood correlation. FIG. 9 is a functional block diagram of the name identification support processing device. FIG. 10 is a diagram showing a main processing flow in the present embodiment. FIG. 11 is a diagram illustrating an example of the read data. FIG. 12 is a diagram illustrating a processing flow of correlation calculation processing. FIG. 13 is a diagram for explaining similarity calculation. FIG. 14 is a diagram illustrating a calculation result of similarity. FIG. 15 is a diagram illustrating a calculation result of the correlation coefficient. FIG. 16 is a diagram illustrating a process flow of the neighborhood correlation calculation process. FIG. 17 is a diagram illustrating an example of sorting when focusing on a specific column. FIG. 18 is a diagram illustrating an example of a case where neighboring records are extracted for each record. FIG. 19 is a diagram illustrating the calculation result of the similarity. FIG. 20 is a diagram illustrating a process flow of the neighborhood correlation calculation process. FIG. 21 is a diagram illustrating an example of the neighborhood correlation coefficient calculated for one column. FIG. 22A is a diagram illustrating a processing flow of column priority calculation processing. FIG. 22B is a table summarizing the calculation results of the correlation and the neighborhood correlation. FIG. 23 is a table summarizing the calculation results of the usefulness and the priority. FIG. 24 is a diagram illustrating a processing flow of the unification processing of redundant column groups. FIG. 25 is a diagram for explaining the classification of redundant column groups. FIG. 26 is a diagram for explaining selection of a column having priority in a redundant column group. FIG. 27 is a diagram for explaining the final column selection result. FIG. 28 is a diagram illustrating an example of a result output. FIG. 29 is a functional block diagram of the name identification support processing device according to the second embodiment. FIG. 30 is a diagram illustrating a main processing flow according to the second embodiment. FIG. 31A is a diagram illustrating an example of missing data. FIG. 31B is a diagram for explaining a method of calculating the weight d. FIG. 32 is a diagram illustrating an example of an appearance frequency table. FIG. 33 is a diagram showing a processing flow of missing value estimation processing. FIG. 34 is a diagram illustrating a process flow of the estimable column determination process. FIG. 35A is a diagram for explaining an estimable column determination process. FIG. 35B is a diagram for explaining an estimable column determination process. FIG. 35C is a diagram for describing an estimable column determination process. FIG. 35D is a diagram for describing an estimable column determination process. FIG. 35E is a diagram illustrating an example of an estimable column pair list. FIG. 36 is a diagram illustrating a processing flow of estimable column determination processing. FIG. 37 is a diagram illustrating a processing flow of name identification possibility estimation processing. FIG. 38A is a diagram for describing name identification possibility estimation processing. FIG. 38B is a diagram for describing name identification possibility estimation processing. FIG. 39 is a diagram showing a processing flow of missing value estimation processing. FIG. 40 is a diagram illustrating a processing flow of the first estimation processing. FIG. 41A is a diagram for describing the first estimation process. FIG. 41B is a diagram for describing the first estimation process. FIG. 41C is a diagram for describing the first estimation process. FIG. 42 is a diagram for describing the second estimation process. FIG. 43 is a diagram for describing the second estimation process. FIG. 44 is a diagram illustrating a process flow of the second name identification possibility estimation process. FIG. 45 is a diagram for describing the second name identification possibility estimation process. FIG. 46 is a functional block diagram of a computer.

[Embodiment 1]
In this embodiment, a column that is useful for name identification, that is, a column that can limit the number of entities to one or a small number (for example, company name, address, etc. in the case of name identification of a customer company) is extracted by introducing a new index. To do.

  Specifically, a well-known correlation (more specifically, a correlation coefficient) and a neighborhood correlation (more specifically, a neighborhood correlation coefficient) are used in combination. The neighborhood correlation, which will be described later in detail, is an index that represents the degree to which the values of a plurality of columns simultaneously match or are similar within a range of a small number of similar records in a specific column. Hereinafter, the relationship between the correlation and the neighborhood correlation will be described.

  First, an overview of correlation and neighborhood correlation will be given with reference to FIG.

  Here, it is assumed that the following typical example holds. (1) In a set of records that can be identified, the column values of pairs of columns useful for name identification are often “both coincide or are very similar”. That is, in this case, the similarity between the two columns is linked. (2) Even when there is a record with a high possibility of name identification, the number of data in that portion is very small compared to the total number of records in the database.

  As shown in the upper left of FIG. 1, the correlation indicates the linkage of the similarity of a specific column pair in the entire analysis target database. When the correlation coefficient of a column pair is large, the main cause is column redundancy. For example, “Company Kanji Name” and “Company Kana Name” have a large correlation coefficient, but they are redundant column pairs for name collation. Will increase. Since the absolute number of records having a high possibility of name identification is very small compared to the typical example (2), they hardly contribute to an increase in correlation. That is, the correlation coefficient alone cannot be extracted because the column useful for name identification is buried.

  On the other hand, the neighborhood correlation is to extract a set of records (lower right in FIG. 1) having similar column values of a specific column (for example, “company kanji name”), such as the hatched portion in the upper left in FIG. This shows the linkage of the similarity of column pairs in a data set with strong local similarity. When the neighborhood correlation coefficient of the column pair is large, it is estimated from the typical example (1) that many records having a high possibility of name identification are included, and there is a high possibility that a column useful for name identification is included. However, redundancy cannot be completely eliminated.

  Next, the correlation will be examined in more detail with reference to FIGS. FIG. 2 is a scatter diagram, in which the horizontal axis represents, for example, the similarity of column values in the transaction product column, the vertical axis represents, for example, the similarity of column values in the address column, and the black dots represent record pairs. As in the example of FIG. 2, when the correlation is almost 0 (non-correlated), the degree of similarity of any column is irrelevant to each other, and such a column pair is not useful for name identification.

  Note that the determination of the magnitude of the correlation uses the determination of the significance of the correlation coefficient in statistics. That is, if the relationship between the limit value α of the significance level and the correlation coefficient r satisfies r> α, the correlation is high, and if not, the correlation is low.

  On the other hand, FIG. 3 is also a scatter diagram, where the horizontal axis represents, for example, the similarity of column values in the company kanji name column, and the vertical axis represents, for example, the similarity of column values in the company kana name column. In the example of FIG. 3, when the similarity of one column is large, the similarity of the other column is also large, and when the similarity of one column is small, the similarity of the other column is also small. Thus, when the behavior of the similarity of both columns is almost the same, this is a redundant column pair. Therefore, almost the same result is expected regardless of which column is used. As described above, the column pair having a large correlation coefficient of similarity as shown in FIG. 3 is a redundant column pair and can be used in relation to the neighborhood correlation as described below.

  Next, the correlation and the neighborhood correlation will be examined in a little more detail with reference to FIGS.

  Note that the correlation coefficient significance test method is also used for the determination of the magnitude of the neighborhood correlation, as in the determination of the magnitude of the correlation.

  FIG. 4 is a scatter diagram, where the vertical axis represents, for example, the similarity of the address column, and the horizontal axis represents the similarity of the transaction product column. For this column pair, it can be seen that the similarity correlation is very small. In addition, since the similarity of the address column of the record having a high similarity of the transaction product column (that is, the hatched portion where the similarity of the transaction product column is close to 1) is also different based on the similarity of the transaction product column. It can be seen that the correlation is also small. Such column pairs are useless for name identification.

  FIG. 5 is also a scatter diagram, where the vertical axis represents, for example, the similarity of the company name, and the horizontal axis represents the similarity of the company kanji name column. It can be seen that this column pair has a large correlation in similarity. In addition, based on the similarity of the company kanji name column, the similarity of the company kana name column of the record having a high similarity of the company kanji name column (that is, the hatched portion where the similarity of the company kanji name column is close to 1) is also increased. ing. That is, the neighborhood correlation is also increased. However, since the magnitude of the correlation of the column pair is simply reflected in the neighborhood correlation value as it is, a column pair with a large correlation and a large neighborhood correlation cannot be said to be a useful pair for name identification by itself.

  FIG. 6 is also a scatter diagram, where the vertical axis represents, for example, the similarity of the company name column, and the horizontal axis represents the similarity of the address column. It can be seen that the overall correlation is small for this column pair. However, on the basis of the similarity of the address column, the similarity of the company Kana name column of the record having a high similarity of the company Kana name column (that is, the hatched portion where the similarity of the address column is close to 1) is high. That is, the neighborhood correlation is large. In this way, if attention is focused on a record with a high degree of similarity for a particular column, if the degree of similarity is also high for the other column, it implies that there is a record to be named, so this Such column pairs can be said to be useful for name identification.

  Furthermore, FIG. 7 is also a scatter diagram, where the vertical axis represents, for example, the similarity of the telephone number column, and the horizontal axis represents the similarity of the company name column. It can be seen that this column pair has a high correlation as a whole. Furthermore, based on the similarity of the Company Kana name column, the linkage of the similarity of the phone number column of the record with a high similarity of the Company Kana name column (that is, the hatched portion where the similarity of the Company Kana name column is close to 1) is You can see that it is getting higher. That is, the magnitude of the neighborhood correlation is larger than the magnitude of the correlation. If such a column pair is focused on a record with a high degree of similarity for a particular column and the degree of similarity is also very high for the other column, it means that there is a record to be identified. It is implied. Therefore, such a column pair can be said to be a column pair useful for name identification. When the company kanji name column and the company kana name column fall under this aspect, there are many cases where the relationship between kanji and reading is many-to-one (same syllables) and one-to-many (words with multiple readings). This includes cases where it is included.

  When the neighborhood correlation is large in this way, focusing on the record with high similarity for a specific column, the other column also has high similarity and linkage, that is, if it is a scatter diagram, it is linearly arranged It is.

  The typical cases described with reference to FIGS. 4 to 7 are summarized as shown in FIG. As will be described in detail later, in the present embodiment, the value obtained by subtracting the correlation value from the neighborhood correlation value is used. Specifically, in the case of “correlation: small and neighborhood correlation: small” as shown in FIG. 4, the column pair has no redundancy, but the column pair is not likely to be useful for name identification. Therefore, the usefulness for name identification is close to zero. Further, in the case of “correlation: large and neighborhood correlation: large” as shown in FIG. 5, there is no possibility that the column pair is useful for name identification because the column pair has only redundancy. Therefore, the usefulness for name identification is close to zero. On the other hand, in the case of “correlation: small and neighborhood correlation: large” as shown in FIG. 6, there is no redundancy in the column pair, and the column pair may be useful for name identification. Therefore, the usefulness for name identification is a positive value. Further, in the case of “correlation: large and neighborhood correlation: very large” as shown in FIG. 7, the column pair has redundancy, but the column pair may be useful for name identification. Therefore, the usefulness for name identification is a positive value.

  Therefore, first, as shown in the lower two rows in the table of FIG. 8, a column pair having a characteristic such that the correlation of similarity is positive and small or large and the neighborhood correlation of similarity is positive and large or very large can be searched. preferable.

  After that, by introducing the indicators described below, it is possible to appropriately extract columns useful for name identification.

  In the present embodiment, a table-like data structure is prepared in the database, and what is grasped as a column and a record will be described in mind.

  FIG. 9 shows a functional block diagram of the name identification support processing apparatus according to the present embodiment. The real name identification support processing device 100 includes (A) a data reading unit 110 that reads data from the name identification target database 500, (B) a first data storage unit 120, (C) a correlation calculation unit 130, and (D) a neighborhood correlation. The calculation unit 140 includes (E) a second data storage unit 150, (F) a column processing unit 160, and (G) an output data storage unit 170.

  The data reading unit 110 stores the data read from the name identification target database 500 in the first data storage unit 120. In addition, the correlation calculation unit 130 performs processing using data stored in the first data storage unit 120 and stores data being processed and processing results in the second data storage unit 150. The neighborhood correlation calculation unit 140 performs processing using the data stored in the first data storage unit 120, and stores data being processed and the processing result in the second data storage unit 150. The column processing unit 160 performs processing using the data stored in the first data storage unit 120 and the second data storage unit 150, and stores the final processing result in the output data storage unit 170. . The column processing unit 160 also stores data being processed in the second data storage unit 150. The column processing unit 160 includes a column priority calculation unit 161, a column extraction processing unit 162, and an output unit 163.

  Note that the name identification target database 500 is managed by, for example, another computer connected via a network. The other computer may be a name identification support processing device.

  Next, processing contents of the name identification support processing apparatus 100 shown in FIG. 9 will be described with reference to FIGS. 10 to 28.

  First, the data reading unit 110 reads processing target data from the name identification target database 500 and stores it in the first data storage unit 120 (FIG. 10: step S1). In this embodiment, it is assumed that a record ID for identifying a record is assigned to each record, and the column of this record ID and the column to be selected are read out. FIG. 11 shows an example of the read data. In the example of FIG. 11, a record ID column, a company kanji name column, a company kana name column, an address column, and a transaction part name column are included, and a total of 11 records are extracted. Is shown. Other columns may be included, and correct processing is performed with a larger number of records.

  Next, the correlation calculation unit 130 performs a correlation calculation process between columns (step S3). This process will be described with reference to FIGS.

  First, the correlation calculation unit 130 identifies one unprocessed record pair in the first data storage unit 120 (FIG. 12: step S21).

  In addition, the correlation calculation unit 130 identifies one unprocessed column (step S23). Then, the correlation calculation unit 130 calculates the similarity between the record pairs in the identified column, and stores it in the second data storage unit 150 (step S25). In the present embodiment, the similarity is based on the similarity based on the edit distance. For example, if a record pair with the record IDs “10000001” and “10000004” is specified, ΔΔΔ Corporation and XX Yayuyo Corporation in the company kanji name column are compared. Then, as shown in FIG. 13, all but the four characters “corporation” are different, so the similarity is calculated as follows. In FIG. 13, double circles indicate coincidence, and x indicates a portion that requires editing (replacement, deletion, insertion).

Similarity = 1.0− (number of characters required for identification / number of longer characters)
= 1.0-5 / 9 = 0.4444

  Then, the correlation calculation unit 130 determines whether all columns have been processed (step S27). If an unprocessed column exists, the process returns to step S23. If all columns have been processed, the correlation calculation unit 130 determines whether all record pairs have been processed (step S29). If there is an unprocessed record pair, the process returns to step S21. On the other hand, if all record pairs have been processed, the process proceeds to step S31.

  At this stage, when the data shown in FIG. 11 is processed, a similarity table as shown in FIG. 14 is obtained. In the example of FIG. 14, a record ID pair column, a company Kanji name similarity column, a company Kana name similarity column, an address similarity column, a transaction part name similarity column, The similarity is registered for each record pair.

  Then, the correlation calculation unit 130 identifies an unprocessed column pair in the similarity table as shown in FIG. 14 (step S31). FIG. 14 shows an example in which a column pair of “company kanji name” and “company kana name” has already been selected.

  After that, the correlation calculation unit 130 calculates a correlation coefficient for the similarity from the similarity of the column pair, and stores it in the second data storage unit 150 (step S33). The calculation method of the correlation coefficient is the same as the conventional method, but the following formula is used.

Incidentally, x _i represents the i-th similarity in column X, y _i denotes the i-th similarity in the column Y. Further, x bar (a symbol with a bar on x) represents an average value of similarity for column X, and y bar (a symbol with a bar on y) represents an average of similarity for column Y. Represents a value.

  Then, the correlation calculation unit 130 determines whether all the column pairs have been processed (step S35). If there is an unprocessed column pair, the process returns to step S31. On the other hand, if all column pairs have been processed, the process returns to the original process.

  When the processing is performed so far, for example, data as illustrated in FIG. 15 is stored in the second data storage unit 150. In the example of FIG. 15, even if the column pair is the same, the values are shown as different column pairs if the order is different. However, since the same correlation coefficient is actually obtained, any one may be calculated. In the above example, the correlation coefficient is calculated for the column pair of “company kanji name” and “company kana name” shown in bold, but the column pair of “company kana name” and “company kanji name” The correlation coefficient for is also the same value.

  Returning to the description of the processing in FIG. 10, next, the neighborhood correlation calculation unit 140 performs neighborhood correlation calculation processing between columns (step S <b> 5). The neighborhood correlation calculation process between columns will be described with reference to FIGS.

  First, the neighborhood correlation calculation unit 140 identifies one unprocessed column in the processing target data stored in the first data storage unit 120 (FIG. 16: step S41). Further, the neighborhood correlation calculation unit 140 sorts the records based on the identified column, and stores the sorting result in, for example, the second data storage unit 150 (step S43). When the processing target data shown in FIG. 11 is sorted in dictionary order based on the company kanji name column, data as shown in FIG. 17 is obtained. When sorting in the dictionary order, when attention is paid to a certain record, a record having a somewhat similar value is arranged in the vicinity of the record in the specified column. Using such a property, it is easy to collect records having similar values in the specified column by relatively light processing.

  However, as shown in FIG. 14, if the similarity for each record pair has already been calculated for each column, the same record ID is used as the ID of one record of the record pair in the specified column. By sorting the rows by similarity, it is possible to specify a row having a high degree of similarity, that is, the other record having a high degree of similarity when focusing on a specific record for the specified column.

  Next, the neighborhood correlation calculation unit 140 extracts a neighborhood record for each record based on the sorting result, and stores the data of the neighborhood record in the second data storage unit 150 (step S45). For example, assume that three records above and below a specific record are extracted. For example, in the example of FIG. 17, when the first record is selected, there is no upper record, so three lower records are specified as neighboring records. The own record is not included in the vicinity. As a method for extracting neighboring records, for example, the edit distance, that is, the similarity described above may be employed, or bi-gram search or the like may be employed. Further, the extraction range in the vicinity may not be a fixed record unit. Only records whose similarity exceeds a threshold value may be extracted. Furthermore, instead of using all records as a reference, the records may be extracted at random.

  For example, when the neighborhood record is extracted based on the criteria described above, for example, data as shown in FIG. 18 is stored in the second data storage unit 150. In the example of FIG. 18, the ID of the comparison source record and the ID of the comparison destination neighboring record are registered in association with each other.

  Then, the neighborhood correlation calculation unit 140 identifies one unprocessed comparison source record in the table as shown in FIG. 18 (step S47). Then, the neighborhood correlation calculation unit 140 calculates the similarity of each column between the identified comparison source record and each comparison destination neighborhood record, and stores it in the second data storage unit 150 (step S49). For example, if the ID of the comparison source record is “10000010”, the comparison destination neighboring records are “1000001”, “10000007”, “10000004”, and “10000009”. When calculated and registered, data (first four records) as shown in FIG. 19 is obtained. In the example of FIG. 19, a row of record ID pairs, a column of company Kanji name similarity, a column of similarity of company Kana name, a column of similarity of address, and a column of similarity of transaction part names Is provided, and the similarity of each column is registered for each combination of the comparison source record and the comparison destination neighboring record. Since the similarity has already been calculated in step S3, the similarity already calculated from the record pair and the column pair may be read and registered in the second data storage unit 150.

  Thereafter, the neighborhood correlation calculation unit 140 determines whether all comparison source records have been processed (step S51). If there is an unprocessed comparison source record, the process returns to step S47. When all the comparison source records are processed, data as shown in FIG. 19 is prepared.

  Then, the neighborhood correlation calculation unit 140 identifies one unprocessed column pair in the data as shown in FIG. 19 (step S53). Thereafter, the neighborhood correlation calculation unit 140 calculates a correlation coefficient as a neighborhood correlation coefficient from the similarity of the identified column pair, and stores it in the second data storage unit 150 (step S55). About the calculation method of a correlation coefficient, it calculates with the method similar to the method demonstrated by step S33. Then, the processing shifts to the processing in FIG.

  Shifting to the description of the processing in FIG. 20, the neighborhood correlation calculation unit 140 determines whether or not all column pairs have been processed (step S57). If there is an unprocessed column pair, the process returns to step S53 via the terminal C. On the other hand, if processing has been performed for all column pairs, the neighborhood correlation calculation unit 140 determines whether processing has been performed for all columns in the data as illustrated in FIG. 11 (step S59). If there is an unprocessed column, the process returns to step S41. On the other hand, if all columns have been processed, the process returns to the original process.

  When processing is performed up to this point, for example, data as shown in FIG. 21 is stored in the second data storage unit 150. In the example of FIG. 21, the neighborhood correlation coefficient is registered for each column pair. Although the neighborhood correlation coefficient between the same columns is not calculated, it is “1” and is shown in FIG. In addition, since records are sorted for each column, unlike the correlation, the value of the neighborhood correlation coefficient differs if the comparison source column in the column pair is different. When steps S53 to S57 are executed once, one neighborhood correlation coefficient in one row in the table as shown in FIG. 21 is calculated, and when steps S53 to S57 are repeated for all the column pairs, a table as shown in FIG. The neighborhood correlation coefficient for one row at is calculated. If Steps S41 to S59 are repeated for all the columns, the neighborhood correlation coefficients of all the rows in the table as shown in FIG. 21 are calculated. In addition, as shown in FIG. 19, different data is generated every time Step S41 to Step S51 are performed.

  In this way, the neighborhood correlation is calculated and stored in the second data storage unit 150. As described above, when the neighborhood correlation calculation process is performed without using the similarity calculated by the correlation calculation unit 130 and stored in the second data storage unit 150, the processing order of step S3 and step S5 is as follows. Are interchangeable and can be executed in parallel.

  Returning to the description of the processing in FIG. 10, the column priority calculation unit 161 of the column processing unit 160 performs column priority calculation processing (step S7). The column priority calculation processing will be described with reference to FIGS. 22A to 23.

  First, the column priority calculation unit 161 calculates the limit value of the significance level for each correlation coefficient and each neighboring correlation coefficient from the number of data, and stores it in the second data storage unit 150, for example (step S61). For example, in the case of a correlation coefficient, the number of data records as shown in FIG. 14 is counted and specified as the degree of freedom. In the case of the neighborhood correlation coefficient, the number of data records as shown in FIG. 19 is counted and specified as the degree of freedom. In addition, 5% and 1% are often used as the significance level, but one of the limit value tables (also referred to as r table. The limit value is also referred to as the risk factor) is prepared in advance, and according to the degree of freedom. Identify limit values. When the limit value corresponding to the degree of freedom is not registered, the limit value having a value with a close degree of freedom may be calculated by interpolation.

  The processing results and limit values of steps S5 and S7 stored in the second data storage unit 150 are summarized as shown in FIG. 22B. In the example of FIG. 22B, for each column, the values of the correlation coefficient and the neighborhood correlation coefficient of the corresponding column are registered, and further, the number of data of the correlation coefficient and the neighborhood correlation coefficient and the number of the data correspond A limit value of 5% significance level is also registered.

  Next, the column priority calculation unit 161 extracts column pairs having a positive correlation coefficient and exceeding the limit value of the corresponding significance level from all the column pairs, and stores the column pairs in, for example, the second data storage unit 150 ( Step S63).

  In the example of FIG. 22B, a column pair of “company kanji name” and “company kana name” corresponds. Specifically, the correlation coefficient “0.8902” of the column pair is compared with the corresponding limit value “0.2649” of the 5% significance level. By executing step S63, a redundant column pair is specified. Such a column pair is used in relation to a column pair to be extracted later.

  Further, the column priority calculation unit 161 extracts, from all column pairs, column pairs in which the correlation coefficient and the neighborhood correlation coefficient are positive and the neighborhood correlation coefficient exceeds the corresponding limit value of the significance level, For example, it is stored in the second data storage unit 150 (step S65).

  In the example of FIG. 22B, hatched portions, that is, a column pair of “company kanji name” and “company kana name”, a column pair of “company kana name” and “company kanji name”, and “company kana name” And a column pair of “address” and a column pair of “address” and “company name” are extracted. There is no distinction between the comparison source and the comparison destination if only the correlation is considered, but in the case of the neighborhood correlation, the comparison source and the comparison destination are distinguished. This identifies column pairs that may be useful for name identification. However, the column pair obtained in step S65 may include a column pair having redundancy as described above.

  Therefore, the column priority calculation unit 161 identifies a column useful for name identification from the column pair obtained in step S63 and the column pair obtained in step S65 (step S67). For example, it is specified by the union of columns included in the column pair obtained in steps S63 and S65. In the example described above, “company kanji name”, “company kana name”, and “address” are specified.

  In the example described above, such a process is possible because the number of columns specified in steps S63 and S65 is small. On the other hand, when the number of columns specified in steps S63 and S65 is large, for example, it is determined whether or not each column of the database is included in any of the column pairs included in steps S63 and S65. If it is included, it may be determined that the column may be useful for name identification. If it is not included, it may be determined that the column is not useful for name identification.

  Then, the column priority calculation unit 161 calculates the usefulness and the priority for each of the identified columns, and stores it in the second data storage unit 150 (step S69). The process then returns to the original process.

In the present embodiment, the following calculation is performed for each combination of columns specified in step S67. For example, consider the comparison source column C _i and the comparison destination column C _j .
Usefulness Δ (C _i , C _j ) = neighbor correlation (C _i , C _i ) −correlation (C _i , C _j )

  Since there is a possibility that the neighborhood correlation coefficient of the column pair includes redundancy, the usefulness of the column pair is used to obtain the net usefulness excluding the redundancy portion that appears in the correlation.

Further, the priority of each column C _i is calculated as follows.
C _i priority = max _j {Δ (C _i , C _j ), Δ (C _j , C _i )}

  This means that the maximum value of the usefulness seen from the own column and the usefulness seen from the other column is obtained from the other columns. Specifically, from the example described above, if “Company Kanji Name” is the own column, the usefulness of other columns “Company Kana Name” and “Address” and “Company Kanji Name” to “Company Kanji Name” The usefulness for “name” is compared with the usefulness for “company kanji name” from “address”.

  FIG. 23 shows the calculation results of the usefulness and the priority. In the example of FIG. 23, for each column, the correlation coefficient, the neighborhood correlation coefficient, and the usefulness are listed for each other corresponding column and the priority is also shown. As described above, if “Company Kanji Name” is the own column, the usefulness to be compared is the numerical value of the hatched part, so the priority “0.0811” is selected as shown by the arrow Is done. Similarly, the basis for the priority of each column is indicated by an arrow.

  The priority calculation method may be another method. For example, the sum of usefulness described above, an average value, or the like may be used.

  As described above, using the neighborhood correlation in addition to the correlation, the usefulness indicating the degree of possibility of being useful for name identification and the priority indicating the degree to be prioritized at the time of comparison are determined.

  Returning to the description of the processing in FIG. 10, the column extraction processing unit 162 of the column processing unit 160 performs the unification processing of the redundant column group (step S <b> 9). This process will be described with reference to FIGS.

  The column extraction processing unit 162 converts, among the useful columns (the column identified in step S67), the common column pair (the column extracted in step S63) having a significant correlation to the redundant column group. Classification is performed, and the classification result is stored in the second data storage unit 150 (FIG. 24: step S71). As schematically shown in FIG. 25, “company kanji name”, “company kana name”, and “address” are extracted as useful columns in step S67. In step S63, “company kanji name” and “company kana name” are extracted. Since the “name” column pair is extracted, the redundant column group classifies the common “company kanji name” and “company kana name” as the redundant column group. In the example described above, since only one column pair is extracted, only one redundant column group is generated. On the other hand, the non-redundant column is an “address” column.

  Next, the column extraction processing unit 162 registers columns that do not belong to the redundant column group among the useful columns to the non-redundant column group, and stores the data about the non-redundant column group in the second data storage unit 150. Store (step S73). Further, the column extraction processing unit 162 selects the column with the highest priority among the redundant column groups as the representative column, and stores the selection result in the second data storage unit 150 (step S75). As schematically shown in FIG. 26, “company name” having high priority is selected as the representative column from “company name” and “company name”.

  Then, the column extraction processing unit 162 extracts a column having a priority level equal to or higher than the threshold from the representative column and the non-redundant column, and stores the data of the column in the second data storage unit 150 as a final column selection result. Store (step S77). As the threshold value, for example, the limit value / 2 of the significance level for the neighborhood correlation is adopted. As schematically shown in FIG. 27, since both the representative column of the redundant column group and the column of the non-redundant column group are larger than the priority threshold value, both “company name” and “address” are selected. Then, the process returns to the original process.

  Returning to the description of the processing in FIG. 10, the output unit 163 reads the data stored in the second data storage unit 150 and stores the data in the output data storage unit 170 or an output device (display device, printing device, or the like). (Step S11). For example, data as shown in FIG. 28 is output. In the example of FIG. 28, each column includes data on priority and redundancy groups. The redundancy group includes the names of other columns belonging to the redundancy group, the correlation coefficient (= redundancy), and the priority.

  By referring to such data, the user can know which column should be used for name identification. If the column thus selected is used, it is expected that the accuracy of name identification can be improved and the processing speed can be increased. In addition, you may make it output to the computer and process which perform a name identification process as it is, without outputting to a user.

[Embodiment 2]
Next, an embodiment in the case where data loss exists in any column of the record will be described. In the present embodiment, the configuration of the first embodiment is used. In the present embodiment, a part of the functions included in the name identification support processing apparatus in FIG. 9 is included as a column selection unit 600 modified as described below, and a name identification support processing apparatus 700 as shown in FIG. 29 is used.

  The name identification support processing device 700 includes a column selection unit 600, a preprocessing unit 710, an estimation unit 720, and a third data storage unit 730. As illustrated in FIG. 9, the column selection unit 600 includes a first data storage unit 120, a second data storage unit 150, and an output data storage unit 170.

  The pre-processing unit 710 performs pre-processing described below on the data stored in the name identification target database 500 stored in the first data storage unit 120 and sends the processing result to the third data storage unit 730. Store. The estimation unit 720 performs processing using the data stored in the first data storage unit 120, the second data storage unit 150, the output data storage unit 170, and the third data storage unit 730, and performs the second data storage unit The processing result is stored in 150. Note that the estimation unit 720 stores data being processed in the third data storage unit 730.

  Next, processing in the present embodiment will be described with reference to FIGS. 30 to 45.

  First, the column selection unit 600 performs a first column selection process (FIG. 30: Step S101). The first column selection process is a process in which the process described in the first embodiment is partially changed. In the first data storage unit 120, for example, a record group including data loss is stored as indicated by “N / A” in FIG. Therefore, when calculating the similarity of a certain record pair for a certain column, there is a case where one or both of the data does not exist. In this case, the similarity is not calculated. That is, even in the similarity data as shown in FIG. 14, a column in which the similarity is not calculated occurs. Then, the record pair related to the column is not taken into account in calculating the correlation coefficient for the combination of the column related to the column and another column.

  Similarly, in the similarity data as shown in FIG. 19, a column in which the similarity is not calculated occurs. However, in the case of FIG. 19, the similarity is calculated for a record pair with a similar record for a specific column (in the case of FIG. 19, “company kanji name”). Does not exist, but since the similarity cannot be calculated for other columns, a column in which the similarity is not calculated occurs. Then, the record pair according to the column is not taken into consideration in calculating the neighborhood correlation coefficient for the combination with the column according to the column.

  Since the number of effective record pairs decreases in this way, the values as they are cannot be used for the correlation coefficient and the neighborhood correlation coefficient. In this embodiment, the significance level is set according to the number of valid record pairs. Therefore, the number of valid record pairs is counted in the data of FIGS. On the other hand, a value obtained by multiplying the correlation coefficient or the neighborhood correlation coefficient by the weight d is used. As the weight d, it is preferable to use a non-missing rate in units of records according to the experiment of the present inventors. In the example of FIG. 31A, when considering a column pair of a company kanji name column and a company kana name column, as shown schematically in FIG. 31B, one or both data are missing in one record. In this case, the record is excluded, and the ratio of records in which both data are registered is calculated and used as the weight d. Specifically, the weight d is calculated by dividing the number of records in which data is registered in both columns by the total number of records. In the case of FIG. 31B, since 3 records out of 8 records are excluded, 0.625 is set to d. Therefore, in the significance determination of the correlation coefficient and the neighborhood correlation coefficient, it is determined whether the significance level <the correlation coefficient or the neighborhood correlation coefficient × d.

  Note that the process of calculating the weight d may be performed in the middle of the process of the column selection unit 600, or the preprocessing unit 710 causes the column selection unit 600 to transfer data from the name identification target database 500 to the first data storage unit 120. May be collectively calculated for each column pair and output to the first data storage unit 120 or the like (second data storage unit 150 or the like) of the column selection unit 600.

  Furthermore, when calculating the similarity as shown in FIG. 14, the average value and the variance of the similarity are calculated for each column and stored in the second data storage unit 150 or the like.

  Next, the preprocessing unit 710 performs preprocessing and stores the processing result in the third data storage unit 730 (step S103). The preprocessing is processing for generating data used in the present embodiment, and includes processing for generating an appearance frequency table for each column. The appearance frequency table is a table as shown in FIG. 32, for example. FIG. 32 is an appearance frequency table for the company name, and the result of counting how many times each data value appears in the company name column is registered.

  Thereafter, the estimation unit 720 performs a missing value estimation process (step S105). The missing value estimation process will be described with reference to FIGS.

  First, the estimation unit 720 performs an estimable column determination process (FIG. 33: step S111). This estimable column determination process will be described with reference to FIGS.

  First, the estimation unit 720 specifies one unprocessed column C1 among non-selected columns (columns other than the column selected in the first column selection process and stored in the output data storage unit 170) (FIG. 34). : Step S131). Then, the estimation unit 720 determines whether or not a missing value exists in the identified column C1 in the name identification target data stored in the first data storage unit 120 (step S133). If there is no missing value in the identified column C1, no estimation is made, and the process proceeds to step S145 in FIG. On the other hand, when there is a missing value in the specified column C1, the estimation unit 720 performs unprocessed processing among the selected columns (the columns selected in the first column selection process and stored in the output data storage unit 170). One column C2 is identified (step S135). Then, the estimation unit 720 determines whether the correlation coefficient between the column C1 and the column C2 stored in the second data storage unit 150 is equal to or greater than a threshold value (step S137). For example, a statistical significance level is used as the threshold value in this step. The statistical significance level is specified by the number of record pairs whose similarity is calculated in both columns. In step S137, the correlation coefficient calculated in the first column selection process is used as it is. This is because this processing is performed to relieve a column that has not been evaluated correctly due to many missing values.

  If the correlation coefficient between the columns C1 and C2 is less than the threshold value, the process proceeds to step S143 in FIG. On the other hand, if the correlation coefficient between the column C1 and the column C2 is equal to or greater than the threshold, the estimation unit 720 has a predetermined number or more (or a predetermined ratio or more) of records in which the data values of the columns C1 and C2 are not missing values. It is determined whether or not there is (step S139). Whether to use the number of records obtained when calculating the weight d or the weight d itself is used. This step is for maintaining statistical reliability. If there are less than a predetermined number of records whose data values in columns C1 and C2 are not missing values, the process proceeds to step S143 in FIG.

  On the other hand, if the number of records in which the data values of the columns C1 and C2 are not missing values is equal to or greater than a predetermined number, the estimation unit 720 adds the columns C1 and C2 to the estimable column pair list in the third data storage unit 730. Registration is performed (step S141). Then, the process proceeds to step S143 in FIG.

  For example, it is assumed that the column C2 that is the processing result of the first column selection process is specified as shown in FIG. 35A, and the remaining non-selected column C1 is specified as shown in FIG. 35B. Here, when it is determined whether or not there is a missing data value as shown in FIG. 35B, the non-selected column C1 as shown in FIG. 35C is processed after step S135. Then, when the correlation coefficient between the selected column C2 and the non-selected column C1 with missing data is compared with its significance level as shown in FIG. 35D, the “company kanji name” column and the “company kana name” column are And the column pair of the “zip code” column and the “address” column are identified as column pairs having a correlation coefficient equal to or higher than the significance level. Then, those column pairs are registered in the estimable column pair list as shown in FIG. 35E.

  36, the estimation unit 720 determines whether there is an unprocessed column for the selected column (FIG. 36: step S143). If there is an unprocessed column in the selected column, the process returns to step S135 in FIG. On the other hand, if there is no unprocessed column in the selected column, it is determined whether there is an unprocessed column for the unselected column (step S145). If there is an unprocessed column for the non-selected column, the process returns to step S131 in FIG. On the other hand, if there is no unprocessed column for the non-selected column, the process returns to the caller process.

  In this way, as shown in FIG. 35E, a non-selectable column that can be estimated in the column of the non-selected column of the estimable column pair list is specified. The selected column registered in the selected column column of the estimable column pair list is used as a basis for estimating the similarity.

  Returning to the description of the processing in FIG. 33, the estimation unit 720 determines the column of the column that can be estimated from the data stored in the second data storage unit 150 (for example, the data of the similarity for each record pair shown in FIG. 14). Regression analysis is performed to calculate a regression coefficient, which is stored in the third data storage unit 730 (step S113). This is because, as described below, for column pairs with a low possibility of name identification, the similarity is estimated using the results of regression analysis. Note that a regression calculation is performed with a selected column registered in the estimable column pair list as a column pair of estimable columns. When there are a plurality of selected columns registered in the estimable column pair list, multiple regression analysis is performed and the regression coefficients are stored in the third data storage unit 730. Note that the regression analysis may be performed using another column.

  Then, the estimation unit 720 specifies one unprocessed record pair in the first data storage unit 120 (step S115). Thereafter, the estimation unit 720 performs a name identification possibility estimation process (step S117). This name identification possibility estimation process will be described with reference to FIG.

  First, the estimation unit 720 determines whether the similarity of the data in the selected column in the identified record pair exceeds a threshold (for example, 0.8 or 0.9) (FIG. 37: step S151). The selection column used here is a column registered in the selection column of the estimable column pair list. This is because, as described in the first embodiment, the similarity of columns useful for name identification reflects the degree of possibility of name identification. When a plurality of selectable columns in the estimable column pair list are included, it is determined whether the sum of similarities exceeds a threshold or whether the minimum value of similarities exceeds the threshold.

  When the similarity of the data in the selected column in the identified record pair exceeds the threshold, the estimation unit 720 registers the identified record pair in the high name identification list in the third data storage unit 730 ( Step S153). Thereafter, the process returns to the calling process. On the other hand, if the similarity of the data in the selected column in the identified record pair is less than or equal to the threshold value, the process returns to the caller process.

  For example, as schematically shown in FIG. 38A, for the record pair with IDs “100001” and “100009”, the similarity of the selection column C2 “enterprise kanji name” is “1.0”, which exceeds the threshold value 0.9. If it is, the “established company name” which is a non-selectable column that can be estimated is also estimated to have a high degree of similarity. That is, it is presumed that the possibility of name identification of these record pairs is high, and such record pairs are registered in the high name identification possibility list.

  On the other hand, as schematically shown in FIG. 38B, for the record pair with IDs “100014” and “100017”, the similarity of the selection column C2 “enterprise kanji name” is “0.36” and the threshold is 0.9 or less. In some cases, it is estimated that “established company name” which is a non-selectable column that can be estimated is also low in similarity. That is, the possibility of name identification of these record pairs is estimated to be low, and such record pairs are not registered in the high name identification possibility list.

  Returning to the description of the processing in FIG. 33, the estimation unit 720 identifies one unprocessed estimable non-selected column (step S <b> 119). Then, the estimation unit 720 determines whether only one of the identified record pairs is missing in the identified non-selectable column that can be estimated (step S121). If both are not missing, estimation is unnecessary, and if both are missing, it is determined that estimation is impossible. Therefore, in the specified non-selectable column that can be estimated, when only one of the specified record pairs is not missing, the process proceeds to step S167 in FIG. On the other hand, if only one of the identified record pairs is missing in the identified non-selectable column that can be estimated, the process proceeds to step S161 in FIG.

  39, the estimation unit 720 determines whether or not this record pair is registered in the high name collation possibility list in step S117 (FIG. 39: step S161). If this record pair is registered in the high possibility collation list, the estimation unit 720 performs the first estimation process (step S163). The first estimation process will be described with reference to FIGS. 40 to 41C. When the first estimation process is completed, the process proceeds to step S167.

  First, the estimating unit 720 specifies the non-missing value v1 of the specified record pair from the appearance frequency table of the non-selectable column C1 that can be estimated (FIG. 40: Step S171). As schematically illustrated in FIG. 41A, when processing is performed on the record pairs with IDs “100001” and “100009”, the non-missing value v1 in the presumable non-selection column “company name” is “ABC show”. Then, in the appearance frequency table of “company name” shown schematically in FIG. 41B, an arrow row “ABC show” is specified.

  Then, the estimation unit 720 calculates the similarity with the non-missing value v1 for a data value in the vicinity of the non-missing value v1 in the appearance frequency table (step S173). In combination with the processing in step S175, the similarity between the nearby data value and the non-missing value v1 may be calculated until the value becomes less than the threshold value. In the example of FIG. 41B, assuming that a = 1, the similarity to the non-missing value v1 “ABC show” is calculated one by one before and after. In the example of FIG. 41B, the similarity “0.60” is calculated for “ABC”, and the similarity “0.82” is calculated for “ABC SHOW TEN”. A value larger than a = 1 may be adopted.

  Then, the estimation unit 720 calculates the similarity addition value weighted by using the appearance frequency of neighboring data values (including itself) for which a similarity degree equal to or greater than a threshold (for example, 0.8) is calculated. The estimated value is calculated and stored in the second data storage unit 150 (step S175).

  In the example of FIG. 41B, since the threshold value is 0.8, “ABC” is not used, and “ABC show” and “ABC show ten” are used. And the similarity weighted by the ratio of the applicable appearance frequency with respect to the total number of appearance frequencies is added. That is, (3 / (3 + 1)) × 1.0 = 0.75 is calculated for “ABC show”, and (1 / (3 + 1)) × 0.82 = 0.205 is calculated for “ABC show ten”. Is done. Therefore, these are added together to obtain 0.955 as the estimated value of similarity.

Generally, it is expressed as follows. Note that W _k represents a weight for the neighborhood data value k having a similarity equal to or greater than a threshold, and (appearance frequency of the corresponding neighborhood data value k) / (sum of appearance frequencies for neighboring data values having a similarity equal to or greater than the threshold). It is. S _k represents the degree of similarity between the neighborhood data value k and the non-missing value v1.

  41A and 41B, in step S175, data as shown in FIG. 41C is stored in the second data storage unit 150. That is, for the record pair with IDs “100001” and “100009”, the estimated similarity value for “company name” is registered as “0.955”.

  In this way, a record pair with a high possibility of name identification is estimated from the similarity of data values in the vicinity of the non-missing value v1 in the appearance frequency table.

  Returning to the description of the processing in FIG. 39, when the identified record pair is not registered in the high name matchability list, the estimation unit 720 performs the second estimation processing (step S165). For the second estimation process, a conventional technique is used. Specifically, the regression coefficient calculated in step S113 and stored in the third data storage unit 730 is used.

  For example, in the case as shown in FIG. 38B, when the only selection column whose correlation coefficient is equal to or greater than the threshold for the estimable non-selected column “company Kana name” is “company kanji name”, as shown in FIG. A simple regression line is obtained by regression analysis. Therefore, from this regression line, as shown in FIG. 38B, an estimated similarity “0.39” corresponding to the similarity “0.36” of the selected columns of the record pairs with IDs “100014” and “100017” is obtained. Further, a normal random number is generated according to the average value and the variance value of the estimable non-selected columns calculated when calculating the similarity as shown in FIG. 14 and added to the estimated similarity. For example, if the random number “0.01” is calculated, 0.39 + 0.01 = 0.40 is calculated as the final estimated similarity. This final estimated similarity is stored in the second data storage unit 150 as shown in FIG. This corrects the deviation of the variance value. Then, control goes to a step S167.

  Proceeding to step S167, the estimating unit 720 determines whether there is an unprocessable estimable non-selected column in the estimable column pair list (step S167). If there is an unprocessable estimable unselected column, the process returns to step S119 via the terminal F. On the other hand, when there is no unprocessable estimable non-selected column, the estimation unit 720 determines whether there is an unprocessed record pair (step S169). If there is an unprocessed record pair, the process returns to step S115 via the terminal G. On the other hand, if there is no unprocessed record pair, the process returns to the caller process.

  As described above, when the possibility of name identification is high, the similarity is calculated based on the similarity and appearance frequency of similar data values. On the other hand, when the possibility of name identification is low, the similarity corresponding to the tendency of other data is calculated while correcting the deviation of the variance value. However, it is not possible to calculate the similarity of record pairs for all the deficiencies.

  Returning to the description of the processing in FIG. 30, the estimation unit 720 converts the estimated similarity data stored in the second data storage unit 150 into the correlation coefficient calculation table (FIG. 14) and the neighborhood correlation coefficient calculation. It sets to a table (FIG. 19) (step S107). In this way, a defective part can be reduced.

  Then, the column selection unit 600 performs the second column selection process (step S109). In the second column selection process, unlike the first column selection process, data as shown in FIG. 14 and FIG. 19 is generated. Therefore, the correlation coefficient and the neighborhood correlation coefficient are calculated for each column pair. Others are the same. Since the defect still exists, the weight d is recalculated, and the significance level is re-specified according to the condition of the defect.

  By performing the above processing, even if there is a missing part in the data to be analyzed in column selection in the name identification field, a more accurate correlation coefficient can be calculated by performing an estimation suitable for the name identification problem. And useful column selection results can be obtained.

  Note that the name identification possibility estimation process (FIG. 37) can be changed to the process shown in FIG. Specifically, the estimation unit 720 determines whether the similarity of the selected column in the identified record pair exceeds a threshold value and the data value of the selected column is unusual (FIG. 44: step S181). Specifically, it is determined whether the appearance frequency is less than a threshold in the appearance frequency table of the selected column. For example, assuming the situation shown in FIG. 38A, the appearance frequency “3” is obtained from the appearance frequency table shown in FIG. 45 for “ABC Trading” in the selection column C2. From the threshold “10” for the appearance frequency, it may be determined that “ABC Trading” is unusual, or the relative frequency (appearance probability) divided by the number of records is calculated, and it is determined that it is unusual in comparison with the threshold for the relative frequency. May be. Note that the value of the selected column C2 may be different in the record pair. In this case, for example, the determination is made after adding both values. You may judge by an average value. Such processing adds a condition that the data value of the record to be identified is unusual because the data value does not appear frequently.

  Although the embodiment of the present technology has been described above, the present technology is not limited to this. For example, although the example which performs the process which calculates a priority was shown above, as long as the column can be narrowed down to a sufficiently small number, it may be enough only to implement to step S67, for example. Similarly, if narrowing down is possible, the user may be able to determine just by presenting only the processing result of step S65.

  Furthermore, the functional block diagrams shown in FIGS. 9 and 29 are examples, and may not necessarily match the actual program module configuration. As for the processing flow, as long as the processing result does not change, the order may be changed or it may be executed in parallel.

  The configuration of the loop in the processing flow can be changed as long as the same processing result is obtained. In the second embodiment, since the non-missing rate in units of records is calculated for each column pair, the processing flow can be changed so that missing records or non-missing records can be easily counted.

  The name identification support processing devices 100 and 700 described above are computer devices, and display control connected to a memory 2501, a CPU 2503, a hard disk drive (HDD) 2505, and a display device 2509 as shown in FIG. A unit 2507, a drive device 2513 for a removable disk 2511, an input device 2515, and a communication control unit 2517 for connecting to a network are connected by a bus 2519. An operating system (OS) and an application program for executing the processing in this embodiment are stored in the HDD 2505, and are read from the HDD 2505 to the memory 2501 when executed by the CPU 2503. If necessary, the CPU 2503 controls the display control unit 2507, the communication control unit 2517, and the drive device 2513 to perform necessary operations. Further, data in the middle of processing is stored in the memory 2501 and stored in the HDD 2505 if necessary. In an embodiment of the present technology, an application program for performing the above-described processing is stored in a computer-readable removable disk 2511 and distributed, and installed from the drive device 2513 to the HDD 2505. In some cases, the HDD 2505 may be installed via a network such as the Internet and the communication control unit 2517. Such a computer apparatus realizes various functions as described above by organically cooperating hardware such as the CPU 2503 and the memory 2501 described above, the OS, and necessary application programs.

  The above-described embodiment can be summarized as follows.

  In the name identification support processing method according to the present embodiment, (A) a step of calculating a first data similarity between records for each column in the database and storing it in a data storage unit; and (B) a combination of columns Calculating a correlation coefficient of the first data similarity stored in the data storage unit for each, and storing the correlation coefficient in the data storage unit; and (C) an arbitrary column in the target column which is an arbitrary column of the columns A record having data similar to or similar to data in the target record that is a record is specified, and a second data similarity for each column in each of the range between the target record and the specified record is calculated or The second data stored in the data storage unit for each combination of the column of interest and other columns is specified and stored in the data storage unit. A neighborhood correlation coefficient calculation step of calculating a correlation coefficient of the similarity as a neighborhood correlation coefficient and storing it in the data storage unit; and (D) a positive correlation coefficient that is calculated and exceeds a predetermined significance level. A first extraction for performing a first extraction process for extracting from the data storage unit a combination of columns for which a positive neighborhood correlation coefficient is multiplied by a neighborhood correlation coefficient or a ratio of records in which no data is missing. Steps.

  Thus, by introducing neighborhood correlation in addition to correlation, columns that may be useful for name identification can be automatically extracted. In addition, the processing speed is increased. Note that the ratio of records in which no data is missing is preferably determined based on the combination of columns to be extracted.

  Note that the neighborhood correlation coefficient calculating step described above includes (c1) a step of sorting records by the data of the target column, and (c2) a predetermined number of records before and after the target record in the sorted sequence. And identifying as a record having data similar to or similar to the data. In this way, the neighborhood correlation coefficient can be calculated with a reduced processing load. Note that similar records may be specified based on the first data similarity.

  Further, the name identification support processing method (E) for each extraction column that is a column included in the combination of extracted columns, the correlation coefficient between the extraction column and another extraction column and the neighborhood correlation coefficient And calculating the difference between the usefulness calculated for the extracted column and the extracted column calculated for the other extracted columns for each of the extracted columns, And calculating the priority from the usefulness of the data and storing the priority in the data storage unit. In this way, it is possible to obtain a degree useful for name identification and a standard for comparing columns included in the extracted column combination. Note that the priority calculation method may be a maximum value, an average value, a sum, or the like.

  Further, the name identification support processing method includes: (G) a second extraction step for extracting a combination of columns for which a positive correlation coefficient exceeding a predetermined significance level is calculated; and (H) a first extraction step. And extracting a column having the highest priority among columns common to the columns included in the column combination extracted in step 2 and the columns included in the column combination extracted in the second extraction step. Also good. In this way, it is possible to specify the column that has the highest priority among the column groups having redundancy.

  Further, the name identification support processing method is (I) a column after excluding columns included in the combination of columns extracted in the second extraction step from columns included in the combination of columns extracted in the first extraction step. You may make it further include the combination production | generation step which produces | generates the combination of a 2nd extraction column and a column with the highest priority. In this way, the column that should be used for name identification is automatically obtained.

  The priority of the second extraction column stored in the data storage unit and the priority of the column with the highest priority may be combined when they are equal to or higher than a predetermined threshold. This is to increase the accuracy of name identification.

  Further, the name identification support processing method includes: (J) a second column that is a column other than the first column that is a column extracted in the first extraction step or a column related to the combination generated in the combination generation step; Identifying a third column that satisfies a first predetermined condition including a condition that data loss has occurred and a correlation coefficient with the first column is equal to or greater than a threshold; and (K) each record pair If the second predetermined condition including the condition that the similarity of the data in the first column is equal to or greater than the second threshold is satisfied, the similarity with the non-missing data of the record pair in the third column Is more than the third threshold value and similar data appearing in the third column (for example, there are cases where the data is extracted only under the similarity condition or there is a limit on the number of data The weighted addition value of the similarity is calculated as the similarity for the third column, and when the second predetermined condition is not satisfied, based on other methods of the first column and the third column The calculated second value is calculated as the degree of similarity for the third column, and the data stored in the data storage unit that stores the estimation step for storing in the data storage unit and (L) the processing result of the estimation step is stored. And calculating again the correlation coefficient of the first similarity and re-calculating the correlation coefficient of the second similarity, and (M) recalculating the correlation coefficient and re-calculating the neighborhood correlation coefficient. A third extraction step of extracting from the data storage unit a combination of columns for which a positive correlation coefficient is calculated from the calculation result and a positive neighborhood correlation coefficient exceeding a predetermined significance level is calculated may be included. .

  In this way, even if there is missing data, the similarity can be estimated by an appropriate method according to the possibility of name identification, and if the estimation result is used, an appropriate column as a whole is extracted. It becomes like this.

  Note that the first predetermined condition described above may further include a condition that the number of records in which data is not missing in both the first column and the second column is greater than or equal to the fourth threshold. . This is to ensure the reliability of the estimation.

  Furthermore, the second predetermined condition described above may further include a condition in which the appearance frequency of data in the first column for the record pair is less than a fifth threshold. This is because when the possibility of name identification is high, the data value of the column is not a data value that appears frequently.

  Further, the weight value of the weighted addition value described above may be a ratio of the appearance frequency of the similar data to the sum of the appearance frequencies of the similar data. In this way, the appearance frequency can be appropriately reflected in the estimated similarity.

  Further, the second value described above calculates the similarity in the third column corresponding to the similarity of the data in the first column from the regression calculation result of the first column and the third column, It may be the result of adding a random value to the calculated similarity. This makes it possible to correct the variance of the estimated value.

  A program for causing a computer to perform the processing described above can be created, such as a flexible disk, a CD-ROM, a magneto-optical disk, a semiconductor memory (for example, ROM), a hard disk, etc. Stored in a computer-readable storage medium or storage device. Note that data being processed is temporarily stored in a storage device such as a RAM.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Appendix 1)
For each column in the database, a first data similarity between records is calculated, stored in a data storage unit, and the first data similarity stored in the data storage unit for each combination of the columns A correlation coefficient calculating unit that calculates a correlation coefficient and stores the correlation coefficient in the data storage unit;
A record having data similar to or similar to data in an attention record that is an arbitrary record in an attention column that is an arbitrary column of the columns is identified, and the record between the attention record and the identified record is specified. And calculating or specifying the second data similarity for each column and storing it in the data storage unit, and storing each combination of the column of interest and other columns in the data storage unit. A neighborhood correlation coefficient calculation unit that calculates a correlation coefficient of the second data similarity as a neighborhood correlation coefficient and stores it in the data storage unit;
The positive neighborhood correlation coefficient for which the positive correlation coefficient has been calculated and multiplied by the percentage of records that have a positive neighborhood correlation coefficient exceeding the predetermined significance level or no data loss has occurred is calculated. A column processing unit for performing a first extraction process for extracting a combination from the data storage unit;
A name identification support processing apparatus.

(Appendix 2)
The neighborhood correlation coefficient calculating unit
The records are sorted by the data of the column of interest, and a predetermined number of records before and after the record of interest are specified as records having data similar or similar to the data in the record of interest in the sorted sequence. 1. The name identification support processing device according to 1.

(Appendix 3)
The column processing unit
For each extraction column that is a column included in the extracted combination of columns, the difference between the correlation coefficient between the extraction column and another extraction column and the neighborhood correlation coefficient is calculated as a usefulness. , Stored in the data storage unit,
For each of the extraction columns, a priority is calculated from the usefulness calculated for the extraction column and the usefulness of the extraction column calculated for the other extraction column, and stored in the data storage unit ,
The name identification support processing device according to attachment 1 or 2.

(Appendix 4)
The column processing unit
Extracting from the data storage unit a combination of the columns in which the positive correlation coefficient exceeding a predetermined significance level or the positive neighborhood correlation coefficient multiplied by the ratio of records in which no data loss has occurred is calculated. Perform the second extraction process,
The column having the highest priority among the columns common to the columns included in the combination of columns extracted in the first extraction process and the columns included in the combination of columns extracted in the second extraction process Extract,
The name identification support processing device according to attachment 3.

(Appendix 5)
The column processing unit
A second extraction column that is a column after excluding a column included in the combination of columns extracted in the second extraction process from a column included in the combination of columns extracted in the first extraction process; The name identification support processing device according to appendix 4, which generates a combination with a column having the highest priority.

(Appendix 6)
The name identification according to appendix 5, wherein the priority of the second extraction column stored in the data storage unit and the priority of the column having the highest priority are combined when they are equal to or higher than a predetermined threshold. Support processing device.

(Appendix 7)
Of the second column that is a column other than the first column that is the column extracted by the column processing unit or the generated combination, the data loss has occurred and the first column A third column that satisfies a first predetermined condition including a condition that the correlation coefficient is equal to or greater than a threshold is specified, and for each record pair, the data similarity in the first column is equal to or greater than the second threshold. If the second predetermined condition including the condition of being present satisfies the second predetermined condition, the similarity with the non-missing data of the record pair in the third column is equal to or higher than a third threshold and appears in the third column A weighted addition value of the similarity of data is calculated as the similarity for the third column, and when the second predetermined condition is not satisfied, the first column and the third column The second value is calculated as the similarity for said third column, the similarity degree estimation unit for storing in the data storage unit that is based on other methods calculation,
Further comprising
Using the data stored in the data storage unit that stores the processing result of the similarity estimation unit, the correlation coefficient calculation unit calculates the correlation coefficient of the first similarity again, and the neighborhood The correlation coefficient calculation unit calculates the correlation coefficient of the second similarity again,
The name identification support processing device according to any one of supplementary notes 1 to 6, wherein the column processing unit performs processing again.

(Appendix 8)
The name identification support according to appendix 7, wherein the first predetermined condition further includes a condition that the number of records in which no data is missing in the first column and the second column is equal to or greater than a fourth threshold value. Processing equipment.

(Appendix 9)
The name identification support processing device according to appendix 7 or 8, wherein the second predetermined condition further includes a condition in which the appearance frequency of data in the first column for the record pair is less than a fifth threshold.

(Appendix 10)
The name identification support processing device according to any one of appendices 7 to 9, wherein the weight value of the weighted addition value is a ratio of the appearance frequency of the similar data to the sum of the appearance frequencies of the similar data.

(Appendix 11)
The second value is
The similarity in the third column corresponding to the data similarity in the first column is calculated from the regression calculation result of the first column and the third column, and the calculated similarity is calculated with respect to the calculated similarity The name identification support processing device according to any one of appendices 7 to 10, which is a result of adding random numbers.

(Appendix 12)
Calculating a first data similarity between records for each column in the database and storing in a data storage unit;
Calculating a correlation coefficient of the first data similarity stored in the data storage unit for each combination of the columns, and storing the correlation coefficient in the data storage unit;
A record having data similar to or similar to data in an attention record that is an arbitrary record in an attention column that is an arbitrary column of the columns is identified, and the record between the attention record and the identified record is specified. And calculating or specifying the second data similarity for each column and storing it in the data storage unit, and storing each combination of the column of interest and other columns in the data storage unit. Calculating a correlation coefficient of the second data similarity as a neighborhood correlation coefficient and storing it in the data storage unit; and
The positive neighborhood correlation coefficient for which the positive correlation coefficient has been calculated and multiplied by the percentage of records that have a positive neighborhood correlation coefficient exceeding the predetermined significance level or no data loss has occurred is calculated. A first extraction step of extracting a combination from the data storage;
And a name identification support processing method executed by a computer.

(Appendix 13)
The neighborhood correlation coefficient calculating step includes:
Sorting the records by the data of the column of interest;
Identifying a predetermined number of records before and after the record of interest in the sorted sequence as records having data similar or similar to the data in the record of interest;
The name collation support processing method of Additional remark 12 containing.

(Appendix 14)
For each extraction column that is a column included in the extracted combination of columns, the difference between the correlation coefficient between the extraction column and another extraction column and the neighborhood correlation coefficient is calculated as a usefulness. Storing in the data storage unit;
For each of the extraction columns, a priority is calculated from the usefulness calculated for the extraction column and the usefulness of the extraction column calculated for the other extraction column, and stored in the data storage unit Steps,
The name identification support processing method according to appendix 12 or 13, further including:

(Appendix 15)
A second extraction step of extracting a combination of the columns for which a positive correlation coefficient exceeding a predetermined significance level is calculated from the data storage unit;
Of the columns common to the columns included in the column combination extracted in the first extraction step and the columns included in the column combination extracted in the second extraction step, the column having the highest priority Extracting the
The name identification support processing method according to supplementary note 14, further comprising:

(Appendix 16)
A second extraction column that is a column after excluding a column included in the combination of columns extracted in the second extraction step from a column included in the combination of columns extracted in the first extraction step; The name identification support processing method according to supplementary note 15, further including a combination generation step of generating a combination with a column having the highest priority.

(Appendix 17)
17. The supplementary note 16, wherein the priority of the second extraction column stored in the data storage unit and the priority of the column having the highest priority are combined when they are equal to or higher than a predetermined threshold value. Name identification support processing method.

(Appendix 18)
Of the column extracted in the first extraction step or the second column other than the first column that is a column related to the combination generated in the combination generation step, data loss has occurred and Identifying a third column that satisfies a first predetermined condition including a condition that a correlation coefficient with the first column is equal to or greater than a threshold;
For each record pair, if the second predetermined condition including the condition that the data similarity in the first column is greater than or equal to the second threshold is satisfied, the record pair in the third column is not missing Calculating a weighted addition value of the similarity of similar data appearing in the third column, the similarity with the data being equal to or greater than a third threshold, and calculating the second predetermined When the condition is not satisfied, a second value calculated based on another method of the first column and the third column is calculated as a similarity degree for the third column, and the data storage unit stores the second value. An estimation step to store;
Using the data stored in the data storage unit that stores the processing result of the estimation step, the correlation coefficient of the first similarity is calculated again, and the correlation coefficient of the second similarity is again calculated. A calculating step;
From the recalculation result of the correlation coefficient and the recalculation result of the neighborhood correlation coefficient, a combination of the columns in which a positive correlation coefficient is calculated and a positive neighborhood correlation coefficient exceeding a predetermined significance level is calculated. A third extraction step for extracting from the data storage unit;
The name identification support processing method according to any one of appendices 12 to 17, further including:

(Appendix 19)
The name identification support according to appendix 18, wherein the first predetermined condition further includes a condition that the number of records in which no data is missing in the first column and the second column is equal to or greater than a fourth threshold value. Processing method.

(Appendix 20)
The name identification support processing method according to appendix 18 or 19, wherein the second predetermined condition further includes a condition in which the appearance frequency of data in the first column for the record pair is less than a fifth threshold.

(Appendix 21)
The name identification support processing method according to any one of appendices 18 to 20, wherein a weight value of the weighted addition value is a ratio of an appearance frequency of the similar data to a total sum of the appearance frequencies of the similar data.

(Appendix 22)
The second value is
The similarity in the third column corresponding to the data similarity in the first column is calculated from the regression calculation result of the first column and the third column, and the calculated similarity is calculated with respect to the calculated similarity The name identification support processing method according to any one of appendices 18 to 21, which is a result of adding a random value.

(Appendix 23)
Calculating a first data similarity between records for each column in the database and storing in a data storage unit;
Calculating a correlation coefficient of the first data similarity stored in the data storage unit for each combination of the columns, and storing the correlation coefficient in the data storage unit;
A record having data similar to or similar to data in an attention record that is an arbitrary record in an attention column that is an arbitrary column of the columns is identified, and the record between the attention record and the identified record is specified. And calculating or specifying the second data similarity for each column and storing it in the data storage unit, and storing each combination of the column of interest and other columns in the data storage unit. Calculating a correlation coefficient of the second data similarity as a neighborhood correlation coefficient and storing it in the data storage unit; and
The positive neighborhood correlation coefficient for which the positive correlation coefficient has been calculated and multiplied by the percentage of records that have a positive neighborhood correlation coefficient exceeding the predetermined significance level or no data loss has occurred is calculated. A first extraction step of extracting a combination from the data storage;
Name identification support processing program for causing a computer to execute

(Appendix 24)
The neighborhood correlation coefficient calculating step includes:
Sorting the records by the data of the column of interest;
Identifying a predetermined number of records before and after the record of interest in the sorted sequence as records having data similar or similar to the data in the record of interest;
A name identification support processing program according to appendix 23, including:

(Appendix 25)
For each extraction column that is a column included in the extracted combination of columns, the difference between the correlation coefficient between the extraction column and another extraction column and the neighborhood correlation coefficient is calculated as a usefulness. Storing in the data storage unit;
For each of the extraction columns, a priority is calculated from the usefulness calculated for the extraction column and the usefulness of the extraction column calculated for the other extraction column, and stored in the data storage unit Steps,
The name identification support processing program according to appendix 23 or 24, further including:

(Appendix 26)
A second extraction step of extracting a combination of the columns for which a positive correlation coefficient exceeding a predetermined significance level is calculated from the data storage unit;
Of the columns common to the columns included in the column combination extracted in the first extraction step and the columns included in the column combination extracted in the second extraction step, the column having the highest priority Extracting the
The name identification support processing program according to appendix 25, further including:

(Appendix 27)
A second extraction column that is a column after excluding a column included in the combination of columns extracted in the second extraction step from a column included in the combination of columns extracted in the first extraction step; 27. The name identification support processing program according to appendix 26, further including a combination generation step of generating a combination with a column having the highest priority.

(Appendix 28)
28. The combination of claim 27, wherein the priority of the second extraction column stored in the data storage unit and the priority of the column having the highest priority are combined when they are equal to or higher than a predetermined threshold. Name identification support processing program.

(Appendix 29) (Addition)
Of the column extracted in the first extraction step or the second column other than the first column that is a column related to the combination generated in the combination generation step, data loss has occurred and Identifying a third column that satisfies a first predetermined condition including a condition that a correlation coefficient with the first column is equal to or greater than a threshold;
For each record pair, if the second predetermined condition including the condition that the data similarity in the first column is greater than or equal to the second threshold is satisfied, the record pair in the third column is not missing Calculating a weighted addition value of the similarity of similar data appearing in the third column, the similarity with the data being equal to or greater than a third threshold, and calculating the second predetermined When the condition is not satisfied, a second value calculated based on another method of the first column and the third column is calculated as a similarity degree for the third column, and the data storage unit stores the second value. An estimation step to store;
Using the data stored in the data storage unit that stores the processing result of the estimation step, the correlation coefficient of the first similarity is calculated again, and the correlation coefficient of the second similarity is again calculated. A calculating step;
From the recalculation result of the correlation coefficient and the recalculation result of the neighborhood correlation coefficient, a combination of the columns in which a positive correlation coefficient is calculated and a positive neighborhood correlation coefficient exceeding a predetermined significance level is calculated. A third extraction step for extracting from the data storage unit;
The name identification support processing program according to any one of appendices 23 to 28, further including:

(Appendix 30)
The name identification support according to appendix 29, wherein the first predetermined condition further includes a condition that the number of records in which no data is missing in the first column and the second column is equal to or greater than a fourth threshold. Processing program.

(Appendix 31)
The name identification support processing program according to attachment 29 or 30, wherein the second predetermined condition further includes a condition in which the appearance frequency of data in the first column for the record pair is less than a fifth threshold.

(Appendix 32)
32. The name identification support processing program according to any one of supplementary notes 29 to 31, wherein a weight value of the weighted addition value is a ratio of an appearance frequency of the similar data to a sum of the appearance frequencies of the similar data.

(Appendix 33)
The second value is
The similarity in the third column corresponding to the data similarity in the first column is calculated from the regression calculation result of the first column and the third column, and the calculated similarity is calculated with respect to the calculated similarity 34. The name identification support processing program according to any one of supplementary notes 29 to 32, which is a result of adding random number values.

100,700 Name identification support processing device 110 Data reading unit 120 First data storage unit 130 Correlation calculation unit 140 Neighborhood correlation calculation unit 150 Second data storage unit 160 Column processing unit 170 Output data storage unit 161 Column priority calculation unit 162 Column extraction Processing unit 163 Output unit 600 Column selection unit 710 Preprocessing unit 720 Estimation unit 730 Third data storage unit

Claims (13)

For each column in the database, a first data similarity between records is calculated, stored in a data storage unit, and the first data similarity stored in the data storage unit for each combination of the columns A correlation coefficient calculating unit that calculates a correlation coefficient and stores the correlation coefficient in the data storage unit;
A record having data similar to or similar to data in an attention record that is an arbitrary record in an attention column that is an arbitrary column of the columns is identified, and the record between the attention record and the identified record is specified. And calculating or specifying the second data similarity for each column and storing it in the data storage unit, and storing each combination of the column of interest and other columns in the data storage unit. A neighborhood correlation coefficient calculation unit that calculates a correlation coefficient of the second data similarity as a neighborhood correlation coefficient and stores it in the data storage unit;
The correlation coefficient calculated by the correlation coefficient calculation unit is a positive value, and data loss occurs in the neighborhood correlation coefficient or the neighborhood correlation coefficient calculated by the neighborhood correlation coefficient calculation unit. A column processing unit that performs a first extraction process of extracting from the data storage unit a combination of the columns that is a positive value in which a product obtained by multiplying a ratio of records that does not exceed a predetermined significance level ;
A name identification support processing apparatus. The neighborhood correlation coefficient calculating unit
The records are sorted by the data of the target column, and a predetermined number of records before and after the target record are specified as records having data similar to or similar to the data in the target record in the sorted sequence. Item 1. The name identification support processing device according to Item 1. The column processing unit
For each extraction column that is a column included in the extracted combination of columns, the difference between the correlation coefficient between the extraction column and another extraction column and the neighborhood correlation coefficient is calculated as a usefulness. , Stored in the data storage unit,
For each of the extraction columns, a priority is calculated from the usefulness calculated for the extraction column and the usefulness of the extraction column calculated for the other extraction column, and stored in the data storage unit ,
The name identification support processing apparatus according to claim 1 or 2. The column processing unit
A positive value in which the product obtained by multiplying the correlation coefficient calculated by the correlation coefficient calculation unit or the ratio of records in which no data loss has occurred in the correlation coefficient calculation unit from the data storage unit exceeds a predetermined significance level A second extraction process for extracting a combination of the columns,
The column having the highest priority among the columns common to the columns included in the combination of columns extracted in the first extraction process and the columns included in the combination of columns extracted in the second extraction process Extract,
The name identification support processing apparatus according to claim 3. The column processing unit
A second extraction column that is a column after excluding a column included in the combination of columns extracted in the second extraction process from a column included in the combination of columns extracted in the first extraction process; The name identification support processing apparatus according to claim 4, wherein a combination with a column having the highest priority is generated. The combination according to claim 5, wherein the priority of the second extraction column stored in the data storage unit and the priority of the column having the highest priority are combined when they are equal to or higher than a predetermined threshold. Name identification support processing device. Of the second column that is a column other than the first column that is the column extracted by the column processing unit or the generated combination, the data loss has occurred and the first column A third column that satisfies a first predetermined condition including a condition that the correlation coefficient is equal to or greater than a threshold is specified, and for each record pair, the data similarity in the first column is equal to or greater than the second threshold. If the second predetermined condition including the condition of being present satisfies the second predetermined condition, the similarity with the non-missing data of the record pair in the third column is equal to or higher than a third threshold and appears in the third column A weighted addition value of the similarity of data is calculated as the similarity for the third column, and when the second predetermined condition is not satisfied, the first column and the third column A second value calculated as the similarity for said third column, similarity degree estimation unit to be stored in the data storage unit, which is calculated based on other methods,
Further comprising
Using the data stored in the data storage unit that stores the processing result of the similarity estimation unit, the correlation coefficient calculation unit calculates the correlation coefficient of the first similarity again, and the neighborhood The correlation coefficient calculation unit calculates the correlation coefficient of the second similarity again,
The name identification support processing apparatus according to any one of claims 1 to 6, wherein the column processing unit performs the process again. The name collation according to claim 7, wherein the first predetermined condition further includes a condition that the number of records in which no data is missing in the first column and the second column is equal to or greater than a fourth threshold. Support processing device. The name collation support processing apparatus according to claim 7 or 8, wherein the second predetermined condition further includes a condition in which the appearance frequency of data in the first column for the record pair is less than a fifth threshold. The name identification support processing apparatus according to any one of claims 7 to 9, wherein a weight value of the weighted addition value is a ratio of an appearance frequency of the similar data to a sum of the appearance frequencies of the similar data. The second value is
The similarity in the third column corresponding to the data similarity in the first column is calculated from the regression calculation result of the first column and the third column, and the calculated similarity is calculated with respect to the calculated similarity The name identification support processing device according to any one of claims 7 to 10, wherein the name identification support processing device is a result of adding random values. Calculating a first data similarity between records for each column in the database and storing in a data storage unit;
Calculating a correlation coefficient of the first data similarity for each of the combinations of the column is stored in the data storage unit, a correlation coefficient calculating step of storing in the data storage unit,
A record having data similar to or similar to data in an attention record that is an arbitrary record in an attention column that is an arbitrary column of the columns is identified, and the record between the attention record and the identified record is specified. And calculating or specifying the second data similarity for each column and storing it in the data storage unit, and storing each combination of the column of interest and other columns in the data storage unit. Calculating a correlation coefficient of the second data similarity as a neighborhood correlation coefficient and storing it in the data storage unit; and
The correlation coefficient calculated in the correlation coefficient calculation step is a positive value, and data loss occurs in the neighborhood correlation coefficient calculated in the neighborhood correlation coefficient calculation step or the neighborhood correlation coefficient. A first extraction step of extracting from the data storage unit a combination of the columns that is a positive value in which a product obtained by multiplying the percentage of records that is not greater than a predetermined significance level ;
And a name identification support processing method executed by a computer. Calculating a first data similarity between records for each column in the database and storing in a data storage unit;
Calculating a correlation coefficient of the first data similarity for each of the combinations of the column is stored in the data storage unit, a correlation coefficient calculating step of storing in the data storage unit,
A record having data similar to or similar to data in an attention record that is an arbitrary record in an attention column that is an arbitrary column of the columns is identified, and the record between the attention record and the identified record is specified. And calculating or specifying the second data similarity for each column and storing it in the data storage unit, and storing each combination of the column of interest and other columns in the data storage unit. Calculating a correlation coefficient of the second data similarity as a neighborhood correlation coefficient and storing it in the data storage unit; and
The correlation coefficient calculated in the correlation coefficient calculation step is a positive value, and data loss occurs in the neighborhood correlation coefficient calculated in the neighborhood correlation coefficient calculation step or the neighborhood correlation coefficient. A first extraction step of extracting, from the data storage unit, a combination of the columns that is a positive value in which a product obtained by multiplying a ratio of records that is not greater than a predetermined significance level ;
Name identification support processing program that causes a computer to execute

Images