JP5780036B2 - Extraction program, extraction method and extraction apparatus - Google Patents

Description

  The present invention relates to an extraction program, an extraction method, and an extraction apparatus for extracting a predetermined character string from a character string group.

  As information technology advances, information processing such as information collection, processing, and storage is often performed using a computer. While the amount of information handled is increasing, an efficient information processing technique is required.

  For example, a technique is known in which a character string representing the characteristics of contents described in a document specified by a user is extracted, and a document including contents similar to the contents described in the document is searched from a document database. ing. A technique is also known in which, when a desired document is searched from a document database according to a search condition specified by a user, the search results are rearranged and displayed according to the degree to which each document matches the search condition.

JP 11-338883 A JP 2001-109766 A

  As one of information processing, there is a process called “name identification” for collating data and deriving the relationship in a situation where a plurality of data is gathered. In name identification, processing for extracting a character string similar to a certain character string is performed. Since the process of extracting a similar character string includes more condition determination processes than the process of extracting the same character string, the processing load is large. When the amount of data subject to name identification is enormous, the amount of name identification processing also becomes enormous. Accordingly, an object of one aspect of the disclosure content of the present application is to suppress an increase in extraction processing load due to an increase in data to be extracted from similar character strings.

Oite extraction technology disclosed in the present application, a 1 or a plurality of partial character strings included in Jo Tokoro string, than before Symbol quotient obtained by dividing the number of characters in a given string in a predetermined edit distance One or a plurality of partial character strings having a small number of characters are extracted, and among the extracted one or a plurality of partial character strings , the predetermined number of characters and the sum of the predetermined character strings are used to calculate the predetermined edit distance and 1 a string containing one of the following substrings minus the product of the natural number smaller than the quotient sum, extracted from character strings.

  As one aspect of the disclosure content of the present application, an increase in extraction processing load due to an increase in data to be extracted from similar character strings can be suppressed.

It is explanatory drawing of a name collation function. It is explanatory drawing about operation | movement of a name collation function. It is explanatory drawing which shows an example of the data structure of a name identification definition. It is explanatory drawing of name collation by rough | squeezing. It is a flowchart which shows the processing procedure of name identification by rough aperture. It is a flowchart which shows the procedure of collation processing. It is explanatory drawing which shows an example of the data structure of a rough aperture definition. It is explanatory drawing which shows the flow of preparation of a rough aperture definition. It is explanatory drawing which shows the detail of a name collation process and a rough narrowing process. It is explanatory drawing which shows the correspondence of the comparison conditions of a name collation process, and the search conditions of a rough narrowing process. It is explanatory drawing which shows the record which needs to be searched from the collection of records of a name identification destination. It is explanatory drawing which shows the record which needs to be searched from the collection of records of a name identification destination. It is explanatory drawing which shows the record which needs to be searched from the collection of records of a name identification destination. It is explanatory drawing which shows the relationship between the comparison conditions of name collation, and the search conditions of rough aperture. It is explanatory drawing which shows the relational expression of the comparison conditions of name collation, and the search conditions of rough drawing. It is explanatory drawing which shows the detail of tuning of a rough aperture definition. It is explanatory drawing which shows the detail of tuning of a rough aperture definition. It is explanatory drawing which shows the function structural example of information processing apparatus. FIG. 25 is an explanatory diagram illustrating a hardware configuration example of an information processing device. It is explanatory drawing which shows the flow of preparation of the rough aperture definition performed by information processing apparatus.

[1. Summary of name identification]
There is a name identification function as a function of collating records with respect to records including a set of values and determining the identity, similarity, and relationship between records. In the name identification function, for example, a set of records to be identified is referred to as a name identification source, and a set of records to be a name identification partner is referred to as a name identification destination. FIG. 1 is a diagram for explaining the name identification function. As shown in the figure, the name identification process that realizes the name identification function detects the same record as the name identification source, a record similar to the name identification source, or a record related to the name identification source from the name identification destination, and outputs the detection result as the name identification result. .

  In recent years, with the increase in capacity (large scale) of databases, a method for performing name identification at high speed is required. The operation of the conventional name identification function will be described with reference to FIG. FIG. 2 is a diagram for explaining the operation of the name identification function. As shown in the figure, the name identification source is a supplier table, and the name identification source is a customer table. And only one record J1 is illustrated among the some records contained in the supplier table | surface which is a name identification source. Further, all records M (M1 to Mn) are illustrated for the customer table as a name identification destination. The name identification source record J1 includes five columns of ID, company name, postal code, address, and telephone number. On the other hand, the name identification record M (M1 to Mn) includes five columns of ID, customer name, postal code, customer address, and telephone number. In the name identification process for realizing the name identification function, name identification with the record M (M1 to Mn) of the name identification destination is executed for one record J1 of the name identification source.

First, in the name identification process, an evaluation function defined in advance for each name identification item is applied to the value of each name identification item (referred to as “name identification item”) in the name identification source record J1 and the name identification destination record M1. Perform verification. Here, the company name of the name identification source and the customer name of the name identification destination are the first name identification target items, and the postal code of the name identification source and the postal code of the name identification destination are the second name identification target items. Further, the name identification source address and the name identification destination customer address are set as third name identification target items, and the name identification source telephone number and the name identification destination telephone number are set as fourth name identification target items. In the name identification process, the first name identification item is fa (), the second name identification item is fb (), the third name identification item is fc (), and the fourth name identification item is fd (). Match by applying evaluation function. In the name identification process, the evaluation value of each name identification target item derived as a result of collation is weighted for each name identification target item using the weights A to D, and the total value is obtained by adding the obtained values. The value h is derived. Further, in the name identification process, comprehensive evaluation values are derived for all remaining name identification destination records M2 to Mn for the name identification source record J1. In the name identification process, a name identification candidate set including a comprehensive evaluation value for the combination of the name identification source record J1 and the name identification target records M1 to Mn is created.

  In the name identification process, determination regarding name identification is performed for a set of records belonging to the candidate group for name identification based on a predetermined threshold value or determination rule. For example, in the name identification process, the total evaluation value h is compared with a threshold value, a set of records determined to match is set to “White”, and a set of records determined to not match is set to “Black”. Output name identification result. In the name identification process, a pair that does not correspond to either “White” or “Black” is output to the candidate list as “Gray”. Then, the relevance of the pair output to the candidate list is determined by a person. The name identification definition 103a that needs to be set by a person includes selection of a name identification item, selection of an evaluation function, setting of a weight and a threshold.

  A specific example of the name identification definition 103a is shown in FIG. FIG. 3 is a diagram illustrating an example of the data structure of the name identification definition. FIG. 3A illustrates the contents of the name identification definition, and FIG. 3B illustrates a specific example of the name identification definition.

  As shown in FIG. 3A, the name identification definition is defined by associating a name identification method d1, a name identification source designation d2, a name identification destination designation d3, a name identification target item designation d4, and a threshold value d5. A name identification method is designated as the name identification method d1. For example, as a name identification method, there is “self-name identification” in which a single record set is subjected to name identification among all the records in the set, a matching record is detected, and duplicate records are removed. The self-name identification has a feature that the structure (record items) is the same because the name identification source and the name identification target are the same record set. As a name identification method, name identification is performed by combining a name identification source record and a name identification destination record for different record sets as a name identification and a name identification destination, a matching record is detected, and association between the corresponding records is performed. "Other name identification". Other name identification is a set in which a name identification source and a name identification destination are different, and thus generally has a feature that its structure (record item) is different. In the name identification source designation d2, access information such as the name identification source database and items of the name identification source record are designated. In the name identification destination designation d3, access information such as the name identification destination database name and items of the name identification destination record are designated. In the name identification target item specification d4, the name identification target item is specified as a combination of the name identification source item and the name identification target item, and an evaluation function and a weight applied to each name identification target item are specified. As the threshold value d5, an upper threshold value for White determination and a lower threshold value for Black determination are designated.

  As shown in FIG. 3B, for example, “other name identification” is designated in the name identification method d1. In the access information of the name identification source designation d2, “business partner table” is designated, and in the record information of the name identification source designation d2, items of ID (identification), company name, postal code, address and telephone number are designated. Yes. “Customer table” is designated as the access information of the name identification destination designation d3, and items of ID (identification), customer name, zip code, customer address and telephone number are designated as the record information of the name identification destination designation d3. Yes.

In the name identification item designation d4, the name identification item is designated as company name: customer name, postal code: postal code, address: customer address, and telephone number: telephone number. In this method, a name identification target item is designated as a set of name identification source item: name identification destination item. The evaluation function and weight to be applied are specified for this name identification item. For example, when the name identification target item is company name: customer name, the evaluation function specifies “within 1 edit distance difference” and the weight is 0.2. When the name identification item is zip code: zip code, “complete match” is specified for the evaluation function and 0.2 is specified for the weight. As the threshold value d5, 0.72 is designated as the upper threshold value and 0.26 is designated as the lower threshold value.

  The “edit distance” is an evaluation function that represents the minimum number of edits as a distance when the name identification target value is transformed into the name identification source value in the collation of the value of the name identification target item between the name identification source and the name identification destination. For example, when the evaluation function is “less than one edit distance difference”, the evaluation function returns 1.0 if the edit distance is less than one letter difference, and 0 in the other cases. return.

  The “complete match” is an evaluation function that indicates whether or not two values are completely matched in the collation of the value of the name identification target item between the name identification source and the name identification target. Returns 1.0 if the two values match completely, 0 otherwise. Note that “edit distance” and “perfect match” shown in FIG. 3B are merely examples of evaluation functions.

  In the name identification process described with reference to FIGS. 1 to 3, the number of record combinations to be subjected to collation processing related to name identification is obtained by multiplying the number of records of the name identification source and the number of records of the name identification destination. For example, in FIG. 2, when the number of records in the customer table as the name identification source is 100,000 and the number of records in the customer table as the name identification destination is 2 million, 200 billion sets of collation processing are required. . Such a large-scale name identification requires a huge amount of time.

[2. Speeding up name identification by rough drawing]
There is a technique for speeding up large-scale name identification by reducing the number of records to be collated before performing collation processing for collating records with respect to a name identification source record and a name identification destination record. Here, a technique of “rough narrowing” will be described in which the name identification target records that may match the name identification source are roughly narrowed before the matching process. Hereinafter, the rough narrowing is also referred to as “prior search”, and the subsequent matching process is also referred to as “main search”.

  FIG. 4 is a diagram for explaining name identification by “rough aperture”. As shown in the drawing, in the rough narrowing process 102, a record is searched from the name identification destination 101 using a search condition generated for each record of the name identification source 100, and the search result is output as a search result 102b. This search condition is generated based on a rough aperture definition 102a described later.

  Here, if it is assumed that the number of search results 102b serving as name identification destination candidates is an average of 100 with respect to one record of the name identification source 100, in the collation by the name identification processing 103, 100,000 cases of name identification source 100 × name identification destination candidates The average of 100 cases = 10 million sets of collation, which is a significant reduction compared to the 200 billion pairs of collation related to name identification for all records of the name identification destination 101.

  Next, a name identification process procedure based on rough drawing will be described with reference to FIGS. FIG. 5 and FIG. 6 are flowcharts showing a name identification process procedure based on rough aperture.

First, the rough aperture processing 102 reads the rough aperture definition 102a, sets the operating environment (step S100), and selects a name identification source record (hereinafter referred to as “name identification source record”) to be identified from the name identification source 100. It takes out in order (step S101). Then, the rough aperture processing 102 roughly searches the name identification destination 101 for each rough aperture target item defined in the rough aperture definition 102a, using the value of the corresponding item in the name identification source record as a condition (step S102). Specifically, the rough narrowing process 102 performs an ambiguous search of the name collation destination 101 with a search condition obtained by ORing each condition with the value of the corresponding item of the name collation source record as a condition for each rough narrowing target item. here,
The ambiguous search is a search by “n-gram” or the like. Then, the rough-drawing process 102 stores the searched record as a search result 102b.

  Next, the name identification process 103 sequentially extracts each record stored in the search result 102b as a name identification destination (step S103), and performs a collation process between the name identification source record and the name identification destination (step S104). Details of step S104 will be described later with reference to FIG. Then, the name identification process 103 stores the collation result in the name identification candidate set (step S105). The collation result includes a comprehensive evaluation value.

  Subsequently, the name identification process 103 determines whether or not there are remaining search result records in the search result 102b (step S106). If it is determined that there are remaining search result records in the search result 102b (step S106; Yes), the name identification process 103 proceeds to step S103 to extract the remaining search result records.

  On the other hand, when it is determined that there is no remaining search result record in the search result 102b (step S106; No), the name identification process 103 executes determination based on a threshold for each comprehensive evaluation value stored in the name identification candidate set. The determination result is output (step S107). For example, if the overall evaluation value is equal to or higher than the upper threshold value, the name identification process 103 determines that the matched name identification source record and name identification target record group is a matched record group and sets “White”. judge. Further, the name identification process 103 cannot determine whether the pair of the collated name identification source record and the name identification target record matches or does not match when the comprehensive evaluation value is less than the upper threshold value and greater than or equal to the lower threshold value. Judgment is made and “Gray” is determined. Further, when the comprehensive evaluation value is less than the lower threshold, the name identification process 103 determines that the collated name identification source record and name identification target record group is a mismatched record group and determines “Black”. To do. Then, a set of records determined as “White” and “Black” is output as a name identification result, and a set of records determined as “Gray” is output as a candidate list. Whether a record set in the candidate list matches or does not match is determined by a person.

  Then, the rough narrowing process 102 determines whether or not there are remaining name identification source records in the name identification source 100 (step S108). If it is determined that there are remaining name identification source records in the name identification source 100 (step S108; Yes), the rough narrowing process 102 proceeds to step S101 to extract the remaining name identification source records. On the other hand, when it is determined that there are no remaining name identification source records in the name identification source 100 (step S108; No), the rough narrowing process 102 ends the name identification process by the rough narrowing.

  Next, the processing procedure of step S104 shown in FIG. 5 will be described with reference to FIG. FIG. 6 is a flowchart showing the procedure of the collation process. The matching process is a process for deriving a comprehensive evaluation value by matching each set of the name identification source record and the name identification target record.

  First, the name identification process 103 sequentially selects the name identification target items defined in the name identification definition 103a (step S110). Note that the name identification target item is defined in advance in the name identification definition 103a as a pair of items to be compared, including a name identification source item and a name identification target item. Then, the name identification process 103 specifies each value corresponding to the selected name identification item for the name identification source record and the name identification destination record (step S111), and applies the evaluation function to the two specified values (step S112). The evaluation value is calculated. The evaluation function is a function defined in advance for each name identification item in the name identification definition 103a.

Subsequently, the name identification process 103 determines whether or not there are remaining name identification items (step S113). If it is determined that there are remaining name identification items (step S113; Yes), the name identification processing 103 proceeds to step S110 to apply the evaluation function to the remaining name identification items.

  On the other hand, when it is determined that there are no remaining name identification target items (step S113; No), the name identification process 103 weights the evaluation value of each name identification target item for each name identification target item, and results of weighting. Each evaluation value is added (step S114). Then, the name identification process 103 outputs the value of the addition result as a comprehensive evaluation value for the target record set (step S115), and finishes the matching process for one set.

  An example of the data structure of the rough aperture definition is shown in FIG. FIG. 7A shows the contents of the rough aperture definition, and FIG. 7B shows a specific example of the rough aperture definition. The rough aperture definition is created by a person in the same manner as the name identification definition.

  As shown in FIG. 7A, in the rough aperture definition, the target item and the search condition are defined in association with each other, and the maximum number of detections can be defined as necessary. A plurality of target items can be specified as a pair of a name identification source item and a name identification target item to which the search condition is applied in the rough narrowing process, and a corresponding search condition is specified. The maximum number of detections is the maximum number of name identification destination records left as a result of searching for a name identification target for one name identification source record.

  As shown in FIG. 7B, in the rough aperture definition 102a, a target name identification item, a name identification source item, and a search condition to be applied are defined for each rough aperture target item d11. d12 is also defined as necessary. The rough aperture target item d11 is associated with “source” and “search condition”. “Source” indicates the name of an item that is a target for rough narrowing of each of the name identification source record and the name identification destination record as “name identification source item: name identification destination item”. For each target item, the search condition specifies a search method for searching for the corresponding item of the name identification destination by the value of the corresponding item of the name identification source. For example, the search condition includes “n-gram” for searching for a name identification destination record including any N characters whose values are continuous for the target item of the name identification source record, or the value of the target item of the name identification target record. There is a “perfect match” that searches for a name identification record having a target item that is completely matched. When “n-gram” is used, the index match number c can be designated. Here, the index in n-gram is a character string continuous over n characters in the target item of the name identification source record. For example, when the search condition is “Bi-gram / index match number 4”, the name identification destination includes a record including four or more characters in the target item of the name identification source record that are continuous over two characters. Search from the record.

  In the example of FIG. 7B, the search condition for the target item “company name: customer name” is “Bi-Gram / number of matches 2”. Further, the search condition of “address: customer address” indicates that “Bi-Gram / number of matches 4”. It indicates that the search condition of the target item “zip code: zip code” and “phone number: phone number” is “complete match”. The union of each set searched by these four search conditions corresponds to the search result 102b in FIG. The maximum number of name identification destination records for each name identification source record is 1000.

[3. Create rough aperture definition]
So far, the name identification with rough drawing has been described with reference to FIGS. As described above, both the name identification definition used for the name identification process and the rough aperture definition used for the rough extraction process are created by a person. Usually, a name identification definition is created first, followed by a rough drawing definition. Of these, FIG. 8 shows the flow of creating a rough aperture definition.

  In step T11, the initial rough refinement definition is considered in consideration of the record that can be excluded as it is not necessary to collate among the name identification source data while referring to the name identification source data, the name identification destination data, and the already created name identification definition. Create

  In step T12, some name identification source records are sampled, and based on the sampled records, rough drawing processing is tried using the initial rough drawing definition created in step T11.

  In procedure T13, the number of narrowed records of name identification destination data is evaluated based on the processing speed of the computer to be used. If it is evaluated that the narrowing has been made sufficiently, the process proceeds to a later-described procedure T14. Otherwise, the rough aperture definition is corrected and the process returns to step T12.

  In procedure T14, name identification processing is tried for the record sampled in procedure T12.

  In step T15, the result of the name identification process in step T14 is confirmed, and whether or not the name identification target record to be collated with the name identification source record is excluded by the rough narrowing process based on the already created “name identification definition”. To investigate the. If it is excluded, the process returns to step T12.

  As described above, by repeating steps T12 to T15 through trial and error, the rough aperture definition is completed in step T16.

  In creating the rough narrowing definition, if the name identification destination records are narrowed too much in consideration of the processing speed of the computer to be used, the data required for the subsequent name identification processing will be leaked, resulting in low name identification accuracy. On the other hand, if too much emphasis is placed on the accuracy of name identification, a sufficient rough drawing cannot be performed, and therefore the name identification process takes an enormous amount of time. Therefore, the creation of the rough aperture definition involves trial and error. In general, the man-hours for creating a rough drawing definition by a person are several man-months.

[4. Efficiently creating rough aperture definition]
Hereinafter, a technique for efficiently creating a rough aperture definition will be described. This technique efficiently creates a rough drawing definition based on a name identification definition created in advance. First, the name identification process and the rough drawing process will be described in more detail with reference to FIG.

  FIG. 9 shows a set M of name identification source records J1 and name identification destination records. These correspond to the name identification source record J1 and the name identification target set M shown in FIG. Then, a set Ma of retrieved records is created by performing a rough narrowing process on the set M of name identification destination records. In performing the rough narrowing process, as shown in the figure, a record including a specific character string is searched from the set M using a search function. This rough drawing process needs to be performed at a high speed while providing a rough accuracy. Examples of search functions include perfect match as described above, n-gram such as Uni-gram and Bi-gram, forward match, and backward match.

When the set Ma of the retrieved records is created, name identification processing is performed on the name identification source record J1 and the set Ma. At this time, a set of records satisfying at least one of the collation rules defined in the name identification definition is created in relation to the record J1. This set is illustrated as a name identification set Mb. In performing this name identification process, as shown in the figure, the comparison function is used to compare the record J1 with each record in the set Ma. This comparison is in contrast to the rough drawing process in that the process takes time because it needs to be done finely. Examples of comparison functions include perfect match and edit distance. Furthermore, as a specific example of the edit distance, there is a function for viewing different numbers of characters in two character strings to be compared. In addition, there is also a function for viewing the value obtained by dividing the number of different characters in the two character strings to be compared by the number of characters in one of these two character strings as the editing ratio.

It is necessary to pay attention to the following points when performing such rough drawing processing.
The record set Mb whose name identification result with the record J1 is White or Gray is included in the searched record set Ma.
On the other hand, the set Ma is set as small as possible.
-Reduce the cost of the rough drawing process itself (processing time, computational resources used, etc.).

  FIG. 10 shows a correspondence relationship between the comparison condition of the name identification process and the search condition of the rough drawing process. As shown in the figure, when “complete match” is used as the comparison condition in the name identification process, the search condition in the rough narrowing process is also “complete match”. This is because “complete match” is a clear character string comparison function and can be used as it is for rough drawing.

  Next, a case where a “fuzzy match”, that is, a condition based on an edit distance is used as a comparison condition in the name identification process will be described. The edit distance is an evaluation function for ambiguous comparison, and has a characteristic that can be effectively evaluated even when the arrangement order of two character strings to be compared is partially changed. As a search condition including this comparison condition, a search condition based on n-gram and the number of index matches is appropriate.

  Next, a specific method for converting the comparison condition based on the edit distance in the name identification process into n-gram, which is a rough drawing search condition, will be described. In order to evaluate based on “n” of n-gram and “number of matching indexes”, it is necessary to consider the minimum number of characters of the comparison source character string.

  First, a case will be described in which the comparison condition is “within 1 character difference in edit distance”. FIG. 11 shows records that need to be searched from the set M of name identification destination records when the comparison source is a three-character string “ABC”. However, “?” Is a character that is not any of “A”, “B”, and “C”. That is, the character string pattern shown in FIG. 11 is a character string pattern extracted from the comparison-source character string “ABC”.

  As shown in FIG. 11, the records that need to be searched from the name identification destination are the one-character index (that is, “A”, “B”, “C”) in the comparison source value “ABC”. A record containing two or more. Therefore, the search condition for searching the record as shown in the figure from the name identification destination is “1 (uni) -gram, index match number 2 or more”.

  Next, when the comparison condition is “less than one character difference in edit distance” and the comparison source is a character string of four characters “ABCD”, it is necessary to search from the set M of name identification destination records. The record is shown in FIG. However, “?” Is a character that is not any of “A”, “B”, “C”, and “D”. That is, the character string pattern shown in FIG. 12 is a character string pattern extracted from the comparison-source character string “ABCD”.

As shown in FIG. 12, the records that need to be searched from the name identification destination are the two-character indexes (that is, “AB”, “BC”, “CD”) in the comparison source value “ABCD”. A record including one or more records. Therefore, the search condition for searching the record as illustrated from the name identification destination is “2 (bi) -gram, index match number 1 or more”. Alternatively, a record including three or more characters in the index of 1 character (that is, “A”, “B”, “C”, “D”) in the comparison source value “ABCD” is searched from the name identification destination. Also good. The search condition in this case is “1 (uni) -gram, index match number 3 or more”.

From the example shown in FIG. 11 and FIG. 12, an expression for converting the comparison condition “within the edit distance d difference” within the name identification into the search condition “n-gram, index match number c” at the time of rough narrowing. Is as follows.
n = F1 (d, m)
c = F2 (d, m, n)
However, m is the minimum number of characters of the comparison source character string, and F1 and F2 are functions different from each other.

  In addition, when the comparison condition is “edit distance is within two characters or less” and the comparison source is a character string of three characters “ABC”, a record that needs to be searched from the set M of name identification destination records This is shown in FIG. However, “?” Is a character that is not any of “A”, “B”, and “C”. That is, the character string pattern shown in FIG. 13 is a character string pattern extracted from the comparison-source character string “ABC”.

  As shown in FIG. 13, a record that needs to be searched from the name identification destination is an index of 1 character (that is, “A”, “B”, “C”) in the comparison source value “ABC”. A record including one or more records. Therefore, the search condition for searching the record as shown in the figure from the name identification destination is “1 (uni) -gram, index match number 1 or more”.

  The example shown in FIGS. 11 to 13 is an example, but the result of further expanding the range is shown in FIG. FIG. 14A shows the relationship between the comparison conditions for name identification (d and m) and the rough search conditions (n and c). As already described, d corresponds to “d” in the comparison condition “within the edit distance d difference”, and m is the minimum number of characters of the comparison source character string. In addition, n and c correspond to “n” and “c” of the search condition “n-gram, index coincidence number c” of the rough aperture, respectively. In FIG. 14A, for d, three patterns of d = 1, 2, 3 are combined, and for m, 10 patterns of m = 1, 2,. The possible value of c is shown.

  For example, when d = 1 and m = 3, “n = 1 / c = 2”. This corresponds to the example shown in FIG. When d = 1 and m = 4, “n = 1 / c = 3” or “n = 2 / c = 1”. This corresponds to the example shown in FIG. When d = 2 and m = 3, “n = 1 / c = 1”. This corresponds to the example shown in FIG. Note that when d = 1 and m = 1, “None (all)” is obtained, which indicates that all the records in the name identification destination are searched, and thus the meaning of the rough narrowing is lost. .

  FIG. 14B shows a relationship between the comparison conditions for name identification (r and m) and the rough search conditions (n and c). Here, r corresponds to “r” in the comparison condition “edit distance (within ratio r)” of name identification. The rest of the configuration is the same as that in FIG. Note that when r = 0.33 and m = 1, “None (= 0%)” is obtained, but this is the same as specifying r = 0. Indicates that there will be no more.

  From FIGS. 14A and 14B, the relationship between d (or r) and m, and n and c can be expressed by the equation shown in FIG. That is, the rough search condition can be created based on the comparison condition defined in the name identification definition. Hereinafter, m, n, c, and d are also referred to as a first number, a second number, a third number, and a fourth number, respectively.

  FIG. 15A shows an equation for obtaining n and c based on d and m. FIG. 15B shows an equation for obtaining n and c based on r and m, but the result is the same as the equation shown in FIG. This is because r can be converted to d based on the equation d = r × m (decimal point truncation). Further, it can be said that both of FIGS. 15A and 15B are common. When the rough narrowing is actually performed, the search index becomes enormous when n is increased. Therefore, the maximum value of n is determined. It is necessary to keep.

  Until now, with reference to FIG. 9 to FIG. 15, the description has been made of the point that determines the search condition of the rough aperture from the comparison condition of name identification. In this way, it is possible to efficiently create a rough aperture definition that has been created by repeating trial and error in the past. Specifically, it is possible to reduce a human error that may occur at the time of creating a rough aperture definition that has been conventionally performed and leakage of a rough aperture condition that should be originally defined. In addition, it is not necessary to acquire specialized knowledge and experience for considering the balance between the precision of the rough drawing and the processing speed of the computer. Furthermore, it is possible to greatly reduce the human cost necessary for creating the rough drawing definition. On the other hand, the user can intuitively create the “name identification definition” by looking at the actual name identification source data and name identification destination data without considering the rough search conditions.

  Then, by using the expression as shown in FIG. 15, no matter what character string is included in the column corresponding to the minimum number of characters of the comparison source, the named set (corresponding to the set Mb in FIG. 9) , (Also corresponding to the set Ma in FIG. 9), it is included in the set of records searched by the rough drawing.

  However, when actually performing name identification, depending on the character strings included in the name identification source and name identification destination records, n or c of the rough narrowing condition is set to be larger than the value obtained by the expression as shown in FIG. By doing so, there is a possibility of narrowing down to a smaller set. Enlarging n and c in this way is called tuning of the rough aperture definition.

[5. Tuning the coarse aperture definition]
Hereinafter, tuning of the rough aperture definition will be described with reference to FIGS. As a premise, it is assumed that a name identification definition and a rough aperture definition (also referred to as an initial rough aperture definition) based on the formula shown in FIG. 15 have been created. In tuning this rough aperture definition n and c, first, some records of the name identification source are sampled. The sampling rate s at this time is assumed to be 0.1. In this sampling, it is necessary to analyze the record of the name identification source so that the sampled record is as biased as possible. For example, sampling can be performed using a technique based on statistics. Further, as illustrated in FIG. 16A, the set P of name identification destination records includes 14 records R1 to R14. This is expressed as v = 14 in FIG.

  Next, based on the comparison condition defined in the name identification definition, name identification without rough narrowing is performed on the record sampled from the name identification source and the name identification destination. This is called name identification trial. By this name identification trial, a name identification destination record that satisfies at least one of the comparison conditions defined in the name identification definition in relation to the record sampled from the name identification source is recorded for each comparison condition. A set of records recorded in this manner is shown as a name identification set Pc in FIG. The set Pc includes four records, records R11 to R14. This is expressed as x = 4 in FIG.

Subsequently, the rough sampling is performed on the records sampled from the name identification source and all the records of the name identification destination by using the rough aperture definition created in advance. This is called a rough drawing trial. The record of the name identification destination searched by this rough narrowing trial is recorded for each column. FIG. 16A shows the set Pa created by the rough drawing trial. The set Pa includes 12 records R3 to R14. That is, the records R1 and R2 are excluded due to the rough drawing trial. This is expressed as w = 12 in FIG.

  Next, it is checked whether or not the set Pc of name identification destination records determined as White or Gray by the name identification trial is included in the set Pa of name identification destination records searched by the rough narrowing trial. Specifically, it is performed as follows.

  First, the product set of the set Pa and the set Pc includes four records R11 to R14. This is expressed as y = 4 in FIG. Further, the ratio that the set Pc is included in the set Pa is referred to as a rough drawing matching rate. The rough drawing precision is calculated as y / w = 4/12 = 33%. Further, the rate at which the set Pc is leaking from the set Pa is referred to as a leakage rate. The leak rate is calculated as 1-y / x = 1-4 / 4 = 0%.

  Note that the number of predicted records to be searched when the name identification destination is roughly narrowed down for all the name identification source records is referred to as a predicted rough narrowing number. The predicted rough aperture number is predicted to be within w / s = 12 / 0.1 = 120. In addition, the number of records at the name identification destination that will be determined as White or Gray when the names are identified for all the records at the name identification source is referred to as a predicted name identification number. The number of predicted names is predicted to be within x / s = 4 / 0.1 = 40.

  That is, in the check, it is determined whether the leakage rate is 0%. If the leak rate is 0%, it is expected that the search condition for the rough aperture can be narrowed a little more, so the process proceeds to a search condition change described later. It should be noted that the leak rate after performing a rough drawing trial based on the rough drawing definition based on the equation as shown in FIG. 15 is always 0%. This is because the values of n and c are determined so that the leakage rate is 0%.

  In the search condition change, in order to make the set of records searched by the rough narrower smaller, the rough narrow search condition, that is, the values of n and c are changed. For example, only the value of c is changed to c ′ = c + 1. Alternatively, n is changed to n ′ = n + 1, and c is changed to c ′. The relationship between n 'and c' at this time is shown in FIG. That is, c ′ = c− (n′−n) × (d + 1). From this equation, c ′ when n is changed to n ′ = n + 1 can be obtained. If n ′ is equal to or less than the maximum value of n of n-gram described with reference to FIG. 15 and c ′ is greater than 0, the search condition using n ′ and c ′ is used to The coarse aperture trial, the check, and the search condition change are repeated. As a result of the check, for example, when the leak rate is not 0% as shown in FIG. 16B, the values of n and c are determined as final values instead of n ′ and c ′. Alternatively, when c ′ becomes a value of 0 or less as a result of changing the search condition, the values of n and c are determined as final values instead of n ′ and c ′.

  In this way, the search condition is changed to n ′ and c ′, and a rough narrowing trial is performed using them, and a series of processes of checking the result of the rough narrowing trial and the result of the name identification trial are repeated. Thereby, a final value n larger than the initial value n by a certain amount is determined.

  As an example, the flow of changing search conditions when the initial value n = 1 and the initial value c = 3 will be described. First, n ′ is obtained. If n ′ is a value larger by 1 than n, then n ′ = n + 1 = 1 + 1 = 2. Next, c ′ is obtained using this n ′. That is, c ′ = c− (n′−n) × (d + 1) = 3- (2-1) × (1 + 1) = 1. Using these search conditions with n ′ = 2 and c ′ = 1, the rough aperture trial, the check, and the search condition change are repeated.

So far, the tuning of the rough aperture definition has been described with reference to FIGS. 16 and 17. With this tuning, the search conditions defined in the rough narrowing definition can be changed to narrower search conditions according to the actual name identification source data and name identification target data, so that rough narrowing can be performed more efficiently. become.

[6. Example]
Next, an example of creating and tuning a rough aperture definition will be described with reference to FIGS. FIG. 18 illustrates a functional configuration example of the information processing apparatus 1 that performs name identification. The information processing apparatus 1 includes a control unit 11, a nonvolatile storage unit 12, and a volatile storage unit 13. The nonvolatile storage unit 12 includes a name identification source DB 121 that stores name identification source data, and a name identification destination DB 122 that stores name identification source data. Furthermore, the non-volatile storage unit 12 stores a name identification definition 123 created in advance by the user. In addition, the non-volatile storage unit 12 stores a rough aperture definition 124 created by a rough aperture definition generation unit 111 described later.

  The control unit 11 includes a rough aperture definition generation unit 111, a rough aperture processing unit 113, and a name identification unit 114. The rough aperture definition generating unit 111 creates a tuned rough aperture definition 124 based on the name identification definition 123 and stores it in the nonvolatile storage unit 12. The rough narrowing processing unit 113 refers to the records in the name identification source DB 121 and the rough narrowing definition 124 and performs rough narrowing of records in the name identification target DB 122. The result of rough drawing is stored in the volatile storage unit 13 as the result of rough drawing processing 131. Based on the rough narrowing processing result 131, the name collation unit 114 performs name collation between each record in the name collation source DB 121 and a narrowed record among the records in the name collation destination DB 122. The name identification result is stored in the volatile storage unit 13 as the name identification process result 132.

  FIG. 19 illustrates a hardware configuration example of the information processing apparatus 1 that performs name identification. The information processing apparatus 1 includes a CPU 1010, an interface device 1020, a display device 1030, an input device 1040, a drive device 1050, an auxiliary storage device 1060, and a memory device 1070, which are mutually connected via a bus 1080. It is connected to the.

  A program for realizing the functions of the information processing apparatus 1 is provided by a recording medium 1090 such as a CD-ROM. When the recording medium 1090 on which the program is recorded is set in the drive device 1050, the program is installed from the recording medium 1090 to the auxiliary storage device 1060 via the drive device 1050. Alternatively, the program does not necessarily have to be installed using the recording medium 1090, and can be downloaded from another computer via a network. The auxiliary storage device 1060 stores the installed program and also stores necessary files and data.

  The memory device 1070 reads the program from the auxiliary storage device 1060 and stores it when there is an instruction to start the program. The CPU 1010 realizes the function of the information processing apparatus 1 according to a program stored in the memory device 1070. The interface device 1020 is used as an interface for connecting to another computer through a network. The display device 1030 displays a GUI (Graphical User Interface) or the like by a program. The input device 1040 is a keyboard, a mouse, or the like, and is used by a name identification user to input various instructions.

FIG. 20 shows a flow of processing for creating the rough aperture definition 124 performed by the information processing apparatus 1. First, in step S400, the rough aperture definition generator 111 reads the name identification definition 123 stored in advance in the nonvolatile storage unit 12. A specific example of the name identification definition 123 is as shown in FIG. Subsequently, the rough aperture definition generation unit 111 generates an initial rough aperture definition (not shown) based on the read name identification definition 123 (also referred to as a current rough aperture definition), and the nonvolatile storage unit 1
Save to. The method for generating the initial rough aperture definition at this time is as described with reference to FIGS. That is, for each of one or more comparison conditions defined in the name identification definition 123, if the comparison condition is “complete match”, the rough search condition is also “complete match”, and the comparison condition is “fuzzy match”. For example, a rough search fuzzy search condition using n-gram and index matching number c is defined.

  In step S401, the rough aperture definition generation unit 111 determines whether the generated initial rough aperture definition includes an ambiguous condition. If the determination result is “NO”, the process is terminated, and the initial rough aperture definition is used as the final rough aperture definition 124 as it is.

  If the determination result in step S401 is “YES”, the name identification unit 114 samples several records in the name identification source DB 121 in step S402. As described above, the sampling is performed by analyzing the name identification source record so that the sampled records are not biased as much as possible. Although not shown, the sampled name identification source record is stored in the volatile storage unit 13.

  In step S <b> 403, the name identification unit 114 performs name identification for some of the sampled records without rough narrowing, that is, name identification trial. The name identification unit 114 has a volatile storage unit as a name identification trial result (not shown) for each comparison condition for a name identification destination record that satisfies at least one of the comparison conditions defined in the name identification definition in relation to the sampled records. 13 to save.

  In step S404, the rough aperture processing unit 113 refers to the record sampled by the name identification unit 114 and the current rough aperture definition, and performs a rough aperture trial. The result of the rough drawing trial is stored in the volatile storage unit 13 as a rough drawing trial result (not shown).

  In step S405, the rough aperture definition generation unit 111 refers to the name identification trial result and the rough aperture trial result stored in the volatile storage unit 13. Then, the rough aperture definition generating unit 111 calculates the rough aperture matching rate and the leak rate as shown in FIGS. 16 (A) and 16 (B).

  In step S406, the rough aperture definition generator 111 checks whether or not the leakage rate calculated in step S405 is 0%. If the check result is “YES”, the process proceeds to step S407. If the check result is “NO”, the process proceeds to step S409, and the rough aperture definition immediately before the current rough aperture definition is set as the current rough aperture definition, and the process proceeds to step S410. Note that the leak rate is always 0% for the initial rough aperture definition, so the check result in step S406 is always "YES".

  In step S407, the rough aperture definition generator 111 checks whether or not the rough aperture matching ratio calculated in step S405 is 100%. If the check result is “YES”, the process proceeds to step S410; otherwise, the process proceeds to step S408.

  In step S408, the rough aperture definition generation unit 111 determines whether there is a fuzzy search condition defined in the current rough aperture definition that is a tuning target. If the determination result is “YES”, the process proceeds to step S412; otherwise, the process proceeds to step S410.

  In step S410, the rough aperture definition generating unit 111 determines whether there is a fuzzy search condition defined in the current rough aperture definition that has not been tuned yet. If this determination is “YES”, the flow proceeds to step S411.

  In step S411, the rough aperture definition generation unit 111 selects one of the fuzzy search conditions that have not been tuned as a tuning target.

  In step S <b> 412, the rough aperture definition generation unit 111 changes the fuzzy search condition to be tuned to a more narrow condition. Specifically, as described with reference to FIG. 17, n ′ is calculated by adding 1 to n of n-gram included in the fuzzy search condition to be tuned. Subsequently, c ′ is calculated based on the n ′ and the index coincidence number c.

  In step S413, the rough aperture definition generation unit 111 determines whether or not the ambiguous search condition can be changed. Specifically, when n ′ = n + 1 is equal to or less than the maximum value of n of n-gram described with reference to FIG. 15 and c ′ is greater than 0, it is determined that the change is possible. In this case, a new fuzzy search condition is created with n 'and c' as new n and c. If the change is impossible, a new fuzzy search condition is created with c '= c + 1. Then, a new rough aperture definition including this new fuzzy search condition is created, and the steps after step S404 are repeated. That is, how much n and c in the initial rough aperture definition can be increased is determined based on the name collation trial result in step S403 and the rough aperture trial result in step S404. Then, the determination result in step S413 eventually becomes “NO”, and the process proceeds to step S410.

  Then, the determination result in step S410 eventually becomes “NO”, and the process ends. The rough aperture definition at this time becomes the final rough aperture definition 124 as it is.

[7. Other Embodiments]
In the name identification definition shown in FIG. 3, one comparison condition is defined for each combination of a name identification source column and a name identification destination column to be compared. However, in another embodiment, a plurality of comparison conditions can be defined for one combination of a name identification source column and a name identification target column to be compared.

  In step S412, FIG. 20 is not limited to increasing n by 1, but may increase n by 2, 3 or more.

  The functional configuration and physical configuration of the information processing apparatus described above are not limited to the above-described aspects. For example, the functions and physical resources are integrated and mounted, or on the contrary, the information processing apparatus is further distributed and mounted. It is also possible to do.

Regarding the above embodiment, the following additional notes are disclosed.
(Appendix 1)
An extraction program for extracting from a character string group a character string having an edit distance between a predetermined character string and a predetermined number or less,
A quotient obtained by dividing the number of characters in the predetermined character string by the predetermined number, which is one or a plurality of partial character strings that are continuous character strings in the predetermined character string. Extract one or more substrings that are smaller than
Extracting a character string including any one of the extracted partial character strings from the character string group;
For a character string extracted from the character string group, it is determined whether an edit distance between the character string and the predetermined character string is equal to or less than a predetermined distance.
An extraction program that causes a computer to execute processing.
(Appendix 2)
A character string having an edit distance between the predetermined character string and the predetermined number or less is:
It is a character string obtained by performing editing of the predetermined number of times or less with respect to the predetermined character string.
The extraction program according to supplementary note 1, characterized in that:
(Appendix 3)
In the computer,
Of the one or a plurality of partial character strings, a partial character string equal to or smaller than a difference obtained by subtracting a product of the number of characters and 1 of the predetermined character string and a sum of the predetermined number and 1 and a natural number smaller than the quotient. A character string including the character string group,
The extraction program according to supplementary note 1, wherein the extraction program is executed.
(Appendix 4)
An extraction method for extracting a character string having a predetermined number or less of edit distances from a predetermined character string from a character string group,
A quotient obtained by dividing the number of characters in the predetermined character string by the predetermined number, which is one or a plurality of partial character strings that are continuous character strings in the predetermined character string. Extract one or more substrings that are smaller than
Extracting a character string including any one of the extracted partial character strings from the character string group;
For a character string extracted from the character string group, it is determined whether an edit distance between the character string and the predetermined character string is equal to or less than a predetermined distance.
An extraction method that causes a computer to execute processing.
(Appendix 5)
A character string having an edit distance between the predetermined character string and the predetermined number or less is:
It is a character string obtained by performing editing of the predetermined number of times or less with respect to the predetermined character string.
The extraction method according to supplementary note 4, characterized by:
(Appendix 6)
In the computer,
Of the one or a plurality of partial character strings, a partial character string equal to or smaller than a difference obtained by subtracting a product of the number of characters and 1 of the predetermined character string and a sum of the predetermined number and 1 and a natural number smaller than the quotient. A character string including the character string group,
The extraction method according to supplementary note 4, wherein:
(Appendix 7)
An extraction device that extracts a character string having an edit distance between a predetermined character string and a predetermined number or less from a character string group,
A quotient obtained by dividing the number of characters in the predetermined character string by the predetermined number, which is one or a plurality of partial character strings that are continuous character strings in the predetermined character string. A first extraction means for extracting one or more partial character strings that are smaller than
Second extraction means for extracting a character string including any one of the extracted partial character strings from the character string group;
Determination means for determining whether an edit distance between the character string extracted from the character string group and the predetermined character string is equal to or less than a predetermined distance;
The extraction apparatus characterized by including.
(Appendix 8)
A character string having an edit distance between the predetermined character string and the predetermined number or less is:
It is a character string obtained by performing editing of the predetermined number of times or less with respect to the predetermined character string.
The extraction device according to appendix 7, characterized by:
(Appendix 9)
The second extraction means comprises:
Of the one or a plurality of partial character strings, a partial character string equal to or smaller than a difference obtained by subtracting a product of the number of characters and 1 of the predetermined character string and a sum of the predetermined number and 1 and a natural number smaller than the quotient. A character string including the character string group,
The extraction device according to appendix 7, characterized by:

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 11 Control part 12 Non-volatile memory | storage part 13 Volatile memory | storage part 100 Name collation source 101 Name collation target 102 Rough narrowing process 102a Rough narrowing definition 102b Search result 103 Name collation process 103a Name collation definition 111 Rough narrowing definition production | generation part 113 Rough narrowing process Part 114 Name identification part 121 Name identification source DB
122 name identification DB
123 Name identification definition 124 Coarse aperture definition 131 Coarse aperture processing result 132 Name identification processing result 1010 CPU
1020 Interface device 1030 Display device 1040 Input device 1050 Drive device 1060 Auxiliary storage device 1070 Memory device 1080 Bus 1090 Recording medium m Minimum number of characters of comparison source character string (first number)
n Value of n of “n-gram” (second number)
c Number of index matches (third number)
d d (fourth number) of “edit distance within d-character difference”
r “Edit distance ratio within r” r

Claims (5)

A one or more partial character strings included in Jo Tokoro string, before Symbol less characters than the quotient obtained by dividing the number of characters of a given character string with a predetermined edit distance, one or more substrings Extract and
The difference obtained by subtracting the product of the predetermined edit distance and the sum of 1 and the natural number smaller than the quotient from the sum of the number of characters of the predetermined character string and the sum of 1 among the extracted partial character strings. a string containing one of the following substrings extracted from character strings,
An extraction program that causes a computer to execute processing. For a character string extracted from the character string group, it is determined whether an edit distance between the character string and the predetermined character string is equal to or less than the predetermined edit distance.
The extraction program according to claim 1, further executing processing. The character string whose edit distance between the predetermined character string is equal to or less than the predetermined edit distance is:
A character string obtained by editing the predetermined character string for the number of editing times equal to or less than the predetermined editing distance ;
The extraction program according to claim 2 , wherein: A one or more partial character strings included in Jo Tokoro string, before Symbol less characters than the quotient obtained by dividing the number of characters of a given character string with a predetermined edit distance, one or more substrings Extract and
The difference obtained by subtracting the product of the predetermined edit distance and the sum of 1 and the natural number smaller than the quotient from the sum of the number of characters of the predetermined character string and the sum of 1 among the extracted partial character strings. a string containing one of the following substrings extracted from character strings,
An extraction method that causes a computer to execute processing. A one or more partial character strings included in Jo Tokoro string, before Symbol less characters than the quotient obtained by dividing the number of characters of a given character string with a predetermined edit distance, one or more substrings First extracting means for extracting;
The difference obtained by subtracting the product of the predetermined edit distance and the sum of 1 and the natural number smaller than the quotient from the sum of the number of characters of the predetermined character string and the sum of 1 among the extracted partial character strings. A second extraction means for extracting a character string including any of the following partial character strings from the character string group ;
The extraction apparatus characterized by including.

Images