JP6143638B2 - Data processing apparatus and data processing method - Google Patents

Description

  The present invention relates to a data processing apparatus, and more particularly to a technique for extracting data including a metacharacter expression meaning that one numerical value is selected from a plurality of numerical values.

  In general, when a customer places an order for a product by referring to a product catalog, the customer often places an order with the product model number. When an order entry server receives an order, if there is an error in the order (for example, a product that is not in the catalog is specified), the correct product name candidate is extracted from the catalog database by searching using the character string similarity. (For example, refer to Patent Document 1).

  In the product catalog of industrial parts, some of the product model numbers represent processing specifications, and some of the product model numbers are displayed in accordance with the width of the conditions that can be processed. Some of the processing conditions are expressed by “metacharacter expression” that “a single numerical value is selected as the processing condition”.

  As an example of the model number of a product having “meta-character expression”, if A {1, 1.5, 2, 2.5, 3} B, {1, 1.5, 2, 2.5, 3} Means that one numerical value is selected from five numerical values 1, 1.5, 2, 2.5, and 3. The customer inputs, for example, A1.5B when placing an order.

  In addition, as an example of the model number of a product having another “metacharacter expression”, A {1. . 3 (0.5)} B, {1. . 3 (0.5)} means that one of the numbers from 1 to 3 is selected from 0.5. In this case, {1. . 3 (0.5)} has the same meaning as {1, 1.5, 2, 2.5, 3}.

International Publication WO2007 / 132564

  In general, the similarity between character strings is calculated for a character string representing a single product. For this reason, when there is data including a metacharacter expression in the database, it is very difficult to calculate a similarity for a character string expressed in the metacharacter expression.

  Therefore, when calculating the character string similarity for the model number of a product having a meta-character expression, the model number of the product having the meta-character expression has to be divided into a character string representing a single product and calculated. . That is, the model number of the product having “metacharacter expression” is A {1. . In the case of 3 (0.5)} B, it is necessary to decompose into A1B, A1.5B, A2B, A2.5B, and A3B.

  However, if the product model number with meta-character representation is decomposed into a single product model number string, the number of product items increases explosively, so this decomposition is not realistic from the viewpoint of computational resources and calculation time. .

  Also, when searching for similar character strings, using a binary search tree with a distance as an index, such as a VP (Vantage-Point) tree, may be possible to search faster than an all-case search. However, in reality, whether or not a VP tree can be used with a character string having a metacharacter representation has not yet been examined.

  The present invention has been made to solve the above-described problem, and is a data process capable of extracting data including a metacharacter expression that means that one numerical value is selected from a plurality of numerical values. An object is to provide an apparatus and a data processing method.

  In order to achieve the above object, a data processing apparatus according to the present invention is a data processing apparatus that extracts candidate data similar to input target data from a database, and the database includes one numerical value among a plurality of numerical values. When data including a metacharacter expression meaning that is selected and data not including a metacharacter expression are stored, and candidate data similar to the target data is extracted from data including the metacharacter expression (for example, Data 74B including the metacharacter expression of FIG. 4), and among the character string including the metacharacter expression, a numerical value other than the metacharacter expression is set as the metacharacter expression (that is, the number selection expression having one option), and the metacharacter expression Each unit is treated as one unit character, and one numerical value and characters other than the numerical value included in the target data and database data are When the candidate data similar to the target data is extracted from data that does not include the metacharacter expression (for example, the data 74A including the metacharacter expression of FIG. 4), the target data and the data of the database are divided. A unit character dividing unit that divides each character of the included character string as one unit character and calculates the number of divided unit characters as the number of unit characters, data including meta character expression from database data, and meta character expression A binary search tree creation unit (for example, a VP tree creation unit 60) that classifies the data into not-contained data, creates a binary search tree (for example, a VP tree) for each number of unit characters based on the classified data, and target data A binary search tree extraction unit (for example, a VP tree extraction unit 32) that extracts a binary search tree corresponding to a predetermined range based on the number of unit characters as candidate data; Of candidate data extracted by the distance calculation unit for calculating the distance between the data and the data in the database and the candidate data extracted by the binary search tree extraction unit as candidate data And a data extraction unit.

  According to the present invention, it is possible to extract data including a metacharacter expression meaning that one numerical value is selected from a plurality of numerical values.

It is a figure which shows the data processor which concerns on embodiment of this invention. It is a figure which shows an example of catalog model number DB. It is a figure which shows an example of incorrect input correction log | history DB. It is a figure which shows an example of catalog model number DB in order of unit character number. It is a flowchart which shows an input determination process. It is a figure which shows an example of an input determination process. It is a figure which shows the example of creation of VP tree. It is a figure which shows the intermediate value of the edit distance of VP tree. It is a figure which shows the relationship of the edit distance of the character strings q, p, and l. It is a figure which shows the relationship of the edit distance of character strings q, p, and r. It is a figure which shows the example which cannot be pruned in case the left pruning condition is satisfied. It is a figure which shows the example which cannot be pruned in case the right pruning condition is satisfied. It is a flowchart which shows an erroneous input process. It is a figure which shows the example of extraction of VP tree. It is a flowchart which shows VP tree search processing. It is a figure which shows the example of a search of VP tree. It is a figure which shows an example of the display screen of a client.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a diagram showing a data processing apparatus according to an embodiment of the present invention. The data processing apparatus 100 for authenticating the product model number input from the client 200 searches the product catalog database 70 and whether the input target data is stored in the database 70, and the target data is the database 70. The processing unit 10 for extracting candidate data similar to the target data from the database 70, the input unit 81 for inputting a threshold value used for the extraction, and the display unit for displaying the processing result 82, and a communication unit 85 that communicates with the client 200 and the like via the network 300.

  The database 70 is configured by an HDD (Hard disk drive) device or the like. The processing unit 10 is realized by executing a program on a RAM (Random Access Memory) or HDD by a CPU (Central Processing Unit). The input unit 81 is a device for inputting an instruction to a computer such as a keyboard and a mouse, and inputs an instruction for starting a program. The display unit 82 is a display or the like, and displays an execution status and an execution result of the processing by the data processing apparatus 100. The communication unit 85 exchanges various data and commands with other servers via the network 300.

  In the database 70, the product name, the product model number, the manufacturer name, the price, the delivery date, the product characteristics, and the like, which are detailed information of the product catalog, are stored in the catalog DB 71 and the catalog DB 71 for each predetermined classification. A catalog model number DB 72 (see FIG. 2) that is classified information, an erroneous input correction history DB 73 (see FIG. 3), and a catalog model number DB 72 that are records of the history of the number of times of input corrected when there is an erroneous input, Catalog model number DB 74 (see FIG. 4), which is information sorted in the order of the number of unit characters, and extraction condition information 75, which is information (for example, a threshold) used when extracting candidate data similar to the target data from the database 70 The catalog model number VP tree DB 76 (catalog model binary search tree DB) created from the unit model number order catalog model DB 74 It is. The VP tree is one of binary search trees, and has the most basic tree structure among the search trees.

  The processing unit 10 includes a search unit 20 (input determination processing unit) that searches whether or not the input target data is stored in the database 70, and the target data when the target data is not stored in the database 70. An extraction unit 30 (error input processing unit) that extracts candidate data similar to the database 70, a candidate presentation unit 40 that presents the extracted candidate data, and a history update unit 50 that updates the history of the erroneous input correction history DB 73 And a plurality of programs of the VP tree creation unit 60 (binary search tree creation unit) that creates the catalog model number VP tree DB 76 for each number of unit characters.

  When the extraction destination of candidate data similar to the target data is data including a metacharacter expression, the search unit 20 uses a numerical value other than the metacharacter expression as a metacharacter expression among character strings including the metacharacter expression (that is, Each option is treated as one unit character for each meta-character representation, and one numeric value and other characters included in the target data and database data are divided as one unit character. When candidate data similar to the data is extracted from data that does not contain meta-character representation, it is divided as one unit character for each character of the character string contained in the target data and database data, and the number of divided unit characters is Whether the unit character dividing unit 21 calculated as the number of unit characters matches the unit character of the target data and the unit character of the comparison target data And a unit character comparison unit 22 for comparing.

  The extracting unit 30 includes a unit character dividing unit 21, a VP tree extracting unit 32 (binary search tree extracting unit) that extracts a VP tree in which the number of unit characters falls within a predetermined range, and between the target data and the comparison target data. A distance calculation unit 33 that calculates the distance of the candidate data, and a candidate data extraction unit 34 that extracts, as candidate data, data whose calculated distance is equal to or less than a predetermined value from the candidate data extracted by the VP tree extraction unit 32 ing.

The data processing apparatus 100 according to the present embodiment has the following characteristics although details will be described later.
(1) The VP tree creation unit 60 classifies the data in the database 70 into data including a metacharacter expression and data not including a metacharacter expression, and generates a VP tree for each of the classified data.
(2) When constructing a VP tree of data including a metacharacter expression, an edit distance that is a Hausdorff distance is used by treating a numerical value as one unit character. In this case, the unit character dividing unit 21 treats a numerical value other than the metacharacter expression as a metacharacter expression (that is, a numerical value selection expression with one option) from the character string including the metacharacter expression as one unit character. At the same time, each character of the character string is treated as one unit character.

  FIG. 2 is a diagram illustrating an example of the catalog model number DB. In the catalog model number DB 72, data representing model numbers of many products is registered. For example, there are 123345, P10-5,... As data not including the metacharacter expression, and C {10,11} -D3, C {1,10} -B {2 as the data including the metacharacter expression. , 3},. There are many product model numbers that are already described in the metacharacter representation.

  More specifically, the product model number is D51-M {0.1. . In the case of data of 100 (0.01)}-LS {10, 20, 45, 50, 90}, a numerical value (for example, 51) other than the metacharacter expression is used as the metacharacter in the character string including the metacharacter expression. Numeric values other than expressions are treated as one-unit characters as meta-character representations (that is, as one-choice numeric selection representations), and each character of the character string is treated as one unit character. That is, in the case of the data, D {51} -M {0.1. . 100 (0.01)}-LS {10, 20, 45, 50, 90}, the character “D”, the metacharacter expression “{51}”, the character “−”, the character “M”, the metacharacter expression “ {0.1 ... 100 (0.01)} ", character"-", character" L ", character" S ", and metacharacter expression" {10, 20, 45, 50, 90} " The number of unit characters is “9”. Note that the portion of the metacharacter expression is one unit character because it represents one numerical value.

  The metacharacter expression part is {0.1. . 100 (0.01)} means that one of the numerical values in increments of 0.01 from 0.1 to 100 is designated. Further, {10, 20, 45, 50, 90}, which is a metacharacter expression part, means that one of the listed numerical values is designated.

  FIG. 3 is a diagram illustrating an example of the erroneous input correction history DB. The incorrect input correction history DB 73 is correction history information that records the history of the number of inputs corrected when there was an incorrect input, and includes the model number and the number of erroneous inputs. More specifically, when the model number is C {10,11} -3D, the number of erroneous inputs is 50, and when the model number is C {1,10} -B {2,3}, the number of erroneous inputs is 100. You can see that there were times.

  FIG. 4 is a diagram illustrating an example of a catalog model number DB in unit character number order. The unit character division unit 21 (see FIG. 1) classifies the data in the database into data 74A that does not include a metacharacter expression and data 74B that includes a metacharacter expression, and catalogs each classified data in advance. The model number DB 72 is classified in the order of the number of unit characters. The unit character number order catalog model number DB 74 includes the number of unit characters, the model numbers after the unit character division (separated by a comma (,) for each unit character), and the original model number.

  In the case of data 74A that does not include a metacharacter expression, one character is used for each character in the character string. For example, when the model number is P10-5, it is divided into P, 1, 0,-, 5 and the number of unit characters is "5". When the model number is 123345, it is divided into 1, 2, 3, 3, 4, and 5 and the number of unit characters is “6”.

  In the case of the data 74B including the meta character expression, a numerical value other than the meta character expression is set as the meta character expression (that is, the numerical value selection expression having one option) in the character string including the meta character expression, and for each meta character expression. While treating as one unit character, one numerical value and characters other than the numerical value included in the target data and database data are divided as one unit character. For example, the model numbers after unit character division are A, B, C,-, L, {1. . 10 (0.1)}, −, W, {2. . In the case of 5 (0.5)}, −, H, {1, 2, 3}, the number of unit characters is “12”. When the model number after the unit character division is C, {10, 11},-, D, 3, the number of unit characters is “5”.

  Here, in FIG. 4, when the model number is C {10,11} -D3, the model number after the unit character division is C, {10,11},-, D, {3}, which includes a metacharacter expression. In the character string, the numerical value “3” other than the metacharacter expression is corrected to “{3}” which is the metacharacter expression.

(Input judgment processing)
FIG. 5 is a flowchart showing the input determination process. FIG. 6 is a diagram illustrating an example of the input determination process. This will be described with reference to FIG. The input determination process S110 is a process for searching whether or not the model number included in the order data is stored in the database 70 as the target data when the search unit 20 receives the order data from the client 200. The target data is, for example, “A10-B2” shown in FIG.

The unit character dividing unit 21 divides the target data into unit characters based on the following (processing S111).
(1) When the search destination does not include a metacharacter expression (for example, when the data 74A does not include a metacharacter expression): The target data is divided into one unit character for each character of the character string. (2) The search destination is a metacharacter. In the case of including an expression (for example, in the case of data 74B including a metacharacter expression): The numerical value of the target data is divided as one unit character. That is, as shown in FIG. , 0,-, B, 2 ", and in the case of (2)," A, 10,-, B, 2 ".

  The unit character dividing unit 21 reads the corresponding catalog model number DB 72 into the memory and stores it in the work list (processing S112).

  The unit character comparison unit 22 repeats the processes S114 to S116 for each unit character from the beginning of the target data (process S113). First, it is determined whether or not a unit character corresponding to the unit character of the target data is in the list (step S114). If there is a corresponding unit character (step S114, Yes), the list is narrowed down (step S115). To the unit character (step S116). If there is no corresponding unit character (No at Step S114), the process proceeds to an erroneous input process (Step S118).

  Specifically, referring to FIG. 6, when narrowing down by the first unit character, it becomes as shown in FIG. Data corresponding to “A” of the target data is narrowed down, for example, A10-XY, A {10,13} -D {2,3}, ABC-L {1. . 10 (0.1)}-W {2. . 5 (0.5)}-H {1, 2, 3}. Next, when narrowing down by the second unit character, it becomes as shown in FIG. In the case of process S111 (1), data corresponding to “1” is narrowed down to, for example, A1-XY. In the case of process S111 (2), data corresponding to “10” is narrowed down to, for example, A {10, 13} −D {2, 3}. Note that “10” is included in {10, 13} of the metacharacter expression.

  Further, when narrowing down by the third unit character, it becomes as shown in FIG. Data corresponding to “0” in the case of the process S111 (1) and “-” in the case of the process S111 (2) is narrowed down, but A10-XY, A {10 that is not changed compared to FIG. , 13} −D {2, 3}. Furthermore, when narrowing down by the 4th unit character, it becomes A10-XY as shown in FIG. 6 (f), and when narrowing down by the 5th unit character, as shown in FIG. There is no corresponding data. In the case of the example shown in FIG. 6, “A10-B2”, which is the target data, is determined as an erroneous input by narrowing down the fifth unit character.

  On the other hand, if there is “A10-B2” which is the target data, the model number is correct, and the process proceeds to the order receiving process (process S117). The order receiving process is handled by an order receiving server (not shown) in charge of the order receiving process, but the description thereof is omitted.

(VP tree creation processing)
The creation of a VP tree of data that does not include a metacharacter expression, which is the basis of this embodiment, will be described. FIG. 7 is a diagram illustrating an example of creating a VP tree. FIG. 8 is a diagram showing an intermediate value of the edit distance of the VP tree. The VP tree is one of binary search trees with the edit distance as an index. Here, the character string is referred to as a node.

  FIG. 7 shows an example of a VP tree including seven nodes “ABCDEFG”, “ABC4EFG”, “ABC45FG”, “AB345FG”, “A23456G”, “A234567”, “1234567”. In the VP tree of FIG. 7, the VP tree is created with the reference model number “ABCDEFG” among the seven nodes.

  When the intermediate value M of the edit distances of the node “ABCDEFG” and all six child nodes is obtained, the edit distance is less than 4 on the left side, and the edit distance of 4 or more is placed on the right side. Then, when the intermediate value M of the edit distance between the node “ABC4EFG” and its two child nodes is 1.5, the edit distance of less than 1.5 is arranged on the left side, and the edit distance of 1.5 or more is placed on the right side. Place things. Further, when the intermediate value M of the edit distance between the node “A23456G” and its two child nodes is obtained, it is 1.5, the edit distance less than 1.5 is arranged on the left side, and the edit distance of 1.5 or more is placed on the right side. Place things. FIG. 8 illustrates the relationship between each node and the intermediate value M of the edit distance. In calculating the distance between the data, for example, existing techniques such as a Hamming distance (signal distance) and a Levenshtein distance (editing distance) can be used.

As a general definition of edit distance,
(1) The edit distance becomes 0 only when the two character strings are equal and is limited to that time.
(2) Symmetric. That is, the edit distance d between the two character strings s and t is
d (s, t) = d (t, s) is established.
(3) Triangular inequality of edit distance holds. When u is an arbitrary character string,
It satisfies d (s, t) ≦ d (s, u) + d (u, t).

This holds for all nodes that have child nodes of the VP tree created above. Hereinafter, when a child node is described, it also includes nodes below grandchildren.
(A) A node has an intermediate value M of edit distances with all child nodes of the node.
(B) The edit distance between the node and the left child node of the node is less than the median value M of the node.
(C) The edit distance between the node and the right child node of the node is equal to or greater than the median value M of the node.

  FIG. 9 is a diagram showing the relationship between the edit distances of the character strings q, p, and l. Now, let q be a search character string as target data, p be a character string of a node having a VP tree, and l be an arbitrary left child node character string of a node of the character string p. When a VP tree is searched for a character string whose edit distance to the search character string is D or less, if D + M <d (q, p) is established, the left branch of the node may not be searched (hereinafter, the left side (This is called pruning.)

  When D + M <d (q, p) holds, there is no node character string on the left side of the node of the character string p that has an edit distance of D or less with respect to the search character string q. In FIG. 9, all the child nodes on the left side of the node of the character string p are inside the radius M from the node of the character string p. For this reason, when D + M <d (q, p) holds, the edit distance from the character string q for any character string on the left side of the character string p is larger (far) than D.

  FIG. 10 is a diagram illustrating the relationship between the edit distances of the character strings q, p, and r. Now, let q be a search character string as target data, p be a character string of a node having a VP tree, and r be an arbitrary right child node character string of a node of the character string p. When searching for a character string whose edit distance to the search character string is D or less using a VP tree, if D + d (q, p) ≦ M holds, the right branch of the node may not be searched (hereinafter, the right side Called pruning)

  When D + d (q, p) ≦ M holds, there is no right-side node character string of the node of the character string p whose edit distance from the search character string q is less than or equal to D. As shown in FIG. 10, since all the child nodes on the right side of the node of the character string p are outside the radius M from the node of the character string p, when D + d (q, p) ≦ M holds, In any character string on the right side, the editing distance from the character string q is larger (far) than D.

  When the left pruning condition shown in FIG. 9 and the right pruning condition shown in FIG. 10 are satisfied, the edit distance calculation count of the search in the VP tree is smaller than the edit distance calculation count in the all-case search, For this reason, the search using the VP tree is faster than the all-case search. If no pruning can be performed with the VP tree, all cases are searched.

  The pruning method described above is applicable to the case of data 74A (see FIG. 4) that does not include a metacharacter expression. However, in the case of data including a metacharacter expression, there is no examination example of whether it can be applied to 74B (see FIG. 4), and in this embodiment, detailed examination including examination of edit distance is performed.

(Edit distance)
The edit distance calculated by the distance calculation unit 33 in this embodiment will be described. For the edit distance, Levenshtein distance, Damerau-Levenshtein distance or the like, which is one of methods for measuring the similarity between one character string and another character string, is used. However, there has been no example of handling the edit distance for the metacharacter expression of the present embodiment. Since the portion of the metacharacter expression can be regarded as a set of a plurality of numerical values, the method for dealing with it has been examined in this embodiment.

  The edit distance indicates that the two character strings are different as the edit distance is larger. The edit distance is determined when the character string that is the target data is divided into unit characters, and the unit character of one character string is inserted, deleted, or replaced (considering transposition in some cases) to generate the other character string. It is defined by the minimum number of operations. Since the part of the meta character expression represents one numerical value, the edit distance is calculated by using each of the sub strings of the meta character expression as unit characters. As already described, when the search destination includes a meta-character expression, the numerical value part in the model number of the catalog product as the target data is calculated as one unit character to calculate the edit distance.

The edit distance calculation rule for the meta selection expression is as follows.
(1) Comparison of the meta-selection expression part {m1, m2, m3} and the letter n: If n matches any of m1, m2 or m3, the same unit character is used.
(2) Comparison between {m1, m2, m3} and {n1, n2, n3, n4}, which are parts of meta selection expressions: If the meta selection expressions are exactly the same contents, the same unit characters and different contents If there is a different unit character.

  For example, in the case of calculating the edit distance between A1-B2 and C {10,11} -D3, the character “A” is replaced with the character “C”, and the numerical value “1” is replaced with the metacharacter expression part “{10 , 11} ”, the character“ − ”is the same, the character“ B ”is replaced with the character“ D ”, and the numerical value“ 2 ”is replaced with the numerical value“ 3 ”. “4”.

  Similarly, when calculating the edit distance between A1-B2 and C {1,10} -B {2,3}, the character “A” is replaced with the character “C”, and the numerical value “1” The character expression part “{1, 10}” is the same for both the character “−” and the character “B”, and the numerical value “2” is the meta character expression part “{2, 3”. } ". Therefore, the edit distance is “1”.

In other words, the unit character comparison rules are as follows.
(A) Comparison of numerical unit characters:
If the numbers are the same, the edit distance is +0 (no addition).
If the numbers are different, the editing distance is +1 (add 1).
(B) Comparison of numerical unit characters and metacharacter expression:
If the numerical value is within the range of the metacharacter expression, the editing distance is set to +0.
If the numerical value is not within the range of the metacharacter expression, the editing distance is set to +1.
(C) Comparison of metacharacter expression parts:
If {m1,..., _{M M} } and {n1,..., _{N N} }, which are parts of the meta selection expression, are completely coincident numerical values, the editing distance is set to +0.
If {m1,..., _{M M} } and {n1,..., _{N N} }, which are parts of the meta selection expression, are not completely matching numerical values, the editing distance is set to +1.

Pruning conditions were examined based on the calculation rules.
FIG. 11 is a diagram illustrating an example in which pruning is not possible when the left pruning condition is satisfied. Here, the character string of the metacharacter expression is capitalized. Referring to FIG. 11, the condition “D (= 1) + M (= 2) <d (q, P) (= 4)” that enables pruning on the left side is established. The fact that L at a distance of 1 is missed will be explained.

  A character string including a meta character expression (hereinafter referred to as a meta character string) P “C {10,11} -D3” includes character strings “C10-D3” and “C11-D3” (hereinafter referred to as elements) that do not include a meta character. It can be viewed as a set of non-metastrings).

The distance between the non-meta character string and the meta character string is the minimum value of the distance between the non-meta character string and the element non-meta character string constituting the meta character string, based on the definition of the calculation rule.
d (A1-B2, C10-D3) = 4,
d (A1-B2, C11-D3) = 4
Therefore, d (A1-B2, C {10,11} -D3) = 4.

Similarly, the meta character string L “C {1, 10} -B {2, 3}” includes element non-meta character strings “C1-B2”, “C1-B3”, “C10-B2”, “C10-”. It can be seen as a set of “B3”.
d (A1-B2, C1-B2) = 1,
d (A1-B2, C1-B3) = 2
d (A1-B2, C10-B2) = 2,
d (A1-B2, C10-B3) = 3
Therefore, d (A1-B2, C {1,10} -B {2,3}) = 1.

On the other hand, since the distance between the meta character string P and the meta character string L is determined based on whether or not the comparison between the meta character expression parts ({} parts) is a complete match, d (P, L) is
Viewing the meta string L as a set of “C {1,10} -B2” and “C {1,10} -B3”,
d (C {10,11} -D3, C {1,10} -B2) = 3,
d (C {10,11} -D3, C {1,10} -B3) = 2
Therefore, d (P, L) = 2.

  This does not hold the triangular inequality “d (q, P) ≦ d (q, L) + d (L, P)”, which is a premise for considering the pruning condition. In other words, the current calculation rule cannot define the distance between meta character strings.

Note that a distance can be defined between the non-meta character strings a and b and the meta character string C, and the triangle inequality formed at that time is
d (a, C) ≦ d (a, b) + d (b, C)
It is. However, when the meta character string is D, d (a, C) ≦ d (a, D) + d (D, C)
Means not true.

From the above, the VP tree based on the editing rule must not be pruned, and the merit of using the VP tree is lost in this state. Therefore, when building a VP tree, the following rules were improved.
VP tree construction rule (1): A character string including a meta character expression and a character string not including a meta character expression are separated and a VP tree is separately constructed. Since the search character string that is the target data does not include the metacharacter expression, the calculation at the normal editing distance and the pruning condition can be applied to the VP tree that does not include the metacharacter expression.

By the way, the edit distance d (s, t) between the character string s (character string length is LLs) and the character string t (character string length is LLt) is
| LLs−LLt | ≦ d (s, t) ≦ LLs + LLt.
Thus, | LLs−d (s, t) | ≦ LLt ≦ LLs + d (s, t) is obtained.

  Therefore, if the search target character strings are grouped by their lengths, the character string length LL is used when extracting a character string having an edit distance D or less with respect to the character string q (character string length is LLq). However, it is only necessary to calculate the edit distance only for a character string that satisfies | LLq−D | ≦ LL ≦ LLs + D. (Hereafter, this is called a character string length group search.)

Therefore, the following rules are also set in the construction of the VP tree.
VP tree construction rule (2): Character strings including metacharacter expressions and character strings not including them are grouped for each character string length, and a VP tree is created for each group (character string length).

  VP tree construction rule (3): For a VP tree of a character string including a metacharacter expression, a numerical part other than the metacharacter expression is surrounded by metacharacter symbols “{” and “}”.

  According to this rule, when the edit distance between the meta character strings is calculated, the comparison between the (c) meta character expression parts is applied, and the edit distance between the meta character strings becomes the Hausdorff distance. As a result, d (q, U) ≦ d (q, V) + d (V, U) is established as a triangular inequality of the distance between the character string q not including the meta character expression and the meta character strings U and V. become. For this reason, the triangle inequality which becomes a pruning precondition on the left side of the VP tree is established. Therefore, if the left pruning condition is satisfied even in the VP tree of the meta string, the left side can be pruned.

  FIG. 12 is a diagram illustrating an example in which pruning is not possible when the right pruning condition is satisfied. The condition “D (= 1) + d (q, P) (= 1) ≦ M (= 2)” that enables pruning on the right side is established, but when pruning, R at a distance 1 from q Explain what you miss.

The meta character string P “A {1,3} -D {2,3}” is a set of element non-meta character strings “A1-D2”, “A1-D3”, “A3-D2”, “A3-D3”. Can be seen.
d (A1-B2, A1-D2) = 1,
d (A1-B2, A1-D3) = 2,
d (A1-B2, A3-D2) = 2,
d (A1-B2, A3-D3) = 3
Therefore, d (A1-B2, A {1,3} -D {2,3}) = 1.

Similarly, the meta character string R “C {1,10} -B {2,3}” includes element non-metacharacter strings “C1-B2”, “C1-B3”, “C10-B2”, “C10-”. It can be seen as a set of “B3”.
d (A1-B2, C1-B2) = 1,
d (A1-B2, C1-B3) = 2
d (A1-B2, C10-B2) = 2,
d (A1-B2, C10-B3) = 3
Therefore, d (A1-B2, C {1,10} -B {2,3}) = 1.

  On the other hand, the distance between the meta character string P and the meta character string R is d (P, R) = 3 because the comparison between the meta character expression parts ({} parts) is a perfect match or not. .

  This does not hold the triangular inequality “d (P, R) ≦ d (P, q) + d (q, R)”, which is a premise for considering the pruning condition. In other words, the current calculation rule cannot define the distance between meta character strings.

Note that a distance can be defined between the meta strings a and b and the meta string C, and the triangle inequality that holds at that time is
d (a, C) ≦ d (a, b) + d (b, C)
It is. However, when the meta string is D,
This means that d (C, D) ≦ d (C, a) + d (a, D) does not hold.

Therefore, the following rules are set during VP tree search.
VP tree search rule (4): In a search of a meta-character string VP tree including a meta-character expression, if the left pruning condition is satisfied, the left side is pruned, but the right side is pruned. Always search regardless.

  This rule does not mean that the right side of the entire VP tree should be pruned, but only the right side of each node should not be pruned.

  In addition, since there is a VP tree construction rule (2), even if no pruning is possible with the VP tree, there is no search for all cases. The search time using the VP tree this time is the same as the character string length group search, and if pruning is possible, the speed is increased accordingly. The rate at which pruning occurs depends on the data and the configuration of the VP tree at that time.

  Note that it is not necessary to create a VP tree for each search, and a VP tree once created can be used as long as there is no change in the database that is the basis of the VP tree. Further, O (nlogn) is O where the function of time required for constructing the VP tree is O and the number of data used for creating the VP tree is n. Therefore, it is possible to reconstruct the VP tree in a nightly batch if the database changes. Furthermore, since data can be divided like the VP tree construction rule (1) and the VP tree construction rule (2), the construction and search of the VP tree can be performed in parallel in this unit. If parallelization is performed in this unit, the speed can be further increased.

(Incorrect input processing)
FIG. 13 is a flowchart showing an erroneous input process. FIG. 14 is a diagram illustrating an example of extraction of a VP tree. This will be described with reference to FIG. The erroneous input process S <b> 130 is a process of extracting candidate data similar to the target data from the database 70 when the target data is not stored in the database 70.

  When the candidate data extracting unit 34 receives the target data (processing S131), the candidate data extracting unit 34 initializes the correction candidate list (processing S132).

  The VP tree extraction unit 32 reads the threshold value stored in the extraction condition information 75 (processing S133).

  Before the VP tree extraction unit 32 performs the VP tree search process, the VP tree creation unit 60 previously stores data 74A (metacharacters) that does not include a metacharacter expression from the unit character number order catalog model number DB 74 (see FIG. 4). None) and data 74B including meta-character representation (with meta-characters), classification is made for each unit character number (see FIG. 14A), and a VP tree is created for each classified unit character number. (See FIG. 14 (b)).

  The candidate data extraction unit 34 extracts correction candidates by a VP tree search process that does not include a metacharacter expression (process S134) and a VP tree search process that includes a metacharacter expression (process S135). Processing S134 and processing S135 will be described with reference to FIG.

  FIG. 15 is a flowchart showing a VP tree search process, where (a) is a VP tree search process that does not include a metacharacter expression, and (b) is a VP tree search process that includes a metacharacter expression.

  In the VP tree search process S134 that does not include the metacharacter expression shown in FIG. 15A, the unit character dividing unit 21 divides the target data into unit characters (process S151). At this time, one character is used for each character of the character string. For example, when the target data is “A10-B2”, it is divided into “A, 1, 0, −, B, 2”, and the number of unit characters is 6. The VP tree extraction unit 32 narrows down the VP tree based on the number of unit characters by the expression (1) based on the number of unit characters and the threshold (see processing S152, FIG. 14C). Note that the process S151 can be omitted if it has already been executed in the input determination process S110.

Absolute value of (number of target data unit characters-threshold) ≤ number of narrowing unit characters
≦ (Number of unit characters of target data + threshold) Expression (1)
Specifically, in the case where the number of unit characters of the target data is “6” and the threshold is “1”, 5 ≦ number of narrowed-down unit characters ≦ 7 according to Expression (1).

  The candidate data extraction unit 34 repeats the processes S154 to S156 for each narrowed unit character (process S153). In step S154, the candidate data extraction unit 34 performs a depth-first search. When the left pruning condition is satisfied, the candidate data extracting unit 34 prunes the left side of the node, and when the right pruning condition is satisfied, Prunes the right side of the node. In step S155, the candidate data extraction unit 34 registers the correction candidate data in the correction candidate list.

  In the VP tree search process S135 including the metacharacter expression shown in FIG. 15B, the unit character dividing unit 21 divides the target data into unit characters (process S161). In this case, the numerical value is a single character. For example, when the target data is “A10-B2”, it is divided into “A, 10,-, B, 2”, and the number of unit characters is 5. Specifically, in the case where the number of unit characters of the target data is “5” and the threshold is “1”, 4 ≦ the number of narrowed-down unit characters ≦ 6 according to Expression (1). The VP tree extraction unit 32 narrows down the VP tree based on the number of unit characters by the above formula (1) based on the number of unit characters and the threshold (see processing S162, FIG. 14C). Note that step S161 can be omitted if it has already been executed in the input determination step S110.

  The candidate data extracting unit 34 repeats the processes S164 to S166 for each narrowed unit character (process S163). In step S164, the candidate data extraction unit 34 performs a depth-first search. When the left pruning condition is satisfied, the candidate data extraction unit 34 performs pruning on the left side of the node, but whether or not the right pruning condition is satisfied. One of the features of the present embodiment is that the search target is not determined (see FIG. 16). In step S165, the candidate data extraction unit 34 registers the correction candidate data in the correction candidate list.

  In addition, in process S154 and process S164, although it implemented by depth priority search, it is not limited to this. For example, you may implement by breadth priority search.

  FIG. 16 is a diagram illustrating a search example of a VP tree. The contents of the process S164 will be specifically described. In node A, when the left pruning condition is satisfied, pruning after node B can be performed. In node A, the process proceeds to node C without determining whether the right pruning condition is satisfied. Then, in the node C, when the left pruning condition is satisfied, the pruning after the node D can be performed. In node C, the process proceeds to node E without determining whether the right pruning condition is satisfied. Furthermore, in the node E, when the left pruning condition is not satisfied, the nodes below the node F are set as search targets. In node E, whether or not the right pruning condition is satisfied is not determined, and nodes G and below are set as search targets.

  Returning to FIG. 13, the candidate data extraction unit 34 determines whether or not there is a correction candidate for the target data in the correction candidate list (processing S136). If there is no correction candidate (No in processing S136), the threshold value increases. (Step S137) and return to Step S134.

  When there is a correction candidate (Yes at Step S136), the candidate presenting unit 40 refers to the erroneous input correction history DB 73, rearranges them in the descending order of the number of erroneous inputs (Step S140), and sorts the corrected candidate data into the client. 200 (processing S141). An output example will be described later with reference to FIG.

  Then, the candidate presentation unit 40 determines whether or not the correction candidate data has been selected by the client 200 (processing S142), and when selected (processing S142, Yes), the number of erroneous inputs of the selected correction candidate data is determined. By adding 1, the erroneous input correction history DB 73 is updated (processing S143), and a series of processing ends. On the other hand, when the correction candidate data is not selected (No in process S142), the process returns to the input determination process S110 (see process S144, FIG. 5), and the process is terminated.

  FIG. 17 is a diagram illustrating an example of a display screen of the client. FIG. 17 shows a display screen of the client 200 presented by the candidate presentation unit 40 (see FIG. 1). On the display screen, for example, in the example shown in FIG. 17A, “A10-B2 does not exist. The model number is not here? C {1,10} -B {2,3}, A {10, 13} -D {2, 3} "is displayed. The customer can quickly notice that there is an input error in the order, for example, C10-B2 is input.

  FIG. 17B is an example in which the candidate presentation unit 40 decomposes the model number including the metacharacter expression of FIG. 17A into the model number not including the metacharacter expression. In the example shown in FIG. 17B, “A10-B2 does not exist. Maybe this? A10-D2, C10-B2, C10-B3, A10-D3, A13-D2, A13- “D3, C1-B2, C1-B3” are displayed. As in FIG. 17A, the customer can quickly notice that there is an input error in the order. In the above example, the output is basically in ascending order of distance and in alphabetical order.

  Although the input from the client 200 has been described in the present embodiment, the administrator may input target data via the input unit 81.

  In some cases, millions of cases are targeted for comparison with the target data. By applying the data processing apparatus of the present embodiment, an input error can be detected at an early stage and a similar model number can be presented, which is an effective means for improving the Internet business. In the present embodiment, the model number of the catalog is shown as the target data. However, the present invention is not limited to this, and the data processing method can be applied to any character string including a numerical value of metacharacter expression.

DESCRIPTION OF SYMBOLS 10 Processing part 20 Search part 21 Unit character division part 22 Unit character comparison part 30 Extraction part 32 VP tree extraction part (binary search tree extraction part)
33 Distance calculation unit 34 Candidate data extraction unit 40 Candidate presentation unit 50 History update unit 60 VP tree creation unit (binary search tree creation unit)
70 Database 71 Catalog DB
72 Catalog DB
73 Wrong input correction history DB
74 Catalog Number DB in Unit Character Order
75 Extraction Condition Information 76 Catalog Model Number VP Tree DB (Catalog Model Number Binary Search Tree DB)
DESCRIPTION OF SYMBOLS 100 Data processing apparatus 200 Client 300 Network S110 Input determination process S130 Incorrect input process S134 VP tree search process which does not include meta character expression S135 VP tree search process which includes meta character expression

Claims (4)

A data processing apparatus that extracts candidate data similar to input target data from a database,
In the database, data including a metacharacter expression meaning that one numerical value is selected from a plurality of numerical values and data not including a metacharacter expression are stored,
When the extraction destination of candidate data similar to the target data is data including a metacharacter expression, a numerical value other than the metacharacter expression is set as a metacharacter expression among the character strings including the metacharacter expression, and each metacharacter expression While treating as one unit character, dividing one numerical value and characters other than the numerical value included in the target data and the data of the database as one unit character, the extraction destination of candidate data similar to the target data is a meta character In the case of data that does not include expression, a unit character dividing unit that divides each character of the character string included in the target data and the data of the database as one unit character and calculates the number of divided unit characters as the number of unit characters When,
A binary search tree creation unit that classifies data including metacharacter expressions and data not including metacharacter expressions from the database data, and generates a binary search tree for each number of unit characters based on the classified data. When,
A binary search tree extraction unit that extracts a binary search tree corresponding to a predetermined range based on the number of unit characters of the target data as the candidate data;
A distance calculation unit for calculating a distance between the target data and the data in the database;
A candidate data extraction unit for extracting, as candidate data, data whose calculated distance is equal to or smaller than a predetermined value among candidate data extracted by the binary search tree extraction unit. apparatus. The candidate data extraction unit, in the case of a binary search tree including the metacharacter representation,
When the distance D between the target data and the candidate data, the intermediate value M of the distance of the binary search tree, and the distance d between the target data and the root node of the binary search tree,
d ≦ MD
2. The data processing apparatus according to claim 1, wherein the binary search tree is searched even when the above condition is satisfied. The candidate data extraction unit is defined as a distance D between the target data and the candidate data, an intermediate value M of the distance of the binary search tree, and a distance d between the target data and a root node of the binary search tree.
D + M <d
The data processing apparatus according to claim 1, wherein when the above condition is satisfied, the binary search tree is not searched. The data processing apparatus includes a database and a processing unit.
The database stores data including a metacharacter expression that means that one numerical value is selected from a plurality of numerical values,
The processor is
Search whether the input target data is stored in the database, and when the target data is not stored in the database,
Among the character strings including the metacharacter expression, a numerical value other than the metacharacter expression is set as the metacharacter expression, and each metacharacter expression is treated as one unit character, and is included in the target data and the database data. A process of dividing a numerical value and a character other than the numerical value as one unit character, and calculating the number of the divided unit characters as the number of unit characters;
A process of creating a binary search tree for each unit character number from the data in the database;
The binary search tree the unit number of characters corresponds to a predetermined range, the process of extracting the candidate data,
A process of calculating a distance between the target data and the data in the database;
The data processing method comprising: executing, among the extracted candidate data, a process of extracting data having the calculated distance equal to or less than a predetermined value as the candidate data.

Images