JP2013120534A - Related word classification device, computer program, and method for classifying related word - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract related words while discriminating words used properly according to different purposes, even when the words subject to be classified are structured data such as database definitions.SOLUTION: A word classification unit 60 optimizes a model parameter for discriminating a class of a word on the basis of a provisional correct answer class allocated to the word appearing in data. The word classification unit 60 selects a maximum likelihood class in classes satisfying a restriction relating to the class allocation on the basis of a likelihood of the class obtained by using a current model parameter, and allocates it to the word as a new provisional correct answer class. The word classification unit 60, after alternately and iteratively processing the two steps, extracts words classified in the same class as the related words.

Description

  The present invention relates to a related word classification device that classifies a plurality of words into groups of related words.

There is a technique of extracting related words by clustering words based on the similarity between feature vectors of words. For example, the appearance frequency of a context word co-occurring with the word is used as an element of the word feature vector. For example, the inner product of feature vectors is used as the similarity between feature vectors of words. As a clustering method, for example, a k-means method, a hierarchical clustering method, or a method combining a plurality of clustering methods is used.
In addition, using a synonym dictionary, a number of synonym pairs in which two words are synonyms and non-synonym pairs that are not synonyms are generated, and the generated pairs are supervised learning such as a support vector machine (SVM). There is a technique for forming a classifier that identifies whether two words are synonyms by having a model learn.

JP 2011-118526 A

Bellegarda, J.A. R. Et al., “A novel word clustering algorithm based on latent semantic analysis,” Proceedings of IEEE International Conference on Acoustics, Speech, and SignA. 1, 172-175 pages, 1996. Akiko Aizawa “Consideration on Calculation of Word Similarity Using Large-scale Text Corpus” IPSJ Transactions, Vol. 49, No. 3, pp. 1426-1436, March 2008. “WordNet-A lexical database for English” http: // wordnet. Princeton. edu / wordnet / Rubin, D.M. B. “Iteratively reweighted least squares” Encyclopedia of Statistical Sciences, Vol. 4, pages 272-275, Wiley, 1983. Bishop, C.I. M.M. I. T.A. Nabney “Pattern Recognition and Machine Learning: A Matlab Companion” Springer, 1998.

In the database, names arbitrarily named by the database designer are used as table names and column names. For this reason, when integrating different databases, a great effort is required to determine the corresponding columns.
For table names and column names, words derived from natural languages are usually used. Therefore, if a column having a similar meaning name can be extracted by applying the related word extraction technique as described above, it is possible to help discriminate the corresponding column.
However, there are the following problems in applying the above-described technique to a table schema that defines a database table and other structured data.
First, in structured data, contextual connections between words are sparse, and related words may not be extracted properly with feature vectors formed based on the context. For this reason, a word that should be treated distinctly and a word that should not be handled cannot be distinguished, and may be extracted as a related word.
Secondly, in the method using the synonym dictionary, it is necessary that the word included in the target structured data is sufficiently registered in the synonym dictionary in advance, and such a synonym dictionary does not exist. , Can not apply.

  An object of the present invention is to extract related words while identifying words that are used properly, even when structured data such as database definitions is targeted.

The related word classification device according to this invention is:
In a related word classification device for classifying a plurality of words into groups of related words,
A processing device for processing data, a text acquisition unit, a feature vector calculation unit, a word classification unit, and a weight vector calculation unit;
The text acquisition unit acquires a plurality of texts including one or more word groups composed of one or more words using the processing device,
The feature vector calculation unit calculates a feature vector representing the feature of the word for each of a plurality of words included in the plurality of texts acquired by the text acquisition unit using the processing device;
The word classification unit classifies each of the plurality of words included in the plurality of texts acquired by the text acquisition unit into any of a plurality of groups using the processing device,
The weight vector calculation unit uses the processing device to determine, for each of the plurality of groups, a weight indicating a likelihood that a word whose feature is represented by the feature vector as an inner product with the feature vector belongs to the group. A vector that calculates a weight vector that best fits the classification by the word classification unit,
The word classification unit further uses the processing device to calculate the feature vector calculated by the feature vector calculation unit for each of the plurality of words and the weight calculated by the weight vector calculation unit for each of the plurality of groups. The likelihood that each of the plurality of words belongs to each of the plurality of groups is calculated based on the vector, and the plurality of words are reclassified based on the calculated likelihood.

  According to the present invention, even when structured data such as a database definition is targeted, related words can be extracted while identifying words that are used properly.

1 is a system configuration diagram illustrating an example of an overall configuration of a related column extraction system 900 according to Embodiment 1. FIG. FIG. 3 is a hardware configuration diagram illustrating an example of hardware resources such as a related column determination device 903 according to the first embodiment. FIG. 3 is a block configuration diagram illustrating an example of a configuration of a related column determination device 903 according to the first embodiment. FIG. 6 is a diagram illustrating an example of related word extraction source data 10 according to the first embodiment. FIG. 4 is a diagram showing an example of conceptual dictionary data 30 in the first embodiment. The figure which shows an example of the hierarchical structure between the concepts which the concept dictionary data 30 in Embodiment 1 represent. The figure which shows an example of the hierarchical structure between the concepts which the concept dictionary data 30 in Embodiment 1 represent. FIG. 10 is a diagram for explaining an example of the operation of the word extraction unit 20 in the first embodiment. FIG. 10 is a diagram for explaining an example of the operation of the context analysis unit 50 according to the first embodiment. FIG. 6 is a diagram for explaining an example of an operation of a feature vector generation unit 40 in the first embodiment. FIG. 6 is a diagram for explaining an example of the operation of the word classification unit 60 in the first embodiment. FIG. 6 is a diagram for explaining an example of the operation of the word classification unit 60 in the first embodiment. FIG. 6 is a diagram for explaining an example of the operation of the word classification unit 60 in the first embodiment. FIG. 6 is a diagram for explaining an example of the operation of the word classification unit 60 in the first embodiment. FIG. 6 is a diagram for explaining an example of an operation of a related word determination unit 70 in the first embodiment. FIG. 6 is a flowchart showing an example of a flow of related extraction processing S800 in the first embodiment. FIG. 7 is a flowchart showing an example of the flow of word classification processing S804 in the first embodiment. FIG. 5 is a flowchart showing an example of a flow of an initialization step S810 in the first embodiment. FIG. 5 is a flowchart showing an example of a flow of a classification calculation step S830 in the first embodiment. The figure for demonstrating an example of operation | movement of the classification | category calculation process S830 word classification | category part 60 in Embodiment 1. FIG. The figure for demonstrating an example of operation | movement of the context analysis part 50 in Embodiment 2. FIG.

Embodiment 1 FIG.
The first embodiment will be described with reference to FIGS.

  FIG. 1 is a system configuration diagram showing an example of the overall configuration of a related column extraction system 900 in this embodiment.

The related column extraction system 900 extracts a plurality of columns that are likely to be related columns from among the columns constituting the table in one or a plurality of databases. The related column is a plurality of columns belonging to different tables and storing corresponding contents. The related column extraction system 900 extracts columns that are highly likely to be related columns based on the names (column names) given to the columns.
The related column extraction system 900 includes, for example, a database definition storage device 901, a concept dictionary storage device 902, and a related column determination device 903.

The database definition storage device 901 stores database definition data that defines a database.
The concept dictionary storage device 902 stores concept dictionary data representing the concept dictionary. The concept dictionary is a dictionary that describes the meaning of words in a natural language and the relationship of meanings between a plurality of words (synonyms, synonyms, antonyms, superordinate concepts, subordinate concepts, etc.). The concept dictionary data stored in the concept dictionary storage device 902 represents, for example, a general-purpose large vocabulary concept dictionary.
A related column determination device 903 (an example of a related word classification device) acquires a column name from the database definition data stored in the database definition storage device 901, and uses the concept dictionary data stored in the concept dictionary storage device 902. Extract columns that are likely to be related columns. The related column determination device 903 displays, for example, the extracted column and presents it to the user of the related column extraction system 900.
The user of the related column extraction system 900 confirms the contents of the column presented by the related column determination device 903 and determines whether or not the column is a related column.

  FIG. 2 is a hardware configuration diagram illustrating an example of hardware resources such as the related column determination device 903 in this embodiment.

The database definition storage device 901, the concept dictionary storage device 902, and the related column determination device 903 are, for example, computers having a processing device 911, an input device 912, an output device 913, and a storage device 914.
The processing device 911 processes the data and controls the entire computer by executing the computer program stored in the storage device 914. Note that the computer may have a plurality of processing devices 911.
The storage device 914 stores a computer program executed by the processing device 911, data processed by the processing device 911, and the like. For example, the storage device 914 is a volatile storage device that stores temporary data during processing. Alternatively, the storage device 914 is a non-volatile storage device that stores a computer program, permanent data, and the like. For example, the memory device 914 is an integrated circuit or a large-scale integrated circuit in which semiconductor memory elements are integrated. Alternatively, the storage device 914 is an external storage device that uses a storage medium such as a magnetic disk or an optical disk. Note that the computer may have a plurality of storage devices 914. The plurality of storage devices 914 may be the same type of storage device or different types of storage devices.
The input device 912 inputs information from outside the computer and converts it into data that can be processed by the processing device 911. For example, the input device 912 is an operation input device such as a keyboard and a mouse for inputting an operation by the operator. Alternatively, the input device 912 is an image input device such as a camera or a scanner device that inputs a still image or a moving image. Alternatively, the input device 912 is a voice input device such as a microphone for inputting voice. Alternatively, the input device 912 is a sensor device such as a temperature sensor, an illuminance sensor, or a pressure sensor that measures physical quantities such as temperature, brightness, and pressure. Alternatively, the input device 912 is an analog-digital conversion device that converts an analog signal into digital data. Alternatively, the input device 912 is a receiving device that receives and demodulates a signal transmitted from an external device to the computer. Note that the computer may have a plurality of input devices 912. The plurality of input devices 912 may be the same type of input device or different types of input devices. The data converted by the input device 912 may be processed directly by the processing device 911, or may be stored temporarily by the storage device 914.
The output device 913 converts the data processed by the processing device 911, the data stored in the storage device 914, and the like, and outputs them to the outside of the computer. For example, the output device 913 is a display device that displays text, images, and the like. Alternatively, the output device 913 is a printing device that prints text, images, and the like. Alternatively, the output device 913 is a sound output device such as a speaker that outputs sound. Alternatively, the output device 913 is a digital-analog conversion device that converts digital data into an analog signal. Alternatively, the output device 913 is a transmission device that transmits a modulated signal to an external device. Note that the computer may be configured to include a plurality of output devices 913. The plurality of output devices 913 may be the same type of output devices or different types of output devices.

The functional blocks described below can be realized by the processing device 911 executing the computer program stored in the storage device 914. In addition, the structure which implement | achieves one apparatus using one computer may be sufficient, and the structure which implement | achieves one apparatus using a some computer may be sufficient. On the contrary, the structure which implement | achieves a some apparatus using one computer may be sufficient.
In addition, some or all of these functional blocks are not realized using a computer, but other configurations such as an electrical configuration such as a digital circuit or an analog circuit, or a mechanical configuration, for example. You may implement | achieve using.

  FIG. 3 is a block configuration diagram showing an example of the configuration of the related column determination device 903 in this embodiment.

  The related column determination device 903 includes, for example, a word extraction unit 20, a feature vector generation unit 40, a context analysis unit 50, a word classification unit 60, and a related word determination unit 70.

The word extraction unit 20 (an example of a text acquisition unit) uses the processing device 911 to extract words to be classified from the related word extraction source data 10 (an example of text).
For example, the word extraction unit 20 acquires a table schema as the related word extraction source data 10 from the database definition data stored in the database definition storage device 901. The table schema is data that defines a database table. The word extraction unit 20 uses the processing device 911 to extract words constituting the column name of the table from the acquired related word extraction source data 10.
The table schema includes one or more column names (an example of word groups). The column name is a character string arbitrarily named by the database designer, but a natural language word related to the contents or a combination of a plurality of words is often used. For example, a column name is formed by concatenating a plurality of words using a delimiter such as “_” (underscore).
For example, the word extraction unit 20 acquires a column name from the related word extraction source data 10. The word extraction unit 20 determines whether or not a delimiter is included in the acquired column name. When the delimiter is not included, the word extraction unit 20 determines that the acquired column name is composed of one word, and extracts the acquired column name as a word as it is. When a delimiter is included, the word extraction unit 20 decomposes the acquired column name into a plurality of character strings with the delimiter as a boundary, and extracts the plurality of decomposed character strings as words.

The feature vector generation unit 40 (an example of a feature vector calculation unit) converts each word extracted by the word extraction unit 20 into a concept vector.
For example, the feature vector generation unit 40 generates a concept vector for each word extracted by the word extraction unit 20 based on the concept dictionary data 30 stored in the concept dictionary storage device 902. A concept vector is a vector composed of a plurality of numerical values representing the meaning of a word and the relationship of meaning with another word. For example, each element of the concept vector is associated with a different concept and takes a value of “1” or “0”. When the value of the element of the concept vector is “1”, it indicates that the concept represented by the word is a concept to which the element is associated or its subordinate concept, and when the value is “0”, the word is Indicates that the concept to be represented is neither a concept to which the element is associated nor a subordinate concept.

The context analysis unit 50 (an example of a feature vector calculation unit) uses the processing device 911 to analyze the positional relationship on the table structure and column structure of each word extracted by the word extraction unit 20.
For example, the context analysis unit 50 generates a context vector for each word extracted by the word extraction unit 20. The context vector is a vector composed of a plurality of numerical values representing the positional relationship in context between the word and other words. For example, each element of the context vector belongs to one of three groups: a prefix word group, a suffix word group, and a table name group. Each element belonging to the prefix group is associated with a different word and takes a value of “1” or “0”. When the value of the element belonging to the prefix group is “1”, it indicates that the word is connected immediately after the word associated with the element in the column name including the word. When the value of the element belonging to the prefix group is “0”, whether the word is the first word in the column name including the word (including the case where the column name is composed of one word). Alternatively, it indicates that the word is connected immediately after a word other than the word with which the element is associated. Each element belonging to the postfix group is associated with a different word and takes a value of “1” or “0”. When the value of the element belonging to the postfix group is “1”, it indicates that in the column name including the word, the word associated with the element is connected immediately after the word. If the value of the element belonging to the postfix group is “0”, whether the word is the last word in the column name including the word (including the case where the column name is composed of one word). Alternatively, it indicates that a word other than the word associated with the element is connected immediately after the word. Each element belonging to the table name group is associated with a different word and takes a value of “1” or “0”. If the value of an element belonging to the table name group is “1”, the name of the table (table name) to which the column having the column name including the word belongs includes the word associated with the element. Represent. When the value of the element belonging to the table name group is “0”, this indicates that the word associated with the element is not included in the table name of the table to which the column having the column name including the word belongs.

The word classification unit 60 uses the processing device 911 to perform clustering of words based on the word concept vector and the word appearance context to determine the class likelihood of each word. The word classification unit 60 extracts words from each class based on the calculated class likelihood.
For example, the word classification unit 60 clusters a plurality of words extracted by the word extraction unit 20 based on the concept vector generated by the feature vector generation unit 40 and the context vector generated by the context analysis unit 50. For example, the word classification unit 60 calculates the likelihood belonging to each of a predetermined number of classes for each word extracted by the word extraction unit 20. The word classification unit 60 determines, for each word extracted by the word extraction unit 20, which class the word belongs to based on the calculated likelihood.

The related word determination unit 70 uses the processing device 911 to extract a column that is likely to be a related column based on the word group extracted by the word classification unit 60 for each class.
For example, the related word determination unit 70 has a class set in which words constituting the column name of a column belonging to a certain table are classified and a class set in which words constituting a column name of a column belonging to another table are classified. If they match, it is determined that the two columns are likely to be related columns.
Using the processing device 911, the related word determination unit 70 outputs the determined result as related word data 80.

  FIG. 4 is a diagram showing an example of the related word extraction source data 10 in this embodiment.

In this example, the related term extraction source data 10 is a table schema included in the database definition data stored in the database definition storage device 901. Each of the three related word extraction source data 10a to 10c shown in this example defines one table.
The related term extraction source data 10a defines a table having a table name “PURCHASE_ORDER” (hereinafter referred to as “PURCHASE_ORDER table”, and so on). The PURCHASE_ORDER table has an ORDER_DATE column, a LINE_NO column, and a PHONE_NO column. The ORDER_DATE column is a date type. The LINE_NO column is an 8-digit numeric type. The PHONE_NO column is a 10-character fixed-length character string type. For example, the ORDER_DATE column stores the order date. The LINE_NO column stores the product number. The PHONE_NO column stores a telephone number.
The related word extraction source data 10b defines an ORDER table. The ORDER table has an ODR_DAT column, an ITEM_NO column, a TEL column, and a FAX column. The ODR_DAT column is a date type. The ITEM_NO column is a 6-digit numeric type. The TEL column is a fixed-length character string type of 8 characters. The FAX column is a fixed-length character string type of 8 characters. For example, the ODR_DAT column stores the order date. The ITEM_NO column stores the product number. The TEL column stores the telephone number. The FAX column stores the facsimile number.
The related word extraction source data 10c defines a SHIPMENT table. The SHIPMENT table has a PRODUCT column and a PROD_TYPE column. The PRODUCT column is an 8-digit numeric type. The PROD_TYPE column is a variable-length character string type with a maximum of 128 characters.

  In this example, the ORDER_DATE column of the PURCHASE_ORDER table and the ODR_DAT column of the ORDER table are both columns that store the order date and are related columns. The LINE_NO column of the PURCHASE_ORDER table, the ITEM_NO column of the ORDER table, and the PRODUCT column of the SHIPMENT table are all columns that store product numbers and are related columns. Also, the PHONE_NO column of the PURCHASE_ORDER table and the TEL column of the ORDER table are both columns that store telephone numbers and are related columns.

As described above, table schemas defined for different systems and table schemas defined by different system designers may include columns for storing corresponding contents although the names are different.
Analyzing the correspondence between these columns is essential when integrating and optimizing existing business systems or building new applications that link existing databases. It is. The related column extraction system 900 automatically extracts a column that is likely to be a related column by automatically estimating the correspondence of words included in the column name, and analyzes the correspondence between a large number of columns in the existing system. Increase efficiency.

  FIG. 5 is a diagram showing an example of the concept dictionary data 30 in this embodiment.

The concept dictionary data 30 includes a plurality of concept data 31, for example. Each concept data 31 represents one concept.
The concept data 31 includes, for example, a concept identifier 32, a concept display word 33, and a higher concept identifier 34.

The concept identifier 32 is an identifier for uniquely identifying the concept data 31 such as an identification number.
The concept display word 33 represents a word or a collocation representing the concept. When there are a plurality of words or collocations representing the same concept, the concept display word 33 is a list in which a plurality of words or collocations are connected by a delimiter such as “,” (comma). Further, when one word or multiple words have a plurality of meanings, the same word may appear in the concept display word 33 of different concept data 31.
The superordinate concept identifier 34 is a concept identifier 32 of the concept data 31 representing the superordinate concept of the concept. It can be said that a superordinate concept of a superordinate concept of a concept is a superordinate concept of the concept. However, since the superordinate concept of the superordinate concept can be discriminated by tracing the superordinate concept identifier 34, the superordinate concept identifier 34 Represents only the concept. In the case of a concept that does not have a superordinate concept, the superordinate concept identifier 34 may be blank. In addition, when there are a plurality of superordinate concepts, the superordinate concept identifier 34 may be a list in which a plurality of concept identifiers 32 are connected by a delimiter such as “,” (comma).

For example, a concept whose concept identifier 32 is “000014040” (hereinafter referred to as “concept {0000001740}”, and so on) is a concept corresponding to one of the meanings represented by the word “entity”. The concept {00001440} is the highest level concept and has no superordinate concept.
Further, one of the meanings represented by the word “object” and one of the meanings represented by the collocation “physical object” have the same meaning and correspond to the concept {0000002684}. The superordinate concept immediately above the concept {000000264} is the concept {0000001930}.
Also, the word “line” has many meanings. For example, one of them is the same as one of the meanings of the word “cable” and the collocation “transmission line”, and corresponds to the concept {029375552}. Another one is the same as one of the meanings of the collocation “railway line” and collocation “rail line”, and corresponds to the concept {0367598}.

  Since English words are often used for column names due to restrictions on the types of characters that can be used for database column names, the concept dictionary represented by the concept dictionary data 30 in this example is an English dictionary. However, the concept dictionary represented by the concept dictionary data 30 is not limited to an English dictionary, and may be a dictionary of another language such as Japanese or a dictionary including a plurality of languages.

  6 and 7 are diagrams showing an example of a hierarchical structure between concepts represented by the concept dictionary data 30 in this embodiment.

  The two figures show a hierarchical structure having the concept {00001440} (entity) as a vertex. In addition, although the figure is divided | segmented, the whole makes one hierarchical structure. “Matter” in the upper part of FIG. 6 is the same as “Matter” in the lower part of FIG. Further, “abstraction / abstract entity” in FIG. 6 is the same as “abstraction / abstract entity” in FIG. 7 and is connected. Further, “document” in FIG. 6 is the same as “document” in FIG. 7 and is connected. Further, “relation” in FIG. 6 is the same as “relation” in FIG. 7 and is connected. 7 is the same as “psychological feature” in the lower part of FIG. 7 and is connected to the “pysyological feature” in the upper part of FIG.

  An ellipse represents a concept. A line extending upward from the ellipse indicates a superordinate concept. A line extending downward from the ellipse indicates a subordinate concept. When a word or collocation is described in the ellipse, it represents a word or collocation representing the concept. When a plurality of words or collocations are described in one ellipse, they are synonyms representing the same concept.

The hierarchical structure between concepts is generally a tree structure, but there are cases where a single concept has a plurality of higher-level concepts (for example, “substance” at the bottom of FIG. 6). It is not a tree structure.
Further, according to this figure, the word “line” also has a meaning that is relatively close to the word “product” (such as a subordinate concept of “merchandise / ware / product” in FIG. 6), and the word “phone”. It is understood that there is a meaning that is relatively close to (such as a subordinate concept of “instrumentality / instrumentation” in FIG. 6).

  FIG. 8 is a diagram for explaining an example of the operation of the word extraction unit 20 in this embodiment.

The table name constituting word data 21 is data representing words constituting the table name. The table name constituent word data 21 includes, for example, a table identifier 22 and one or more table name constituent words 23. The table identifier 22 is an identifier for uniquely identifying a table. The word extraction unit 20 generates one table name constituent word data 21 for one table. The table name constituent word 23 represents a word constituting the table name of the table identified by the table identifier 22.
The column name constituting word data 24 is data representing words constituting the column name. The column name constituent word data 24 includes, for example, a table identifier 25, a column identifier 26, and one or more column name constituent words 27. The table identifier 25 is the table identifier 22 of the table to which the column belongs. The column identifier 26 is an identifier for uniquely identifying a column. The word extraction unit 20 generates one column name constituent word data 24 for one column. The column identifier 26 may be an identifier that can uniquely identify a column by the column identifier 26 alone, or may be an identifier that can uniquely identify a column in combination with the table identifier 25. The column name constituent word 27 represents a word constituting the column name of the column identified by the column identifier 26.

The word extraction unit 20 performs word division on the table name and column name of the table schema, and extracts words.
For example, for the PURCHASE_ORDER table, the word extraction unit 20 assigns the identifier “T0001” to the table identifier 22. The word extraction unit 20 divides the table name “PURCHASE_ORDER” into the word “PURCHASE” and the word “ORDER” by the delimiter “_” (underscore) to obtain a table name constituent word 23.
In addition, for the ORDER_DATE column of the PURCHASE_ORDER table, the word extraction unit 20 assigns an identifier “C0011” as a column identifier 26. The word extraction unit 20 divides the column name “ORDER_DATE” into the word “ORDER” and the word “DATE” by the delimiter “_” (underscore), and forms the column name constituting word 27.

  FIG. 9 is a diagram for explaining an example of the operation of the context analysis unit 50 in this embodiment.

  The prefix word 51 is a word that appears immediately before the column name constituent word 27 in the column name including the column name constituent word 27. The postfix word 52 is a word that appears immediately after the column name constituent word 27 in the column name including the column name constituent word 27. The table name constituent word 53 is a table name constituent word 23 constituting the table name of the table to which the column including the column name constituent word 27 belongs.

For example, the context analysis unit 50 extracts the prefix word 51 from the column name constituent words 27 extracted by the word extraction unit 20. The context analysis unit 50 extracts only the words in which the trailing word is present from the column name constituent words 27 and sets them as prefix words 51.
Further, the context analysis unit 50 extracts a postfix word 52 from the column name constituent words 27 extracted by the word extraction unit 20. The context analysis unit 50 extracts only the words in which the front word is present from the column name constituent words 27 and uses them as postfix words 52.
The context analysis unit 50 sets the table name constituent word 23 extracted by the word extraction unit 20 as the table name constituent word 53.

The context vector 54 is a context vector generated by the context analysis unit 50. The context analysis unit 50 generates one context vector 54 for one word. The context analysis unit 50 distinguishes words with the same spelling that appear at different positions (columns or tables). For example, the column name constituent word 27 “NO” includes a column {C0012} (LINE_NO column) of the table {T0001} (PURCHASE_ORDER table), a column {C0013} (PHONE_NO column) of the table {T0001} (PURCHASE_ORDER table), and It appears in three places with column {C0222} (ITEM_NO column) of table {T0002} (ORDER table). The context analysis unit 50 distinguishes the column name constituent word 27 “NO” appearing at these three locations, and generates a context vector 54 for each.
Note that, for the prefix word 51, the suffix word 52, and the table name constituent word 53, the context analysis unit 50 treats words having the same spelling as one word without distinguishing them.

Each element of the context vector 54 is associated with any one of the prefix word 51, the suffix word 52, and the table name constituent word 53.
For example, the context analysis unit 50 determines the order of the context vector 54 based on the numbers of the extracted prefix word 51, postfix word 52, and table name constituent word 53. The context analysis unit 50 adds up the number of extracted prefix words 51, the number of postfix words 52, and the number of table name constituent words 53 to obtain the order of the context vector 54.
Based on the column name constituent word data 24 generated by the word extraction unit 20, for each column name constituent word 27, the context analysis unit 50 determines whether there is a word located immediately before the column name constituent word 27. Determine. When it is determined that there is a word located immediately before the column name constituent word 27, the context analysis unit 50 selects a word located immediately before the column name constituent word 27 from the prefix words 51. The context analysis unit 50 sets the value of the element associated with the selected prefix word 51 among the elements of the context vector 54 to “1”, and sets the value of the element associated with the other prefix word 51 as the value of the element. Set to “0”.
Based on the column name constituent word data 24 generated by the word extraction unit 20, the context analysis unit 50 determines whether there is a word located immediately after the column name constituent word 27. When it is determined that there is a word located immediately after the column name constituent word 27, the context analysis unit 50 selects a word located immediately after the column name constituent word 27 from the postfix words 52. The context analysis unit 50 sets the value of the element associated with the selected postfix word 52 among the elements of the context vector 54 to “1”, and sets the value of the element associated with the postfix word 52 other than that. Set to “0”.
Based on the table name constituent word data 21 generated by the word extraction unit 20, the context analysis unit 50 acquires a table name constituent word 23 that constitutes a table identified by the table identifier 25. The context analysis unit 50 selects the acquired table name constituent word 23 from the table name constituent words 53. The context analysis unit 50 sets the value of the element associated with the selected table name constituent word 53 among the elements of the context vector 54 to “1”, and sets the values of the elements associated with the other table name constituent words 53. Set the value to “0”.

  For example, for the word “ORDER” appearing in the column {C0011} (ORDER_DATE column) of the table {T0001} (PURCHASE_ORDER table), the context analysis unit 50 positions the word “ORDER” at the beginning of the column name of the column {C0011}. Since there is no immediately preceding word, all the values of the elements associated with the prefix word 51 among the elements of the context vector 54 are set to “0”. In addition, since the word “DATE” is located immediately after the word “ORDER” in the column name of the column {C0011}, the context analysis unit 50 includes the elements of the context vector 54 associated with the postfix word 52. The value of the element associated with “DATE” is set to “1”, and the values of the other elements are set to “0”. In addition, since the table name constituent word 23 of the table {T0001} is “PURCHASE” and “ORDER”, the context analysis unit 50 selects “PURCHASE” and “ORDER” from among the elements of the context vector 54 associated with the table name constituent word 53. The values of the elements associated with “PURCHASE” and “ORDER” are set to “1”, and the values of the other elements are set to “0”.

  FIG. 10 is a diagram for explaining an example of the operation of the feature vector generation unit 40 in this embodiment.

The concept identifier 41 is a concept identifier 32 in the concept dictionary data 30 for the concept represented by the column name constituent word 27.
For example, the feature vector generation unit 40 extracts, from the concept dictionary data 30, the concept data 31 that includes any one of the column name constituent words 27 extracted by the word extraction unit 20 in the concept display word 33. The concept identifier 32 included in the extracted concept data 31 is a concept identifier 41. Further, the feature vector generation unit 40 extracts the concept data 31 representing the superordinate concept of the concept from the concept dictionary data 30 based on the superordinate concept identifier 34 included in the extracted concept data 31. The feature vector generation unit 40 adds the concept identifier 32 included in the extracted concept data 31 to the concept identifier 41. The feature vector generation unit 40 repeats this until the highest concept is reached.

  The concept vector 44 is a concept vector generated by the feature vector generation unit 40. The feature vector generation unit 40 generates one concept vector 44 for one word. However, unlike the context analysis unit 50, the feature vector generation unit 40 does not distinguish between words having the same spelling. The value of the element of the concept vector 44 does not change depending on the appearance position. For example, the column name constituent word 27 “NO” appears in three places, but the feature vector generation unit 40 generates one concept vector 44 for the column name constituent word 27 “NO”.

Each element of the concept vector 44 is associated with a concept identifier 41.
For example, the feature vector generation unit 40 determines the order of the concept vector 44 based on the number of extracted concept identifiers 41. The feature vector generation unit 40 sets the number of extracted concept identifiers 41 as the order of the concept vector 44.
The feature vector generation unit 40 extracts, for each column name constituent word 27, the concept data 31 in which the column name constituent word 27 is included in the concept display word 33 from the concept dictionary data 30. The feature vector generation unit 40 selects a concept identifier 41 that matches the concept identifier 32 of the extracted concept data 31 from the concept identifiers 41. The feature vector generation unit 40 extracts the concept data 31 representing the superordinate concept of the concept from the concept dictionary data 30 based on the superordinate concept identifier 34 included in the extracted concept data 31. The feature vector generation unit 40 further selects a concept identifier 41 that matches the concept identifier 32 of the extracted concept data 31 from the concept identifiers 41. The feature vector generation unit 40 repeats this until the highest concept is reached. In this manner, the feature vector generation unit 40 selects all the concept identifiers 41 of the concepts included in the concept category path in the concept dictionary of the column name constituent word 27.
The feature vector generation unit 40 sets the value of the element associated with the selected concept identifier 41 among the elements of the concept identifier 41 to “1”, and sets the value of the element associated with the other concept identifier 41 to “ 0 ”.

  FIG. 11 is a diagram for explaining an example of the operation of the word classification unit 60 in this embodiment.

The upper part of this figure is a context management graph. The context management graph represents the compositional relationship between tables, columns, and words.
The highest level of the network is an individual table identified by the table identifier 25. The second hierarchy is an individual column identified by the column identifier 26. The third hierarchy is a column name constituent word 27. However, in the third hierarchy, nodes are generated as different vocabularies.
A line connecting a table and a column indicates that the column belongs to the table. Each table is connected to one or more columns. Each column is connected to any one table.
A line connecting the column and the column name constituent word 27 indicates that the column name of the column includes the column name constituent word 27. Each column is connected to one or more column name constituent words 27. Since different columns may contain the same spelling word, each column name component word 27 may be connected to multiple columns.

The lower part of the figure represents the class. The class is a group for classifying the column name constituent words 27. The number of classes is predetermined. For example, the related column determination device 903 inputs the number of classes set by the user of the related column extraction system 900 using the input device 912, and stores the input number of classes using the storage device 914.
The class identifier 61 is an identifier for uniquely identifying a class. For example, the word classification unit 60 uses the processing device 911 to generate a class identifier 61 that identifies each class based on the stored number of classes.

The word classification unit 60 classifies the column name constituent words 27 into any class. That is, the word classification unit 60 draws a pair of other branches that connect the column name constituent word 27 (different word) and the class shown in FIG. Each column name constituent word 27 is connected to one class. Each class is connected to one or more column name constituent words 27.
The word classification unit 60 searches for an optimal class assignment that satisfies the condition based on the given condition.

  FIG. 12 is a diagram for explaining an example of the operation of the word classification unit 60 in this embodiment.

The class identifier 62 is a class identifier 61 of a class in which the column name constituent word 27 is classified.
The word classification unit 60 uses the processing device 911 to classify the column name constituent words 27 extracted by the word extraction unit 20 into any class. The word classification unit 60 sets the class identifier 61 of the class into which the column name constituent words 27 are classified as the class identifier 62.
At this time, the word classification unit 60 does not distinguish between the same spelled words appearing at different positions. That is, the word classifying unit 60 always classifies the same spelled words appearing at different positions into the same class.

The word classification unit 60 generates a class vector 63 for each column of each table. The word classification unit 60 generates the same number of class vectors 63 as the total number of columns. The class vector 63 represents a class into which the column name constituent words 27 included in the column name of the column are classified.
Each element of the class vector 63 is associated with a class identified by the class identifier 61. Each element of the class vector 63 represents whether or not there is a word classified into the class among the column name constituent words 27 included in the column name of the column.
For example, the word classification unit 60 generates a class vector 63 whose degree is a predetermined number of classes for each column identified by the column identifier 26 of the table identified by the table identifier 25. Based on the column name constituent word data 24 generated by the word extraction unit 20 and the class identifier 62 of the classified class, the word classification unit 60 classifies the words belonging to the column from the class identifier 61. A class identifier 61 that matches the class identifier 62 is selected. The word classification unit 60 sets the value of the element associated with the selected class identifier 61 among the elements of the class vector 63 to “1” (in the column name constituting word 27 included in the column name of the column, An example of a value indicating that there is a word classified into the class.), And the value of the other elements are classified as “0” (the column name constituting word 27 included in the column name of the column is classified into the class). An example of a value indicating that there is no added word.).

For example, the word classification unit 60 classifies the word “ORDER” into the class {y3}, classifies the word “DATE” into the class {y3}, classifies the word “LINE” into the class {y4}, and adds the word “NO”. Is classified into class {y1}, and the word “TEL” is classified into class {y1}.
For column {C0011} (ORDER_DATE column) of table {T0001} (PURCHASE_ORDER table), word classification unit 60 classifies word “ORDER” into class {y3} and classifies word “DATE” into class {y3}. Therefore, among the elements of the class vector 63, the value of the element associated with the class {y3} is set to “1”, and the value of the element associated with the other class is set to “0”.

Further, for the column {C0012} (LINE_NO column) of the table {T0001}, the word classification unit 60 classifies the word “LINE” into the class {y4} and classifies the word “NO” into the class {y1}. Among the elements of the class vector 63, the value of the element associated with the class {y1} and the class {y4} is set to “1”, and the value of the element associated with the other class is set to “0”. .
In addition, for the column {C0023} (TEL column) of the table {T0002} (ORDER table), the word classification unit 60 classifies the word “TEL” into the class {y1}, so among the elements of the class vector 63, The value of the element associated with the class {y1} is set to “1”, and the value of the element associated with the other class is set to “0”.

The word classification unit 60 classifies the column name constituent words 27 into classes so that the class vectors 63 generated for the columns belonging to the same table are different from each other.
For example, the word classification unit 60 generates the class vector 63 after classifying all the column name constituent words 27. If there is a match among the class vectors 63 generated for the columns belonging to the same table, the word classification unit 60 performs classification again.
Or the word classification | category part 60 classify | categorizes the column name structure word 27 in order, and produces | generates the class vector 63 in an order from the column which classified all the words contained in a column name. When the newly generated class vector 63 matches the class vector 63 that has already been generated for other columns belonging to the same table, the word classifying unit 60 selects the class of the column name constituting word 27 that has been classified last. change.

  FIG. 13 is a diagram for explaining an example of the operation of the word classification unit 60 in this embodiment.

The word classification unit 60 (an example of a feature vector generation unit) uses the processing device 911 to combine the context vector 54 generated by the context analysis unit 50 and the concept vector 44 generated by the feature vector generation unit 40. , A feature vector 64 is generated. If the degree of the prefix word 51 is m and the degree of the concept identifier 41 is n, the degree of the feature vector 64 is m + n.
At this time, like the context analysis unit 50, the word classification unit 60 distinguishes words with the same spelling that appear at different positions. The word classification unit 60 generates the same number of feature vectors 64 as the number of context vectors 54 generated by the context analysis unit 50.
Each element of the feature vector 64 is associated with one of the prefix word 51, the suffix word 52, the table name constituent word 53, and the concept identifier 41.

  For example, the word classification unit 60 selects, for each of the context vectors 54 generated by the context analysis unit 50, the concept vector 44 that matches the column name constituent word 27 from the concept vectors 44 generated by the feature vector generation unit 40. To do. The word classification unit 60 generates a feature vector 64 having a total of m + n elements including m elements of the context vector 54 and n elements of the selected concept vector 44.

  FIG. 14 is a diagram for explaining an example of the operation of the word classification unit 60 in this embodiment.

The word classification unit 60 (an example of a weight vector generation unit) uses the processing device 911 to generate a weight parameter vector 65 (an example of a weight vector) for each class identified by the class identifier 61. The word classification unit 60 generates the same number of weight parameter vectors 65 as the number of classes determined in advance.
The order of the weight parameter vector 65 is the same as the order of the feature vector 64. Each element of the weight parameter vector 65 is associated with any one of the prefix word 51, the suffix word 52, the table name constituent word 53, and the concept identifier 41.

  The weight parameter vector 65 is used to calculate the likelihood that the column name constituent word 27 belongs to the class. The word classification unit 60 calculates the inner product of the feature vector 64 generated for a certain column name constituent word 27 and the weight parameter vector 65 generated for a certain class, whereby the likelihood that the column name constituent word 27 belongs to that class. Calculate the degree. Each element of the weight parameter vector 65 indicates which item (prefix word 51, postfix word 52, table name constituent word 53 or concept identifier 41) to which the element is associated corresponds to the likelihood that the word belongs to the class. How it affects. If the value of the element is positive, it means that when the word corresponds to the item, the likelihood that the word belongs to the class is high. Conversely, when the value of the element is negative, it means that the likelihood that the word belongs to the class is high when the word does not correspond to the item. The larger the absolute value of the element value, the greater the influence. If the value of the element is 0, whether or not the word corresponds to the item means that it is irrelevant whether or not the word belongs to the class.

  First, the word classification unit 60 classifies the column name constituent words 27 randomly within a range that satisfies the condition.

Next, the word classification unit 60 generates a weight parameter vector 65 that is optimal for the classification. The word classification unit 60 uses, for example, supervised learning of multiclass logistic regression. The word classification unit 60 calculates the weight parameter vector 65 by, for example, obtaining an optimal solution by an iterative weighted least squares method.
The word classification unit 60 uses the calculated weight parameter vector 65 to calculate the likelihood that the column name constituent word 27 belongs to each class. The word classification unit 60 reclassifies the column name constituent word 27 using the calculated likelihood. The word classification unit 60 classifies the column name constituent words 27 by, for example, constrained unsupervised clustering using a multi-class logistic regression model.

The word classification unit 60 compares the newly reclassified classification with the classification before reclassification. When there is a change in classification, the word classification unit 60 recalculates the weight parameter vector 65 and reclassifies the column name constituent word 27 using the recalculated weight parameter vector 65. The word classification unit 60 repeats this until the classification does not change.
Since the classification may not converge, a configuration in which the maximum number of repetitions is determined in advance may be used. The word classification unit 60 aborts the process when the number of repetitions reaches the maximum number.

  FIG. 15 is a diagram for explaining an example of the operation of the related word determination unit 70 in this embodiment.

  The related word determination unit 70 generates related column data 71 based on the result of classification by the word classification unit 60. The related column data 71 represents a set of a plurality of columns that are likely to be related columns. The related column data 71 includes a plurality of sets of table identifiers 25 and column identifiers 26, for example. The related column data 71 indicates that the column identified by the column identifier 26 of the table identified by the table identifier 25 is highly likely to be a related column.

  The related word determination unit 70 includes, for example, a plurality of class vectors 63 matching the class vector 63 based on the class vector 63 generated by the word classification unit 60 based on the class in which the word classification unit 60 finally classifies the column name constituent word 27. It is determined whether or not a column exists. When there are a plurality of columns that match the class vector 63, the related word determination unit 70 generates related column data 71 based on the table identifier 25 of the table to which each column belongs and the column identifier 26 of each column. And output.

For example, the related word determination unit 70 includes the class vector 63 generated by the word classification unit 60 for the column {C0011} (ORDER_DATE column) of the table {T0001} (PURCHASE_ORDER table), and the column {C0002} (ORDER table) of the table {T0002} (ORDER table). Since the class vector 63 generated by the word classification unit 60 for C0021} (ODR_DAT column) matches, it is determined that these two columns are columns that are likely to be related columns, and the table of these two columns The related column data 71 including the identifier 25 and the column identifier 26 is generated.
In addition, the related word determination unit 70 generates a word for the class vector 63 generated by the word classification unit 60 for the column {C0012} (LINE_NO column) of the table {T0001} and the column {C0022} (ITEM_NO column) of the table {T0002}. Since the class vector 63 generated by the classification unit 60 matches, it is determined that these two columns are likely to be related columns, and includes the table identifier 25 and the column identifier 26 of these two columns. Related column data 71 is generated.

The related word determination unit 70 may further include the table name of the table identified by the table identifier 25, the column name of the column identified by the column identifier 26, and the like in the related column data 71. .
Further, the related word determination unit 70 acquires the data type of each column from the related word extraction source data 10, and only the columns where the acquired data types match or are close are highly likely to be related columns. May be determined.

  Further, the related word determination unit 70 may be configured to output a set of column name constituent words 27 classified by the word classification unit 60 into the same class, instead of outputting the related column data 71.

  FIG. 16 is a flowchart showing an example of the flow of the related extraction process S800 in this embodiment.

In the related extraction process S800, the related column determination device 903 extracts a column that is highly likely to be a related column from a plurality of columns of a plurality of tables represented by the database definition data stored in the concept dictionary data 30.
The association extraction process S800 includes, for example, a word extraction process S801, a context analysis process S802, a concept analysis process S803, a word classification process S804, and an association determination process S805. The related column determination apparatus 903 starts the related extraction process S800 from the word extraction process S801.

In word extraction processing S <b> 801, the word extraction unit 20 acquires the related word extraction source data 10 from the database definition storage device 901. The word extraction unit 20 extracts column name constituent words 27 from the acquired related word extraction source data 10.
In the context analysis process S802, the context analysis unit 50 generates a context vector 54 for each column name constituent word 27 extracted by the word extraction unit 20 in the word extraction process S801.
In the concept analysis process S803, the feature vector generation unit 40 uses the concept dictionary data 30 stored in the concept dictionary storage device 902, and uses the concept dictionary data 30 for each column name constituent word 27 extracted by the word extraction unit 20 in the word extraction process S801. A vector 44 is generated.
In the word classification process S804, the word classification unit 60, based on the context vector 54 generated by the context analysis unit 50 in the context analysis process S802 and the concept vector 44 generated by the feature vector generation unit 40 in the concept analysis process S803, The column name constituent words 27 extracted by the word extraction unit 20 in the word extraction process S801 are classified.
In the related determination process S805, the related word determination unit 70 determines a column that is likely to be a related column based on the classification result classified by the word classification unit 60 in the word classification process S804.

  FIG. 17 is a flowchart showing an example of the flow of the word classification process S804 in this embodiment.

  The word classification process S804 includes, for example, an initialization step S810, a weight optimization step S820, a classification calculation step S830, and a convergence determination step S840.

In the initialization step S810, the word classification unit 60 uses the processing device 911 to randomly assign a provisional correct answer class without satisfying the constraints. The word classification unit 60 randomly assigns a provisional correct answer class to each word while satisfying the following matching constraint and exclusion constraint.
(1) Match constraint: Words with the same notation are assigned the same correct class regardless of the appearance context of the table / column.
(2) Exclusive constraint: correct classes are allocated so that one or more words of different classes exist between columns constituting the same table.

ただし、モデルパラメータｗは、Ｋ個の重みパラメータベクトル｛ｗ_ｋ｜ｋ＝１，２，…，Ｋ｝を並べたＭ行Ｋ列の行列（ｗ_１，ｗ_２，…，ｗ_Ｋ）を表わす。Ｋは、あらかじめ定めれらたクラスの数を表わす。重みパラメータベクトルｗ_ｋは、Ｍ次の縦ベクトルである。Ｍは、特徴ベクトルの次数を表わす。ａｒｇ ｍａｘは、下に記した変数（この場合、ｗ。）を変化させた場合に、引数である関数（この場合、ｌｏｇ ｐ（ｔ，ｗ，Ａ_ｔ ^{（η−１）}｜Φ）。）の値が最大になる変数の値を表わす。ｌｏｇは、オイラー数を底とする対数（自然対数）を表わす。ｐ（ｔ，ｗ，Ａ_ｔ ^{（η−１）}｜Φ）は、ΦにおけるｔかつｗかつＡ_ｔ ^{（η−１）}の条件付き確率を表わす。ｔは、Ｎ個の正解クラス｛ｔ_ｉ｜ｉ＝１，…，Ｎ｝を要素とするＮ次のベクトルを表わす。Ｎは、同じ綴りを区別したカラム名構成単語２７の数を表わす。ｔ_ｉは、ｉ番目のカラム名構成単語ｘ_ｉ（ｉ＝１，…，Ｎ）に割り当てる正解クラスを表わす。Ａ_ｔは、ｔの割り当てを表わす。ηは、１以上の整数である。ηは、重み最適化工程Ｓ８２０〜収束判定工程Ｓ８４０のループを実行した繰り返し回数を表わす。例えば、ｗ^（η）は、η回目の重み最適化工程Ｓ８２０で算出するｗを表わす。Ａ_ｔ ^（η）は、η回目の分類算出工程Ｓ８３０で算出するＡ_ｔを表わす。Ａ_ｔ ^（０）は、初期化工程Ｓ８１０で単語分類部６０が割り当てた暫定正解クラスの割り当てを表わす。Φは、Ｎ個の特徴ベクトル｛φ_ｉ｜ｉ＝１，２，…，Ｎ｝を並べたＭ行Ｎ列の行列（φ_１，φ_２，…，φ_Ｎ）を表わす。特徴ベクトルφ_ｉは、Ｍ次の縦ベクトルである。特徴ベクトルφ_ｉは、ｉ番目のカラム名構成単語ｘ_ｉについて、単語分類部６０が生成した特徴ベクトル６４を表わす。 In the weight optimization step S820, the word classification unit 60 uses the processing device 911 to fix the provisional correct answer class and optimize model parameters such as the weight parameter vector 65. Word classification unit 60, for example using the following formula, the allocation A _t the correct class is fixed to A _{t ^(eta-1),} optimizing the model parameters w.

Here, the model parameter w represents an M-row / K-column matrix (w ₁ , w ₂ ,..., W _K ) in which _K weight parameter vectors {w _k | k = 1, ₂ ,. . K represents a predetermined number of classes. The weight parameter vector w _k is an M-th order vertical vector. M represents the order of the feature vector. arg max is an argument function (in this case, log p (t, w, A _t ^(η−1) | Φ)) when a variable described below (in this case, w.) is changed. Represents the value of the variable that has the maximum value. log represents a logarithm (natural logarithm) based on the Euler number. p (t, w, A _t ^(η−1) | Φ) represents the conditional probability of t and w and A _t ^(η−1) in Φ. t represents an Nth-order vector having N correct answer classes {t _i | i = 1,..., N} as elements. N represents the number of column name constituent words 27 that distinguish the same spelling. t _i represents a correct class assigned to the i-th column name constituent word x _i (i = 1,..., N). A _t represents the assignment of t. η is an integer of 1 or more. η represents the number of repetitions of executing the loop of the weight optimization step S820 to the convergence determination step S840. For example, w ^(η) represents w calculated in the η-th weight optimization step S820. A _t ^(eta) represents a _{A t} to calculate the classification calculation step S830 of eta time. A _t ⁽⁰⁾ represents the assignment of the temporary correct class word classification unit 60 assigned in the initialization step S810. Φ represents an M-row N-column matrix (φ ₁ , φ ₂ ,..., Φ _N ) in which _N feature vectors {φ _i | i = 1, ₂ ,. The feature vector φ _i is an M-th order vertical vector. The feature vector φ _i represents the feature vector 64 generated by the word classification unit 60 for the i-th column name constituent word x _i .

The word classification unit 60 obtains the value of log p (t, w, A _t | Φ) by the following equation, for example.

ただし、Ｃ_１は、あらかじめ定めた定数（ハイパーパラメータ）である。
例えば、関連カラム判定装置９０３は、入力装置９１２を用いて、Ｃ_１の設定値を入力し、記憶装置９１４を用いて、入力したＣ_１の設定値を記憶しておく。単語分類部６０は、記憶したＣ_１の設定値を使って、ｌｏｇ ｐ（ｗ）の値を算出する。
あるいは、Ｃ_１は、固定値ではなく、繰り返し回数ηが多くなるにしたがって減衰する値であってもよい。例えば、関連カラム判定装置９０３は、入力装置９１２を用いて、Ｃ_１の初期値Ｃ_１ ^（１）と、Ｃ_１の収束値Ｃ_１ ^（∞）と、減衰係数ε_１とを入力する。関連カラム判定装置９０３は、記憶装置９１４を用いて、入力したＣ_１の初期値Ｃ_１ ^（１）と、Ｃ_１の収束値Ｃ_１ ^（∞）と、減衰係数ε_１とを記憶しておく。単語分類部６０は、η回目の繰り返し（ただし、η≧２。）において、前回の繰り返しで使ったＣ_１の値Ｃ_１ ^{（η−１）}と、記憶したＣ_１の収束値Ｃ_１ ^（∞）と減衰係数ε_１とに基づいて、例えば次の式により、今回の繰り返しで使う定数Ｃ_１ ^（η）を算出する。

このように、Ｃ_１の値を徐々に減衰させることにより、学習の強度を徐々に強めることができる。これにより、不安定な学習の初期段階において誤ったクラス割り当てをしてしまったとしても、その後の反復において大域的なクラス割り当てに基づいて誤ったクラス割り当ての変更が起きやすくなる。 Here, log p (w) is a quadratic regularization term for the model parameter w, and prevents the model from overfitting the learning data using an excessively large weight. The word classification unit 60 obtains the value of log p (w) by the following formula, for example.

However, _{C 1} is a predetermined constant (hyper parameters).
For example, associated column determination unit 903 uses the input device 912 inputs the set value of C _1, using the storage device 914 stores the set values of C ₁ entered. Word classification unit 60 uses the set value of the stored _{C 1,} to calculate the value of log p (w).
Alternatively, C ₁ is not a fixed value, but may be a value that attenuates as the number of repetitions η increases. For example, associated column determination unit 903 uses the input device 912 to input the initial value _C ¹ of _{C 1} ^(1), and the convergence value _{C 1} of _{C 1} ^(∞), and a damping coefficient epsilon _1. Related column determination unit 903, using the storage device 914, the initial value _C ¹ of _{C 1} input ^(1), and the convergence value _{C 1} of _{C 1} ^(∞), and stores the attenuation coefficients epsilon ₁ . In the η-th iteration (where η ≧ 2), the word classifying unit 60 uses the C ₁ value C ₁ ^(η−1) used in the previous iteration and the stored C ₁ convergence value C ₁ ^{(∞). )} And the attenuation coefficient ε ₁ , for example, a constant C ₁ ^(η) used in the current iteration is calculated by the following equation.

In this way, by gradually attenuate the value of C _1, it can be increased gradually the intensity of the learning. As a result, even if an incorrect class assignment is made in the initial stage of unstable learning, an incorrect class assignment change is likely to occur in the subsequent iteration based on the global class assignment.

ただし、Ｃ_２は、あらかじめ定めた定数（ハイパーパラメータ）である。例えば、関連カラム判定装置９０３は、入力装置９１２を用いて、Ｃ_２の設定値を入力し、記憶装置９１４を用いて、入力したＣ_２の設定値を記憶しておく。単語分類部６０は、記憶したＣ_２の設定値を使って、ｌｏｇ ｐ（Ａ_ｔ｜ｗ）の値を算出する。
あるいは、Ｃ_２は、固定値ではなく、繰り返し回数ηが多くなるにしたがって減衰する値であってもよい。例えば、関連カラム判定装置９０３は、入力装置９１２を用いて、Ｃ_２の初期値Ｃ_２ ^（１）と、Ｃ_２の収束値Ｃ_１ ^（∞）と、減衰係数ε_２とを入力する。関連カラム判定装置９０３は、記憶装置９１４を用いて、入力したＣ_２の初期値Ｃ_２ ^（１）と、Ｃ_２の収束値Ｃ_２ ^（∞）と、減衰係数ε_１とを記憶しておく。単語分類部６０は、η回目の繰り返し（ただし、η≧２。）において、前回の繰り返しで使ったＣ_２の値Ｃ_２ ^{（η−１）}と、記憶したＣ_２の収束値Ｃ_２ ^（∞）と減衰係数ε_２とに基づいて、例えば次の式により、今回の繰り返しで使う定数Ｃ_２ ^（η）を算出する。

このように、Ｃ_２の値を徐々に減衰させることにより、学習の強度を徐々に強めることができる。これにより、不安定な学習の初期段階において誤ったクラス割り当てをしてしまったとしても、その後の反復において大域的なクラス割り当てに基づいて誤ったクラス割り当ての変更が起きやすくなる。 Also, log p (A _t | w) maximizes the sum of weights received from all learning data in each class. Combined with the penalty for size according to the regularization term described above, it works as a penalty that balances class size (number of types of words) among classes. The word classification unit 60 obtains the value of log p (A _t | w) by the following formula, for example.

However, _{C 2} is a predetermined constant (hyper parameters). For example, associated column determination unit 903 uses the input device 912 inputs the set value of the C _2, using the storage device 914 stores the set value of C ₂ entered. The word classification unit 60 calculates the value of log p (A _t | w) using the stored setting value of C ₂ .
Alternatively, C ₂ is not a fixed value, but may be a value that attenuates as the number of repetitions η increases. For example, associated column determination unit 903 uses the input device 912, the initial value of _{C 2} and _C ^{2 (1),} and the convergence value _{C 1} of _{C 2} ^(∞), and inputs the damping coefficient epsilon _2. Related column determination unit 903, using the storage device 914, the initial value _C ² of _{C 2} input ^(1), the convergence value _{C 2} of _{C 2} ^(∞), and stores the attenuation coefficients epsilon ₁ . In the η-th iteration (where η ≧ 2), the word classifying unit 60 uses the C ₂ value C ₂ ^(η−1) used in the previous iteration and the stored C ₂ convergence value C ₂ ^{(∞ )} And the attenuation coefficient ε ₂ , for example, the constant C ₂ ^(η) used in the current iteration is calculated by the following equation.

In this way, by gradually attenuate the value of C _2, it can be increased gradually the intensity of the learning. As a result, even if an incorrect class assignment is made in the initial stage of unstable learning, an incorrect class assignment change is likely to occur in the subsequent iteration based on the global class assignment.

Also, log p (t | Φ; w, A _t ) is the likelihood of the learning data by the model. For example, the word classification unit 60 obtains the value of log p (t | Φ; w, A _t ) by the following equation.

ただし、ｅｘｐは、オイラー数を底とする指数関数を表わす。Ｚ_ｉは、正規化係数である。単語分類部６０は、例えば次の式により、正規化係数Ｚ_ｉを求める。

When multi-class logistic regression is used, the word classification unit 60 uses, for example, the following expression to estimate the likelihood p (y _k | φ _i ; w, A that the i th word x _i belongs to the k th class y _{k. t} ).

Here, exp represents an exponential function based on the Euler number. Z _i is a normalization coefficient. The word classification unit 60 obtains the normalization coefficient Z _i by the following equation, for example.

ただし、Ｓは、制約条件を充足するＡ_ｔの集合を表わす。 In the classification calculation step S830, the word classification unit 60 uses the processing device 911 to fix the model parameters and obtain a constrained maximum likelihood class. The word classifying unit 60 fixes the model parameter w to w ^(η) , for example, by the following formula, searches for an optimal class with constraints, and reassigns it as a correct class A _t ^(η) .

However, S represents the set of A _t that satisfy the constraints.

For example, the word classifying unit 60 solves the 0-1 integer programming problem using whether each column name component word 27 is classified into each class as a variable, thereby obtaining an optimal class selection solution (a constrained maximum likelihood solution). ) This method is effective when the number of classes and words is small.
Or the word classification | category part 60 calculates | requires a suboptimal solution by assigning a class in an order from a word with a high average likelihood by a greedy method. This method is effective when the number of classes and words is large.

In the convergence determination step S840, the word classification unit 60 uses the processing device 911 to determine the convergence of the clustering process. For example, the word classification unit 60 determines that the clustering process has converged if the provisional correct answer class assigned to any word does not differ from the class assigned in the previous iteration, and aborts the iteration. Alternatively, the word classification unit 60 forcibly terminates the repetition when the number of repetitions η reaches a predetermined maximum number of repetitions.
If it is determined that the iteration is not terminated, the word classification unit 60 returns the process to the weight optimization step S820.
When it is determined that the repetition is to be terminated, the word classification unit 60 ends the word classification process S804.

  FIG. 18 is a flowchart showing an example of the flow of the initialization step S810 in this embodiment.

  The initialization step S810 includes, for example, a word selection step S811, a provisional class assignment step S812, a column selection step S813, a sibling column selection step S814, and an exclusion constraint determination step S815.

In the word selection step S811, the word classification unit 60 selects one word ζ from the column name constituent words 27 to which no provisional correct answer class has been assigned yet.
If the provisional correct answer class has been assigned to all the vocabularies (column name constituent words 27) and there are no words to be selected, the provisional correct answer class is assigned to all the words. Exit.
When there is a vocabulary to which no provisional correct answer class has been assigned, the word classification unit 60 extracts one word to which no provisional correct answer class has been assigned and sets it as a word ζ. The word classification unit 60 proceeds to the provisional class assignment step S812.

In the provisional class assignment step S812, the word classification unit 60 selects one provisional correct answer class from the classes not yet assigned to the word ζ selected in the word selection step S811. For example, the word classification unit 60 generates a list in which all classes are rearranged in a random order, and sets it as a provisional correct class candidate list. The word classification unit 60 selects the class at the head of the provisional correct answer class candidate list as the provisional correct answer class, and deletes the selected class from the provisional correct answer class candidate list. The word classification unit 60 stores a provisional correct answer class candidate list for the word ζ. When the next provisional class assignment step S812 is executed for the same word ζ, the word classification unit 60 uses the provisional correct class candidate list stored for the word ζ and is at the head of the provisional correct class candidate list. The class is selected as a provisional correct answer class, and the selected class is deleted from the provisional correct answer class candidate list.
When there is no class not yet assigned to the word ζ, for example, when the temporary correct class candidate list stored for the word ζ is empty, the word classification unit 60 assigns a class to the word ζ. Cannot be assigned. The word classification unit 60 gives up assigning a class to the word ζ and rolls back to the word selected in the previous word selection step S811. The word classification unit 60 returns the word ζ selected in the word selection step S811 to unselected, and changes the provisional correct answer class assigned to the word selected in the previous word selection step S811. The word classification unit 60 selects one provisional correct answer class from the classes not yet assigned to the word ζ selected in the previous word selection step S811. For example, the word classification unit 60 selects the class at the head of the provisional correct class candidate list stored for the word ζ selected in the previous word selection step S811 as the provisional correct class, and selects the selected class as the provisional correct class. Remove from the candidate list.

In the column selection step S813, the word classification unit 60 extracts a column including the word ζ to which the provisional correct answer class is assigned in the provisional class assignment step S812 from all the columns. The word classification unit 60 extracts, from the extracted columns, a column to which a provisional correct answer class is assigned to all words (all constituent words) included in the column name of the column, and sets it as a column set C1.
When the column set C1 is an empty set and there is no column belonging to the column set C1, the word classification unit 60 returns to the word selection step S811 and selects the next word ζ.
When the column set C1 is not an empty set and there is a column belonging to the column set C1, the word classification unit 60 selects one column from the columns belonging to the column set C1, and selects the selected column from the column set C1. delete. The word classification unit 60 stores a column set C1 for the word ζ. When the next column selection step S813 is executed for the same word ζ, the word classification unit 60 uses the stored column set C1 to select one column from the columns belonging to the column set C1. The selected column is deleted from the column set C1.
The word classification unit 60 generates a class vector 63 for the selected column, and stores the generated class vector 63. The word classification unit 60 proceeds to the brother column selection step S814.

In the sibling column selection step S814, the word classification unit 60 extracts columns belonging to the same table as the column selected in the column selection step S813 from all the columns. The word classification unit 60 extracts, from the extracted columns, a column to which a provisional correct answer class is assigned to all words included in the column name of the column, and sets it as a column set C2. Since the column set C2 always includes the column selected in the column selection step S813, the word classification unit 60 deletes the column selected in the column selection step S813 from the column set C2.
When the column set C2 is an empty set and there is no column belonging to the column set C2, the word classification unit 60 returns the process to the column selection step S813 and selects the next column from the column set C1.
When the column set C2 is not an empty set and there is a column belonging to the column set C2, the word classification unit 60 selects one column from the columns belonging to the column set C2, and selects the selected column from the column set C2. delete. The word classification unit 60 stores a column set C2 for the column. When the next sibling column selection step S814 is performed on the same column, the word classification unit 60 selects one column from the columns belonging to the column set C2 using the stored column set C2. The selected column is deleted from the column set C2.
The word classification unit 60 advances the processing to the exclusion constraint determination step S815.

In the exclusion constraint determination step S815, the word classification unit 60 compares the class vector 63 generated for the column selected in the column selection step S813 with the class vector 63 stored for the column selected in the sibling column selection step S814. To do.
When the two class vectors 63 match, the word classifying unit 60 determines that the restriction is violated, and returns to the provisional class assignment step S812 to assign the next provisional correct answer class to the word ζ.
When the two class vectors 63 are different, the word classification unit 60 determines that the constraint is satisfied, returns the process to the sibling column selection step S814, and selects the next column from the column set C2.

  FIG. 19 is a flowchart showing an example of the flow of the classification calculation step S830 in this embodiment.

  The classification calculation step S830 includes, for example, a word selection step S831, a provisional class assignment step S832, a column selection step S813, a sibling column selection step S814, and an exclusive constraint determination step S815.

ただし、Ｘ_ζは、単語ｘ_ｉがζである添え字ｉの集合を表わす。 In the word selection step S831, the word classification unit 60 calculates the average likelihood for all classes for all words. The word classification unit 60 calculates an average likelihood p￣ (y _k | ζ) _obtained by, for example, averaging the likelihood that the word ζ belongs to the class y _k with respect to all appearance locations by the following formula.

Here, X _ζ represents a set of subscripts _i whose word x _i is ζ.

The word classification unit 60 generates a list in which classes are arranged in descending order of average likelihood for each word ζ, and sets it as a provisional correct answer class list for the word ζ.
The word classifying unit 60 generates a list in which the words ζ are arranged in descending order of the average likelihood (that is, the maximum value of the average likelihood) for the class at the head of the provisional correct answer class list, and sets it as the assignment order word list.
The word classification unit 60 selects one unprocessed word from the allocation order word list.
When there is no unprocessed word in the allocation order word list, the class allocation for all the words is completed, and the word classification unit 60 ends the classification calculation step S830.
When there is an unprocessed word in the allocation order word list, the word classification unit 60 selects the word that is the foremost among the unprocessed words (that is, the word having the largest maximum average likelihood). Let ζ be the word ζ. The word classification unit 60 proceeds to the provisional class assignment step S832.

In the provisional class assignment step S832, the word classification unit 60 assigns a class to the word ζ selected in the word selection step S831, using the provisional correct answer class list generated in the word selection step S831. The word classification unit 60 selects one unselected class from the provisional correct answer class list.
If no unselected class exists in the provisional correct answer class list, a class cannot be assigned to the word ζ, and the word classification unit 60 rolls back to the previous word. The word classification unit 60 returns all provisional correct answer class lists for the word ζ selected in the word selection step S831 to the unselected state. The word classification unit 60 selects the last word from the processed words in the assigned order word list, sets it as the word ζ, and returns it to the unprocessed state. The word classification unit 60 selects one unselected class from the provisional correct answer class list for the rolled back word ζ.
When there is an unselected class in the provisional correct answer class list, the word classification unit 60 selects the class that is at the forefront among the unselected classes (that is, the class with the highest average likelihood).

  The column selection step S813, the sibling column selection step S814, and the exclusion constraint determination step S815 are steps for verifying whether the class assignment does not violate the exclusion constraint, and are the same as the initialization step S810.

  In the initialization step S810, the word selection order and the class selection order are random, whereas in the classification calculation step S830, the words are selected in descending order of the maximum average likelihood, and the classes are selected in descending order of the average likelihood. To do. As a result, it is possible to find a class assignment having a high likelihood as a whole while satisfying the condition that the exclusive constraint is not violated.

  FIG. 20 is a diagram for explaining an example of the operation of the classification calculation step S830 word classification unit 60 in this embodiment.

In the word selection step S831, the word classification unit 60 distinguishes between the same spelled words appearing at different positions, and calculates the likelihood 66 for each class for each column name constituent word 27.
The word classification unit 60 calculates the average likelihood 67 based on the likelihood 66 calculated for the same spelled word appearing at different positions. In this example, since the word “NO” appears in three places, the word classification unit 60 calculates an average likelihood 67 from the calculated three likelihoods 66 for each class.
The word classification unit 60 does not distinguish between words with the same spelling that appear at different positions, and generates a provisional correct answer class list 68 in which classes are arranged in descending order of the calculated average likelihood 67 for each word. In addition, the word classification unit 60 does not distinguish between the same spelling words appearing at different positions, and generates an allocation order word list 69 in which words are arranged in descending order of the maximum average likelihood 67 among the calculated average likelihoods 67. To do.

  The word classification unit 60 assigns classes in order from the word positioned at the top of the assignment order word list 69. The word classifying unit 60 assigns the class located at the head of the provisional correct answer class list 68, and if it violates the exclusion constraint, assigns the next class.

For example, the word classification unit 60 selects the word “NO” at the top of the allocation order word list 69 at the word selection step S831, and at the top of the provisional correct answer class list 68 for the word at the temporary class allocation step S832. Select class {y4} and assign class {y4} to word “NO”.
As described above, the word “NO” appears in three places. According to the likelihood 66 calculated separately for each appearance position, the class {y4} has the largest likelihood 66 in the column {C0012} of the table {T0001}, and the column {C0013] of the table {T0001}. }, The class {y4} has the largest weight parameter vector 65, but the class {y2} has the largest likelihood 66 in the column {C0022} of the table {T0002}. However, the word classification unit 60 assigns the class {y4} to the word “NO” without distinguishing the appearance position. Thereby, it is possible to find an assignment satisfying the matching constraint.

  Next, the word classification unit 60 selects the second word “ITEM” in the allocation order word list 69 in the word selection step S831, and in the provisional class assignment step S832, the head of the provisional correct answer class list 68 of the word. Class {y1} is selected and class {y1} is assigned to the word “ITEM”. At this point, since the classes are assigned to all the words included in the column name for the column {C0022} of the table {T0002}, the word classification unit 60 generates the class vector 63 in the column selection step S813. In this way, the word classification unit 60 generates the class vector 63 in order from the column in which classes are assigned to all words. For the other columns belonging to the table {T0002}, the class vector 63 has not been generated yet, so the word classification unit 60 determines that it is not an exclusive constraint violation.

  In this way, the word classification unit 60 assigns classes to words in the order of the assignment order word list 69.

Assume that the classification calculation step S830 ends up to the word “DATE”. At this point, the word classification unit 60 includes all the column names {C0012} of the table {T0001}, the columns {C0022} of the table {T0002}, and the columns {C0032} of the table {T0003} that are included in the column names. Class assignment is finished for the word, and a class vector 63 is generated.
Next, in the word selection step S831, the word classification unit 60 selects the word “PHONE” next to the assigned order word list 69, and selects the class {y1} at the head of the provisional correct answer class list 68 of the word. Class {y1} is assigned to the word “PHONE”. For the column {C0013} of the table {T0001}, classes are assigned to all the words included in the column name, so the word classification unit 60 generates a class vector 63 in the column selection step S813. However, since the generated class vector 63 matches the class vector 63 generated for the column {C0012} of the same table {T0001}, the word classifying unit 60 determines that the exclusive constraint is violated. Therefore, the word classification unit 60 returns the process to the provisional class assignment step S832, selects the second class {y2} in the provisional correct answer class list 68 of the word “PHONE”, and class {y2} to the word “PHONE”. Reassign Thereby, it is possible to find an assignment satisfying the exclusive constraint.

  In this way, even for samples that are difficult to identify by unsupervised learning alone, the correct class can be assigned based on the global identification situation and task heuristics, and high-precision clustering can be realized.

The related word extraction method (related column determination device 903, related word classification device) in this embodiment extracts related words from data including text using a multi-class identification model.
The word classification unit 60 (weight vector calculation unit) optimizes the model parameters for identifying the word class based on the provisional correct answer class assigned to the word appearing in the data.
The word classifying unit 60 selects the most likely class among the classes satisfying the restrictions on the class assignment based on the class likelihood obtained using the current model parameter, and sets the word as a new provisional correct answer class. assign.
The word classification unit 60 extracts the words classified into the same class as related words after alternately repeating the two.

  The data including the text is database schema definition data (table schema).

  The constraint on the class assignment is an assignment such that sister data elements in the data structure have different classes of data elements.

  According to the related word extraction method in this embodiment, related words can be automatically extracted from a table schema group with high accuracy.

The related word extraction method in this embodiment is based on the assumption that “similar words appear in similar contexts” and is weighted appropriately based on the appearance frequency vector of co-occurrence words as feature vectors. Then, related words are extracted based on the similarity.
However, the amount of data that can be used for structured data such as a table schema is limited compared to newspaper articles and web pages. For this reason, the co-occurrence frequency vector becomes sparse, and the similarity between words may not be sufficiently explained only by the co-occurrence words.
Therefore, the related word extraction method in this embodiment further adds feature amounts using a concept dictionary.

However, the feature amount added using the concept dictionary is affected by ambiguity and ambiguity, which are problems that generally exist in natural language processing.
Therefore, the related word extraction method in this embodiment introduces the following three assumptions to determine the proper use of words.

Assumption (1): The related word is close to the feature of the appearance context or the registration hierarchy in the concept dictionary.
Assumption (1) is a basic assumption for extracting related words by clustering feature vectors. Assumption (1) is reflected in the configuration of the feature vector 64. By using the feature vector 64 including the context vector 54 and the concept vector 44, clustering according to the assumption (1) is realized.

Assumption (2): Words consistently represent the same concept regardless of appearance context.
Assumption (2) is a rather strong assumption that is introduced to aggregate frequency with sparse schema data. Assumption (2) is reflected in the above-mentioned coincidence constraint. The word classification unit 60 implements clustering according to assumption (2) by assigning the same class to words of the same spelling that appear at different positions.

Assumption (3): The column name represents a different concept in the table.
Assumption (3) is the strongest assumption based on heuristics of table design. Assumption (3) is reflected in the exclusion constraint described above. When the column name is composed of a plurality of words, the concept represented by the column name is considered to be a combination of the concepts represented by the words constituting the column name. Therefore, based on Assumption (3), the word classification unit 60 assigns classes to words so that combinations of classes assigned to words constituting column names are different among a plurality of columns belonging to the same table.

  In the above example, the “PURCHASE_ORDER” table includes a “LINE_NO” column and a “PHONE_NO” column. If the word “LINE” and the word “PHONE” are classified into the same class, “LINE_NO” and “PHONE_NO” cannot be conceptually distinguished. Assumption (3) is an assumption for avoiding such unsuitable clustering.

As described above, the related word extraction method according to the present embodiment uses not only the similarity of feature vectors as a measure in clustering of word feature vectors but also the heuristics acquired from the original data as class classification matching constraints. Use as an exclusive constraint for clustering.
Unlike the ad hoc method that adds a penalty to the similarity of feature vectors, the method that uses heuristics as a constraint condition is reflected in anonymous class assignment in clustering using an identification model used in supervised learning. Discriminative learning method. For this reason, it is possible to effectively improve the accuracy of clustering.

  Furthermore, the related term extraction scheme in this embodiment assumes, as heuristics, that sister data elements (columns belonging to the same table) in the schema structure are designed to represent different concepts, and therefore These data elements are used to have different classes of data elements. This assumption can be easily obtained from schema data, and is expected to hold widely in schema design. As a result, clustering can be performed without confusing the use of similar words.

The related word extraction method in this embodiment realizes high-precision related word extraction in which a word that should be treated distinctly and a word that should not be handled are discriminated and are not extracted as related words.
In addition, the related word extraction method in this embodiment does not need to generate a sample for learning based on a combination of related words registered in the dictionary, so the words of the target domain are sufficiently registered in the dictionary in advance. Even if it is not, it can be applied, and even when the words of the target domain are not registered in the dictionary, it is possible to realize high-precision related word extraction.

Note that the context analysis unit 50 not only sets the value of the element of the context vector 54 associated with the word located immediately before or after the column name constituent word 27 to “1”, but also separates them within the same column name. The value of the element of the context vector 54 associated with the word at the position may also be “1”.
Alternatively, the context analysis unit 50 may be configured to give a numerical value that becomes smaller as the distance between the words is longer, to the element of the context vector 54 that is associated with the word at a distant position. For example, the context analysis unit 50 associates the value of the element of the context vector 54 associated with the word before or after two by “½”, the word before or after three, The configuration may be such that the value of the element of the context vector 54 is “1/3”.

  The feature vector generation unit 40 not only sets the value of the element of the concept vector 44 associated with the concept represented by the column name constituent word 27 and its superordinate concept to “1”, but also represents the column name constituent word 27. The configuration may be such that the value of the element associated with the concept subordinate concept is also “1”.

  Alternatively, the feature vector generation unit 40 may be configured to give a numerical value that decreases as the hierarchy difference increases to the elements of the concept vector 44 that are associated with concepts that are separated from each other. For example, the feature vector generation unit 40 sets the value of the element associated with the superordinate concept immediately above to “1/2”, and sets the value of the element associated with the superordinate concept on the second hierarchy to “1/3”. It may be configured as follows.

  Alternatively, the feature vector generation unit 40 may be configured to apportion the numerical values given to the elements of the concept vector 44 according to the frequency of use of the meaning when the column name constituent word 27 is a polysemy. For example, the feature vector generation unit 40 sets “0.5” as the value of the element associated with the primary concept of the word and “0” as the value of the element associated with the secondary concept. .3 ”, the value of the element associated with the third meaning concept may be“ 0.2 ”.

  The class vector 63 matches the combination of the class assigned to the word included in the column name of the column and the combination of the class assigned to the word included in the column name of the other column in order to find an exclusive constraint violation. It is for determining whether to do. The word classification unit 60 may be configured to find an exclusion condition violation using other data instead of generating the class vector 63. For example, the word classification unit 60 generates a list or set of classes assigned to the words included in the column name, and uses the generated list or set to combine the classes assigned to the words included in the column name of the column. And a combination of classes assigned to words included in column names of other columns may be determined.

The word classification unit 60 (weight vector calculation unit) may be configured to calculate the weight parameter vector 65 (weight vector) using another optimization method instead of multi-class logistics regression.
Alternatively, the word classification unit 60 may be configured to cluster words using other clustering methods instead of the greedy method.

Embodiment 2. FIG.
Embodiment 2 will be described with reference to FIG.
In addition, about the part which is common in Embodiment 1, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The overall configuration of the related column extraction system 900 in this embodiment, the hardware configuration of the related column determination device 903, and the block configuration of the related column determination device 903 are the same as those in the first embodiment.

  FIG. 21 is a diagram for explaining an example of the operation of the context analysis unit 50 in this embodiment.

The context analysis unit 50 (an example of a feature vector generation unit) generates a context vector 54 for each column name constituent word 27 without distinguishing between the same spelling words appearing at different positions.
Of the elements of the context vector 54, the elements associated with the prefix word 51 and the suffix word 52 represent the appearance frequency of the prefix word and the suffix word of the word. Of the elements of the context vector 54, the element associated with the table name constituent word 53 is the table to which the column including the word belongs, and the number of tables having the table name including the table name constituent word 53 is indicated. Represent.

For example, for each column name constituent word 27, the context analysis unit 50 extracts a column having a column name including the word from all the columns, and forms a column set. The context analysis unit 50 extracts the prefix word that precedes the word from the column name constituent words 27 that constitute the column names of the columns that belong to the extracted column set, and sets it as the prefix word set. In the prefix word set, the context analysis unit 50 distinguishes words having the same spelling. That is, when there are a plurality of columns in which the word preceding the word has the same spelling, the prefix word set includes a plurality of words having the same spelling. For each prefix word 51, the context analysis unit 50 extracts a word that matches the prefix word 51 from the prefix word set. The context analysis unit 50 sets the number of extracted words as the value of the element of the context vector 54 associated with the prefix word 51.
Similarly, the context analysis unit 50 extracts a suffix word that comes after the word from the column name constituent words 27 that constitute the column names of the columns that belong to the extracted column set, and forms the suffix word set. . In the succeeding word set, the context analysis unit 50 distinguishes words having the same spelling. For each postfix word 52, the context analysis unit 50 extracts a word that matches the postfix word 52 from the set of suffix words. The context analysis unit 50 sets the number of extracted words as the value of the element of the context vector 54 associated with the postword 52.
Further, the context analysis unit 50 extracts a table to which a column belonging to the extracted column set belongs from all the tables, and sets it as a table set. In the table set, the context analysis unit 50 does not distinguish between the same tables. That is, even when there are a plurality of columns included in the same table in the column set, only one table is included in the table set. The context analysis unit 50 extracts the words constituting the table names of the tables belonging to the table set and sets them as the table name constituting word set. In the table name constituent word set, the context analysis unit 50 distinguishes words having the same spelling. That is, when the same spelling word is included in the table names of different tables, the table name constituent word set includes a plurality of words having the same spelling. For each table name constituent word 53, the context analysis unit 50 extracts a word that matches the table name constituent word 53 from the table name constituent word set. The context analysis unit 50 sets the number of extracted words as the value of the element of the context vector 54 associated with the table name constituent word 53.

  For example, the word “NO” appears in three places: the “LINE_NO” column of the “PURCHASE_ORDER” table, the “PHONE_NO” column of the “PURCHASE_ORDER” table, and the “ITEM_NO” column of the “ORDER” table. To do.

For the word “NO”, the context analysis unit 50 generates a set of three columns, a “LINE_NO” column, a “PHONE_NO” column, and an “ITEM_NO” column, and sets it as a column set.
The context analysis unit 50 generates a set of three words, the word “LINE”, the word “PHONE”, and the word “ITEM”, and sets the set as a front word set. The context analysis unit 50 includes an element associated with the word “LINE”, an element associated with the word “PHONE”, an element associated with the word “PHONE”, and the word “ITEM” among the elements of the context vector 54 associated with the prefix word 51. The values of the three elements corresponding to the element associated with “” are set to “1”, and the values of the elements associated with the other prefix words 51 are set to “0”.
In addition, the context analysis unit 50 sets an empty set that does not include a word as a trailing word set. The context analysis unit 50 sets all the values of the elements of the context vector 54 associated with the postfix word 52 to “0”.
In addition, the context analysis unit 50 generates a set of two tables, a “PURCHASE_ORDER” table and an “ORDER” table, and sets it as a table set. The context analysis unit 50 generates a set including the word “PURCHASE”, the word “ORDER”, and the word “ORDER” (note that two words “ORDER” are included), To do. The context analysis unit 50 sets the value of the element associated with the word “PURCHASE” among the elements of the context vector 54 associated with the table name constituent word 53 to “1” and is associated with the word “ORDER”. The value of the corresponding element is set to “2”, and the value of the element associated with the other table name constituting word 53 is set to “0”.

  As the context analysis unit 50 generates the context vector 54 that integrates the same spelled words that appear at different positions without distinguishing the appearance positions of the words, the word classification unit 60 (an example of a feature vector generation unit). In addition, the feature vector 64 is generated by integrating the same spelled words appearing at different positions without distinguishing the appearance positions of the words. For each word, the word classification unit 60 combines the context vector 54 generated by the context analysis unit 50 and the concept vector 44 generated by the feature vector generation unit 40 into a feature vector 64.

  Since there is only one feature vector 64 for the same spelled word, the word classifying unit 60 calculates the likelihood 66 for each class without distinguishing the appearance position of the word. Therefore, the word classification unit 60 does not need to calculate the average likelihood 67. The word classification unit 60 classifies words into any class based on the calculated likelihood 66.

  As described above, the configuration described in each embodiment is an example, and another configuration may be used. For example, the structure which combined the structure demonstrated in different embodiment may be sufficient, and the structure which replaced the structure of the non-essential part with the other structure may be sufficient.

The related word classification device (related column determination device 903) described above classifies a plurality of words into groups (classes) of related words.
The related word classification device includes a processing device (911) for processing data, a text acquisition unit (word extraction unit 20), a feature vector calculation unit (feature vector generation unit 40, context analysis unit 50, word classification unit 60), And a word classification unit (60) and a weight vector calculation unit (word classification unit 60).
The text acquisition unit acquires a plurality of texts (table schemas) including one or more word groups (column names) made up of one or more words (column name constituent words 27) using the processing device.
The feature vector calculation unit calculates a feature vector (64) representing the feature of the word for each of a plurality of words included in the plurality of texts acquired by the text acquisition unit using the processing device.
The word classification unit classifies each of a plurality of words included in the plurality of texts acquired by the text acquisition unit into one of a plurality of groups using the processing device.
The weight vector calculation unit uses the processing device to determine, for each of the plurality of groups, a weight indicating a likelihood that a word whose feature is represented by the feature vector as an inner product with the feature vector belongs to the group. A vector (weight parameter vector 65) that is most suitable for classification by the word classification unit is calculated.
The word classification unit further uses the processing device to calculate the feature vector calculated by the feature vector calculation unit for each of the plurality of words and the weight calculated by the weight vector calculation unit for each of the plurality of groups. The likelihood that each of the plurality of words belongs to each of the plurality of groups is calculated based on the vector, and the plurality of words are reclassified based on the calculated likelihood.

  As a result, even when structured data or the like is used as a target text, related words can be extracted while identifying words that are used properly.

  The weight vector calculation unit further includes a case where the number of times the word classification unit reclassifies is less than a predetermined number of times, and a classification after the word classification unit reclassifies and before the word classification unit reclassifies In the case where the classification is different from at least one of the classifications, the weight vector is recalculated using the processing device based on the classification after the word classification unit reclassifies.

  By alternately repeating the calculation of the weight vector and the word classification, it is possible to extract related words while identifying the words that are used properly.

  The word classifying unit includes a group group in which one or more words included in a certain word group are classified, and one or more words included in another word group belonging to the same text as the word group. The plurality of words are classified so that the group of the classified group is different.

  Since words are classified based on features of structured data such as a database definition, related words can be extracted while identifying different words.

  The word classifying unit is configured such that a group in which a certain word is classified and a group in which the same word as the word appearing at a position different from the word in the plurality of texts are the same are the same. Classify words.

  Since words are classified based on features of structured data such as a database definition, related words can be extracted while identifying different words.

  The feature vector calculation unit, for each of the plurality of words, the concept represented by the word, other words (prefix word 51, postfix word 52) belonging to the word group to which the word belongs, and the word The feature vector is calculated based on at least one of the words (table name constituting word 53) constituting the name of the word group (table name).

  Since the words are classified based on the general concept represented by the word, the context in which the word appears, and the like, it is possible to extract related words while identifying the words that are used properly.

The text acquired by the text acquisition unit is table definition data that defines a table in the database. The word group is a column name of columns constituting the table.
The related word classification device further includes a related column determination unit (related word determination unit 70).
The related column determination unit uses the processing device to group a group of one or more words constituting the column name of the column in which the word classification unit is located, and the word classification unit in a different column. When a pair of groups in which one or more words constituting a column name are respectively matched, it is determined that there is a possibility that the two columns are columns (related columns) that store the same contents. .

  Since a column that may be a related column is determined based on the word classification result, the efficiency of the work of determining the related column can be improved.

  10 related word extraction source data, 20 word extraction unit, 21 table name constituent word data, 22, 25 table identifier, 23, 53 table name constituent word, 24 column name constituent word data, 26 column identifier, 27 column name constituent word, 30 concept dictionary data, 31 concept data, 32, 41 concept identifier, 33 concept display word, 34 superordinate concept identifier, 40 feature vector generation unit, 44 concept vector, 50 context analysis unit, 51 prefix word, 52 suffix word, 54 Context Vector, 60 Word Classifier, 61, 62 Class Identifier, 63 Class Vector, 64 Feature Vector, 65 Weight Parameter Vector, 66 Likelihood, 67 Average Likelihood, 68 Temporary Correct Answer Class List, 69 Allocation Order Word List, 70 Related word determination unit, 71 Related column data, 80 Related word data , 900 related column extraction system, 901 database definition storage device, 902 conceptual dictionary storage device, 903 related column determination device, 911 processing device, 912 input device, 913 output device, 914 storage device, S800 related extraction processing, S801 word extraction processing , S802 context analysis processing, S803 concept analysis processing, S804 word classification processing, S805 association determination processing, S810 initialization step, S811, S831 word selection step, S812, S832 provisional class assignment step, S813 column selection step, S814 sibling column selection Step, S815 Exclusive constraint determination step, S820 weight optimization step, S830 classification calculation step, S840 convergence determination step.

Claims (8)

In a related word classification device for classifying a plurality of words into groups of related words,
A processing device for processing data, a text acquisition unit, a feature vector calculation unit, a word classification unit, and a weight vector calculation unit;
The text acquisition unit acquires a plurality of texts including one or more word groups composed of one or more words using the processing device,
The feature vector calculation unit calculates a feature vector representing the feature of the word for each of a plurality of words included in the plurality of texts acquired by the text acquisition unit using the processing device;
The word classification unit classifies each of the plurality of words included in the plurality of texts acquired by the text acquisition unit into any of a plurality of groups using the processing device,
The weight vector calculation unit uses the processing device to determine, for each of the plurality of groups, a weight indicating a likelihood that a word whose feature is represented by the feature vector as an inner product with the feature vector belongs to the group. A vector that calculates a weight vector that best fits the classification by the word classification unit,
The word classification unit further uses the processing device to calculate the feature vector calculated by the feature vector calculation unit for each of the plurality of words and the weight calculated by the weight vector calculation unit for each of the plurality of groups. A related word characterized by calculating a likelihood that each of the plurality of words belongs to each of the plurality of groups based on the vector, and reclassifying the plurality of words based on the calculated likelihood Classification device. The weight vector calculation unit further includes a case where the number of times the word classification unit reclassifies is less than a predetermined number of times, and a classification after the word classification unit reclassifies and before the word classification unit reclassifies The weight vector is recalculated on the basis of the classification after the word classification unit reclassifies using the processing device in at least one of cases where the classification is different from The related word classification device according to claim 1. The word classifying unit includes a group group in which one or more words included in a certain word group are classified, and one or more words included in another word group belonging to the same text as the word group. The related word classification device according to claim 1 or 2, wherein the plurality of words are classified so that a group of the classified groups is different. The word classifying unit is configured such that a group in which a certain word is classified and a group in which the same word as the word appearing at a position different from the word in the plurality of texts are the same are the same. The related word classification device according to claim 1, wherein words are classified. The feature vector calculation unit includes, for each of the plurality of words, a concept represented by the word, another word belonging to the word group to which the word belongs, and a word constituting the name of the word group to which the word belongs. 5. The related word classification device according to claim 1, wherein the feature vector is calculated based on at least one of the above. The text acquired by the text acquisition unit is table definition data that defines a table in the database, and the word group is a column name of a column constituting the table,
The related word classification device further includes a related column determination unit,
The related column determination unit uses the processing device to group a group of one or more words constituting the column name of the column in which the word classification unit is located, and the word classification unit in a different column. It is characterized in that it is determined that there is a possibility that the above two columns are columns storing the same contents when a set of groups in which one or more words constituting the column name are respectively matched. The related word classification device according to any one of claims 1 to 5.   A computer program for causing a computer having a processing device for processing data to function as the related word classifying device according to any one of claims 1 to 6 when executed by a computer. A related word classification device having a processing device that processes data, a text acquisition unit, a feature vector calculation unit, a word classification unit, and a weight vector calculation unit classifies a plurality of words into groups of related words. In the related term classification method,
The text acquisition unit acquires a plurality of texts including one or more word groups composed of one or more words using the processing device,
The feature vector calculation unit calculates a feature vector representing the feature of the word for each of a plurality of words included in the plurality of texts acquired by the text acquisition unit using the processing device;
The word classification unit classifies each of a plurality of words included in the plurality of texts acquired by the text acquisition unit into any of a plurality of groups using the processing device,
The weight vector calculation unit uses the processing device to determine, for each of the plurality of groups, a weight representing a likelihood that a word whose feature is represented by the feature vector as an inner product with the feature vector belongs to the group. A vector that calculates a weight vector that best fits the classification by the word classification unit,
The word classifying unit using the processing device, the feature vector calculated by the feature vector calculating unit for each of the plurality of words, and the weight vector calculated by the weight vector calculating unit for each of the plurality of groups; Based on the above, the related word classification method characterized by calculating the likelihood that each of the plurality of words belongs to each of the plurality of groups, and reclassifying the plurality of words based on the calculated likelihood .

Images