JP6817246B2 - Data processing equipment, data processing method and data processing program - Google Patents

Description

The present invention relates to devices, methods and programs for creating structural data and schemas from semi-structured data.

Conventionally, for example, local governments have released information about the city such as nursery school information, public facility information, and demographic information as open data. By integrating and analyzing these data, a house search support service that displays information on the city and comparison information with other cities when the user searches the Web for the house he / she wants to live in is provided. In addition, the company has data such as customer information, product information, and purchase information. By integrating and analyzing these data, the tendency of customers to purchase products can be grasped, and it is utilized for recommending other similar products or developing new products.

In this way, by integrating and analyzing multiple distributed data, value that could not be provided by individual data alone is created.
However, the format of the data published by each local government, etc. to be integrated may be different. For example, some local governments publish structural data in CSV (Comma-Separated Values) format with attribute names placed in the first line so that they can be handled by RDB (Relational Data), but other local governments publish structural data in RDB. Semi-structured data is open to the public in a format such as PDF (Portable Document Forum) or XML (eXtensible Markup Language) that cannot be handled. Therefore, when integrating data, it is necessary to create a schema of structural data and RDB from semi-structured data in PDF or XML format.
The semi-structured data is data that includes both structural data that can be expressed in a tabular format such as a list of childcare facilities and unstructured data that cannot be expressed in a tabular format such as titles and prefaces included in public materials.

For example, the following method has been proposed as a method for creating a schema of structural data and RDB from semi-structured data.
In the method of Patent Document 1, the RDB schema is created from the description of <xsd: element name = "(attribute name)"> in the XML schema.
In the method of Patent Document 2, from the description of <(attribute name)> attribute value </ / (attribute name)> in the tag of the XML data, the word in <> is recognized as the attribute name and the structural data and the RDB schema are generated. Will be created.
In the method of Patent Document 3, the sentence structure of a text that is a natural sentence is assumed in advance, and for example, an attribute name such as "[action] was performed for [purpose] in [event]" and a place where the attribute value is entered are specified. Structural data and RDB schema are created based on the structured conversion rules.

In addition, the library of Non-Patent Document 1 uses XML as one information unit for each of text, table, image, etc., and represents PDF data by dividing it into information units with <ahp: frame> </ ahp: frame> tags. Convert to data. For example, in the places separated by the description of <ahp: frame ahp: frame-type = "table"> </ ahp: frame>, the description regarding the tabular data is performed.

Japanese Unexamined Patent Publication No. 2006-350924 Japanese Unexamined Patent Publication No. 2003-271444 Japanese Unexamined Patent Publication No. 2003-288332

"Antenna House PDFXML Conversion Library V2.0", [online], Antenna House Co., Ltd., [Searched on March 13, 2018], Internet <http://www. antenna. co. jp / pdfxml />

However, in the conventional method described above, by using the XML schema or XML data in which the attribute name is specified, or the structured conversion rule in which the attribute name is specified in advance, the semi-structured data can be used as the structural data and the structural data that can be used in the RDB. The schema was created automatically.

In the method of Patent Document 1, if the XML schema in which the attribute name is defined does not exist in the target semi-structured data, the attribute name cannot be recognized and the RDB schema cannot be automatically created.
In the method of Patent Document 2, in the target semi-structured data, a description indicating the style of the document (for example, <P> or <TD>) is included in the XML tag, and the attribute name is not described. The attribute name cannot be recognized from the information in the tag, and the RDB schema cannot be created automatically.
In the method of Patent Document 3, it is necessary to manually create a structured conversion rule that specifies the place where the attribute name and the attribute value are entered for each semi-structured data in which the attribute name and the attribute value are not distinguished. Yes, the RDB schema cannot be created automatically.

Further, in the method of Non-Patent Document 1, although the structural data in CSV format can be extracted from the semi-structured data, if the value of each cell cannot be recognized as the attribute name or the attribute value, the correct structural data and the RDB schema are automatically executed. Cannot be created.

The semi-structured data does not have a schema, and although the attribute name representing the property such as "implementation facility" and the attribute value such as "first nursery school" corresponding to the value of this attribute are described, the attribute name And attribute values may not be clearly distinguished. It should be noted that the fact that the attribute name and the attribute value are not distinguished means that both are described in the data in the same format, and the distinction cannot be determined by this description alone.
As described above, when the attribute name and the attribute value are not clearly distinguished in the target semi-structured data, it is difficult to create the correct structural data and RDB schema by the conventional method.

An object of the present invention is to provide a data processing apparatus, a data processing method, and a data processing method capable of automatically creating structural data and a schema from semi-structured data in which attribute names and attribute values are not clearly distinguished. ..

The data processing apparatus according to the present invention determines the frequency with which each of the data acquisition unit that converts semi-structured data into general-purpose data and extracts the table structure part from the general-purpose data and the word included in the table structure part appears. The attribute candidate is a combination of a predetermined row or column based on the format of the table structure and an attribute candidate extraction unit that calculates and extracts the word having the highest frequency as an attribute candidate from the histogram of the frequency. An attribute estimation unit that discriminates words arranged in the combination with the highest occurrence rate as an attribute name, and the attribute name is associated with the general-purpose data excluding the attribute name as an attribute value, and has a structure. It includes a structural data creation unit that creates data and a schema creation unit that creates a schema that defines the attribute names.

The attribute estimation unit may adopt only the attribute candidates whose similarity with words existing in a predetermined common vocabulary is equal to or higher than the threshold value.

When the attribute candidate is not adopted by the attribute estimation unit, the attribute candidate extraction unit may extract a word whose frequency is one step lower as the attribute candidate.

The attribute estimation unit may determine the format of the table structure based on the frequency of words extracted as the attribute candidates.

The data acquisition unit may convert the semi-structured data into the general-purpose data including the XML format, and extract the part of the table structure by a specific tag in the XML format.

In the data processing method according to the present invention, a data acquisition step of converting semi-structured data into general-purpose data and extracting a part of the table structure from the general-purpose data, and a frequency at which each word included in the part of the table structure appears. The attribute candidate among the combination of the attribute candidate extraction step that calculates and extracts the word with the highest frequency as the attribute candidate from the histogram of the frequency, and a predetermined row or column based on the format of the table structure. An attribute estimation step that determines a word arranged in the combination with the highest occurrence rate as an attribute name, and the attribute name is associated with the general-purpose data excluding the attribute name as an attribute value, and has a structure. The computer executes a structural data creation step for creating data and a schema creation step for creating a schema in which the attribute name is defined.

The data processing program according to the present invention determines the data acquisition step of converting semi-structured data into general-purpose data and extracting the part of the table structure from the general-purpose data, and the frequency with which each word included in the part of the table structure appears. The attribute candidate among the combination of the attribute candidate extraction step that calculates and extracts the word with the highest frequency as the attribute candidate from the histogram of the frequency, and a predetermined row or column based on the format of the table structure. An attribute estimation step that determines a word arranged in the combination with the highest occurrence rate as an attribute name, and the attribute name is associated with the general-purpose data excluding the attribute name as an attribute value, and has a structure. This is for causing a computer to execute a structural data creation step for creating data and a schema creation step for creating a schema in which the attribute name is defined.

According to the present invention, structural data and schema can be automatically created from semi-structured data in which attribute names and attribute values are not clearly distinguished.

It is a figure which shows the functional structure of the data processing apparatus which concerns on embodiment. It is a figure which illustrates the semi-structured data of the 1st pattern which concerns on embodiment. It is a figure which illustrates the semi-structured data of the 2nd pattern which concerns on embodiment. It is a figure which illustrates the general-purpose data with respect to the semi-structured data of the 1st pattern which concerns on embodiment. It is a figure which illustrates the histogram with respect to the semi-structured data of the 1st pattern which concerns on embodiment. It is a figure which illustrates the result of having calculated the degree of similarity between the attribute candidate and a common vocabulary with respect to the semi-structured data of the 1st pattern which concerns on embodiment. It is a figure which shows the method of discriminating the attribute name and the attribute value in the semi-structured data of the 1st pattern which concerns on embodiment. It is a figure which shows the output example of the structural data and the schema with respect to the semi-structured data of the 1st pattern which concerns on embodiment. It is a figure which illustrates the general-purpose data with respect to the semi-structured data of the 2nd pattern which concerns on embodiment. It is a figure which illustrates the histogram with respect to the semi-structured data of the 2nd pattern which concerns on embodiment. It is a figure which illustrates the result of having calculated the degree of similarity between the attribute candidate and a common vocabulary with respect to the semi-structured data of the 2nd pattern which concerns on this embodiment. It is a figure which shows the method of discriminating the attribute name and the attribute value in the semi-structured data of the 2nd pattern which concerns on embodiment. It is a figure which shows the output example of the structural data and the schema with respect to the semi-structured data of the 2nd pattern which concerns on embodiment. It is a 1st flowchart which shows the creation process of the structural data and the schema which concerns on embodiment. 2 is a second flowchart showing a process of creating structural data and a schema according to an embodiment. It is a 3rd flowchart which shows the creation process of the structural data and schema which concerns on embodiment.

Hereinafter, an example of the embodiment of the present invention will be described.
FIG. 1 is a diagram showing a functional configuration of the data processing device 1 according to the present embodiment.
The data processing device 1 is an information processing device such as a server device or a personal computer, and various functions of the present embodiment are realized by the control unit executing predetermined software (data processing program) stored in the storage unit. To do. Further, the data processing device 1 includes an input / output device and a communication interface, receives semi-structured data to be processed as input, and creates and outputs structural data and a schema by various functions of the present embodiment.

For example, the local government publishes information on facilities that provide childcare services in a PDF file that includes some tabular data. In the data processing device 1, although the tabular data is described in this way, the attribute name and the attribute value are not clearly distinguished, and the semi-structured data that cannot be directly imported into the RDB is divided into the structural data. Create a schema automatically so that it can be imported into RDB.

The data processing device 1 is common to the data acquisition unit 11, the attribute candidate extraction unit 12, the attribute estimation unit 13, the structural data creation unit 14, the schema creation unit 15, the memory unit 16, and the rule management unit 17. A vocabulary storage unit 18 and a setting unit 19 are provided.

When the data acquisition unit 11 acquires semi-structured data from a data resource including the Internet, the data acquisition unit 11 converts the semi-structured data into general-purpose data using an OCR (Optical Character Recognition / Reader) tool. The general-purpose data is, for example, XML format data, and is preferably converted into text format data.
Further, the data acquisition unit 11 extracts a part of the table structure from the converted general-purpose data using a specific tag (for example, <table>) in the XML format as a mark, and transfers it to the attribute candidate extraction unit 12 and the memory unit 16. To do.

The attribute candidate extraction unit 12 calculates the frequency at which each word included in the table structure part of the data transferred from the data acquisition unit 11, that is, the general-purpose data appears, and from the histogram of this frequency, the frequency is the highest. Extract a word as an attribute candidate. The attribute candidate extraction unit 12 transfers the extracted word to the attribute estimation unit 13.
Further, when the attribute candidate is not adopted by the attribute estimation unit 13 described later, the attribute candidate extraction unit extracts a word having a frequency one step lower in frequency as a new attribute candidate and provides it to the attribute estimation unit 13. ..

The attribute estimation unit 13 is being developed by the attribute candidate words transferred from the attribute candidate extraction unit 12 and the common vocabulary stored in the common vocabulary storage unit 18 (for example, the Information-technology Promotion Agency (IPA)). Find the degree of similarity with the words in the glossary for unifying the notation), and estimate that the words above the threshold are attribute names. The common vocabulary is, for example, a glossary of terms used by the Information-technology Promotion Agency (IPA) to unify the notation.

Further, the attribute estimation unit 13 refers to the XML format data stored in the memory unit, and updates the word estimated as the attribute name by the method described later based on the table structure.
Specifically, the attribute estimation unit 13 determines that the word arranged in the combination having the highest occurrence rate of the word estimated as the attribute name among the predetermined row or column combinations based on the format of the table structure is determined as the attribute name. To do. Here, the format of the table structure is determined based on, for example, the frequency of appearance of the words extracted as attribute candidates.
The attribute estimation unit 13 transfers the word estimated as the attribute name to the structural data creation unit 14.

The structure data creation unit 14 arranges the word estimated as the attribute name transferred from the attribute estimation unit 13 on the first line, and sets the data excluding the attribute name from the general-purpose data stored in the memory unit 16 as the attribute value. Judge and arrange in association with the attribute name to create structural data. The structural data creation unit 14 transfers the structural data to the schema creation unit 15.

The schema creation unit 15 defines the word in the first line of the structural data transferred from the structural data creation unit 14 as an attribute name, determines the data format by referring to a common vocabulary as necessary, and creates an RDB schema. To do. The schema creation unit 15 outputs the created RDB schema and structural data.

The memory unit 16 stores general-purpose data created by the data acquisition unit 11 and provides it to the attribute estimation unit 13, the structural data creation unit 14, and the like.

The rule management unit 17 resembles, for example, the acquisition URL, acquisition interval, conversion rule, etc. of the semi-structured data in the data acquisition unit 11 and the common vocabulary in the attribute estimation unit 13 as rules for automatic generation processing of structural data and schema. It manages the threshold value of the degree.

The common vocabulary storage unit 18 stores a common vocabulary that serves as a template for the RDB schema. As the common vocabulary, in addition to the standardized one, similar words may be set by referring to a dictionary or the like on the Internet.

The setting unit 19 provides an interface for the administrator to set rules and a common vocabulary, and provides the input rules to the rule management unit 17 and the common vocabulary to the common vocabulary storage unit 18.

Here, a pattern of semi-structured data to be input to the data processing device 1 will be illustrated.
Semi-structured data can be classified into the following cases according to the correspondence between the described attribute name and the attribute value.

FIG. 2 is a diagram illustrating semi-structured data of the first pattern that is input to the data processing device 1 according to the present embodiment.
In this example, in the list on the second page following the text on the first page, the attribute name and the attribute value have a one-to-many correspondence. For example, a plurality of attribute values ("A nursery school", "B nursery school", ...) Are described for one attribute name "implementation facility".
However, in this data, although the attribute name and the attribute value are described, there is no definition as to which word is the attribute name and which word is the attribute value, so the attribute name and the attribute value are clearly distinguished. Semi-structured data that has not been created.

FIG. 3 is a diagram illustrating semi-structured data of a second pattern that is input to the data processing device 1 according to the present embodiment.
In this example, there is a one-to-one correspondence between the attribute name and the attribute value in the plurality of tables following the title of the information. For example, for one attribute name "name" appearing in the first table from the top, one corresponding attribute value "AA nursery school" is described. Similarly, the attribute value "BB nursery school" is associated with the attribute name "name" appearing in the second table from the top on a one-to-one basis.
However, in this data, although the attribute name and the attribute value are described, there is no definition as to which word is the attribute name and which word is the attribute value, so the attribute name and the attribute value are clearly distinguished. Semi-structured data that has not been created.

In addition, the input semi-structured data can be classified as follows depending on the number of table structures included in the data file and whether or not the attribute name is shared.
(Case 1) A single table structure is described in one file (semi-structured data).
(Case 2) A plurality of table structures are described in one file.
(Case 2-1) A plurality of table structures sharing attribute names are described in one file.
(Case 2-2) A plurality of table structures that do not share attribute names are described in one file.

Next, the procedure for creating the structural data and the schema by the data processing device 1 will be described in detail for each pattern of the table structure.

[Example 1]
The first embodiment is a case where the semi-structured data (FIG. 2) of the first pattern in which the attribute name and the attribute value are one-to-many is input.
First, the data acquisition unit 11 extracts a part of the table structure from the semi-structured data.

FIG. 4 is a diagram illustrating general-purpose data for the semi-structured data of the first pattern according to the present embodiment.
Here, the data obtained by extracting the part of the table structure from the data in the text format and the XML format in which the attribute name and the attribute value are described in a one-to-many pattern (FIG. 2) is shown.
In this example, the part specified by the <table> tag in the XML format data is extracted as the part of the table structure.
The text format data may be created by deleting the tag from the XML format data.

Next, the attribute candidate extraction unit 12 extracts words in the table structure and creates a histogram based on the frequency of occurrence for each word.

FIG. 5 is a diagram illustrating a histogram for the semi-structured data of the first pattern according to the present embodiment.
Words such as "implementation facility" and "location / contact" are extracted from the table structure part (FIG. 4) in the general-purpose data, and the frequency of appearance in the data is recorded for each word (A).
Then, the number of words and the frequency (frequency × number of words) are totaled for each frequency of once, twice, ..., And a histogram (B) is created.

In the one-to-many pattern of the attribute name and the attribute value, the attribute name appears only once and the attribute value often appears a plurality of times, so that the number of words and the frequency of 1 are high. Therefore, in this example, the attribute candidate extraction unit 12 extracts 46 words having a frequency of 1 having the highest frequency of 46 as attribute candidates.

Subsequently, the attribute estimation unit 13 refers to the common vocabulary storage unit 18, and the attribute candidates are narrowed down to words having a high degree of similarity to the common vocabulary.

FIG. 6 is a diagram illustrating the result of calculating the similarity between the attribute candidate and the common vocabulary with respect to the semi-structured data of the first pattern according to the present embodiment.
In the figure, words such as "implementation facility" and "location / contact" arranged horizontally are attribute candidates extracted by the attribute candidate extraction unit 12, and vertically arranged "identification information" and "group code". Words such as are common vocabularies.
In this example, the attribute estimation unit 13 calculates the number of characters that match any character in the common vocabulary among the character strings of the attribute candidates, and sets them as the degree of similarity. The index of similarity is not limited to this, and may be another index such as the Levenshtein distance.

The attribute estimation unit 13 estimates, for example, a word having two or more matching characters as an attribute name. In this example, among multiple attribute candidates, the attributes are "implementation facility", "target age", "implementation date and childcare hours", and "usage fee", which have a similarity of 2 or more to any of the common vocabularies. Presumed to be the name.

Then, the attribute estimation unit 13 finally determines the attribute name and the attribute value by using the feature of the table structure that the attribute name and the attribute value are one-to-many.

FIG. 7 is a diagram showing a method of determining an attribute name and an attribute value in the semi-structured data of the first pattern according to the present embodiment.
In the semi-structured data of the first pattern, the attribute name is described in either k = 1 (first row) or l = 1 (first column) of each data A _kl in the table structure.
Therefore, the attribute estimation unit 13 determines from the first row or the first column, the one having the higher estimated attribute ratio indicating the ratio of the data estimated to be the attribute name is the attribute name. In this example, the word group on the first line (estimated attribute rate = 80%) is finally determined as the attribute name, and a new "location / contact destination" is added to the estimated attribute name.

FIG. 8 is a diagram showing an output example of structural data and a schema for the semi-structured data of the first pattern according to the present embodiment.
The structure data creation unit 14 arranges the word determined to be the attribute name in the first line, and arranges the data obtained by removing the attribute name from the general-purpose data as the attribute value in association with the attribute name. The correspondence between the attribute name and the attribute value is determined based on the tag described in the XML data.

Further, the schema creation unit 15 creates an RDB schema by referring to the data format defined in the common vocabulary in order to define the attribute name arranged in the first line of the structural data.
Specifically, as the data format of the attribute name, the data format of the common vocabulary calculated to have high similarity may be adopted. For example, as shown in FIG. 6, the “target age” has a high degree of similarity to the common vocabulary “service_target [available age]”. Therefore, the "character string" which is the data format of "service_target [usable age]" is adopted. In addition, when the degree of similarity with multiple common vocabularies is high, such as "implementation facility", the data format of the common vocabulary with the highest degree of similarity is used, and when there are multiple highest degree of similarity, these data are used. Many of the formats may be adopted. If there are no similar words in the common vocabulary, a specific data format (for example, a character string) may be adopted as a default.

[Example 2]
The second embodiment is a case where the semi-structured data (FIG. 3) of the second pattern in which the attribute name and the attribute value are one-to-one is input.

FIG. 9 is a diagram illustrating general-purpose data for the semi-structured data of the second pattern according to the present embodiment.
Here, the data obtained by extracting the part of the table structure from the data in the text format and the XML format in which the attribute name and the attribute value are described in a one-to-one pattern (FIG. 3) is shown.

FIG. 10 is a diagram illustrating a histogram for the semi-structured data of the second pattern according to the present embodiment.
Words such as "facility information" and "group code" are extracted from the table structure portion (FIG. 9) of the general-purpose data, and the frequency of appearance in the data is recorded for each word (A).
Then, the number of words and the frequency (frequency × number of words) are totaled for each frequency of once, twice, ..., And a histogram (B) is created.

In a pattern in which the attribute name and the attribute value are one-to-one, the attribute name often appears n times, which is the same as the number of table structures, so that the number of words and the frequency of two or more specific frequencies n are high. Therefore, in this example, the attribute candidate extraction unit 12 extracts 14 words having a frequency of 4 having the highest frequency of 56 as attribute candidates.

FIG. 11 is a diagram illustrating the result of calculating the similarity between the attribute candidate and the common vocabulary with respect to the semi-structured data of the second pattern according to the present embodiment.
In this example, as in the first embodiment, the attribute estimation unit 13 calculates the number of characters that match any character in the common vocabulary among the character strings of the attribute candidates, and sets the degree of similarity.
The attribute estimation unit 13 estimates, for example, a word having two or more matching characters as an attribute name. In this example, "group code", "group name", "type", "installer", "name", "group code", "group name", "type", "group code", "group name", "type", "group code", "group name", "type", "group code", "group name", "type", "group code", "group name", "type", "group code", "group name", "type", "group code", "group name", "type", "group code", "group name", "type", "group code", "group name", "type", "group code", "group name", "type", "group code", "group name", "name" The attribute names are presumed to be "address", "phone number", "acceptance age", "capacity", "temporary custody", "school lunch", and "allergic food".

FIG. 12 is a diagram showing a method of determining an attribute name and an attribute value in the semi-structured data of the second pattern according to the present embodiment.
In the semi-structured data of the second pattern, of each data A _kl in the table structure, k is an odd row, k is an even row, l is an odd column, and l is an even column. The attribute name is described.
Therefore, the attribute estimation unit 13 uses the combination of rows or columns having the highest estimated attribute ratio indicating the ratio of data estimated to be the attribute name among the odd rows, even rows, odd columns, and even columns as the attribute name. Determine. In this example, a word group in an odd number sequence (estimated attribute rate = 86%) is finally discriminated from the attribute name.

FIG. 13 is a diagram showing an output example of structural data and a schema for the semi-structured data of the second pattern according to the present embodiment.
Similar to the first embodiment, the structural data creation unit 14 arranges the word determined to be the attribute name in the first line, and arranges the data obtained by removing the attribute name from the general-purpose data as the attribute value in association with the attribute name.
Further, the schema creation unit 15 creates an RDB schema with reference to the data format defined in the common vocabulary in order to define the attribute name arranged in the first line of the structural data, as in the first embodiment.

14 to 16 are flowcharts showing the process of creating the structural data and the schema according to the present embodiment.

In step S1, the data acquisition unit 11 acquires semi-structured data from a data resource including the Internet.

In step S2, the data acquisition unit 11 converts the acquired data into general-purpose data in text format and XML format.

In step S3, the data acquisition unit 11 determines whether or not the converted XML format data includes a table structure. If this determination is YES, the process proceeds to step S4, and if the determination is NO, the process ends.

In step S4, the data acquisition unit 11 extracts only the part of the table structure from the general-purpose data in the text format and the XML format. At this time, a plurality (n) table structures may be extracted. In this case, the data acquisition unit 11 separates the plurality of table structures.

In step S5, the data acquisition unit 11 saves the conversion result of step S2 and the individual table structure extracted in step S4 in the memory unit 16 and transfers them to the attribute candidate extraction unit 12.

In step S6, the attribute candidate extraction unit 12 extracts words from the data of the table structure, calculates the frequency at which each word appears, and creates a histogram based on this frequency.

In step S7, the attribute candidate extraction unit 12 extracts the word having the highest frequency in the created histogram, and transfers the extracted word as an attribute candidate to the attribute estimation unit 13.

In step S8, the attribute estimation unit 13 calculates the degree of similarity between the extracted word and each word of the common vocabulary storage unit 18.

In step S9, the attribute estimation unit 13 estimates and extracts words having a calculated similarity equal to or greater than a threshold value as attribute names.

In step S10, the data processing device 1 determines whether or not the attribute name has been extracted by the attribute estimation unit 13. If this determination is YES, the process proceeds to step S12, and if the determination is NO, the process proceeds to step S11.

In step S11, since the data processing device 1 does not have a word presumed to be an attribute name among the attribute candidates, the word that has already been investigated is excluded, the process is returned to step S7, and the word with the next highest frequency is used. Is extracted as an attribute candidate.

In step S12, the data processing device 1 first reads the table structure extracted in step S4 in order to create structure data, and first initializes the index i to 1.

In step S13, the data processing device 1 determines whether or not the index i exceeds the number n of the table structures. If this determination is YES, the process proceeds to step S20, and if the determination is NO, the process proceeds to step S14.

In step S14, the attribute estimation unit 13 determines whether or not the value with the highest frequency determined in step S7 is 1, and thereby classifies the pattern of the target table structure into cases. If this determination is YES, it is estimated that the attribute name and the attribute value are the first pattern of one-to-many, and the process proceeds to step S15. On the other hand, when the determination is NO, it is estimated that the attribute name and the attribute value are one-to-one in the second pattern, and the process proceeds to step S16.

In step S15, since the table structure is the first pattern, the attribute estimation unit 13 compares the first row and the first column, and determines the one with the higher estimated attribute ratio as the attribute name. The attribute estimation unit 13 transfers the word determined to be the attribute name to the structural data creation unit 14.

In step S16, since the table structure is the second pattern, the attribute estimation unit 13 compares odd-numbered rows, even-numbered rows, odd-numbered columns, and even-numbered columns, and determines that the one with the highest estimated attribute ratio is the attribute name. The attribute estimation unit 13 transfers the word determined to be the attribute name to the structural data creation unit 14.

In step S17, the structural data creation unit 14 arranges a word determined to be an attribute name on the first line, and creates structural data in a predetermined format conforming to the RDB.

In step S18, the structural data creation unit 14 determines the data obtained by removing the attribute name from the general-purpose data stored in the memory unit 16 as the attribute value, and arranges the data in the structural data to be created. The structural data creation unit 14 transfers the created structural data to the schema creation unit 15.

In step S19, the data processing device 1 counts up the index i, and the process returns to step S13.

In step S20, the structural data creation unit 14 determines whether or not the number n of the table structures extracted in step S4 is 1, and thereby, the semi-structured data to be processed is converted into the above-mentioned case 1 (single table). ) And case 2 (plurality of tables). If this determination is YES, case 1 occurs and the process proceeds to step S23. On the other hand, if the determination is NO, case 2 occurs and the process proceeds to step S21.

In step S21, the structural data creation unit 14 determines whether or not the attribute names of the structural data (tables) created for each of the n table structures are common. If this determination is YES, the semi-structured data corresponds to the above-mentioned case 2-1 and the process proceeds to step S22. On the other hand, when the determination is NO, the semi-structured data corresponds to the above-mentioned case 2-2, and the structural data creating unit 14 determines that each has independent structural data, and the process proceeds to step S23.

In step S22, the structural data creation unit 14 sets the common attribute name as the first line, sets the attribute value in the second and subsequent lines, and based on the tags (for example, <TR> and <TD>) described in the XML data. Arrange in order and integrate n structural data.

In step S23, the schema creation unit 15 creates a schema with reference to the data format of the common vocabulary, with the first line of the structural data as the attribute name.

In step S24, the schema creation unit 15 outputs the created structural data and the schema.

According to the present embodiment, the data processing device 1 calculates the frequency at which each word appears in the part of the table structure of the semi-structured data, and from the histogram of this frequency, the word with the highest frequency is selected as an attribute candidate. Extract as. Then, the data processing device 1 determines the word arranged in the combination having the highest appearance rate of the attribute candidate among the predetermined row or column combinations based on the format of the table structure as the attribute name.
As a result, the data processing device 1 can automatically determine whether the word included in the table structure is an attribute name or an attribute value, and from the semi-structured data in which the attribute name and the attribute value are not clearly distinguished, Structural data and schema can be created automatically.
As a result, semi-structured data that does not have a schema and has attribute names and attribute values but is not clearly distinguished can be automatically imported into an RDB and used for analysis.

The data processing device 1 can extract more reliable candidates as attribute names by adopting only the attribute candidates whose similarity with the standard word existing in the common vocabulary storage unit 18 is equal to or higher than the threshold value, and the attribute names and attributes can be extracted. The accuracy of value discrimination can be improved.
At this time, when the data processing device 1 cannot obtain an attribute candidate similar to the common vocabulary, the data processing device 1 newly extracts a word having a frequency one step lower in the histogram as an attribute candidate.
As a result, the data processing device 1 can correctly adjust and determine the attribute name even if there is an error in the estimation of the attribute name based on the frequency.

The data processing device 1 determines the format of the table structure, such as whether the attribute name and the attribute value are one-to-many or one-to-one, based on the frequency of appearance of words extracted as attribute candidates from the table structure. ..
Therefore, the data processing device 1 can automatically extract the features of the table structure from the histogram, and can distinguish between the attribute name and the attribute value without limiting the format of the table structure that can be processed.

The data processing device 1 converts the semi-structured data into general-purpose data including the XML format, and extracts a part of the table structure by a specific tag in the XML format.
As a result, the data processing device 1 can efficiently extract the part of the table structure by using the XML tag, and can easily grasp the correspondence between the attribute name and the attribute value to create the structural data. ..

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. In addition, the effects described in the above-described embodiments are merely a list of the most preferable effects arising from the present invention, and the effects according to the present invention are not limited to those described in the embodiments.

In the above-described embodiment, the standard attribute name is stored in advance in the common vocabulary storage unit 18 as a common vocabulary, but the reference form of the common vocabulary is not limited to this. For example, it may be in the form of referencing an external database.
In addition, an existing schema may be used as a common vocabulary.

The data processing method by the data processing device 1 is realized by software. When realized by software, the programs that make up this software are installed in the information processing device (computer). Further, these programs may be recorded on a removable medium such as a CD-ROM and distributed to the user, or may be distributed by being downloaded to the user's computer via a network. Further, these programs may be provided to the user's computer as a Web service via a network without being downloaded.

1 Data processing device 11 Data acquisition unit 12 Attribute candidate extraction unit 13 Attribute estimation unit 14 Structural data creation unit 15 Schema creation unit 16 Memory unit 17 Rule management unit 18 Common vocabulary storage unit 19 Setting unit

Claims (7)

A data acquisition unit that converts semi-structured data into general-purpose data and extracts the table structure part from the general-purpose data.
An attribute candidate extraction unit that calculates the frequency at which each word included in the table structure appears and extracts the word with the highest frequency as an attribute candidate from the histogram of the frequency.
Of the predetermined row or column combinations based on the table structure format, the attribute estimation unit that determines the word arranged in the combination with the highest occurrence rate of the attribute candidates as the attribute name, and
A structural data creation unit that creates structural data by associating the general-purpose data excluding the attribute name with the attribute name as an attribute value.
A data processing device including a schema creation unit that creates a schema in which the attribute name is defined. The data processing device according to claim 1, wherein the attribute estimation unit employs only the attribute candidates whose similarity with words existing in a predetermined common vocabulary is equal to or higher than a threshold value. The data processing device according to claim 2, wherein the attribute candidate extraction unit extracts a word having a frequency one step lower as the attribute candidate when the attribute candidate is not adopted by the attribute estimation unit. The data processing device according to any one of claims 1 to 3, wherein the attribute estimation unit determines the format of the table structure based on the frequency of words extracted as the attribute candidates. The data acquisition unit converts the semi-structured data into the general-purpose data including the XML format, and extracts a part of the table structure by a specific tag in the XML format according to any one of claims 1 to 4. The data processing device described. A data acquisition step that converts semi-structured data into general-purpose data and extracts a part of the table structure from the general-purpose data.
An attribute candidate extraction step that calculates the frequency at which each word included in the table structure part appears and extracts the word with the highest frequency as an attribute candidate from the histogram of the frequency.
Among the predetermined row or column combinations based on the table structure format, the attribute estimation step for determining the word arranged in the combination having the highest occurrence rate of the attribute candidate as the attribute name, and
A structural data creation step of creating structural data by associating the general-purpose data excluding the attribute name with the attribute name as an attribute value.
A data processing method in which a computer executes a schema creation step for creating a schema in which the attribute name is defined. A data acquisition step that converts semi-structured data into general-purpose data and extracts a part of the table structure from the general-purpose data.
An attribute candidate extraction step that calculates the frequency at which each word included in the table structure part appears and extracts the word with the highest frequency as an attribute candidate from the histogram of the frequency.
Among the predetermined row or column combinations based on the table structure format, the attribute estimation step for determining the word arranged in the combination having the highest occurrence rate of the attribute candidate as the attribute name, and
A structural data creation step of creating structural data by associating the general-purpose data excluding the attribute name with the attribute name as an attribute value.
A data processing program for causing a computer to execute a schema creation step for creating a schema in which the attribute name is defined.

Images