JP6804913B2 - Table structure estimation system and method - Google Patents

Description

The present invention relates to structure estimation systems and methods for tables with unclear structures.

Data is being released as a public asset. This is the so-called openness of data. For example, useful data released by the national government, ministries, and public organizations includes statistical data such as sales, purchasing information, and investment information by prefecture and industry. On the other hand, there is an increasing need for effective use of publicly available data. For example, in a company or the like, useful findings can be obtained by searching publicly available statistical data and analyzing it in combination with data in an organization. Currently published data is often provided in tabular format such as Excel (trademark) or comma-separated values (CSV). Hereinafter, the data provided in the tabular format will be simply referred to as a "table".

The published table is created by a structure unilaterally determined by the data creator or provider. Also, the table itself often does not contain information about the structure. Therefore, a third party who intends to use the data must use the data of various structures without having knowledge about the structure of the table. However, it is difficult to use the data in a table whose structure is unclear without human intervention. That is, it is difficult to mechanically perform processing such as searching the published data by an information processing device such as a computer.

Here, the "structure" of the table means what the table represents "what" in "what viewpoint". The structure can be defined by "dimension" and "measure". The above "what" corresponds to "measure", and "which viewpoint" corresponds to "dimension". If the structure of the table is clear, the data in the table can be searched mechanically based on the structure of the table. In addition, by associating the table structure with the natural language expression, it is considered that data retrieval in the table using natural language can be realized relatively easily. On the other hand, if it is unclear, there are multiple possible associations between the table structure and natural language expressions, making it difficult to search using natural language. In order to realize table data retrieval, a technique for automatically estimating and clarifying the structure is required.

As a technique for estimating the structure of a table whose structure is unclear, there is one described in Non-Patent Document 1. In this document, it is stated that for each column of the table, what is represented is estimated by a machine learning method using the notation appearing in the table and the positional relationship with other columns.

JP-A-2015-41225

Goel, A., Knoblock, C.A., Lerman, K .: Exploiting Structure within Data for Accurate Labeling using Conditional Random Fields. In: Proceedings of the 14th international conference on Artificial Intelligence, ICAI (2012)

Non-Patent Document 1 does not assume that anything other than the table is used. Therefore, only the table is used to estimate the structure. For example, the notation and positional relationship that appear in the table. In this case, if the table does not include features for estimating the structure, the estimation becomes difficult.

Further, Patent Document 1 discloses a technique for associating highly co-occurrence text information with an image, but it is not a technique for handling a table.

An object of the present invention is to provide a technique for automatically estimating and clarifying the structure of a table with high accuracy even when the table does not include features for estimating the structure.

One aspect of the present invention that solves the above problems is a table structure estimation system including an input device, an output device, a storage device, and a processing device. In this system, the storage device stores feature definition data that defines features related to both tables and sentences. The input device receives the data to be analyzed. The processing device acquires a table from the data to be analyzed and also acquires sentences related to the table as related texts. The feature is a feature from the acquired table and related texts using the related text information extraction unit and the feature definition data. It is provided with a feature extraction unit for extracting data and an identification unit for estimating the structure of the table based on the feature extraction result of the feature extraction unit.

Another aspect of the present invention is a method of estimating a table structure using an input device, an output device, a storage device, and a processing device. In this method, the storage device prepares feature definition data that defines features related to both tables and sentences. Input the analysis target data from the input device. The processing device acquires a table from the data to be analyzed, acquires sentences related to the table as related texts, and extracts features from the acquired table and related texts using the feature definition data. Estimate the distinction between "dimension" and "measure" of table items based on their characteristics.

According to the present invention, it is possible to estimate the structure of the table, and as a result, the accuracy of the estimation is improved. Issues, configurations and effects other than those described above will be clarified by the description of the following embodiments.

Conceptual diagram showing an example of the table to be processed Conceptual diagram of table structure Conceptual diagram of processing performed by the table structure estimation system of the embodiment Hardware configuration block diagram of the system of the embodiment Block diagram of the pre-processing unit Conceptual diagram explaining the correspondence between tables and sentences Table diagram showing an example of feature definition data Flow diagram showing the flow of training data processing Conceptual diagram showing the discriminative results and discriminative model of training data Flow diagram showing the flow of feature extraction processing Flow diagram showing the processing flow of the data to be analyzed Table diagram showing other examples of discriminative models Table diagram showing an example of identification processing using an identification model Conceptual diagram of search processing in natural language

The embodiment will be described in detail with reference to the drawings. However, the present invention is not construed as being limited to the description of the embodiments shown below. It is easily understood by those skilled in the art that a specific configuration thereof can be changed without departing from the idea or purpose of the present invention.

In the configuration of the invention described below, the same reference numerals may be used in common among different drawings for the same parts or parts having similar functions, and duplicate description may be omitted.

The notations such as "first", "second", and "third" in the present specification and the like are attached to identify the components, and do not necessarily limit the number or order. In addition, numbers for identifying components are used for each context, and numbers used in one context do not always indicate the same composition in other contexts. Further, it does not prevent the component identified by a certain number from having the function of the component identified by another number.

The position, size, shape, range, etc. of each configuration shown in the drawings and the like may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings and the like.

Components represented in the singular form herein shall include the plural form unless explicitly stated in the context.

In order to solve the problems described above, in a typical example of the method and system of the present invention, the structure is estimated by a method using a feature that combines a table and related documents of the table. Examples include a table structure estimation method and a table structure estimation system. In the examples described below, in addition to the table, documents related to the table are also used to estimate the structure of the table.

<1. Explanation of table data>
Prior to the specific description of the embodiment, an example of the table to be processed by the present embodiment will be described.

FIG. 1 is an example of a table to be processed by the table structure estimation system of the embodiment. FIG. 1 shows an example of report 100, which is the data to be analyzed. Report 100 includes Table 101, Table Title 102, and Description 103. Table 101 shows the population of each age group corresponding to the year. Title 102 shows that the table shows the population by year and age. The description 103 contains descriptions related to Table 101. Hereinafter, information including text accompanying the table 101, such as the title 102 and the explanatory text 103, will be referred to as "text" or "text information". When the term "sentence" or "text information" is used in the present specification, it may be a plurality of sentences or a single sentence.

Here, the data format of the table, title, and description is not particularly limited as long as it can be recognized as text and can identify the positional relationship, correspondence relationship, or arrangement relationship with the table. For example, it is data including texts such as Excel (trademark) and CSV. Alternatively, it may be image data or paper data in bitmap format. In the case of image data, the character portion may be processed by optical character recognition (OCR) or the like to obtain text data.

FIG. 2 shows the concept of the structure in Table 101. The cells (frames) 201 to 203 at the top of Table 101 are "items". In the "item", the "dimension" and "measure" which are the structures of the table are stored. The "measure" is the observation target, and the "dimension" is the observation condition. In this example, the "year" in cell 201 and the "age" in cell 202 are "dimensions". The "population" in cell 203 is a "measure". "Item values" are stored in cells other than "items" in Table 101. For the item "year" in cell 201, "dimensional values (year values)" are stored in cells 204 and 205. For the item "age" in cell 202, "dimensional value (age value)" is stored in cells 206 and 207. For the item "population" in cell 203, the observed value "measure value (population value)" is stored in cells 208 and 209. Note that the item cell is generally the row of the top cell of the table or the column of the leftmost cell, so this part may be estimated as an item by default.

The purpose of the table structure estimation system of this embodiment is to automatically estimate the above structure from the table. For example, from the items of "year", "age", and "population", "population by year and age" and "age (average) and population by year" can be considered. In the former case, "year" and "age" are "dimensions" and "population" is "measure". In the latter case, the "year" is the "dimension" and the "age" and "population" are the "measures".

In this embodiment, even when the structure of Table 101 is unknown, from Table 101, the title 102, and the explanatory text 103, for example, the “year” and “age” in FIG. 2 are “dimensions” and the “population”. Estimate that is a "measure". Even if the item is omitted, it is estimated whether the item corresponding to each column is "dimension" or "measure".

<2. Outline of processing according to this embodiment>
FIG. 3 shows the concept of processing performed by the table structure estimation system of this embodiment. A typical example of this system employs a supervised machine learning method that uses not only tables but also items in table titles and explanations (text information) and how their values appear. This improves the estimation accuracy of the table structure.

In the table structure estimation system of this embodiment, training data 301 is used. The training data 301 includes a table 101T, a table title 102T, and a description 103T. Further, structural data 302 is added to the training data 301. The structural data 302 includes the contents of the target data, items that are dimensions, values of dimensions, items that serve as measures, and the like. The structural data can be stored as data by the operator visually analyzing and inputting the training data 301.

In this embodiment, a plurality of training data 301 and a plurality of structural data 302 corresponding thereto will be used. In general, the larger the number of training data 301, the better the estimation accuracy.

Using the training data 301 and the structural data 302 prepared as described above, features are extracted and an identification model is generated by a series labeling method. The discriminative model defines the relationship between the feature extraction results and the structure of the table. The discriminative model 303 includes, for example, a dimensional model relating to the dimensional structure and a measure model relating to the structure of the measure. After preparing the discriminative model 303 as described above, the analysis target data 304 can be input and the analysis can be performed based on the discriminative model 303 to obtain the structure estimation data 305. The configuration of the embodiment will be specifically described below.

<3. Overall system configuration of the embodiment>
FIG. 4 is a block diagram showing a configuration of a table structure estimation system, which is an example of an embodiment of the present invention. As a specific example, the table structure estimation system 1 is configured by an information processing device such as a computer. Similar to a normal information processing device, the table structure estimation system 1 includes a central processing unit (CPU) 11, a known input / output device 13 such as a keyboard and an image monitor, and a memory 15 composed of a magnetic disk device and a semiconductor storage device. Be prepared. Further, the data communication unit 12 may be provided as an interface for exchanging data with the outside. The data communication unit 12 is connected to, for example, an external network 16. The term "input / output device" does not mean only a device having both input / output functions, but also includes a device having only an input function, a device having only an output function, and a device having both input / output functions. It shall mean.

In this embodiment, functions such as calculation and control are realized by executing a program stored in the memory 15 by the CPU 11 in cooperation with other hardware. A program executed by the CPU 11, a function thereof, or a means for realizing the function may be referred to as a "function", a "means", a "part", a "unit", a "module", or the like. In FIG. 4, a function conceptually executed by the CPU 11 based on software is shown as a control unit 14. The control unit 11 includes a preprocessing unit 141, a learning unit 142, an identification unit 143, and a display unit 144. The program for realizing these functions is stored in the memory 15. Further, as data, training data 301, structure data 302, identification model 303, analysis target data 304, structure estimation data 305 which is an identification result, feature definition data 306, dictionary 307, and the like are stored in the memory 15.

The training data 301, the structural data 302, the analysis target data 304, the feature definition data 306, and the dictionary 307 stored in the memory 15 can be input via the input / output device 13 and the data communication unit 12. Further, the identification model 303 and the structure estimation data 305 can be output via the input / output device 13 and the data communication unit 12.

The above configuration may be configured by a single computer as shown in FIG. 4, or another computer in which any part of the input device, output device, processing device, and storage device is connected by a network. It may be composed of. Further, in this embodiment, the same function as the function configured by software can be realized by hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit). Such aspects are also included in the scope of the present invention.

<4. Description of the data structure of the system of the embodiment>
The data stored in the memory 15 will be described.

The training data 301 is data including the table 101T and the text information accompanying the table 101T as described in FIG. The text information is, for example, a table title 102T, a description 103T, and the like. In this embodiment, the training data 301 is text-based data, and includes information indicating the positional relationship between Table 101T and the text information accompanying the table 101T. The information indicating the positional relationship is, for example, the coordinate information of Table 101T and the text information accompanying the table 101T. Alternatively, it is information indicating that Table 101T and the text information accompanying the table 101T are included in the same file. Alternatively, it is information indicating that Table 101T and the text information accompanying the table 101T have a specific relationship, for example, are included in the same page of an Excel ™ file.

The structural data 302 is data showing the structure of Table 101T of the training data, as described in the concept in FIG. The structural data 302 defines, for example, the distinction between "dimension" and "measure" for the "item name" in Table 101T. Further, the values of "dimension" and "measure" may be specified. It may also include information about the name and content of the table. The structural data 302 is generated by the operator visually and analyzing the training data 301.

The discriminative model 303 is a model generated by using the training data 301 and the structural data 302. The generation and use of the discriminative model will be described later.

The analysis target data 304 is table data for which the table structure for which the structure should be estimated is unknown, as shown in FIG. The data structure may be the same as that of the training data 301, and an example is the structure shown in the report 100 of FIG.

The structure estimation data 305 is data showing the table structure of the analysis target data 304, which is the result of analyzing the analysis target data 304 based on the identification model 303. The data structure may be the same as that of the structural data 302.

The feature definition data 306 is data that defines features used for estimating the structure of the table. For example, the operator may generate it based on experience and input it from the input / output device 13. Alternatively, it may be automatically generated.

The dictionary 307 is used for processing the related text information extraction unit 1412, which will be described later.

<4. Description of the control unit of the system of the embodiment>
The control unit 14 will be described.

<4-1. Pretreatment section>
FIG. 5 is a functional block diagram of the preprocessing unit 141. The preprocessing unit 141 includes a table-text correspondence unit 1411, a related text information extraction unit 1412, a related text information interpretation unit 1413, a feature definition unit 1414, and a feature extraction unit 1415.

The table-sentence correspondence unit 1411 extracts the text (text information) corresponding to the table. Extraction can be performed, for example, on the condition that the table and the text have a specific arrangement relationship. For example, when a table and a sentence are arranged in contact with each other or are on the same page, the sentence can be extracted. Alternatively, when a sentence is embedded inside the table, such as a title, the sentence can be extracted. The table-text correspondence unit 1411 performs the above processing on the training data 301 and the analysis target data 304. By associating the table with the text, it is possible to select and extract the part corresponding to the table from the text information attached to the table. However, this process can be omitted and all the text information attached to the table can be used in the subsequent processes.

The related text information extraction unit 1412 extracts as related text information the item in the table or the sentence in which the value of the item appears in all the text information extracted by the table-sentence correspondence unit 1411 or attached to the table. .. At this time, synonyms of items are treated in the same way as items. For this purpose, a dictionary (thesaurus) 307 can be used to extract synonyms and synonyms. Similarly, abbreviations for items and corresponding foreign languages can be treated in the same way as items. Further, as the value of the item, the one with the unit added and the one without the unit can be treated in the same way. In addition, units exhibiting the same physical quantity and properties, such as kilogram (kg) and ton (t), yen and US dollar, can be treated in the same manner. In addition, the presence or absence of a multiplier can be treated in the same way. These related words may be registered in the dictionary 307.

FIG. 6 shows an example of extracting related text information. The title 102 is extracted because there is a sentence 601 including the items "year", "age", and "population". From the explanatory text 103, a sentence 602 including the "population" and a sentence 603 including the item "population" and the value of the item "2010", "65 years old", and "610 people" are extracted. The extracted sentences may be one or more. Further, the extracted sentence may be composed of one sentence or may be composed of a plurality of sentences.

The related text information interpretation unit 1413 specifies the word order, subject, predicate, object, modifier, and the like of the text information sentence. It can be configured with known document analysis software that performs parsing of natural language, such as used in text mining.

The feature definition unit 1414 defines the features of the table or text and generates the feature definition data 306.

FIG. 7 shows an example of the feature definition data 306. The example of FIG. 7 is definition data created by the operator based on an empirical rule. The feature definition data 306 includes information such as an ID 701 that uniquely indicates the feature, a feature target 702, and a feature content 703. As shown in FIG. 7, the features include features targeting tables, such as "feature 1" and "feature 2", and features targeting tables and sentences, such as "feature 3" and "feature 4". ..

For example, "feature 1" for a table is a feature relating to the position of an item in the table, and includes "the item appears on the right side of the table". For example, "feature 3" for a table and a document is a feature relating to the structure of an item and a sentence, and includes "the item is the subject and is modified". Further, for example, "feature 4" for a table and a sentence is a feature relating to an item value and a sentence structure, and includes "the item value is included in the predicate".

The feature definition unit 1414 defines the above features and stores them in the memory 15 as feature definition data 306. The feature can be created by the operator based on an empirical rule and input from the input / output device 13 or the data communication unit 12, for example. Alternatively, it is also possible to automatically generate features comprehensively and automatically by deleting features that do not contribute to estimation in the processing of the learning unit described later.

The feature extraction unit 1415 also uses the interpretation result of the related text information interpretation unit 1413 to determine whether or not the feature defined by the feature definition unit 1414 applies to each item of the training data 301 or the analysis target data 304. Features are judged for each item in the table.

<4-2. Learning Department>
The learning unit 142 performs a process of generating the discriminative model 303 by using the training data 301 and the structural data 302.

FIG. 8 shows a processing flow of processing performed by the learning unit 142.

In the process S801, for example, the training data 301 as shown in FIG. 3 is associated with the table 101T, the title 102T, and the explanatory text 103T by using the related text information extraction unit 1412. The concept of association has been described in FIG. At this time, as described above, only the necessary part may be selected and extracted from the text information of the training data 301 by using the table-text correspondence unit 1411 and processed.

In the process S802, a correct answer is given to each training data. That is, for example, for each item in Table 101T of FIG. 3, structural data 302 for specifying the "dimension" and "measure" is input. As described above, the operator may visually determine the table 101T and input the input from the input / output device 13. The structure data 302 is stored in the memory 15. The process S802 may be performed before the process S801 or after the process S803.

In the process S803, the features of the training data 301 are extracted by using the related text information interpretation unit 1413 and the feature extraction unit 1415 based on the feature definition data 306.

FIG. 9 is a conceptual diagram illustrating the feature extraction of the training data 301. For each item in the table included in the training data 301, it is determined whether or not the feature content 703 of the feature definition data 306 of FIG. 7 is applicable, and the feature is extracted. In the example of FIG. 9, in the feature extraction result 901, the number 911 is added to each item of the training data 301 for convenience, and the presence or absence of feature extraction is determined by “True” and “False” for each feature ID 701. When the feature is extracted, it is set as "True", and when the feature is not extracted, it is set as "False". At this time, since the correct answer for each item of the training data 301 is known as the structural data 302, the discriminative model 303 can be obtained by statistically processing the feature extraction result 901.

In the example shown in FIG. 9, in the discriminative model 303, models A, B, C ... Are defined as the model definition table 902 based on the appearance pattern of the feature. When there are multiple items corresponding to each model, it can be determined from the structural data 302 whether the item is "dimension" or "measure", so statistically "dimension" and "measure" in the model. Occurrence frequency 903 can be determined. The created identification model 303 is stored in the memory 15.

In the discriminative model 303 of FIG. 9, the appearance frequency 903 is used as it is for estimating the table structure as the judgment result of the judgment, but the one with the higher application frequency is selected and either "dimension" or "measure" is selected. It may be the result of the alternative.

As mentioned above, in the case where features are comprehensively and automatically generated, features that do not correlate with the appearance frequency 903 are judged to be features that do not contribute to estimation and deleted. The number can be reduced and the processing amount can be reduced.

<4-3. Feature extraction process>
FIG. 10 is a diagram showing details of the training data feature extraction process S803. The flow of FIG. 10 shows the processing for one item, and the same processing is performed for all the items in the table of the training data 301.

FIG. 10 shows an example in which feature extraction is performed with the feature definition data 306 of FIG. 7 and the feature extraction result 901 of FIG. 9 is obtained.

In process S8031, 1 is assigned to the variable N.

In the process S8032, the contents of the Nth feature are acquired from the feature definition data 306. The first feature is (feature 1).

In the process S8033, a branch process is performed according to the content of the feature. If the feature relates to both a table and a sentence, the process proceeds to S8036. If the feature is not related to both the table and the text, the process proceeds to S8034. In the example of the feature definition data 306 of FIG. 7, the features relating to both the table and the text are (feature 3) (feature 4) (feature 7) (feature 8). Features that are not related to both tables and sentences, that is, features related only to tables, are (feature 1), (feature 2), (feature 5), and (feature 6).

In the process S8034, which is a feature determination process relating only to the table, the positions of the items in the table are determined. For example, with respect to (feature 1) and (feature 5) of FIG. 7, the feature can be extracted by the process S8034.

In the process S8035, the item in the table or the value of the item is determined. For example, with respect to (feature 2) and (feature 6) of FIG. 7, the feature can be extracted by the process S8035.

In the process S8036, which is a feature determination process for both the table and the text, the related text information extracted in the process S801 is acquired.

In the process S8037, the related text information interpretation unit 1413 analyzes the sentence of the related text information.

In the process S8038, it is determined how to use the item or the value of the item in the text. That is, in the example of FIG. 7, (feature 3) is extracted by determining "whether or not the item is the subject and is modified". Further, (feature 4) is extracted by determining "whether or not the value of the item is included in the predicate".

As a result of the above processing, the feature can be extracted in the processing S8039, and the results of "True" and "False" can be obtained for the feature.

In the process S8040, the value of N is incremented, and in the process S8041, the process ends when the process is completed up to the last feature, and returns to the process S8032 when the process is not completed up to the last feature. The following features are extracted. As a result, the feature extraction result 901 shown in FIG. 9 can be obtained.

In the above description, the processing S803 of the training data 301 has been described, but the feature extraction processing S1102 of the analysis target data 304 can also be performed in the same manner.

<4-4. Identification unit>
The identification unit 143 performs a process of generating the structure estimation data 305 by using the analysis target data 304 and the identification model 303.

FIG. 11 shows a processing flow of processing performed by the identification unit 143.

In the process S1101, the table 101 is associated with the title 102 and the explanatory text 103 by using the related text information extraction unit 1412 with respect to the analysis target data 304. The specific process may be the same as that of process S801.

In the process S1102, the features of the analysis target data 304 are extracted by using the related text information interpretation unit 1413 and the feature extraction unit 1415 based on the feature definition data 306. The specific process may be the same as that of process S803. Specifically, it may be the same as that described with reference to FIG. The feature extraction result of the analysis target data 304 has the same structure as the feature extraction result 901 of the training data 301 of FIG.

In the process S1103, the feature extraction result of the analysis target data 304 is applied to the identification model 303, and the table items included in the analysis target data 304 are identified. For example, in the alternative method, the item from which the feature corresponding to the model A in FIG. 9 is extracted is determined as "dimension".

In the process S1104, the identification unit 143 sends the identification result to the display unit 144, and under the control of the display unit 144, displays the result on, for example, an image display device included in the input / output device 13. Further, the result may be transmitted from the data communication unit 12 at the same time or instead.

In the above-described embodiment, not only the table but also the table items and their values appear in the table titles and explanations as features. According to this embodiment, by using both the table and the document related to the table, the structure is estimated in the related document of the table even if the table does not contain the features for estimating the structure. If the feature is included, the structure can be estimated. As a result, the accuracy of estimation is improved. Moreover, since the discriminative model for estimating the structure can be automatically generated, the part that requires manual labor can be reduced.

The discriminative model 303 used in Example 1 uses all of the defined features. However, it is also possible to generate a discriminative model using only the features that are highly correlated with the result without using all the features. By giving a predetermined algorithm, a desired discriminative model can be generated.

For example, in FIG. 9, when the feature extraction results 901 of the training data 301 are aggregated, the model can be generated by the algorithm of "generating a predetermined discriminative model when a predetermined condition is satisfied". ..

For example, if the condition that "if a certain feature X is" True ", 90% or more of the item is" dimension "" is satisfied regardless of the result of the other feature, "feature" is satisfied. If X is "True", the discriminative model that "the item is a dimension" can be generated.

Although the above is a simple example, various conditions and discriminative models can be adopted, and a complicated discriminative model may be generated under more complicated conditions.

In the first embodiment, the learning unit 142 automatically generated the discriminative model 303 using the training data 301 and the structural data 302.

However, if it is a simple discriminative model 303, the user of the data can empirically create it manually. The created identification model 303 is input from the input / output device 13 and the data communication unit 12 and stored in the memory 15. The system configuration in this case is basically the same as that of the first embodiment, but the data used by the learning unit 142 and the learning unit 142 in FIG. 4 becomes unnecessary.

FIG. 12 shows an example of the identification model 303-2 of the third embodiment. The discriminative model 303-2 of FIG. 12 is created by the operator based on the experience side, classified and displayed. In FIG. 12, features (feature 1) (feature 2) (feature 5) (feature 6) for tables and features (feature 3) (feature 4) (feature 7) (feature 8) for tables and documents. ) Is defined. In addition, the feature (feature 1) (feature 2) (feature 3) (feature 4) for determining "measure" and the feature (feature 5) (feature 6) (feature 7) (feature 8) for determining "dimension" There is.

For example, targeting a table (feature 1) is a feature relating to the position of an item in the table, and includes "the item appears on the right side of the table". Since the item indicating "measure" tends to appear on the right side of the table, the item from which (feature 1) is extracted is determined to be "measure".

For example, targeting a table and a document (feature 3) is a feature relating to the items of the table and the structure of the sentence, and includes that "the item is the subject and is modified in the sentence". Items from which such features have been extracted are determined to be "measures".

For example, targeting a table and a sentence (feature 4) is a feature relating to the value of an item in the table and the structure of the sentence, and includes "the value of the item is included in the predicate". Items from which such features have been extracted are determined to be "measures".

The processing flow when the identification unit 143 identifies using the identification model 202-2 as shown in FIG. 12 is the same as that described in FIG.

FIG. 13 shows an example of the identification process S1003 of the analysis target data 304 in the third embodiment. In this example, each item of the table included in the analysis target data 304 is numbered 1311 for convenience. For each item, it is counted whether or not the eight features shown in FIG. 12 are extracted. For example, in item # 001, the count of the feature indicating the measure is 3, and the count of the feature indicating the dimension is 1, and in this example, the judgment is made by a majority vote, so the result is "measure". Of course, it is possible to make changes such as giving different weights to each feature rather than a simple majority vote. In addition, the eight features described are examples, and it is possible to increase or decrease the types.

FIG. 14 shows an application example of a search system in natural language using the structural information of the table obtained in Examples 1 to 3.

According to Examples 1 to 3, structural data 302 relating to the structure is added to Table 101 of the analysis target data 304. According to this structure, a searchable natural language expression 1401 is prepared. The natural language expression 1401 may be automatically generated based on a predetermined production rule, or may be created by the operator. For example, when the dimension is "prefecture" and the measure is "age" and "population", the natural language expression indicating this is "age and population by prefecture". Therefore, by presenting such a natural language expression to the user 1402 or having the user 1402 input it, it is possible to support the search for a desired table in the natural language.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add / delete / replace the configurations of other examples with respect to a part of the configurations of each embodiment.

Table: 101
Table title: 102
Description: 103

Claims (9)

A table-structured estimation system with input devices, output devices, storage devices, and processing devices.
The storage device is
Stores feature definition data that defines features related to both tables and sentences,
The input device is
It accepts data to be analyzed and
The processing device
A related text information extraction unit that acquires a table from the analysis target data and also acquires sentences related to the table as related texts.
A feature extraction unit that extracts the feature from the acquired table and the related text using the feature definition data,
Based on the feature extraction result of the feature extraction unit, the identification unit that estimates the distinction between "dimension" and "measure" of the items in the table, and the identification unit.
Equipped with a,
The related text information extraction unit
The text including the table items included in the analysis target data is acquired as the related text, and the text is obtained.
The above features
It defines how table items are used in a document.
The processing device
It has a related text information interpretation unit that analyzes the syntax of the related text.
The feature extraction unit
By determining how the items in the table are used in the related text, the features are extracted.
How to use the items in the above table
(1) The item is the subject and is qualified
(2) The item modifies the subject
Including at least one of
The storage device is
A discriminative model that defines the relationship between the feature extraction result of the feature extraction unit and the distinction between "dimension" and "measure" of the items in the table is stored.
The identification unit
Using the discriminative model, the distinction between "dimension" and "measure" of the items in the table is estimated.
Table structure estimation system. A table-structured estimation system with input devices, output devices, storage devices, and processing devices.
The storage device is
Stores feature definition data that defines features related to both tables and sentences,
The input device is
It accepts data to be analyzed and
The processing device
A related text information extraction unit that acquires a table from the analysis target data and also acquires sentences related to the table as related texts.
A feature extraction unit that extracts the feature from the acquired table and the related text using the feature definition data,
Based on the feature extraction result of the feature extraction unit, the identification unit that estimates the distinction between "dimension" and "measure" of the items in the table, and the identification unit.
Equipped with a,
The related text information extraction unit
A sentence including the value of the item in the table included in the analysis target data is acquired as the related text, and the text is obtained.
The above features
It defines how the values of table items are used in a document.
The processing device
It has a related text information interpretation unit that analyzes the syntax of the related text.
The feature extraction unit
The feature is extracted by determining how the values of the items in the table are used in the related text.
How to use the values of the items in the above table
(1) Item values are included in the predicate
(2) The item value modifies the subject
Including at least one of
The storage device is
A discriminative model that defines the relationship between the feature extraction result of the feature extraction unit and the distinction between "dimension" and "measure" of the items in the table is stored.
The identification unit
Using the discriminative model, the distinction between "dimension" and "measure" of the items in the table is estimated.
Table structure estimation system. A table-structured estimation system with input devices, output devices, storage devices, and processing devices.
The storage device is
Stores feature definition data that defines features related to both tables and sentences,
The input device is
It accepts data to be analyzed and
The processing device
A related text information extraction unit that acquires a table from the analysis target data and also acquires sentences related to the table as related texts.
A feature extraction unit that extracts the feature from the acquired table and the related text using the feature definition data,
Based on the feature extraction result of the feature extraction unit, the identification unit that estimates the distinction between "dimension" and "measure" of the items in the table, and the identification unit.
Equipped with a,
The related text information extraction unit
A sentence including a word related to a table item included in the analysis target data is acquired as the related text.
The storage device is
Stores a dictionary that defines words related to a given word as related words,
The words related to the items in the table above are:
It is a related word defined by the dictionary.
The above features
It defines how table items are used in a document.
The processing device
It has a related text information interpretation unit that analyzes the syntax of the related text.
The feature extraction unit
By determining how the items in the table are used in the related text, the features are extracted.
How to use the items in the above table
(1) The item is the subject and is qualified
(2) The item modifies the subject
Including at least one of
The storage device is
A discriminative model that defines the relationship between the feature extraction result of the feature extraction unit and the distinction between "dimension" and "measure" of the items in the table is stored.
The identification unit
Using the discriminative model, the distinction between "dimension" and "measure" of the items in the table is estimated.
Table structure estimation system. A table-structured estimation system with input devices, output devices, storage devices, and processing devices.
The storage device is
Stores feature definition data that defines features related to both tables and sentences,
The input device is
It accepts data to be analyzed and
The processing device is
A related text information extraction unit that acquires a table from the analysis target data and also acquires sentences related to the table as related texts.
A feature extraction unit that extracts the feature from the acquired table and the related text using the feature definition data,
Based on the feature extraction result of the feature extraction unit, the identification unit that estimates the distinction between "dimension" and "measure" of the items in the table, and the identification unit.
Equipped with a,
The related text information extraction unit
A sentence including a word related to the value of the item in the table included in the analysis target data is acquired as the related text.
The storage device is
Stores a dictionary that defines words related to a given word as related words,
The words associated with the values of the items in the table above are:
It is a related word defined by the dictionary.
The above features
It defines how the values of table items are used in a document.
The processing device is
It has a related text information interpretation unit that analyzes the syntax of the related text.
The feature extraction unit
The feature is extracted by determining how the values of the items in the table are used in the related text.
How to use the values of the items in the above table
(1) Item values are included in the predicate
(2) The item value modifies the subject
Including at least one of
The storage device is
A discriminative model that defines the relationship between the feature extraction result of the feature extraction unit and the distinction between "dimension" and "measure" of the items in the table is stored.
The identification unit
Using the discriminative model, the distinction between "dimension" and "measure" of the items in the table is estimated.
Table structure estimation system. The input device is
It accepts training data and
The processing device includes a learning unit, and the learning unit
Information regarding the distinction between "dimension" and "measure" of the items in the table of the training data is stored in the storage device as structural data.
Using the training data as the analysis target data, the related text information extraction unit and
Let the feature extractor execute the process
The identification model is generated based on the features extracted by the feature extraction unit and the structural data, and stored in the storage device.
The table structure estimation system according to any one of claims 1 to 4 . Equipped with a table-text correspondence section
The correspondence part between the table and the text is
Wherein the analysis target data, out extract the text in the table and text specific positional relationship, a sentence the extracted, sent to the related text information extracting unit,
The table structure estimation system according to any one of claims 1 to 4 . A method for estimating a table structure using an input device, an output device, a storage device, and a processing device.
In the storage device
Prepare feature definition data that defines features related to both tables and sentences,
From the input device
Enter the data to be analyzed and
The processing device
A table is acquired from the analysis target data, and sentences related to the table are acquired as related texts.
Using the feature definition data, the feature is extracted from the acquired table and the related text.
Based on the extracted features, and estimate the distinction of "measure" and "dimension" of the item of the table,
The text including the table items included in the analysis target data is acquired as the related text, and the text is obtained.
The above features
It defines how table items are used in a document.
The processing device
By determining how the items in the table are used in the related text, the features are extracted.
How to use the items in the above table
(1) The item is the subject and is qualified
(2) The item modifies the subject
Including at least one of
When estimating the distinction between "dimension" and "measure" of the items in the table, a discriminative model is used that defines the relationship between the extracted features and the distinction between "dimension" and "measure" of the items in the table.
How to estimate the table structure. A method for estimating a table structure using an input device, an output device, a storage device, and a processing device.
In the storage device
Prepare feature definition data that defines features related to both tables and sentences,
From the input device
Enter the data to be analyzed and
The processing device
A table is acquired from the analysis target data, and sentences related to the table are acquired as related texts.
Using the feature definition data, the feature is extracted from the acquired table and the related text.
Based on the extracted features, and estimate the distinction of "measure" and "dimension" of the item of the table,
A sentence including the value of the item in the table included in the analysis target data is acquired as the related text, and the text is obtained.
The above features
It defines how the values of table items are used in a document.
The processing device is
The feature is extracted by determining how the values of the items in the table are used in the related text.
How to use the values of the items in the above table
(1) Item values are included in the predicate
(2) The item value modifies the subject
Including at least one of
When estimating the distinction between "dimension" and "measure" of the items in the table, a discriminative model is used that defines the relationship between the extracted features and the distinction between "dimension" and "measure" of the items in the table.
How to estimate the table structure. From the input device
Enter the training data and
From the input device
Input the structural data including the information that distinguishes the "dimension" and "measure" of the table items included in the training data.
The processing device
A table is acquired from the training data, and sentences related to the table are acquired as related texts.
Using the feature definition data, the feature is extracted from the acquired table and the related text.
Based on the extracted features and the structural data, the discriminative model is generated and stored in the storage device.
The method for estimating the table structure according to claim 7 or 8 .

Images