JP6480380B2 - Table cell search apparatus, method, and program - Google Patents

Description

  The present invention relates to a table cell search apparatus, method, and program, and more particularly, to a table cell search apparatus, method, and program for indexing tabular data and answering a search query.

  With the development of computer technology, a large amount of table data described in HTML on the Web has come to exist. In table data, there are many tables from which triples of entities, attributes, and attribute values can be extracted. By accurately reading the table structure, search requests that query entity attribute values and specific attributes It is possible to directly answer a search request for an entity having a value.

  In order to extract a triple of entity, attribute, and attribute value from table data, it is necessary to understand the type of the table and the type of each row and each column included in the table. The method proposed in 2 etc. can be used.

Eric Crestan, Patrick Pantel: Web-scale table census and classification.WSDM 2011: 545-554 Jing Fang, Prasenjit Mitra, Zhi Tang, C. Lee Giles: Table Header Detection and Classification. AAAI 2012

  However, only the information in the table is insufficient as knowledge to be used as a search target. For example, even if you can extract the triple (Taro Yamada, score, 123.45) from the table that summarizes the scores of each player who participated in Sports Tournament A, you must supplement that these triples are knowledge in Sports Tournament A. For example, it is not possible to directly answer a user's search request. Until now, it has not been realized that information obtained from the inside of the table is appropriately expressed by out-of-line information and stored as searchable knowledge.

  The present invention has been made in order to solve the above-described problems, and obtains cell knowledge using out-of-line knowledge and can search for an answer to a search query, a table cell search device, It is an object to provide a method and a program.

  In order to achieve the above object, a table cell search apparatus according to a first aspect of the present invention extracts and indexes knowledge contained in the tabular data from the tabular data included in the HTML document, and sets a keyword set or natural sentence. A table cell search apparatus that ranks and returns table cells that can be directly answered to the search query given by the search score, and includes a table format composed of tabular data described by table tags from the set of HTML documents. A tabular data extraction unit that acquires a data set; and a tabular information extraction unit that extracts, for each of the tabular data, table-related information related to the tabular data from the HTML document including the tabular data; For each of the tabular data, a table type of the tabular data is determined based on the structure and contents of the tabular data. A table type classification unit for classifying each of the tabular data, and a matrix type classification for classifying each row and each column type included in the table of the tabular data based on the structure and contents of the tabular data The table-related data extracted by the table-related information extraction unit, the classification result in the table type classification unit, and the classification result in the matrix type classification unit for each of the table format data In the table for extracting cell knowledge composed of a set of information in the table including entities, attributes and attribute values extracted from the format data and the table related information, and storing the extracted cell knowledge in the search database For the knowledge extractor and the keyword set or natural sentence that is the given search query, label the keywords corresponding to the attributes in the search query. A query interpreting unit, a knowledge search unit that outputs the cell knowledge corresponding to the search query from the search database based on the search query to which the label is given by the query interpreting unit, and A search result generation unit that returns a search result to the user based on the output.

  Further, in the table cell search device according to the first invention, the table related information extraction unit, for each of the table format data, from the HTML document, a page title, a table caption, a section heading in which the table is described, The text immediately before the table or the text information immediately after the table is extracted as table related information, and the table knowledge extracting unit extracts the table related information extracted by the table related information extracting unit for each of the tabular data. Based on information, a classification result in the table type classification unit, and a classification result in the matrix type classification unit, an entity in the tabular data is obtained from the tabular data according to a table type of the tabular data. Consists of a set of in-table information including type, attribute, and attribute value, and the table-related information Extracting Le knowledge, it extracted the cell knowledge may be the to be stored in the search database.

  Further, in the table cell search device according to the first invention, the query interpretation unit further gives a label to a keyword other than an attribute in the search query, and the knowledge search unit reads the search query from the search database. The cell knowledge in which the keyword corresponding to the attribute matches the attribute of the information in the table, and the keyword other than the attribute of the search query matches the entity, entity type, or the table related information in the information in the table May be output.

  In the table cell search method according to the second invention, knowledge contained in the tabular data is extracted from the tabular data included in the HTML document and indexed, and the search query given directly by the keyword set or natural sentence is directly applied. A table cell search method in a table cell search apparatus that ranks and returns answerable table cells according to a search score, wherein a table format data extraction unit describes table format data described by table tags from the set of HTML documents. A table type data set comprising: a table type classification unit classifying a table type of the tabular data based on the structure and contents of the tabular data for each of the tabular data And the matrix type classification unit, for each of the tabular data, based on the structure and content of the tabular data The step of classifying each row and each column type included in the table of the tabular data, and the table related information extraction unit, for each of the tabular data, the table related information related to the tabular data, A step of extracting from the HTML document including format data, and an in-table knowledge extraction unit, for each of the table format data, the table related information extracted by the table related information extraction unit, the classification in the table type classification unit Cell knowledge composed of a set of the table-related information and the in-table information including the entity, the attribute, and the attribute value extracted from the tabular data based on the result and the classification result in the matrix type classification unit And the extracted cell knowledge is stored in a search database, and the query interpreter includes a keyword set as a given search query. Or, for a natural sentence, a step of assigning a label to a keyword corresponding to an attribute in a search query, and a knowledge search unit from the search database based on the search query to which the label is given by the query interpretation unit The step of outputting the cell knowledge corresponding to the search query and the step of the search result generation unit returning the search result to the user based on the output of the knowledge search unit are performed. To do.

  In the table cell search method according to the second invention, the table-related information extracting unit extracts the page title, table caption, and table from the HTML document for each of the table format data. The step of extracting the section heading, the text immediately before the table, or the text information immediately after the table as the table related information, and extracting by the in-table knowledge extracting unit is the table related information extraction for each of the tabular data. Based on the table related information extracted by the table, the classification result in the table type classification unit, and the classification result in the matrix type classification unit, according to the table type of the tabular data, from the tabular data, In-table information including entities, entity types, attributes, and attribute values in the tabular data Extract the cell knowledge consists of a set of said table-related information, it may be stored the extracted the cell knowledge search database.

In the table cell search method according to the second invention, the step of interpreting by the query interpreter further adds a label to a keyword other than an attribute in the search query,
The step of searching by the knowledge search unit includes, from the search database, a keyword corresponding to the attribute of the search query, a keyword that matches the attribute of the information in the table, and a keyword other than the attribute of the search query, and the table You may make it output the said cell knowledge in which the entity of internal information, entity type, or the said table relevant information corresponds.

  A program according to the third invention is a program for causing a computer to function as each part of the table cell search device according to the first invention.

  According to the table cell search apparatus, method, and program of the present invention, for each tabular data, table related information related to the tabular data is extracted from the HTML document including the tabular data, and a table of tabular data is obtained. In-table information including entities, attributes, and attribute values extracted from tabular data based on table-related information and classification results, and table-related Extracts cell knowledge that consists of a set of information, stores the extracted cell knowledge in the search database, and labels the keyword set corresponding to the attribute in the search query for the keyword set or natural sentence that is the given search query The cell knowledge corresponding to the search query is output from the search database based on the search query to which the label is assigned, and the user is output based on the output. By returning the search results, acquired cell information by using the knowledge of the outside tables, it is possible to find an answer for a search query, the effect is obtained that.

It is a block diagram which shows the structure of the table cell search apparatus concerning embodiment of this invention. It is a flowchart which shows the table cell search process routine in the table cell search apparatus concerning embodiment of this invention. It is a figure which shows an example of the table type classified by the table type classification | category part. It is a flowchart which shows the table format data extraction process routine in the table cell search apparatus concerning embodiment of this invention. It is a figure which shows the example of extraction of tabular data from a table tag. It is a flowchart which shows the table relevant-information extraction process routine in the table cell search apparatus concerning embodiment of this invention. It is a figure which shows the example of extraction of table relevant information. It is a flowchart which shows the knowledge extraction process routine in a table | surface in the case of the "vertical list" in the table cell search apparatus concerning embodiment of this invention. It is a figure which shows the example of extraction of the knowledge in a table | surface when the class of a table type is a "vertical list". It is a flowchart which shows the knowledge extraction process routine in a table | surface in the case of the "horizontal list" in the table cell search apparatus concerning embodiment of this invention. It is a figure which shows the example of extraction of the knowledge in a table | surface when the class of a table type is a "horizontal list". It is a flowchart which shows the knowledge extraction process routine in a table | surface in the case of "longitudinal attribute" in the table cell search apparatus concerning embodiment of this invention. It is a figure which shows the example of extraction of the knowledge in a table | surface in case a table type class is a "vertical attribute." It is a flowchart which shows the knowledge extraction process routine in a table | surface in the case of "horizontal attribute" in the table cell search apparatus concerning embodiment of this invention. It is a figure which shows the example of extraction of the knowledge in a table | surface when the class of a table type is a "horizontal attribute". It is a flowchart which shows the knowledge extraction process routine in a table | surface in the case of the "matrix" in the table cell search apparatus concerning embodiment of this invention. It is a figure which shows the example of extraction of the knowledge in a table | surface in case a table type class is a "matrix." It is a figure which shows the example of extraction of the knowledge in a table | surface in case a table type class is a "matrix." It is a flowchart which shows the table cell search process routine in the table cell search apparatus concerning embodiment of this invention. It is a figure which shows an example of the search result of a table cell search device.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Configuration of Table Cell Search Device According to Embodiment of the Present Invention>

  First, the configuration of the table cell search device according to the embodiment of the present invention will be described. The table cell search apparatus according to the embodiment of the present invention extracts and indexes knowledge included in tabular data from tabular data included in an HTML document, and directly searches a search query given by a keyword set or a natural sentence. The table cells that can be answered are ranked by search score and returned.

  As shown in FIG. 1, a table cell search apparatus 100 according to an embodiment of the present invention includes a CPU, a RAM, a search indexing processing routine, and a program and various data for executing a table cell search processing routine to be described later. And a ROM including a ROM that stores information. Functionally, the table cell search apparatus 100 includes an input unit 10, a calculation unit 20, and an output unit 50 as shown in FIG.

  The input unit 10 accepts a set of HTML documents as knowledge extraction targets. The HTML document includes tabular data. The input unit 10 accepts a keyword set or a natural sentence as a search query.

  The calculation unit 20 includes a search database 28, a tabular data extraction unit 30, a table type classification unit 32, a matrix type classification unit 34, a table related information extraction unit 36, an in-table knowledge extraction unit 38, and a query interpretation. A unit 40, a knowledge search unit 42, and a search result generation unit 44 are configured. The specific processing of each processing unit will be described in the operation of the table cell search device described later.

  The tabular data extraction unit 30 acquires a tabular data set composed of tabular data described by the table tag from the set of HTML documents.

  The table type classification unit 32 classifies the table type of the tabular data for each of the tabular data acquired by the tabular data extraction unit 30 based on the structure and contents of the tabular data.

  For each of the tabular data acquired by the tabular data extraction unit 30, the matrix type classification unit 34 determines the type of each row and each column included in the table of the tabular data based on the structure and contents of the tabular data. Classify.

  The table related information extraction unit 36 extracts, for each of the table format data acquired by the table format data extraction unit 30, table related information related to the table format data from the HTML document including the table format data. Here, as the table related information, the page title, the caption of the table, the section heading in which the table is described, the text immediately before the table, or the text information immediately after the table is extracted from the HTML document.

  The in-table knowledge extracting unit 38, for each of the tabular data, based on the table related information extracted by the table related information extracting unit 36, the classification result in the table type classification unit 32, and the classification result in the matrix type classification unit 34. Depending on the table type of the tabular data, the cell is composed of a set of tabular data, in-table information including entities, entity types, attributes, and attribute values in the tabular data, and table related information. Knowledge is extracted, and the extracted cell knowledge is stored in the search database 28.

  The search database 28 stores the cell knowledge extracted by the in-table knowledge extraction unit 38. The cell knowledge includes a knowledge record table and a knowledge cell ID table. The knowledge record table is a table in which an entity type, an entity, an entity 2, an attribute, an attribute value, a table type, and related information (title, caption, section heading, front text, out-of-line text) are recorded as one record. The knowledge cell ID table includes entity type row ID, entity type column ID, entity row ID, entity column ID, entity 2 ID, entity 2 ID, attribute row ID, attribute column ID, attribute value row ID, and attribute value column ID. It is a table as a record.

  The query interpretation unit 40 assigns a label to a keyword corresponding to an attribute in the search query for a keyword set or a natural sentence that is a search query received by the input unit 10, and labels a keyword other than the attribute in the search query. Give.

  The knowledge search unit 42 outputs cell knowledge corresponding to the search query from the search database 28 based on the search query to which the label is given by the query interpretation unit 40. Here, from the search database 28, the keyword corresponding to the attribute of the search query matches the attribute of the information in the table, and the keyword other than the attribute of the search query and the entity, entity type, or table related information in the table Output cell knowledge with matching information.

  Based on the cell knowledge output from the knowledge search unit 42, the search result generation unit 44 outputs a set of field “attribute values” of the acquired knowledge record of cell knowledge to the output unit 50 as a search result, so that the user Return search results to.

<Operation of Table Cell Search Device According to Embodiment of the Present Invention>

  Next, the operation of the table cell search device 100 according to the embodiment of the present invention will be described.

  The table cell search apparatus 100 executes a search indexing process routine and a table cell search process routine described below.

  First, the search indexing processing routine will be described.

  When the input unit 10 receives a set of HTML documents, the table cell search device 100 executes a search indexing process routine shown in FIG.

  First, in step S100, the tabular data extraction unit 30 acquires a tabular data set including tabular data described by table tags from the HTML document of the HTML document set received by the input unit 10.

  Next, in step S102, the table type classification unit 32 inputs the attribute-attribute value set of the table format data acquired in step S100 for the table format data of the table tag to be processed, and classifies the table type. Do. For the table type classification, a method described in Non-Patent Document 1 using an attribute-attribute value set as a feature can be used. The table types in the present embodiment are vertical list, horizontal list, vertical attribute, horizontal attribute, matrix, or other six classes. If the classification result in step S102 is “others”, the process returns to step S100, the next table tag is selected, and the subsequent processing is continued.

  FIG. 3 shows an example of each table type. In the process of step S106 described later, the following assumptions are made for each table. The vertical list and the vertical attribute table have a header line in which attributes are described. The horizontal list and the horizontal attribute table have a header column in which attributes are described. The matrix has a header row and a header column in which entities are described. The vertical list has a column describing entities. The horizontal list has a line describing the entity.

  In step S104, the matrix type classification unit 34 uses the attribute-attribute value set of the tabular data extracted in step S100 as input for the tabular data of the table tag to be processed, and inputs each row (row number) of the tabular data. type classification of a cell set having i) and each column (cell set having a column number j). For the classification of the row type and the column type, a method described in Non-Patent Document 2 using an attribute-attribute value set as a feature can be used. The matrix types in the present embodiment are two classes of header or data. If there is no row and column determined as the header, the process returns to step S100, the next table tag is selected, and the subsequent processing is continued.

  In step S106, the table related information extraction unit 36 analyzes the HTML document including the table format data of the table tag to be processed, and extracts the attribute-attribute value set that is the table related information regarding the table format data of the HTML document. To do.

  In step S108, based on the table related information extracted in step S106, the table type classification result in step S102, and the matrix type classification result in step S104, the table format of the table tag data to be processed is determined. Depending on the data table type, cell knowledge consisting of a set of table-related information and table-related information including entities, entity types, attributes, and attribute values in tabular data is extracted from tabular data. The extracted cell knowledge is stored in the search database 28.

  The above steps S100 to S108 are repeated for each table tag included in the HTML document.

  Further, the repetition for each table tag is repeated for each HTML document included in the set of HTML documents.

  The processing in step S100 is executed by a tabular data extraction processing routine shown in FIG.

  In step S200, the table tag that appears as a child element of the table tag is replaced with a character string set included in the table tag, and all HTML structures are deleted.

  In step S201, the line number variable i is initialized with zero. Also, the tabular data is initialized as a two-dimensional array.

  Next, every time a tr tag appears, the following processing is repeated.

First, in step S202, the row number variable i is incremented by 1 and the column number variable j is set to 0.
Then, every time a tag representing a td tag or a th tag (hereinafter referred to as a cell tag) appears as a child element of the tr tag, the process of step S204 is repeated.

  In step S204, an attribute-attribute value set is acquired from a cell tag for a tag representing a td tag or a th tag (hereinafter referred to as a cell tag). The detailed attribute-attribute value set is as follows.

  As the attribute “row size (symbol n)”, the rowspan attribute is an attribute value. If the rowspan attribute value is not specified, n = 1.

  As an attribute “column size (symbol m)”, a colspan attribute is an attribute value. If no colspan attribute value is specified, m = 1.

  As an attribute “character string”, a text in a cell tag child element is used as an attribute value.

  As an attribute “tag name”, a cell tag name (‘th’ or ‘td’) is an attribute value.

  For the attribute “image flag”, the attribute value is 1 if the ‘img’ tag exists in the child element of the cell tag, and 0 otherwise.

  For the attribute “numeric flag”, 1 is used as the attribute value if the character string in the cell is a numeric value, and 0 otherwise.

  For the attribute “date flag”, the attribute value is 1 if the character string in the cell is a date, and 0 otherwise.

  For the attribute “symbol flag”, 1 is used as the attribute value if the character string in the cell is only a symbol, and 0 otherwise.

  For the attribute “long character string flag”, 1 is used as the attribute value if the number of characters of the character string in the cell is equal to or greater than the threshold value, and 0 otherwise. For example, 50 is set as the threshold.

  For the attribute “format flag”, the attribute value is set to 1 if there is a tag that changes the appearance when browsing, such as a b tag or strong tag, and 0 otherwise.

  In addition, as information on the corresponding cell tag from the HTML document, for example, background color, presence / absence of character position designation, and the like may be added as attributes.

  In step S206, the attribute-attribute value set acquired in step S204 is stored in the tabular data [i] [j].

  In the processing of step S206, j is incremented by 1 until the column number in which the attribute-attribute value set is not stored in the tabular data [i] [j]. Thereafter, for all combinations of row numbers and column numbers satisfying i ≦ row number ≦ i + n and j ≦ column number ≦ j + m, the attribute-attribute value set obtained in step S204 is displayed in tabular data [row number] [column Number]. That is, here, a process of copying the contents of the linked cells to the corresponding tabular data [row number] [column number] is performed. The tabular data in which the row number and the column number are designated is hereinafter referred to as “cell”.

  The processes in steps S204 to S206 are repeated in the order of appearance until there is no unprocessed cell tag.

  As for the cell tag processing, the tr tag that appears in the child element of the table tag is repeated in the order of appearance until there is no unprocessed tr tag.

  In step S208, the largest of the row number and column number in which the attribute-attribute value set is stored in the tabular data is held as the number of rows N and the number of columns M of the tabular data. For all combinations of row number and column number satisfying 1 ≦ row number ≦ N, 1 ≦ column number ≦ M, empty if no attribute-attribute value set is stored in tabular data [row number] [column number] Add a set.

  FIG. 5 shows an example of tabular data extraction. As shown in FIG. 5, the tabular data is assigned a row number and a column number, and has a character string, a tag name, a numerical value, a row size, a column size, and a starting point as elements.

  The processing in step S106 is executed by a table related information extraction processing routine shown in FIG.

  In step S300, the text information of the title tag is acquired as the attribute value as the title, and stored as the attribute “title”. If the title tag does not exist, a null character string is set as the attribute value.

  In step S302, when the caption tag is included in the child element of the table tag from which the tabular data is extracted, the text included in the child element of the caption is acquired as the attribute value, and the attribute “caption” is acquired. Store. If the caption tag does not exist, a null character string is set.

  In step S304, the text included in the child element of the p or div tag that appears immediately before the start of the table tag from which the tabular data is extracted is acquired as the previous text, and is stored as the attribute “previous text”. If there is no p or div tag that appears immediately before the table tag, a null character string is set.

  In step S306, the text included in the child element of the p or div tag that appears immediately after the end of the table tag from which the tabular data is extracted is acquired in the attribute “following text”. If there is no p or div tag that appears immediately after the table tag, a null character string is set.

  In step S308, the header array (element numbers 1 to 6) is initialized with an empty character string. When the extracted elements are h1 to h6 tags, the heading level of each tag (1 for h1 and 6 for h6) is acquired, and the heading level of the heading array is initialized with an empty character string. For example, in the case of h2, 2 to 6 after the heading array are deleted. Then, in steps S310 to S312, the h1 to h6 tags appearing in the HTML document and the table tag of the tabular data are extracted in the order of appearance as extraction elements.

  In step S310, the character string of the extracted element is acquired, and the character string of the extracted element is stored in the heading array [heading level].

  In step S312, when the extracted element is a table tag, the repetition is skipped, and a character string in which each element of the header array is combined with a blank as an attribute value of the attribute “headline” is stored. If no character string is stored in the heading array, the null character string is stored as the attribute value of the attribute “heading”.

  By performing the processing of steps S300 to S312, the table related information extracting unit 36 includes, as the table related information, the page title, the table caption, the section heading in which the table is written, the text immediately before the table, or the table The text information immediately after is extracted.

  FIG. 7 shows an example of extracting table related information.

  As described below, the processing in step S108 includes in-table knowledge corresponding to each of the vertical list, horizontal list, vertical attribute, horizontal attribute, and matrix according to the table type class classified in step S102. An extraction processing routine is executed.

  First, the case where the table type class is “vertical list” will be described. In the case of “vertical list”, the in-table knowledge extraction processing routine shown in FIG. 8 is executed.

In step S400, when the row type classification result of row number i is all headers in the range of 1 ≦ i ≦ n, the range of 1 ≦ i ≦ n is acquired as the header row number set. A set of row numbers that are not included in the header row number set and satisfy 1 ≦ i ≦ N is acquired as a data row number set. If the header row number set is an empty set, the process returns to step S100 of the search indexing processing routine, selects the next table tag, and continues the subsequent processing.

In step S402, a set of column numbers satisfying the following condition 1 is acquired as an entity column number set.

(Condition 1) The type of the attribute “character string” of the attribute-attribute value set of the tabular data [row number] [column number] in the row whose row type classification result is “data” among the row numbers 1 to N Number K1 ÷ Number of rows whose classification result of row type is “data” K2 <threshold θ

  When K2 is 0, condition 1 is not satisfied. Specifically, the value of θ is set to 0.75 or the like. If the entity column number set is an empty set, the process returns to step S100 of the search indexing processing routine, selects the next table tag, and continues the subsequent processing.

  The following processing is repeated for each column number j included in the entity column number set. However, if k = j, the process proceeds to the next column number j.

  First, the following steps S404 and S406 are repeated for each row number i included in the data row number set for each attribute column number k from 1 to M.

  In step S404, in-table knowledge is extracted for column number j, attribute column number k included in the entity column number set, and row number i included in the data row number set. The in-table knowledge to be extracted is the following entity type, entity, entity 2, attribute, and attribute value.

  The entity type is a character obtained by acquiring the attribute “character string” of the tabular data [h] [j] in ascending order of the line number for each of the line numbers h included in the header line number set, and then removing the duplicate character string. The string set is a string concatenated with a space character. The entity type row ID is a header row number set, and the entity type column ID is j.

  The entity is an attribute “character string” of the tabular data [i] [j]. The entity row ID is i and the entity column ID is j.

  Entity 2 is a null character string. The entity 2 row ID and the entity 2 column ID are each a null character string.

  The attribute is a character string obtained by removing the duplicate character string after acquiring the attribute “character string” of the tabular data [h] [k] in ascending order of the line number for each of the line numbers h included in the header line number set. It is a character string that concatenates sets with a space character. The attribute row ID is a header row number set, and the attribute column ID is k.

  The attribute value is the attribute “character string” of the tabular data [i] [k]. The attribute value row ID is i, and the attribute column ID is k.

  In step S406, the entity type, entity, entity 2, attribute, attribute value acquired in step S404, table type acquired in step S102, and table related information acquired in step S106 (title, caption, section heading, (Front text and out-of-line text) are stored in the knowledge record table of the search database 28 as one record. In addition, the entity type row ID, entity type column ID, entity row ID, entity column ID, entity 2 ID, entity 2 ID, attribute row ID, attribute column ID, attribute value row ID, and attribute value column ID acquired in step S404. Is stored in the knowledge cell ID table of the search database 28 as one record.

  The above steps S404 to S406 are repeated for each row number i included in the data row number set.

  Further, the repetition for each row number i as described above is repeated for each attribute column number k from 1 to M.

  In addition, the repetition for each attribute column number k as described above is repeated for each column number j included in the entity column number set.

  FIG. 9 shows an example of extracting in-table knowledge when the table type class is “vertical list”.

  Next, a case where the table type class is “horizontal list” will be described. In the case of “horizontal list”, the in-table knowledge extraction processing routine shown in FIG. 10 is executed.

  In step S500, when all the column type classification results of column number j are headers in the range of 1 ≦ j ≦ m, the range of 1 ≦ j ≦ m is acquired as a header column number set. A set of row numbers that are not included in the header column number set and satisfy 1 ≦ j ≦ M is acquired as a data column number set. If the header column number set is an empty set, the process returns to step S100 of the search indexing processing routine, selects the next table tag, and continues the subsequent processing.

  In step S502, a set of row numbers satisfying the following condition 2 is acquired as an entity row number set.

(Condition 2) Type of attribute “character string” of attribute-attribute value set of tabular data [row number] [column number] in a column whose column type classification result is “data” among column numbers 1 to M Number K1 ÷ number of columns whose column type is “data” K2 <threshold θ

  When K2 is 0, condition 2 is not satisfied. Specifically, the value of θ is set to 0.75 or the like. If the entity row number set is an empty set, the process returns to step S100 of the search indexing processing routine, selects the next table tag, and continues the subsequent processing.

  The following processing is repeated for each row number i included in the entity row number set.

  First, the following steps S504 and S506 are repeated for each column number j included in the data column number set for each row number k from 1 to N.

  In step S504, in-table knowledge is extracted for row number i, row number k included in the entity row number set, and column number j included in the data column number set. The in-table knowledge to be extracted is the following entity type, entity, entity 2, attribute, and attribute value.

  The entity type is a character obtained by obtaining the attribute “character string” of the tabular data [i] [h] in ascending order of the column number for each of the column numbers h included in the header column number set, and then removing the duplicate character string. The string set is a string concatenated with a space character. The entity row ID is i, and the entity column ID is a header column number set.

  The entity is an attribute “character string” of the tabular data [i] [j]. The entity row ID is i and the entity column ID is j.

  Entity 2 is a null character string. The entity 2 row ID and the entity 2 column ID are each a null character string.

  The attribute is a character string obtained by removing the duplicate character string after obtaining the attribute “character string” of the tabular data [k] [h] in ascending order of the column number for each column number h included in the header column number set. It is a character string that concatenates sets with a space character. The attribute row ID is k, and the attribute column ID is a header column number set.

  The attribute value is the attribute “character string” of the tabular data [k] [j]. The attribute row ID is k and the entity column ID is j.

  In step S506, the entity type acquired in step S504, entity, entity 2, attribute, attribute value, table type acquired in step S102, and table related information acquired in step S106 (title, caption, section heading, (Front text and out-of-line text) are stored in the knowledge record table of the search database 28 as one record. In addition, the search database includes the entity type row ID, entity type column ID, entity row ID, entity column ID, entity 2 ID, entity 2 ID, attribute row ID, attribute column ID, attribute value row ID, and attribute value column ID as one record. Stored in the knowledge cell ID table.

  The above steps S504 to S506 are repeated for each column number j included in the data column number set.

  The repetition for each column number j as described above is repeated for each row number k from 1 to N.

  Further, the repetition for each row number k as described above is repeated for each row number i included in the entity row number set. However, when k = i, the process proceeds to the next line number i.

  FIG. 11 shows an example of extraction of in-table knowledge when the table type class is “horizontal list”.

  Next, a case where the table type class is “vertical attribute” will be described. In the case of “vertical attribute”, the in-table knowledge extraction processing routine shown in FIG. 12 is executed.

  In step S600, when the row type classification result of row number i is all headers in the range of 1 ≦ i ≦ n, the range of 1 ≦ i ≦ n is acquired as the header row number set. A set of row numbers that are not included in the header row number set and satisfy 1 ≦ i ≦ N is acquired as a data row number set. If the header row number set is an empty set, the process returns to step S100 of the search indexing processing routine, selects the next table tag, and continues the subsequent processing.

  In step S602, when the column type classification result of column number j is a header in the range of 1 ≦ j ≦ m, the range of 1 ≦ j ≦ m is acquired as a header column number set. A set of row numbers not included in the header column number set and satisfying 1 ≦ j ≦ M is acquired as a data column number set.

  Then, the following steps S604 to S606 are repeated for each column number j included in the data column number set.

  In step S604, knowledge in the table is extracted for the column number j included in the data column number set, and the record is stored in the search database 28. The in-table knowledge to be extracted is the following entity type, entity, entity 2, attribute, and attribute value.

  The entity type is a null string. Each of the entity type row ID and the entity type column ID is a null character string.

  When an attribute other than a null character string is set in the attribute “caption” of the table related information, the entity sets the attribute value, and sets the entity row ID and the entity column ID as a null character string.

  When the attribute “caption” is a null character string and the header row number set is not an empty set, the column numbers h are included in the header column number set and the row numbers k of 1 to N are in ascending order. Next, after acquiring the attribute “character string” of the tabular data [k] [h] in ascending order of the line numbers, a character string obtained by concatenating a character string set from which duplicate character strings are removed with a blank character is used as an entity. The entity row ID is [1, 2,..., N], and the entity column ID is a header column number set.

  When the attribute “caption” is a null character string and the header row number set is an empty set, the attribute “title” of the table related information is an entity. Each of the entity row ID and the entity column ID is a null character string.

  Entity 2 is a null character string. The entity 2 row ID and the entity 2 column ID are each a null character string.

  The attribute is a character string obtained by obtaining the attribute “character string” of the tabular data [h] [j] in ascending order of the line numbers for each of the line numbers h included in the header line number set, and then removing the duplicate character strings. It is a character string that concatenates sets with a space character. The attribute row ID is a header row number set, and the attribute column ID is j.

  The attribute value is a character obtained by acquiring the attribute “character string” of the tabular data [d] [j] in ascending order of the column number for each row number d included in the data row number set, and then removing the duplicate character string. The string set is a string concatenated with a space character. The attribute value row ID is a data row number set, and the attribute value column ID is j.

  In step S606, the entity type acquired in step S604, entity, entity 2, attribute, attribute value, table type acquired in step S102, and related information acquired in step S106 (title, caption, section heading, table) The previous text and out-of-line text) are stored in the knowledge record table of the search database 28 as one record. In addition, the search database includes the entity type row ID, entity type column ID, entity row ID, entity column ID, entity 2 ID, entity 2 ID, attribute row ID, attribute column ID, attribute value row ID, and attribute value column ID as one record. It is stored in 28 knowledge cell ID tables.

  FIG. 13 shows an example of extracting in-table knowledge when the table type class is “vertical attribute”.

  Next, a case where the table type class is “horizontal attribute” will be described. In the case of “horizontal attribute”, the in-table knowledge extraction processing routine shown in FIG. 14 is executed.

  In step S700, when the row type classification result of row number i is all headers in the range of 1 ≦ i ≦ n, the range of 1 ≦ i ≦ n is acquired as the header row number set. A set of row numbers that are not included in the header row number set and satisfy 1 ≦ i ≦ N is acquired as a data row number set.

In step S702, when the column type classification result of the column number j is a header in the range of 1 ≦ j ≦ m, the range of 1 ≦ j ≦ m is acquired as the header column number set. A set of row numbers not included in the header column number set and satisfying 1 ≦ j ≦ M is acquired as a data column number set. If the header column number set is an empty set, the process returns to step S100 of the search indexing processing routine, selects the next table tag, and continues the subsequent processing.

  Then, the following steps S704 to S706 are repeated for each row number i included in the data row number set.

  In step S704, in-table knowledge is extracted for the row number i included in the data row number set, and the record is stored in the search database unit. The in-table knowledge to be extracted is the following entity type, entity, entity 2, attribute, and attribute value.

  The entity type is a null string. Each of the entity type row ID and the entity type column ID is a null character string.

  When a non-null character string is set in the attribute “caption” of the table related information, the attribute value of the attribute “caption” is set in the entity, and the entity row ID and the entity column ID are set to the null character string, respectively.

  If the attribute “caption” is a null character string and the header row number set is not an empty set, the row numbers h included in the header row number set and the column numbers k of 1 to M are in ascending order, Next, after obtaining the attribute “character string” of the tabular data [h] [k] in ascending order of the column number, a character string obtained by concatenating a set of character strings from which duplicate character strings are removed with a blank character is used as an entity. The entity row ID is a header row number set, and the entity column ID is an array of [1, 2,..., M].

  When the attribute “caption” is a null character string and the header row number set is an empty set, the attribute “title” of the table related information is an entity. Each of the entity row ID and the entity column ID is a null character string.

  Entity 2 is a null character string. The entity 2 row ID and the entity 2 column ID are each a null character string.

  The attribute is a character string obtained by removing the duplicate character string after obtaining the attribute “character string” of the tabular data [i] [h] in ascending order of the column number for each column number h included in the header column number set. It is a character string that concatenates sets with a space character. The attribute row ID is i, and the attribute column ID is a header column number set.

  The attribute value is a character obtained by obtaining the attribute “character string” of the tabular data [i] [d] in ascending order of the column number for each of the column numbers d included in the data column number set, and then removing the duplicate character string. The string set is a string concatenated with a space character. The attribute value row ID is i, and the attribute column ID is a data column number set.

  In step S706, the entity type acquired in step S704, entity, entity 2, attribute, attribute value, table type acquired in step S102, and table related information acquired in step S106 (title, caption, section heading, (Front text and out-of-line text) are stored in the knowledge record table of the search database 28 as one record. In addition, the search database includes the entity type row ID, entity type column ID, entity row ID, entity column ID, entity 2 ID, entity 2 ID, attribute row ID, attribute column ID, attribute value row ID, and attribute value column ID as one record. It is stored in 28 knowledge cell ID tables.

  FIG. 15 shows an example of extraction of in-table knowledge when the table type class is “horizontal attribute”.

  Next, a case where the table type class is “matrix” will be described. In the case of “matrix”, the in-table knowledge extraction processing routine shown in FIG. 16 is executed.

  In step S800, when the row type classification result of row number i is all headers in the range of 1 ≦ i ≦ n, the range of 1 ≦ i ≦ n is acquired as the header row number set. A set of row numbers that are not included in the header row number set and satisfy 1 ≦ i ≦ N is acquired as a data row number set.

  In step S802, when the column type classification result of the column number j is a header in the range of 1 ≦ j ≦ m, the range of 1 ≦ j ≦ m is acquired as the header column number set. A set of row numbers that are not included in the header column number set and satisfy 1 ≦ j ≦ M is acquired as a data column number set. If the header column number set is an empty set, the process returns to step S100 of the search indexing processing routine, selects the next table tag, and continues the subsequent processing.

  And about repeating for every column number j, the following processes are repeated for every row number i included in a data row number set.

  The following steps S804 to S806 are repeated for each column number j included in the data column number set.

  In step S804, in-table knowledge is extracted for the row number i included in the data row number set and the column number j included in the data column number set, and the record is stored in the search database 28. The in-table knowledge to be extracted is the following entity type, entity, entity 2, attribute, and attribute value.

  The entity type is a null string. Each of the entity type row ID and the entity type column ID is a null character string.

  The entity acquires the attribute “character string” of the tabular data [i] [h] in ascending order of the column number for each of the column numbers h included in the header column number set, and then removes the duplicate character string. It is a character string that concatenates sets with a space character. The entity row ID is i, and the entity column ID is a header column number set.

  The entity 2 obtains the attribute “character string” of the tabular data [h] [j] in ascending order of the line numbers for each of the line numbers h included in the header line number set, and then removes the duplicate character strings. The string set is a string concatenated with a space character. The entity 2 row ID is a header row number set, and the entity 2 column ID is j.

  The attribute specifies an attribute value that is not a null character string among the attribute values “caption”, attribute “section heading”, and attribute “title” of the table related information. If there are a plurality of attribute values that are not null character strings, the caption has the highest priority, and the section heading has the highest priority. Each of the attribute row ID and attribute column ID is a null character string.

  The attribute value is the attribute “character string” of the tabular data [i] [j]. The attribute value row ID is i, and the attribute value column ID is j.

In step S806, the entity type, entity, entity 2, attribute, attribute value acquired in step S804, the table type acquired in step S102, and the related information acquired in step S106 (title, caption, section heading, (Front text and out-of-line text) are stored in the knowledge record table of the search database 28 as one record. In addition, the search database includes the entity type row ID, entity type column ID, entity row ID, entity column ID, entity 2 ID, entity 2 ID, attribute row ID, attribute column ID, attribute value row ID, and attribute value column ID as one record. It is stored in 28 knowledge cell ID tables.
Steps S804 to S806 are repeated for each column number j included in the data column number set.

  The repetition for each column number j as described above is repeated for each row number i included in the data row number set.

  FIG. 17 and FIG. 18 show examples of extracting in-table knowledge when the table type class is “matrix”.

  The above is the description of the search indexing processing routine.

  Next, the table cell search processing routine will be described.

  When the input unit 10 receives a keyword set or a natural sentence as a search query, the table cell search device 100 executes a table cell search processing routine shown in FIG.

  In step S900, the query interpretation unit 40 assigns labels to keywords corresponding to attributes in the search query and keywords other than the attributes in the search query for the keyword set or natural sentence that is the search query received by the input unit 10. .

  The labeling method varies depending on the content of the search query.

  First, if the search query is given as a keyword set that matches a defined keyword template (blank-separated character strings are called keywords), each keyword is labeled with "attribute" or "other" according to the given definition. Give.

The predefined keyword templates are
"(Attribute) of (other)"
`` (Other) (Attribute) ''
For example, for the keyword set “Yamada's birthday”, the label “Yamada”, which is a keyword other than the attribute, “Other”, and the attribute keyword “Birthday”, “Attribute” Is granted. In addition, freely defined keyword templates may be added.

  Next, when the search query is given as a keyword set that does not match the defined keyword template set, each keyword searches the “attribute” field of the knowledge record table of the search database, and the keyword with the largest number of search hits is “ The “attribute” label is assigned, and the “other” label is assigned to keywords other than the attribute.

  When the query is given as a natural sentence that matches the defined natural sentence template, a label is assigned to each keyword in the natural sentence according to the given definition. Predefined natural sentence templates are

"When is (. +: Attribute) (. +: Other)"?

Etc. are available. Here, the parentheses in the template mean (regular expression pattern: label name).

  For example, the natural sentence "When is Yamada's birthday" matches the template by regular expression matching, so "Yamada" is labeled "Other" and "Birthday" is "Attribute" it can. After that, among the morphemes obtained by morphological analysis of the natural sentence, a “other” label is assigned to a morpheme that does not match the template, thereby obtaining a labeled keyword set.

  Next, when the search query is given as a natural sentence that does not match the defined natural sentence template, after the natural sentence is converted into a keyword set after morphological analysis, the search query does not match the defined keyword template set. A labeled keyword set is obtained by performing processing when given as a set.

  In addition, for each keyword in the labeled keyword set, those that match the synonym dictionary are given the same label as the keyword that matched the synonym and added to the labeled keyword set. For example, if the labeled keyword set is “Yamada: Other” and “Birthday: Attribute”, and “Birthday” and “Birthday” are registered in the synonym dictionary, the labeled keyword set is “Yamada: Other”, “Birthday: Attribute”, and “Birthday: Attribute”.

  In step S902, the search database 28 is searched using the labeled keyword set obtained in step S202, and a knowledge record is acquired. Specifically, the search is performed as follows.

  One of the keywords assigned with the “attribute” label acquires a knowledge record that matches the full-text search in the “attribute” field of the knowledge record table of the search database 28.

  In addition, any of the keywords assigned with the “other” label acquires a knowledge record that matches the full-text search in the “entity” field of the knowledge record table.

  In addition, a knowledge record in which any of the keywords to which the “other” label is assigned matches the full-text search of a field other than the “entity” field is also acquired. Here, we get knowledge records that match the full-text search to either the “entity type” or “title” or “section heading” or “caption” or “before text” or “after text” field of the knowledge record table. To do. For knowledge records with the “table type” of the acquired knowledge record “matrix”, determine whether any of the keywords with the “other” label matches the “entity 2” field in the full-text search. If it does not match, it is removed from the acquired knowledge record.

In step S904, the set of field “attribute values” of the knowledge record acquired in step S902 is output to the output unit 50 as a search result. Further, as shown in the example of FIG. 20, a record obtained by searching the knowledge cell ID table of the search database 28 corresponding to the field “knowledge ID” of the acquired knowledge record may be included in the search result.

  As described above, according to the table cell search device according to the embodiment of the present invention, for each table format data, the table related information related to the table format data is extracted from the HTML document including the table format data. A table containing entities, attributes, and attribute values extracted from tabular data based on table-related information and classification results, classifying table types of tabular data, classifying each row and column type Cell knowledge that consists of a set of internal information and table-related information is extracted, the extracted cell knowledge is stored in the search database, and for the keyword set or natural sentence that is the given search query, the attribute in the search query A label is assigned to the keyword corresponding to, and cell knowledge corresponding to the search query is output from the search database based on the search query to which the label is assigned. There, by returning the search results to the user, acquire the cell information by using the knowledge of the outside tables, it is possible to find answers to search queries.

  In addition, according to the method according to the embodiment of the present invention, information composed of entities, attributes, and attribute values included in a table of tabular data, supplemented by out-of-line information, is extracted as a knowledge record. This makes it possible to directly answer a search request, so that it can be used for information retrieval, question answering, a dialogue system, and the like.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 Input part 20 Operation part 28 Search database 30 Tabular data extraction part 32 Table type classification | category part 34 Matrix type classification | category part 36 Table related information extraction part 38 In-table knowledge extraction part 40 Query interpretation part 42 Knowledge search part 44 Search result generation part 50 Output unit 100 Table cell search device

Claims (7)

From the tabular data contained in the HTML document, knowledge contained in the tabular data is extracted and indexed, and the table cells that can be directly answered to the search query given by the keyword set or the natural sentence are ranked by the search score. A table cell search device to be returned,
A tabular data extraction unit that acquires a tabular data set composed of tabular data described by a table tag from the set of HTML documents;
For each of the tabular data, a table related information extracting unit that extracts table related information related to the tabular data from the HTML document including the tabular data;
For each of the tabular data, based on the structure and contents of the tabular data , the tabular data is defined in advance as a vertical list, a horizontal list, a vertical attribute, a horizontal attribute, and a matrix. A table type classifier that classifies any of the table types;
For each of the tabular data, based on the structure and content of the tabular data, a matrix type classification unit that classifies the type of each row and each column included in the table of the tabular data;
For each of the table format data, based on the table related information extracted by the table related information extraction unit, the classification result in the table type classification unit, and the classification result in the matrix type classification unit, the type of the table In accordance with a predetermined extraction rule, cell knowledge composed of a set of in-table information including entities, attributes, and attribute values extracted from the tabular data and the table related information is extracted and extracted. An in-table knowledge extractor for storing the cell knowledge in a search database;
A query interpreter that gives a label to a keyword corresponding to an attribute in the search query for a keyword set or a natural sentence that is a given search query;
A knowledge search unit that outputs the cell knowledge corresponding to the search query from the search database based on the search query to which the label is given by the query interpretation unit;
Based on the output of the knowledge search unit, a search result generation unit that returns a search result to the user;
Table cell search device including The table-related information extraction unit extracts, for each of the tabular data, a page title, a table caption, a section heading in which the table is written, text immediately before the table, or text information immediately after the table from the HTML document. Extract as table related information,
The in-table knowledge extraction unit, for each of the table format data, the table-related information extracted by the table-related information extraction unit, the classification result in the table type classification unit, and the classification result in the matrix type classification unit based on, according to the extraction rule which is predetermined according to the type of the table, from the table-format data, and tables in the information including entities in said tabulated data, entity types, attributes, and attribute values, the table-related information The table cell search device according to claim 1, wherein cell knowledge composed of a pair is extracted and the extracted cell knowledge is stored in a search database. The query interpreter further gives a label to keywords other than the attribute in the search query,
The knowledge search unit includes, from the search database, a keyword corresponding to an attribute of the search query, an attribute of the in-table information, and a keyword other than the attribute of the search query, and an entity of the in-table information The table cell search device according to claim 1, wherein the cell knowledge that matches the entity type or the table related information is output. From the tabular data contained in the HTML document, knowledge contained in the tabular data is extracted and indexed, and the table cells that can be directly answered to the search query given by the keyword set or the natural sentence are ranked by the search score. A table cell search method in the table cell search device to be returned,
A tabular data extraction unit obtaining a tabular data set composed of tabular data described by a table tag from the set of HTML documents;
A table type classifying unit converts, for each of the tabular data, the tabular data based on the structure and contents of the tabular data, a vertical list, a horizontal list, and a vertical attribute that are predetermined for the tabular data. Categorizing into one of the table types:, horizontal attribute, and matrix ;
A matrix type classification unit for each of the tabular data, classifying the type of each row and each column included in the table of the tabular data based on the structure and content of the tabular data;
A table-related information extraction unit for each of the tabular data, extracting table-related information related to the tabular data from the HTML document including the tabular data;
Based on the table related information extracted by the table related information extraction unit, the classification result in the table type classification unit, and the classification result in the matrix type classification unit for each of the tabular data In accordance with an extraction rule determined in advance according to the type of the table, the table is composed of a set of in-table information including entities, attributes, and attribute values extracted from the tabular data, and the table-related information. Extracting cell knowledge and storing the extracted cell knowledge in a search database;
A step in which a query interpreter assigns a label to a keyword corresponding to an attribute in a search query for a keyword set or a natural sentence that is a given search query;
A knowledge search unit, based on the search query to which the label is given by the query interpretation unit, outputting the cell knowledge corresponding to the search query from the search database;
A step in which the search result generation unit returns the search result to the user based on the output of the knowledge search unit;
Table cell search method including. The step of extracting the table-related information extracting unit includes, for each of the tabular data, a page title, a caption of the table, a section heading in which the table is described, a text immediately before the table, or immediately after the table. Text information is extracted as table related information,
The step of extracting the in-table knowledge extracting unit includes the table-related information extracted by the table-related information extracting unit, the classification result in the table type classifying unit, and the matrix type classifying unit for each of the tabular data based on the classification results in accordance with the extraction rule which is predetermined according to the type of the table, from the table-format data, and tables in the information including entities in said tabulated data, entity types, attributes, and attribute values, The table cell search method according to claim 4, wherein cell knowledge composed of a pair with the table related information is extracted, and the extracted cell knowledge is stored in a search database. The step of interpreting by the query interpreter further adds a label to a keyword other than the attribute in the search query,
The step of searching by the knowledge search unit includes, from the search database, a keyword corresponding to the attribute of the search query, a keyword that matches the attribute of the information in the table, and a keyword other than the attribute of the search query, and the table The table cell search method according to claim 4 or 5, wherein the cell knowledge in which the entity of the internal information, the entity type, or the table related information matches is output. The program for functioning a computer as each part of the table cell search apparatus of any one of Claims 1-3.