JP2014142769A - Text extraction device, text extraction method and text extraction program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately extract a text even when it is not possible to acquire another document which can be grouped with a document as the extraction object of the text.SOLUTION: A text extraction device 10 comprises: a division data creation part 120 for extracting one or more elements each including a character string from an object document as a text extraction object; a text identifier creation part 170 for extracting a morpheme from a related document related to the object document; a featured value generation part 210 for determining whether or not the extracted morpheme is included in the element, and for, when determining that the extracted morpheme is included in the element, increasing scores indicating a level at which the text of the object document is included in the element to calculate the scores about the element; and a text determination part 220 for specifying the element including the text from among the respective elements on the basis of the scores.

Description

  The present invention relates to a text extracting device, a text extracting method, and a text extracting program for extracting text from a document.

  In recent years, countless contents such as blogs, news, photos, videos, and music are generated every day due to the spread of the Internet. The spread of smartphones is likely to increase the content in the future. By using a search service on the Internet, an Internet user can find out a content desired by the user from an enormous amount of content. In order to discover such content, it is necessary to appropriately extract text information indicating the content body from content described in, for example, HTML (HyperText Markup Language). In view of the above, there is an increasing need for a technique for identifying a text from content as a service.

  The need for text extraction from content is not limited to search services. In recent years, smartphone applications and web services that show web content like a magazine have become popular. In these services, in order to display content in a magazine style, it is necessary to specify and extract the text from the content by a general-purpose method.

  It is technically difficult to remove the HTML tag from the content described in a markup language such as HTML and extract only the text. This is because the HTML text often includes text that is not directly related to the advertisement or the text, and it is difficult to determine where the text is located on the HTML because there is a lot of noise.

  In Patent Document 1, all HTML tags are removed from HTML text, converted to text data, HTML texts of the same site are grouped, a statistic is calculated for each character string, and the text is extracted using the statistic. ing.

JP 2006-338364 A

  However, in the technique described in Patent Document 1, in order to extract the text from the document, another document that can be grouped with the document from which the text such as HTML text of the same site is extracted is required. That is, according to the method described in Patent Document 1, if another document that can be grouped with the document from which the text is to be extracted cannot be obtained, the text cannot be extracted from the document.

  The present invention has been made in view of the above-described problems, and even when another document that can be grouped with a document from which the text is to be extracted cannot be obtained, the text can be appropriately extracted. It is an object to provide a text extracting device, a text extracting method, and a text extracting program.

  In order to achieve the above object, a text extraction apparatus according to the present invention includes a target document input unit that inputs a target document that is a text extraction target, and a character string from each of the target documents input by the target document input unit. Element extracting means for extracting one or more elements, related document input means for inputting a related document related to the target document, and character string extracting means for extracting a character string from the related document input by the related document input means And whether or not the character string extracted by the character string extracting unit is included in the element extracted by the element extracting unit, and if it is determined that the element is included, the element is the text of the target document. A score calculating means for calculating a score by increasing a score indicating the degree of inclusion, and a pair of elements based on the score calculated by the score calculating means Comprising specifying means for specifying an element that contains the body of the document, and output means for outputting information corresponding to the identified element by a particular means.

  For example, a microblog comment on content on the Internet often has a description relating to the text of the content. By using this comment, it is possible to grasp what the content is described. Some of the metadata of the document to be analyzed similarly indicates the contents of the document. In the text extracting device according to the present invention, a score indicating the degree of inclusion of the text of the target document is calculated with respect to the elements extracted from the target document using the microblog comment, metadata, or the like as a related document. The containing element is specified. The related document is not a document that can be grouped with a document from which a body text such as HTML text of the same site is extracted. That is, according to the text extracting apparatus of the present invention, the text can be appropriately extracted even when another document that can be grouped with the text from which the text is to be extracted cannot be obtained.

  The element extraction means may store tag information indicating tags that can be included in the target document in advance as an extraction target, and extract an element indicated by the stored tag information. According to this configuration, elements can be appropriately extracted when the target document is an HTML document.

  The element extraction means stores in advance tag information indicating a tag that can be included in the target document as an exclusion target, and can exclude an element indicated by the stored tag information from the extraction target. According to this configuration, an element that is considered not to include a text in advance can be an analysis target, and the present invention can be implemented efficiently and appropriately.

  The character string extraction means can extract a character string from a related document by performing morphological analysis. According to this configuration, a character string can be appropriately extracted from a related document, and the present invention can be appropriately implemented.

  The related document input unit inputs the metadata of the target document as the related document, and the score calculation unit stores the feature word score corresponding to the character string in advance and corresponds to the character string extracted by the character string extracting unit The score can be calculated based on the feature word score. According to this configuration, the present invention can be appropriately implemented using metadata as a related document.

  The related document input unit inputs a document including information indicating the target document as a related document in the character string, and the score calculation unit calculates a score based on the appearance frequency of the character string extracted by the character string extraction unit. , That can be. According to this configuration, the present invention can be appropriately implemented using a microblog comment including information indicating a target document such as a URL (Uniform Resource Locator) as a related document.

  The image processing apparatus further includes a feature amount generation unit that generates a feature amount necessary for machine learning with respect to the element extracted by the element extraction unit, and the specifying unit includes the score calculated by the score calculation unit and the feature generated by the feature amount generation unit. It is possible to specify an element including the text of the target document among the elements by machine learning based on the quantity. According to this configuration, the text can be appropriately determined by machine learning.

  The feature quantity generation means can generate the feature quantity based on whether or not the element extracted by the element extraction means includes a link to another document. According to this configuration, it is possible to appropriately determine the text based on links to other documents included in the target document.

  The feature quantity generation means may generate the feature quantity based on the number of punctuation marks or the length of the character string in the character string included in the element extracted by the element extraction means. According to this configuration, it is possible to appropriately determine the text based on the text likeness of the character string included in the target document.

  By the way, the present invention can be described as an invention of a text extracting device as described above, and can also be described as an invention of a text extracting method and a text extracting program as follows. This is substantially the same invention only in different categories, and has the same operations and effects.

  That is, the text extraction method according to the present invention includes a target document input step for inputting a target document that is a text extraction target, and one or more elements each including a character string from the target document input in the target document input step. In the element extraction step to extract, the related document input step to input the related document related to the target document, the character string extraction step to extract the character string from the related document input in the related document input step, and the character string extraction step It is determined whether or not the extracted character string is included in the element extracted in the element extraction step, and when it is determined that the extracted character string is included, a score indicating a degree that the element includes the body of the target document is determined for the element. A score calculation step for calculating a score by increasing the score, and a score calculated in the score calculation step Te including a specifying step of specifying an element that contains the body of the target document of each element, and an output step of outputting information corresponding to the specified elements in a specific step.

  In the text extraction program according to the present invention, the computer includes a target document input unit that inputs a target document that is a text extraction target, and one or more character strings each including a character string from the target document input by the target document input unit. Element extracting means for extracting the elements of the document, related document input means for inputting a related document related to the target document, character string extracting means for extracting a character string from the related document input by the related document input means, and a character string It is determined whether or not the character string extracted by the extraction unit is included in the element extracted by the element extraction unit, and when it is determined that the character string is included, the degree to which the element includes the text of the target document is determined. A score calculation means for calculating a score by increasing the score shown, and a pair of elements based on the score calculated by the score calculation means Specifying means for specifying an element that contains the body of the document, to function as, and output means for outputting information corresponding to the identified element by a particular means.

  In the present invention, a comment indicating the degree to which the text of the target document is included is calculated for the elements extracted from the target document using microblog comments and metadata as related documents, and the element including the text is specified. The The related document is not a document that can be grouped with a document from which a body text such as HTML text of the same site is extracted. That is, according to the present invention, the text can be appropriately extracted even when another document that can be grouped with the document from which the text is to be extracted cannot be obtained.

It is a figure which shows the function structure of the text extraction apparatus which concerns on embodiment of this invention. It is a figure which shows the example of the document of the extraction object of a text. It is a figure which shows the information used in order to extract an element from a target document. It is a table which shows the information memorized by a document division data storage part. It is a figure which shows the information in the case of using metadata as a related document. It is a figure which shows the information in the case of using the information of a microblog as a related document. It is a table which shows the information produced | generated in order to determine whether an element contains the text from a target document. It is a figure which shows the hardware constitutions of the text extraction apparatus which concerns on embodiment of this invention. It is a flowchart which shows the process which concerns on the input of an object document among the processes (text extraction method) performed with the text extraction apparatus which concerns on embodiment of this invention. It is a flowchart which shows the process which produces | generates the text identifier at the time of using metadata as a related document among the processes (text extraction method) performed with the text extraction apparatus which concerns on embodiment of this invention. It is a flowchart which shows the process which produces | generates the text identifier at the time of using the comment of a microblog as a related document among the processes (text extraction method) performed with the text extraction apparatus which concerns on embodiment of this invention. It is a flowchart which shows the process which extracts the text among the processes (text extraction method) performed with the text extraction apparatus which concerns on embodiment of this invention. It is a figure which shows the structure of the text extraction program which concerns on embodiment of this invention with a recording medium.

  Hereinafter, a text extracting device, a text extracting method, and a text extracting program according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 shows a text extracting apparatus 10 according to the present embodiment. The text extraction device 10 is a device that extracts a text from a target document that is a text extraction target. The target document that is the subject of text extraction is, for example, Web content published on the Internet. The content includes, for example, a character string to be displayed. The content is described in, for example, HTML. The Web content includes a main body that is a main part, and includes many character strings other than the main body such as advertisements in addition to the main body. The text extraction apparatus 10 according to the present embodiment is an apparatus that extracts a text from a target document in which a character string that is a text and a character string other than the text can be mixed. The text extracted in this way is used, for example, for generating the magazine-like content described above.

  The text extracting device 10 is connected to a device (for example, a Web server) that outputs a target document so as to acquire (receive) the target document via a network such as the Internet. Further, as will be described later, a related document related to the target document is used for extracting the text. The text extracting device 10 is connected to a device (for example, a server that provides a microblog service) that outputs the relevant document so that the relevant document can be acquired (received) via a network such as the Internet. .

  As shown in FIG. 1, the text extracting apparatus 10 includes a document input unit 100, a document storage unit 110, a divided data creation unit 120, a document divided data storage unit 130, a metadata input unit 140, and a microblog input unit. 150, a microblog storage unit 160, a text identifier creation unit 170, a text identifier storage unit 180, a noise determination unit 190, a text noise storage unit 200, a feature value generation unit 210, a text determination unit 220, The divided data combining unit 230 is provided.

  The document input unit 100 is a target document input unit that inputs a target document that is a text extraction target. For example, the document input unit 100 acquires (receives) a target document by requesting a document by specifying a URL to a Web server via the Internet. The document input unit 100 stores a site from which the target document is acquired in advance, and acquires the target document from the site at a designated timing. Information indicating the timing for acquiring the site and the target document is input to the text extracting device 10 in advance by an administrator of the text extracting device 10 or the like. A plurality of target documents may be acquired.

  For example, the target document is described in HTML as shown in FIG. As shown in FIG. 2, a document (HTML document) described in HTML includes tags indicated by “<” and “>”, and is divided into elements by tags. There are a plurality of types of tags depending on the contents of the elements.

  The document input unit 100 associates a URL (for example, http://xxx.com) with the acquired target document and inputs it to the document storage unit 110. The document storage unit 110 is a unit that stores the target document input from the document input unit 100.

  The divided data creation unit 120 is an element extraction unit that extracts one or more elements each containing a character string from the target document input by the document input unit 100 and stored in the document storage unit 110. The divided data creation unit 120 includes a text extraction target area storage unit 121, a text extraction target area specifying unit 122, a removal target element storage unit 123, an element removal unit 124, a split target element storage unit 125, and a document division unit. 126. Specifically, the extraction of elements is performed by dividing an HTML document based on tags as follows. Note that the elements in the present invention do not necessarily match the elements of the HTML document.

  The text extraction target area storage unit 121 stores tag information indicating a tag of a part (element) as a text extraction target in the target document as an extraction target. This tag information is input to the text extracting device 10 in advance by an administrator of the text extracting device 10 or the like. The tag information stored in the text extraction target area storage unit 121 is, for example, information illustrated in FIG.

  The text extraction target area specifying unit 122 reads the target document stored in the document storage unit 110 and specifies the text extraction target area (part) from which the text is to be extracted from the target document. Specifically, the text extraction target area specifying unit 122 reads tag information stored in the text extraction target area storage unit 121 and extracts a text extraction target area that is an element indicated by the tag information (corresponding Leave only the elements you want). FIG. 2B shows an example of information (element indicated by a tag of <target_a>) extracted as a text extraction target area from the target document.

  Depending on the HTML document, a part that has a high possibility of including the text may be indicated by a specific tag. For example, in a page that employs an advertisement of a specific company, a text area that is desired to be analyzed in order to place an optimal advertisement is set as an element indicated by a tag including the description “AdSense” (enclosed by the element). ) Often. The element indicated by this tag is very likely to contain the text. This improves the accuracy of text extraction. The text extraction target area specifying unit 122 outputs the extracted information to the element removing unit 124.

  The removal target element storage unit 123 stores tag information indicating a tag of a portion (element) to be excluded from the text extraction target in the target document as an exclusion target. This tag information is input to the text extracting device 10 in advance by an administrator of the text extracting device 10 or the like. The tag information stored in the removal target element storage unit 123 is, for example, information illustrated in FIG. For example, a tag of <script> that is not related to the text is tag information to be excluded.

  The element removal unit 124 excludes (removes) unnecessary elements (portions) not to be extracted from the text from the information on the text extraction target area input from the text extraction target area specifying unit 122. Specifically, the element removal unit 124 reads the tag information stored in the removal target element storage unit 123, and excludes the element indicated by the tag information from the text extraction target area. FIG. 2C shows an example of information on the text extraction target area that has been removed (information in which the element indicated by the tag <script> is excluded). In this way, noise can be reduced by removing elements indicated by unnecessary tags. The text extraction target area specifying unit 122 outputs the information that has been excluded as described above to the document dividing unit 126.

  The division target element storage unit 125 stores tag information indicating a tag of a portion (element) that is a unit for dividing the target document. This tag information is input to the text extracting device 10 in advance by an administrator of the text extracting device 10 or the like. The tag information stored in the division target element storage unit 125 is, for example, information illustrated in FIG. For example, tags such as <div> and <p> are tag information that is a unit to be divided.

  The document dividing unit 126 divides the text extraction target area information input from the element removing unit 124. Specifically, the document dividing unit 126 reads the tag information stored in the division target element storage unit 125 and divides the text extraction target area information in units of elements indicated by the tag information. By dividing the target document in this way, it can be determined whether or not the element includes the text in an appropriate unit. The document dividing unit 126 outputs the information divided as described above to the document divided data storage unit 130.

  Note that the processing by the text extraction target area specifying unit 122, the element removing unit 124, and the document dividing unit 126 does not necessarily have to be performed all, and any of them may be performed.

  The document division data storage unit 130 is a unit that stores the target document after division input from the document division unit 126. The document division data storage unit 130 stores information by storing information in a table as shown in FIG. As shown in the table of FIG. 4, the document division data storage unit 130 stores URLs, paths, and texts in association with each other. The URL is information indicating the acquisition source of the target document. The path is information that uniquely identifies the divided element. The path is described using, for example, an XPath expression method. The path is generated, for example, when the document is divided by the document dividing unit 126. The path need not be XPath as long as it is information that can uniquely identify an element in the document.

  The text is an element of the divided target document including the character string to be extracted from the body. Information stored in the document division data storage unit 130 may have a tree structure. That is, a certain text may have a structure including another text. For example, the text of the path of html / body / div [3] shown in FIG. 4 includes the text of the path of html / body / div [3] / div [1]. Note that a path must be identifiable when there are multiple elements with the same name at the same level. Further, as shown in FIG. 4, the document division data storage unit 130 may store a plurality of divided target documents. The unit stored in the document divided data storage unit 130 is a unit for determining whether or not the text of the target document is included.

  The metadata input unit 140 is one of related document input means for inputting a related document related to the target document. As described above, the related document is information used to extract the text from the target document. The metadata input unit 140 inputs metadata of the target document as a related document. The metadata is data about the target document, and includes a character string in this embodiment. For example, when the target document is an HTML document, the metadata is text data explaining an outline of the HTML document. Such metadata is prepared, for example, by a creator of an HTML document. The metadata is an element indicated by a tag including a description “description” in an HTML document, for example. For example, when the target document is as shown in FIG. 5A, the element 300 corresponds to metadata.

  The metadata input unit 140 stores in advance a path or query necessary for specifying an area for extracting metadata from the target document as shown in FIG. The path or query is input to the text extracting device 10 in advance by an administrator of the text extracting device 10 or the like. In the example shown in FIG. 5B, the information is described using the XPath query description method, but may be described by another method as long as the description method can extract the area including the metadata. The metadata input unit 140 reads the target document stored in the document storage unit 110 and extracts metadata using the above-described path or query. Although the metadata is extracted from the target document, the metadata itself can be regarded as a related document related to the independent target document.

  In addition to acquiring the metadata included in the HTML document as described above, the metadata input unit 140, for example, has an RSS (RDF site) attached to the HTML document as shown in FIG. summary, rich Site summary, and really simple syndication) may be acquired as metadata. The metadata input unit 140 may acquire an element indicating a specific tag from the acquired RSS as metadata. The specific tag is, for example, a description tag that is a tag of an element including an article description for an item having the same URL. The metadata input unit 140 inputs the extracted metadata to the body identifier creation unit 170.

  The microblog input unit 150 is one of related document input means for inputting a related document related to the target document. The microblog input unit 150 inputs microblog information as a related document. For ease of understanding, here is an example of microblogging Twitter. Twitter often adds a URL and writes a comment for that URL. If this URL indicates the acquisition source of the target document, a text can be extracted from the target document using a comment for the URL.

  The microblog input unit 150 may acquire and receive (receive) a document from a server that provides a microblog service via the Internet and requests a server that stores the document, for example. It is also possible to receive document data. The document input unit 100 stores a site from which a related document is acquired in advance, and acquires a target document from the site at a designated timing. Information indicating the timing for acquiring the site and the target document is input to the text extracting device 10 in advance by an administrator of the text extracting device 10 or the like. A plurality of target documents may be acquired.

  The microblog input unit 150 extracts and acquires a document including the URL of the target document from the document acquired from the server. The URL of the target document is stored in advance in the microblog input unit 150. That is, the microblog input unit 150 inputs a document (microblog comment) including information indicating the target document as a related document in a character string. The microblog input unit 150 outputs the input microblog comment to the microblog storage unit 160.

  The microblog storage unit 160 is means for storing a microblog comment input from the microblog input unit 150. The microblog storage unit 160 stores information by storing the information in a table as shown in FIG. As shown in the table of FIG. 6A, the microblog storage unit 160 stores URLs, UserIDs, and texts in association with each other. The URL is information indicating the URL of the target document included in the microblog comment. UserID is information that identifies the user who posted the comment. The UserID is acquired when the microblog input unit 150 acquires a microblog comment. The text is a character string that is a comment body of the microblog.

  The text identifier creating unit 170 is a character string extracting unit that extracts a character string from related documents that are metadata or microblog comments. The body identifier creating unit 170 creates a body identifier used for extracting the body of the target document based on the extracted character string. As shown in FIG. 1, the body identifier creating unit 170 includes a body identifier creating unit 171 based on metadata and a body identifier creating unit 172 based on related comments.

  The body identifier creation unit 171 based on metadata extracts a character string that is a morpheme (word) from the metadata by performing morphological analysis on the metadata input from the metadata input unit 140. The extracted morpheme is used as a body identifier for extracting the body from the target document. Further, the text-body identifier creating unit 171 based on metadata stores in advance a feature word score corresponding to the morpheme for each morpheme that can be obtained by the morpheme analysis. As described above, the metadata-based body text identifier creation unit 171 is a part of score calculation means described later in more detail. The feature word score is preset by, for example, an administrator of the text extracting device 10 and input to the text extracting device 10. This feature word score is set so that, for example, the higher the degree that the sentence including the morpheme is the body of the target document, the higher the value is. The text identifier creation unit 171 based on metadata stores the feature word score corresponding to the morpheme by storing information in a table (default identifier score dictionary) shown in FIG. 5C, for example.

  The body identifier creating unit 171 using metadata associates the URL of the target document related to the metadata, the obtained morpheme, and the feature word score stored in advance corresponding to the morpheme, and stores them in the body identifier storage unit 180. The text identifier storage unit 180 stores the information by storing the information in a table shown in FIG.

  The text identifier creating unit 172 based on the related comments reads the microblog comment information stored in the microblog storage unit 160. Subsequently, the body identifier creating unit 172 based on the related comment uses the text (information in the text column in FIG. 6A) of the read information, and the user name starting from the URL, hash tag, and at-sign that is not used for analysis. Etc. are excluded (normalization processing is performed). The information on which the normalization process has been performed is, for example, information illustrated in FIG.

  Subsequently, the body identifier creating unit 172 based on the related comments performs a morphological analysis on the normalized text to extract a character string that is a morpheme (word) from the text. The extracted morpheme is used as a body identifier for extracting the body from the target document. Subsequently, the body identifier creating unit 172 based on the related comments calculates a score for each morpheme as a TFIDF value based on the appearance frequency of the morpheme as follows. First, a TF (Term Frequency) value is obtained for each target document (URL of the target document) and each extracted morpheme. First, the body identifier creating unit 172 based on the related comment sets the morpheme obtained from the text of the microblog comment associated with the URL of the target document as the processing target. Subsequently, for each morpheme, a TF value is calculated by dividing the number of morphemes obtained by morpheme analysis by the number of all morphemes obtained by morpheme analysis. The calculated TF value is a value for each URL and morpheme as shown in FIG.

  In addition, the body identifier creating unit 172 based on related comments calculates an IDF (Inverse Document Frequency) value by regarding a comment group associated with one URL (one target document) as one document. For the calculation of the IDF value, morphemes obtained from microblog comment text associated with URLs of a plurality of target documents are used. A plurality of target documents are set in advance by an administrator of the text extracting apparatus 10 or the like. For each morpheme, the body identifier creation unit 172 based on the related comment calculates an IDF value by taking the logarithm by dividing the total number of documents (the number of URLs of the target document) by the number of documents in which the comment includes the morpheme. The calculated IDF value is a value for each morpheme as shown in FIG.

  The body identifier creating unit 172 based on the related comment calculates the TFIDF value by taking the product of the TF value calculated for each morpheme and the IDF value. As described above, the body identifier creation unit 172 based on the related comments regards a comment group associated with one URL (one target document) as one document and obtains TFIDF. Further, the TFIDF value may be calculated by calculating the IDF value by regarding a comment group associated with one user as one document. This TFIDF value indicates, for example, the degree to which the morpheme is a characteristic word, that is, the degree to which the text containing the morpheme is the text of the target document. As described above, the body identifier creating unit 172 based on the related comment is a part of score calculation means described later in more detail.

  The body identifier creation unit 172 based on the related comment stores the URL of the target document related to the comment, the obtained morpheme, and the calculated TFIDF value in the body identifier storage unit 180 in association with each other. The text identifier storage unit 180 stores the above information by storing the information in a table shown in FIG.

  The noise determination unit 190 is a means for acquiring information for determining noise included in the target document. For example, in this embodiment, noise is determined using the structure of an HTML document. In the case of a web page, if it is an advertisement, a collection of links to other sites, or links to other documents (related web pages) are linked, that is, if it is a linked list, it is not related to the main text. There is a high possibility of noise. By determining that an element (HTML tag) including a link list is not subject to text extraction or has a low possibility of being a text, the accuracy of text extraction is improved.

  Specifically, the noise determination unit 190 acquires the elements of the target document stored in the document division data storage unit 130. The noise determination unit 190 determines whether an <a> tag indicating a link is included in the text of the element. The noise determination unit 190 acquires the number (number of links) including the <a> tag for each element as information related to noise. Further, the noise determination unit 190 calculates the value of the ratio between the length of the character string included in the <a> tag and the length of the character string not included in the <a> tag for each element, and obtains it as information related to noise. To do.

  The noise determination unit 190 stores the information related to the acquired noise in the body noise storage unit 200. Information related to noise is used as a feature amount in machine learning described later (a tuning point is provided). As described above, the noise determination unit 190 is a part of a feature amount generation unit described later in more detail.

  The feature quantity generation unit 210 is a unit that generates information for determining whether or not the element of the target document stored in the document division data storage unit 130 includes the target document.

  The feature quantity generation unit 210 determines whether or not the morpheme extracted from the related document by the body identifier creation unit 170 is included in the element, and if it is determined that the element is included, It is a score calculation means for calculating a score by increasing the score indicating the degree of including the text of the target document.

  The feature quantity generation unit 210 acquires the elements of the target document stored in the document division data storage unit 130. In addition, information for extracting the text stored in the text identifier storage unit 180 (for example, information shown in FIGS. 5D and 6E) is acquired. The information for extracting the text acquired here is information related to the same URL as the element of the target document. The feature quantity generation unit 210 determines whether or not the morpheme stored in the body identifier storage unit 180 is included in the element of the target document. When the feature quantity generation unit 210 determines that the morpheme stored in the body identifier storage unit 180 is included in the element of the target document, the feature amount generation unit 210 uses the score (feature word score or TFIDF value) associated with the morpheme The total score is calculated by summing up the elements. Note that the initial value of the total score of each element is set to zero, for example.

  For example, when the URLs of the target document have the scores of 0.1, 1.0, and 4.0 stored in the body identifier storage unit 180 for the morphemes “Today”, “Weather”, and “Sunny”, respectively, The total score of the element “Today's weather <p> sunny <p>” is 0.1 + 1.0 + 4.0 = 5.1.

  The score for each element may be separated from the score based on metadata and the score based on microblog comments. The total score for each element (path) calculated is acquired as information as shown in FIG. The feature quantity generation unit 210 inputs information indicating the generated total score to the text determination unit 220.

  When machine learning is used for the determination of the text, the feature value generation unit 210 further generates (acquires) feature values necessary for machine learning. For example, the feature amount generation unit 210 acquires information related to noise for each element generated by the noise determination unit 190 and stored in the text noise storage unit 200 and uses it as a feature amount.

  Alternatively, the feature value generation unit 210 generates the feature value of the element based on the number of punctuation marks or the length of the character string in the character string included in the element of the target document. Specifically, the feature value generation unit 210 specifies the number of punctuation marks and the length of the character string from the character string included in the element, and uses the number or length information as the feature value of each element. . The feature amount for each element is acquired as information as shown in FIG. 7B, for example. In the description of the feature quantity “1: 0.7, 2: 1, 3: 3,...” Shown in FIG. 7B, the first feature quantity (feature) has a value of 0.7 and 2 The value of the feature value (feature) of the eye is 1, and the value of the third feature value (feature) is 3. The feature amount includes the above total score. As described above, the feature quantity generation unit 210 is also a feature quantity generation unit.

  Based on the total score for each element input from the feature value generation unit 210, the body determination unit 220 is a specifying unit that specifies the elements including the body of the target document among the elements. For example, the text determination unit 220 stores a threshold value, and determines an element whose total score exceeds the threshold as an element including the text. The threshold value is set in advance by the administrator of the text extracting device 10 and is input to the text extracting device 10. For example, when the total score is an example shown in FIG. 7A and the threshold is 3.5, the paths are html / body / div [3] / div [1] and html / body / div. The element of [3] / div [1] / p [1] is determined as an element including the text.

  The body determination unit 220 may determine an element including the body of the target document among the elements by machine learning based on the feature amount including the total score for each element input from the feature amount generation unit 210. The body determination unit 220 uses a feature amount of each element as an explanatory variable in machine learning, and uses a value indicating whether or not the element is a body text as an objective variable in machine learning. A learning model (machine learner) obtained by machine learning The above determination is made using. In the case of using machine learning, a document in which the text is included in advance is prepared in advance, and learning is performed in advance using the document.

  The learning model described above outputs, for example, a value indicating the degree of the text of the target document for each element. If the value is 0 or more, it is determined that the element includes the text. If it is less than 0, it is determined that the element does not contain a body. As a machine learning method, for example, supervised machine learning such as a support vector machine (SVM) or a neural network can be used.

  The body determination unit 220 inputs, to the divided data combining unit 230, elements determined to include a body (that is, the body) for each target document.

  The divided data combining unit 230 is an output unit that outputs information corresponding to the element input from the text determination unit 220. The divided data combining unit 230 combines the character strings (text) of elements input from the body determination unit 220 for each target document. Note that, when an element having an overlapping character string among the elements of the tree structure is determined to be the text, only the highest element among them may be the element to be combined. The divided data combining unit 230 outputs, for example, a character string (text) combined with another device connected to the body extracting device 10 as the body of the target document. Specifically, a character string such as “Today's sunny weather, temperature is sunny at 30 degrees” is output as the text of the target document. Thus, the character string which is the text extracted and output by the text extracting device 10 is used, for example, for generating the magazine-like content described above. The functional configuration of the text extracting device 10 has been described above.

  FIG. 8 shows a hardware configuration of the text extracting apparatus 10. As shown in FIG. 8, a text extracting device 10 includes a central processing unit (CPU) 1001, a random access memory (RAM) 1002 and a read only memory (ROM) 1003, a communication module 1004 for communication, It is configured to include a computer including hardware such as an auxiliary storage device 1005 such as a hard disk. The functions of the text extracting device 10 described above are exhibited when these components are operated by a program or the like. The above is the configuration of the text extracting apparatus 10.

  Subsequently, a text extraction method, which is a process executed by the text extraction apparatus 10 according to the present embodiment, will be described using the flowcharts of FIGS. As shown in FIG. 9, in this process, first, the target document is acquired by the document input unit 100, input to the body extraction device 10, and stored in the document storage unit 110 (S01, target document input step).

  Subsequently, the divided data creation unit 120 extracts an element that is a determination target of whether or not the text is included from the target document stored in the document storage unit 110 (S02 to S05, element extraction step). Specifically, it is performed as follows. First, the text extraction target area specifying unit 122 specifies the text extraction target area from the target document based on the information stored in the text extraction target area storage unit 121 (the text extraction target area is narrowed down) (S02). ).

  Subsequently, the element removal unit 124 excludes unnecessary elements not to be extracted from the text from the text extraction target area based on the information stored in the removal target element storage unit 123 (S03). Subsequently, the document dividing unit 126 divides the text extraction target area based on the information stored in the division target element storage unit 125 (S04). Information on the divided elements is stored in the document divided data storage unit 130 (S05). The above is the processing related to the target document shown in FIG.

  In the text extracting apparatus 10, while processing related to the target document is performed, processing related to related documents as shown in FIGS. 10 and 11 is also performed. The processing shown in FIG. 10 uses metadata as related documents. Process. This process is performed after the process related to the target document shown in FIG. 9 (at least the process of S01) is performed.

  In this process, first, the metadata input unit 140 reads a query for extracting metadata (S11, related document input step). Subsequently, the metadata input unit 140 reads the target document stored in the document storage unit 110 (S12, related document input step). Subsequently, the metadata input area 140 extracts the metadata target area (metadata) from the target document using the above query (S13, related document input step).

  Subsequently, the morpheme is extracted by performing morpheme analysis on the metadata extracted by the metadata input unit 140 by the metadata body identifier creating unit 171 (S14, character string extraction step). Subsequently, the default identifier score dictionary is referred to by the body identifier creation unit 171 based on metadata, and the feature word score is associated with the extracted morpheme (S15, character string extraction step). The obtained morpheme and feature word score are stored in the body identifier storage unit 180 (S16, character string extraction step). The above is the processing when metadata is used as the related document shown in FIG.

  Next, processing when a microblog comment is used as a related document shown in FIG. 11 will be described. This process can be performed independently of the process related to the target document shown in FIG. Further, only one of the processing shown in FIG. 10 and the processing shown in FIG. 11 needs to be performed. In this process, first, a microblog comment including the URL of the target document is acquired by the microblog input unit 150, input to the text extracting device 10, and stored in the microblog storage unit 160 (S21, related document). Input step).

  Subsequently, the comment information of the microblog stored in the microblog storage unit 160 is read out by the body identifier creating unit 172 based on the related comment. Subsequently, information such as a URL, a hash tag, and a user name starting with an at sign that is not used for analysis is excluded from the text of the read information (normalization processing is performed) (S22). Subsequently, the morpheme is extracted by performing morpheme analysis on the text subjected to the normalization process by the body identifier creating unit 172 based on the related comment (S23, character string extraction step).

  Subsequently, the TF value is calculated for each URL and the extracted morpheme of the target document by the body identifier creation unit 172 based on the related comment (S24, character string extraction step). Further, an IDF value is calculated for each extracted morpheme (S25, character string extraction step). Subsequently, the TFIDF value is calculated for each URL and the extracted morpheme of the target document from the calculated TF value and IDF value. The obtained morpheme and TFIDF value are stored in the body identifier storage unit 180 (S26, character string extraction step). The above is the processing when metadata is used as the related document shown in FIG.

  Next, the process of extracting the text based on the information generated as described above shown in FIG. 12 will be described. In this processing, first, the feature quantity generation unit 210 stores each element of the target document stored in the document division data storage unit 130 in the body identifier storage unit 180 in association with the target document. Whether or not is included. Subsequently, the feature amount generation unit 210 calculates the total score of the elements based on the determination (S31, score calculation step). This total score is one of the feature quantities.

  Further, when the text is determined by machine learning, a feature amount other than the score is obtained (S32, feature amount generation step). For example, the feature amount is calculated by the noise determination unit 190 based on the number of links of each element of the target document. The calculated text content is stored in the noise storage unit 200 and acquired by the feature value generation unit 210. Further, the feature quantity generation unit 210 may generate a feature quantity from the number of punctuation marks of the character string included in the element and the length of the character string. Information indicating the feature amount for each element generated by the feature amount generation unit 210 is input to the text determination unit 220.

  Subsequently, the body determination unit 220 determines whether or not the element includes the element of the target document based on the feature amount for each element (S33, specific step). This determination is performed based on the learning model obtained by comparing the total score and the threshold or by machine learning as described above. Information on an element determined to include a text is input from the text determining unit 220 to the divided data combining unit 230. Subsequently, the divided data combination unit 230 combines the character strings of the elements and outputs the combined information as information indicating the text of the target document (S34, output step). The above is the text extraction method that is the process executed by the text extraction apparatus 10 according to the present embodiment.

  In many cases, microblog comments on content on the Internet are described in the body of the content. By using this comment, it is possible to grasp what the content is described. Some of the metadata of the document to be analyzed similarly indicates the contents of the document. In the present embodiment, a score indicating the degree of inclusion of the text of the target document is calculated for the elements extracted from the target document using microblog comments, metadata, and the like as related documents. Identified. The related document is not a document that can be grouped with a target document such as HTML text on the same site. That is, according to the present embodiment, the text can be appropriately extracted even when another document that can be grouped with the document from which the text is to be extracted cannot be obtained.

  In this embodiment, the score is increased when a character string appearing in a related document appears, and it is described in Patent Document 1 that the lower the appearance frequency, the higher the possibility that the text is a text. The method is fundamentally different from the method.

  Further, if elements are extracted using tags as in the present embodiment, elements can be appropriately extracted when the target document is an HTML document. Further, if elements are excluded from the text extraction target as in the present embodiment, elements that are considered not to include the text in advance can be analyzed, and the present invention is implemented efficiently and appropriately. can do.

  If a character string is extracted from a related document by morphological analysis as in the present embodiment, the character string can be appropriately extracted from the related document, and the present invention can be appropriately implemented. Further, according to the above-described method, when metadata or microblog comments are used as related documents, the present invention can be implemented appropriately for each.

  If the text is determined by machine learning, the text can be determined appropriately. In this case, the text can be appropriately determined by generating the feature amount based on the number of links and punctuation marks included in the target document or the length of the character string.

  Subsequently, a text extracting program for causing a computer to execute the above-described series of processing by the text extracting device 10 will be described. As shown in FIG. 13, the text extraction program 40 is stored in a program storage area 31 that is inserted into a computer and accessed, or formed in a recording medium 30 provided in the computer.

  The text extraction program 40 includes a document input module 400, a document storage module 410, a divided data creation module 420, a document divided data storage module 430, a metadata input module 440, a microblog input module 450, and a microblog storage. Module 460, text identifier creation module 470, text identifier storage module 480, noise determination module 490, text noise storage module 500, feature quantity generation module 510, text determination module 520, and divided data combination module 530 It is configured with. Document input module 400, document storage module 410, divided data creation module 420, document divided data storage module 430, metadata input module 440, microblog input module 450, microblog storage module 460, and body identifier Realized by executing the creation module 470, the body identifier storage module 480, the noise determination module 490, the body noise storage module 500, the feature amount generation module 510, the body determination module 520, and the divided data combination module 530. The functions to be performed include the document input unit 100, the document storage unit 110, the divided data creation unit 120, the document divided data storage unit 130, the metadata input unit 140, and the microblog input of the text extracting device 10 described above. 150, a microblog storage unit 160, a text identifier creation unit 170, a text identifier storage unit 180, a noise determination unit 190, a text noise storage unit 200, a feature value generation unit 210, a text determination unit 220, The function is the same as that of the divided data combination unit 230.

  Note that a part or all of the text extraction program 40 may be transmitted via a transmission medium such as a communication line and received and recorded (including installation) by another device. Further, each module of the text extraction program 40 may be installed in any one of a plurality of computers instead of one computer. In that case, a series of processes of the text extraction program 40 described above are performed by the computer system of the plurality of computers.

  DESCRIPTION OF SYMBOLS 10 ... Text extraction apparatus, 100 ... Document input part, 110 ... Document storage part, 120 ... Divided data preparation part, 121 ... Text extraction object area storage part, 122 ... Text extraction object area specification part, 123 ... Removal object element storage part , 124 element removal unit, 125 division target element storage unit, 126 document division unit, 130 document division data storage unit, 140 metadata input unit, 150 microblog input unit, 160 microblog storage unit, 170 ... Text identifier creation unit, 171 ... Text identifier creation unit based on metadata, 172 ... Text identifier creation unit based on related comments, 180 ... Text identifier storage unit, 190 ... Noise determination unit, 200 ... Text noise storage unit, 210 ... Features Quantity generation unit, 220 ... text determination unit, 230 ... divided data combination unit, 1001 ... CPU, 1002 ... RAM, 1003 ... OM, 1004 ... communication module, 1005 ... auxiliary storage device, 30 ... recording medium, 31 ... program storage area, 40 ... text extraction program, 400 ... document input module, 410 ... document storage module, 420 ... divided data creation module, 430 Document division data storage module 440 Metadata input module 450 Microblog input module 460 Microblog storage module 470 Text identifier creation module 480 Text identifier storage module 490 Noise determination module 500 Text noise storage module 510... Feature quantity generation module 520... Text determination module 530.

Claims (11)

A target document input means for inputting a target document from which a text is to be extracted;
Element extraction means for extracting one or more elements each containing a character string from the target document input by the target document input means;
A related document input means for inputting a related document related to the target document;
A character string extracting means for extracting a character string from the related document input by the related document input means;
It is determined whether or not the character string extracted by the character string extraction unit is included in the element extracted by the element extraction unit. If it is determined that the character string is included, the element is included in the target document. Score calculating means for calculating a score by increasing a score indicating the degree of including the body;
Specifying means for specifying an element including the text of the target document among the elements based on the score calculated by the score calculating means;
Output means for outputting information according to the element specified by the specifying means;
A text extracting device comprising:   The text extracting device according to claim 1, wherein the element extraction unit stores in advance tag information indicating a tag that can be included in the target document as an extraction target, and extracts an element indicated by the stored tag information.   3. The element extraction unit according to claim 1, wherein the element extraction unit stores in advance tag information indicating a tag that can be included in the target document as an exclusion target, and excludes an element indicated by the stored tag information from the extraction target. Text extraction device.   The said character string extraction means is a text extraction apparatus as described in any one of Claims 1-3 which extracts a character string from a related document by performing a morphological analysis. The related document input means inputs metadata of the target document as a related document,
The score calculation unit stores a feature word score corresponding to the character string in advance, and calculates a score based on the feature word score corresponding to the character string extracted by the character string extraction unit. The text extraction device according to any one of to 4. The related document input means inputs a document including information indicating the target document as a related document in a character string,
The text extraction device according to any one of claims 1 to 5, wherein the score calculation unit calculates a score based on an appearance frequency of the character string extracted by the character string extraction unit. A feature amount generating means for generating a feature amount necessary for machine learning for the element extracted by the element extracting means;
The specifying unit specifies an element including the main body of the target document by machine learning based on the score calculated by the score calculation unit and the feature amount generated by the feature amount generation unit. The text extracting device according to any one of 1 to 6.   8. The text extracting device according to claim 7, wherein the feature quantity generation unit generates a feature quantity based on whether or not the element extracted by the element extraction unit includes a link to another document.   9. The text extraction according to claim 7 or 8, wherein the feature quantity generation means generates a feature quantity based on the number of punctuation marks or the length of the character string in the character string included in the element extracted by the element extraction means. apparatus. A target document input step for inputting a target document to be subjected to text extraction;
An element extraction step of extracting one or more elements each containing a character string from the target document input in the target document input step;
A related document input step for inputting a related document related to the target document;
A character string extraction step of extracting a character string from the related document input in the related document input step;
It is determined whether or not the character string extracted in the character string extraction step is included in the element extracted in the element extraction step. If it is determined that the character string is included, the element is included in the target document. A score calculating step for calculating a score by increasing a score indicating a degree of including the body;
A specifying step of specifying an element including the body of the target document among the elements based on the score calculated in the score calculating step;
An output step of outputting information according to the element specified in the specifying step;
Text extraction method. Computer
A target document input means for inputting a target document from which a text is to be extracted;
Element extraction means for extracting one or more elements each containing a character string from the target document input by the target document input means;
A related document input means for inputting a related document related to the target document;
A character string extracting means for extracting a character string from the related document input by the related document input means;
It is determined whether or not the character string extracted by the character string extraction unit is included in the element extracted by the element extraction unit. If it is determined that the character string is included, the element is included in the target document. Score calculating means for calculating a score by increasing a score indicating the degree of including the body;
Specifying means for specifying an element including the text of the target document among the elements based on the score calculated by the score calculating means;
Output means for outputting information according to the element specified by the specifying means;
Text extraction program to function as.

Images