JP2011081495A - Document data analysis device, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To more accurately estimate the meaning of a text contained in object data. <P>SOLUTION: A document data analysis device for analyzing document data containing the plurality of object data, includes: a holding part 22 which holds a plurality of search keys defined using regular expressions; a search part 24 which searches the plurality of object data using the plurality of held search keys; a clustering part 28 which sets the information obtained as a result of searching as variables and performs clustering processing to group the similar object data; and an information acquiring/estimating part 27 which acquires the attribute information set in the other object data belonging to the same group as a result of clustering processing as the attribute information of the object data. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for analyzing document data.

  Conventionally, natural sentences are decomposed into morphemes, and the presence or absence of a causal relationship is determined by determining whether or not the arrangement of the morphemes corresponds to a prepared event pattern. There is a technique for performing data clustering in consideration of the causal relationship (see Patent Document 1). Also, a search is performed using predicate keywords, the corresponding predicate keywords and the preceding and following sentences are extracted, and the extracted sentences are subjected to narrowing search and rearrangement to remove noise, and combined with the preceding and following sentences. There is a technique for storing the information in the knowledge base (see Patent Document 2).

JP 2008-203964 A JP 2005-115882 A JP 2005-063470 A JP 2001-84250 A

Akihiro Nineyama, 3 others, “Filtering Free Questionnaire Sentences Using Decision Trees”, Proceedings of National Conference of the Japanese Society for Artificial Intelligence, 2004 1A4-05 Masaaki Tsuchida and two others, “Construction of Dictionary System Using World Wide Web”, Proceedings of National Conference of the Japanese Society for Artificial Intelligence, 2004 1A3-04 Atsushi Nishiyama and 4 others, “Extraction of feature expression for mining technical documents”, Proceedings of National Conference of the Japanese Society for Artificial Intelligence, 2008 3K3-2 Shinichi Kobayashi and five others, “A time-series analysis of technology trends using Internet resources”, IEEJ Transactions C (Electronics, Information and Systems Division), 2005, Vol. 125-C, Number5

  Conventionally, as a method for extracting knowledge from a large amount of document data, a morphological analysis technique is used to extract words from one document and estimate the dependency relationship of words based on independent words, thereby obtaining a syntax tree from the dependency relationship. The technology to build is used. In such morphological analysis, frequent patterns (knowledge) included in many syntax trees in the constructed syntax tree are found based on the given pattern constraints, and Search for documents that match the assignment.

  However, the natural sentences that are actually analyzed are subject to large fluctuations in the notation, and the subject and predicate may be omitted. It was difficult to estimate the exact meaning of the sentence from the results.

  In view of the problems described above, an object of the present invention is to more accurately estimate the semantic content of a sentence included in target data.

The present invention has the following configuration to solve the above-described problems. That is, the present invention is a document data analyzing apparatus for analyzing document data including a plurality of target data, the holding unit holding a plurality of search keys defined using regular expressions, and the holding unit Search means for searching the plurality of target data using the plurality of stored search keys and information obtained as a result of the search by the search means are set as variables to group similar target data together Document data analysis comprising: clustering means for performing clustering processing; and information estimation means for acquiring attribute information set in other target data belonging to the same group in the result of the clustering processing as attribute information of the target data Device.

  According to the present invention, by using a search key defined by a regular expression, a necessary feature portion can be searched without being affected by the fluctuation of the notation of the target data. In order to more accurately search for such a feature portion, it is preferable to use a search key that is a regular expression of a feature that is not affected by notation fluctuations such as colloquial or subject omission.

  Furthermore, in the present invention, clustering processing is performed in which information obtained as a result of the search by the search means is set as a variable and similar target data is grouped. Here, the clustering process is a conventional technique in which object data that are close to each other are aggregated to perform grouping. For example, the distance between cases is calculated, and the two cases with the closest distance are sequentially determined. Grouping can be performed by merging with. By the clustering process, target data having similar meanings are grouped. For this reason, according to the present invention, it is possible to cluster feature patterns of regular expressions and acquire and set attribute information from similar cases. That is, according to the present invention, even for target data for which attribute information is not set, the attribute information is acquired from other target data whose contents are estimated to be similar, in other words, to similar target data. Based on this, it is possible to estimate the semantic content of the text included in the target data.

  Also, the document data analysis apparatus according to the present invention refers to whether or not the target data includes a search key relationship defined in advance as a combination of related search keys by referring to a search result by the search means. When the determination unit determines that the target data includes the search key relationship, attribute information associated in advance with the search key relationship is used as attribute information of the target data. An attribute acquisition unit for acquiring, and the information estimation unit acquires attribute information for target data for which attribute information has not been acquired by the determination unit and the attribute acquisition unit based on a result of the clustering process. Also good.

  Here, the search key relationship is a combination of search keys having relevance as semantic content, in other words, a relationship defined in advance as representing a predetermined semantic content (attribute) when appearing in the target data at the same time. is there. In the present invention, it is possible to extract a dependency relationship that cannot be extracted by conventional language analysis by determining whether or not the target data includes such a search key relationship.

  Further, the information estimation means further includes a representative search key relationship of the group to which the search key relationship included in the target data belongs among the groups grouped by the clustering means as information related to the target data. Related information may be acquired.

  Based on the search key relationship of the representative cases specified by grouping, the search key of the target data is estimated in detail by replacing or supplementing information related to the target data, and the search key is embodied. It is possible to refine or unify.

Furthermore, the present invention can be understood as a method executed by a computer or a program executed by a computer. Further, the present invention may be a program in which such a program is recorded on a recording medium readable by a computer, other devices, machines, or the like. Here, a computer-readable recording medium is a recording medium that stores information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read from a computer or the like. Say.

  According to the present invention, it is possible to more accurately estimate the semantic content of a sentence included in target data.

It is a figure which shows the hardware constitutions of the document data analysis apparatus 1 which concerns on embodiment. It is a figure which shows the outline of a function structure of the document data analysis apparatus 1 which concerns on embodiment. It is a figure which shows the structure of the base dictionary which concerns on embodiment. It is a figure which shows the structure of the relational dictionary which concerns on embodiment. It is a flowchart which shows the flow of the data clustering process which concerns on embodiment. It is a figure which shows the result of the search process using a base dictionary in embodiment. It is a figure which shows the structure of the variable data produced | generated for clustering in embodiment. It is a figure which shows the result of data clustering in embodiment. It is a flowchart which shows the flow of the pattern extraction process which concerns on embodiment. It is a figure which shows the result of the attribute determination process using the relation dictionary in embodiment. It is a figure which shows the result of the pattern extraction process in embodiment.

  Hereinafter, an embodiment of a document data analysis apparatus 1 according to the present invention will be described with reference to the drawings.

<System configuration>
FIG. 1 is a diagram illustrating a hardware configuration of a document data analysis apparatus 1 according to the present embodiment. The document data analysis apparatus 1 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 13 as a main storage device, a ROM (Read Only Memory) 12, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. The auxiliary storage device 14, the display device 15 as a display, and the input device 16 as a computer (information processing device) including a keyboard and a mouse.

  FIG. 2 is a diagram illustrating an outline of a functional configuration of the document data analysis apparatus 1 according to the present embodiment. In the computer having the configuration shown in FIG. 1, the document data analysis program recorded in the auxiliary storage device 14 is read into the RAM 13 and executed by the CPU 11, whereby the input receiving unit 21 and the holding unit 22. , Functioning as a document data analysis apparatus 1 including an output unit 23, a search unit 24, a determination unit 25, a pattern extraction unit 26, an information acquisition / estimation unit 27, a clustering unit 28, and a notation control unit 29.

When the determination unit 25 determines that the target case includes a search key relationship, the information acquisition / estimation unit 27 acquires attribute information associated in advance with the search key relationship as attribute information of the target case, When the attribute information is not acquired by the attribute information associated in advance with the search key relationship, the target case is set as another target case belonging to the same group as the target case attribute information as the target case attribute information. Get attribute information.

  In addition, the holding unit 22 holds a plurality of search keys defined in advance using normal character strings or regular expressions, and various dictionaries (such as a base dictionary and a relational dictionary) including attribute information corresponding to the search keys. It should be noted that it is preferable that different dictionaries are prepared for each field, and a dictionary in an appropriate field is preferentially used according to the field to which the document data belongs.

  FIG. 3 is a diagram showing a configuration of the base dictionary according to the present embodiment. The base dictionary has a pattern having a search key ID for identifying a search key, one search key defined by a normal character string or a character string using a regular expression, and attribute information corresponding to the search key. It is preferable that different dictionary data is prepared for each field of the document data to be analyzed. In principle, the search key is defined by a regular expression, and attribute information indicating the semantic content of a sentence that matches the regular expression is associated with each search key. For example, since a sentence that matches the regular expression “. * Struggled” is considered to mean a claim, attribute information indicating “claim” is associated therewith.

  FIG. 4 is a diagram showing a configuration of the relation dictionary according to the present embodiment. The relational dictionary is dictionary data in which patterns having a search key ID, a plurality of search keys, and attribute information corresponding to a combination of the search keys are accumulated. In principle, search keys are defined by regular expressions, and attribute information indicating the semantic content of sentences matching all regular expressions (sets of search keys) included in one pattern is associated with each set of search keys. It has been. For example, a sentence that matches the regular expression ``. * (Satisfied) '' that means a claim alone, but also matches a regular expression ``. Since it is thought that it means not a claim but an opinion request as a whole, attribute information indicating “opinion request” is associated. The search key of the relation dictionary may not be directly included in the data as a regular expression, but may be set by specifying a search key ID (search key identification information) in the base dictionary.

<Process flow>
Next, the flow of processing executed by the document data analysis apparatus 1 according to this embodiment will be described. In the following, the flow of processing when the document data analysis apparatus 1 analyzes the reception log input and accumulated by the operator at the call center will be described. In the call center, telephone calls such as inquiries from customers are accepted, and the operator inputs customer correspondence records into the computer. A part of the reception log input and accumulated by the operator is input in text. However, the document data analysis apparatus 1 according to the present embodiment can be used for the purpose of analyzing various sentences (for example, questionnaire results and the like) in addition to the call center reception log.

  FIG. 5 is a flowchart showing a flow of data clustering processing according to the present embodiment. The process shown in this flowchart is started when the user specifies the document data to be analyzed and the input of the operation for instructing the start of the data clustering process is accepted. Note that the order of processing and the specific processing shown in this flowchart are examples that can be adopted in carrying out the present invention, and the actual processing order and specific processing are not included in order to implement the present invention. Various processing orders and specific processing that can be employed by a vendor may be employed.

In step S101, input of document data is accepted. The input receiving unit 21 receives input of document data via a network such as a LAN, a portable recording medium such as a USB memory and a CD-ROM, and the like. Document data to be processed in this embodiment is a reception log in a call center, and the reception log includes text input by an operator as data using character codes in units of calls or a series of cases. . Hereinafter, the data in units of calls or cases included in the reception log is referred to as “case”.

  Each case includes a case ID for identifying the case and case attribute information input by the operator. The attribute information is information for indicating the attribute of the case. For example, in order to be able to determine the meaning content of the call or case related to the case, such as “claim”, “request for opinion”, “praise”, etc. This is information set by the operator who made the call. However, the attribute information is not necessarily set for all cases, and there may be cases where the attribute information is not set in the reception log due to forgetting the setting by the operator. When the input document data is recorded in the RAM and the input acceptance is completed, the process proceeds to step S102.

  In step S102, the notation in the document is regulated. The notation control unit 29 controls the notation in the document by executing a search / replacement process using the notation control dictionary for the document data input in step S101. The dictionary for notation control contains a search key that defines notation fluctuation and frequent misprints as a normal character string or a regular expression character string, and a replacement character string corresponding to this search key, that is, a notation The post-control character string is included. By using such a dictionary for notation control, search / replacement processing is applied to document data, so that notation in the document is controlled and there are no notation or typographical errors (or reduced typographical or typographical errors). Document data) is generated.

  Here, the term “control” refers to standardizing expressions used in sentences under a certain standard. Specifically, correction of typographical errors, unification of words and phrases having a plurality of notation methods (for example, replacing all character strings such as “fax” and “facsimile” with “FAX”), synonym control (for example, “ (Easy, quick, and quickly) replaces the string with “immediately”), deletes prefix, corrects phrasing, unifies to key, simplifies, has meaning No end of sentence is deleted and corrected.

  When analyzing call center response records, questionnaires, etc., the sentences to be handled are written in a colloquial style that includes typographical errors and conversion errors, and the notation is greatly shaken. For this reason, the dependency relationship with independent words is often not clearly expressed in the text. For example, in the case of “Den Wataiou is XX”, a person can read “Phone support is XX”, so the dependency relationship between “Phone support” and “XX” can be extracted. It is difficult to extract mechanically. Therefore, in the present embodiment, the shaking of the notation is removed before analysis by the control process. Further, in document data such as questionnaires, the subject and predicate are often omitted, and the dependency relationship may be leaked. For example, in the sentence “telephone correspondence yesterday ○○”, there is a blank character between “telephone correspondence” and “yesterday ○○”, so even though it is a dependency relationship that can be understood by humans, In the morphological analysis, the dependency relationship between “Yesterday” and “XX” can be extracted, but the relationship with “telephone correspondence” is not extracted. According to the text data analysis process according to the present embodiment, the expression in the text is arranged by the control process, so that the dictionary search process and the data clustering process using the regular expression, which will be described later, are more efficient and highly accurate. Analysis results can be expected.

Further, in the document data analysis apparatus 1 according to the present embodiment, by setting the replacement character string with a regular expression, it is possible to replace only a part of the character string searched with the regular expression. For this reason, in this embodiment, a part to be quoted in the replacement character string is indicated in advance in a regular expression for matching by enclosing it with a tag such as “¥ 1”. For example, if you want to unify only the units from “100 Watts, 200 Watts,...” To “100 W, 200 W,. Search with the regular expression defined as “Watt” to find the replacement part, and define the replacement string as “\ 1 [0-9] + \ 1W”, so that only the numeric part is replaced Can be obtained from a string and replaced. In addition, when there are a plurality of parts to be cited at the time of replacement in one regular expression, it can be dealt with by using enclosing tags such as “\ 2”, “\ 3”. According to such a replacement process, it is possible to reduce the probability of notation mistakes and notation without preparing all the replacement patterns in the replacement process. When the notation control ends, the process proceeds to step S103.

  In step S103, search processing using a base dictionary is performed. The search unit 24 searches the document data with the notation controlled generated in step S102 with a search key included in the base dictionary, and searches for a searched character string (hereinafter referred to as “searched character string”), The case ID of the target case including the output character string, the position of the search character string in the document, the search key related to the search, the attribute of the search character string, and the like are associated with each other and stored in the RAM 13 as a search result.

  FIG. 6 is a diagram illustrating a result of search processing using the base dictionary in the present embodiment. In the case of a search using a regular expression, the search is performed by finding a character string that matches the regular expression using a delimiter specified in the system as a data delimiter. The delimiter can be freely specified. For example, “.” Or a line feed can be specified as the delimiter. Further, the position of the searched character string can be specified by using the number of characters from the top of the document, the number of bytes, the number of logical lines, and the like. When the base dictionary search ends, the process proceeds to step S105.

  In step S104, the relationship of the searched character string is extracted. In each case, the pattern extraction unit 26 extracts the relationship between the searched character strings existing in the same case based on the order of the searched character strings that matched the search key in step S103. Here, the pattern extraction unit 26 compares the distance of the appearance position of each searched character string with a predetermined threshold value, and excludes a searched character string that is more than the threshold value from the extraction target because the relationship is low. Further, the pattern extraction unit 26 compares the ratio of the searched character string to the case (number of characters in the searched character string / number of characters in one document) with a predetermined threshold value, and extracts the searched character string whose ratio is equal to or less than the threshold value. Exclude from the target. Through such processing, it is possible to extract the relationship of the searched character string for each case. In addition, it is preferable that different definitions for extraction including the above-described threshold value are prepared for each field of document data to be analyzed. Thereafter, the process proceeds to step S105.

  In step S105, the relationship between searched character strings is narrowed down. The pattern extraction unit 26 calculates the relationship between the searched character strings (regular expression key relationship) extracted in step S104 by the appearance frequency or the appearance rate (for example, “number of appearance cases / number of all cases”). Appearance rate can be used.) At this time, as a narrowing-down method, a method of narrowing down by comparing the calculated appearance frequency or appearance rate with a predetermined threshold and extracting only a relationship having an appearance frequency or appearance rate equal to or higher than the threshold may be employed. Thereafter, the process proceeds to step S106.

  In step S106, a similar case (similar pattern) and its representative case (representative pattern) are extracted by data clustering. The clustering unit 28 extracts the key relationships of the regular expressions extracted and refined up to step S105, the cases, the regular expressions and their appearance ranks, the attributes, the number of corresponding documents, the number of corresponding sentences, the distance between the regular expressions, the appearance frequency. Then, data clustering is performed using the appearance rate as a variable, and a similar case and its representative case are extracted.

FIG. 7 is a diagram illustrating a configuration of variable data generated for clustering in the present embodiment. Note that the distance between regular expressions is the same as the distance in data clustering. For example, in the key relationship between two regular expressions (hereinafter referred to as “binary relationship”), the distance when two regular expressions appear in a sentence is (the beginning position of the second regular expression) −
1 at the end of the sentence of the first regular expression.

  FIG. 8 is a diagram showing the result of data clustering in the present embodiment. Here, the representative case is a case that is closest to each case among similar cases, that is, a case where the result of data clustering is close to the center of gravity. Note that when a regular expression key relationship exists in the relation dictionary, attribute information is set in the extracted case.

  Data clustering is a conventional technique in which objects that are close to each other (cases, variables) are aggregated to perform grouping, and key relationship 1, key relationship 2,... Key relationship k (k is a key) The number of relationship cases), the appearance rank of each key relationship, the attribute, the frequency, and the character length are designated as variables, and clustering processing is performed to group key relationship patterns having similar meanings. For example, by employing clustering based on the Ward method, the distance between cases can be calculated, and the two cases with the closest distance can be grouped by sequentially merging.

More specifically, in data clustering, as an initial case, m variables x _i1 , x _i2 ,..., X _im are set for n cases (i = 1, 2,...). , N). Then, as the merging of the first stage, the distance between cases is calculated from the distance d (x _a , x _b ) (dissimilarity) between cases a and b, and the two cases with the shortest distance are sequentially determined. Merged. At this time, the following equation is used.

Furthermore, as a merge after the second stage, c and d, which merge cases a and b, are merged. At this time, the following equation is used. _{Incidentally, n a, n b, n} c, n d is the case a, b, c, the number of individuals included in the d (number of data).

  When merging is repeated in this way, one group is formed. For this reason, the distance between cases is compared with a predetermined threshold, and those within the threshold can be determined as the final group. In the group, the case closest to each case, that is, the case having the center of gravity is the representative case, and has a representative description of the regular expression of the group.

  The result of data clustering can be used not as a predefined dictionary but as a dictionary automatically generated from accumulated document data (hereinafter referred to as “real-time dictionary”), and for adding dictionary items to a relational dictionary Can be used as candidate data. The method used as the real-time dictionary will be described later in the description of the pattern extraction process.

  FIG. 9 is a flowchart showing a flow of pattern extraction processing according to the present embodiment. The process shown in this flowchart is started when the user specifies the document data to be analyzed and receives an operation input instructing the start of the pattern extraction process. Note that the order of processing and the specific processing shown in this flowchart are examples that can be adopted in carrying out the present invention, and the actual processing order and specific processing are not included in order to implement the present invention. Various processing orders and specific processing that can be employed by a vendor may be employed.

  In steps S201 to S203, input of document data is accepted, notation in the document is regulated, and search processing using a base dictionary is performed. Details of the processing from step S201 to step S203 are substantially the same as the processing described in step S101 to step S103, and thus description thereof is omitted. When the search process using the base dictionary is completed, the process proceeds to step S204.

  In step S204, the semantic content (attribute) of the sentence included in the target case is determined using the relation dictionary. The determination unit 25 and the information acquisition / estimation unit 27 determine the semantic content of the sentence included in the target case by comparing the search result in step S203 with the relation dictionary, and set the attribute information. More specifically, the determination unit 25 refers to the search result in step S203 and determines whether or not a search character string using the same search key as the search key included in the relation dictionary is included in the search result in step S203. It is determined whether or not a plurality of search keys defined in each record of the relational dictionary are simultaneously included in the target case (search target document). When a plurality of search keys are included in the target case at the same time, the determination unit 25 further determines whether the length of the searched character string, the interval between the searched character strings, etc. satisfy the conditions defined in the relational dictionary. If the condition is satisfied, the information acquisition / estimation unit 27 sets the attribute information on the assumption that the target case has the attribute attached to the corresponding record in the relation dictionary.

  FIG. 10 is a diagram illustrating a result of attribute determination processing using a relation dictionary in the present embodiment. For example, in the case of a sentence “I struggled. XX, I should do it.”, Since it is composed of two sentences “XX, I should have done it.” And “XX, I should do it.” In the morphological analysis, the dependency relationship between “XX” and “good” can be extracted, but the relationship between these expressions and “OO” is not extracted. According to the document data analysis processing according to the present embodiment, not only the dependency of morphemes in one sentence but also the analysis considering the context of the sentence and the analysis incorporating characteristic expressions such as human behavior and emotions are possible. It becomes.

  The extraction process is performed in the order defined in the definition dictionary. In addition, a search result having a large number of search keys that match as a result of matching and a search result having a short distance of appearance position between each item are preferentially extracted. It is also possible to set priorities for attributes and perform extraction according to the priorities. For example, a higher priority than “request for opinion” may be set for “claim” among the attributes, and the claims may be extracted preferentially. In the present embodiment, by comparing the search result in step S203 with the related dictionary, it is determined whether or not a plurality of search keys included in each record of the related dictionary are simultaneously included in the target case. However, instead of such a method, by directly searching the target case with the search key included in the relation dictionary, a plurality of search keys included in each record of the relation dictionary are simultaneously included in the target case. It is good also as determining whether it is. When the semantic content determination is completed, the process proceeds to step S205.

In step S205, it is determined whether or not the relationship between the searched character strings in the target case has been extracted using the relationship dictionary. The determination unit 25 determines whether or not the relationship between the plurality of searched character strings searched in step S203 is extracted by the semantic content extraction process using the relation dictionary in step S204. In other words, the determination unit 25 determines whether or not the relationship of the searched character string in the target case exists in the relationship dictionary. If the relationship between the searched character strings in the target case is extracted using the relationship dictionary, the process proceeds to step S207. If the relationship between the searched character strings in the target case is not extracted using the relationship dictionary, the process proceeds to step S206.

  In step S206, the semantic content of the target case is extracted based on the result of data clustering. The information acquisition / estimation unit 27 refers to the result of data clustering (real-time dictionary) to acquire attribute information of a similar case close to the pattern of the target case (a combination of a plurality of searched character strings), and this attribute information Is extracted as attribute information (meaning content) of the target case (see FIG. 10). Here, as for the priority of complementation, the representative case closest to the center of gravity has the highest priority, and the priority becomes lower in order of increasing distance from the center of gravity. Note that the attribute information extracted from the data clustering result may be attribute information of a representative case of a group to which the target case (or a search key relationship searched in the target case) belongs, or attribute information from the representative case. May not be obtained, it may be attribute information of other cases in the same group. In addition, the information acquisition / estimation unit 27 includes, as information related to the target case, general information related to the representative case of the target case or a group to which the search key relationship of the target case belongs (that is, not limited to attribute information, Or a wide range of information such as a search string). As a result, information related to the target case can be replaced or supplemented, and the search key of the target case can be estimated in detail, and the search key can be specified, detailed, or unified. Thereafter, the process proceeds to step S207.

  In step S207, pattern extraction is performed. The pattern extraction unit 26 uses the search result based on the base dictionary in step S203, the search result based on the relation dictionary in step S204, and the estimation result of the meaning content in step S206, and the pattern of cases included in the reception log. Perform extraction.

  FIG. 11 is a diagram illustrating a result of pattern extraction processing in the present embodiment. In this embodiment, a search key and an attribute included in a case are extracted as a pattern included in each case. Further, by summing up the extracted cases, the appearance frequency and the appearance rate of the search key relationship included in the reception log are calculated. For pattern extraction, the appearance frequency or appearance rate (number of appearance cases / number of all cases) and character length ratio (number of characters corresponding to regular expressions / number of characters in one document) are specified in advance for the relationship searched up to step S206. It is performed by narrowing down by comparing with the threshold value. By narrowing down using a threshold, patterns that do not meet the threshold are indicated as candidates for deletion from the base dictionary and relational dictionary, and patterns that are equal to or greater than the threshold but not defined in the dictionary are indicated as candidates for addition to the base dictionary and relational dictionary. Can be. When pattern extraction is completed, the process proceeds to step S208.

  In step S208, the extracted pattern is output. The output unit 23 outputs the pattern extracted in step S207, the calculated aggregation result, and the like to the display device 15 for display, and provides them to the user. The output unit 23 outputs such that the appearance frequency and appearance rate of the key relationship and the search key appearance frequency and appearance rate are high. Furthermore, the output unit 23 presents a pattern with a low appearance frequency or low appearance rate as a candidate for deletion from the dictionary, and is currently defined in the dictionary as a different pattern but has a similarity that is greater than or equal to a predetermined threshold. Are presented as candidates that are grouped together as one entry in the dictionary. The user creates, updates, and deletes a regular expression dictionary such as a base dictionary and a relational dictionary with reference to the output information. Thereafter, the processing shown in this flowchart ends.

In the present embodiment, display output using the display device 15 is used as an output method for the user, but output methods other than display output may be employed. For example, information may be provided to the user by print output using a printer.

  According to the document data analysis apparatus 1 according to the present embodiment, a form that takes into account the context of a sentence from a document composed of a plurality of sentences, which cannot be extracted by a conventional knowledge acquisition technique based on text analysis, or a human intuitive The feature expression of a simple language can be converted into a regular expression and registered in the language analysis dictionary, the range of analysis can be expanded, and analysis, classification, and knowledge acquisition of documents with many abbreviations such as questionnaires can be handled. In addition, since the document data analysis apparatus according to the present embodiment has a document analysis function based on a feature pattern of regular expressions, it is more effective than pattern analysis of dependency relations extracted after finely dividing into conventional morphemes. It is possible to more accurately extract the relationship between the notation patterns and easily extract intuitive expression patterns by humans.

DESCRIPTION OF SYMBOLS 1 Document data analyzer 22 Holding | maintenance part 24 Search part 25 Judgment part 26 Pattern extraction part 27 Information acquisition / estimation part 28 Clustering part 29 Notation control part

Claims (9)

A document data analysis apparatus for analyzing document data including a plurality of target data,
Holding means for holding a plurality of search keys defined using regular expressions;
Search means for searching the plurality of target data using the plurality of search keys held by the holding means;
Clustering means for performing clustering processing for grouping similar target data by setting information obtained as a result of the search by the search means as a variable;
Information estimating means for acquiring attribute information set in other target data belonging to the same group in the result of the clustering process as attribute information of the target data;
Document data analysis apparatus comprising: A determination unit that determines whether or not the target data includes a search key relationship defined in advance as a combination of related search keys by referring to a search result by the search unit;
Attribute determination means for acquiring, as the attribute information of the target data, attribute information associated in advance with the search key relation when the determination means determines that the target data includes the search key relation; In addition,
The information estimation unit acquires attribute information based on a result of the clustering process for target data for which attribute information has not been acquired by the determination unit and the attribute acquisition unit.
The document data analysis apparatus according to claim 1. The information estimation means further relates to a representative search key relation of a group to which the search key relation included in the target data belongs among groups grouped by the clustering means as information related to the target data. Get information,
The document data analysis apparatus according to claim 1. A document data analysis method for analyzing document data including a plurality of target data.
A holding step for holding a plurality of search keys defined using regular expressions;
A search step for searching the plurality of target data using the plurality of search keys held in the holding step;
A clustering step for performing a clustering process for grouping similar target data by setting the information obtained as a result of the search in the search step as a variable;
As attribute information of the target data, obtaining attribute information set in other target data belonging to the same group in the result of the clustering process, an information estimation step;
Document data analysis method to execute. In the computer,
A determination step of determining whether or not the target data includes a search key relationship defined in advance as a combination of related search keys by referring to a search result in the search step;
In the determination step, when it is determined that the target data includes the search key relationship, an attribute acquisition step of acquiring attribute information previously associated with the search key relationship as attribute information of the target data; Let it run further,
In the information estimation step, the attribute information is acquired based on the result of the clustering process for the target data for which the attribute information has not been acquired in the determination step and the attribute acquisition step.
The document data analysis method according to claim 4. In the information estimation step, the information related to the target data is further related to the representative search key relationship of the group to which the search key relationship included in the target data belongs among the groups grouped in the clustering step. Information is retrieved,
The document data analysis method according to claim 4. A document data analysis program for analyzing document data including a plurality of target data, comprising:
Holding means for holding a plurality of search keys defined using regular expressions;
Search means for searching the plurality of target data using the plurality of search keys held by the holding means;
Clustering means for performing clustering processing for grouping similar target data by setting information obtained as a result of the search by the search means as a variable;
Information estimating means for acquiring attribute information set in other target data belonging to the same group in the result of the clustering process as attribute information of the target data;
Document data analysis program to function as The computer,
A determination unit that determines whether or not the target data includes a search key relationship defined in advance as a combination of related search keys by referring to a search result by the search unit;
When the determination unit determines that the target data includes the search key relationship, attribute acquisition unit acquires attribute information associated with the search key relationship in advance as attribute information of the target data. Make it work,
The information estimation unit acquires attribute information based on a result of the clustering process for target data for which attribute information has not been acquired by the determination unit and the attribute acquisition unit.
The document data analysis program according to claim 7. The information estimation means further relates to a representative search key relation of a group to which the search key relation included in the target data belongs among groups grouped by the clustering means as information related to the target data. Get information,
The document data analysis program according to claim 7.

Images