JP5538830B2 - Document analysis system and dictionary creation system - Google Patents

Description

  The present invention relates to a document analysis system and a dictionary creation system that extract a quantitative expression such as a numerical expression from an electronic document and automatically construct a dictionary that associates the linguistically expressed quantity with the numerical expression.

  The need to automatically collect and analyze useful information in a document is increasing as a large amount of digitized documents can be stored and used freely. In order to meet such needs, methods for analyzing document contents have been studied in the study of natural language processing. When analyzing document contents, quantitative expressions represented by numerical expressions often used for the purpose of representing the attributes of things are important expressions. For example, in a numerical limited patent, the invention is limited by the numerical value described. In order to appropriately search for numerical limited patents, it is desirable to be able to appropriately narrow down quantitative expressions.

  There are two types of quantitative expressions for documents: numerical expression and verbal expression. Numeric character strings, units and classifiers are used for numerical expressions. For example, “50 years old” or “5,000 yen”. The expression by words is the expression by words that express quantity. For example, “middle-aged” or “cheap”. When dealing with quantitative expressions in documents, it is desirable to associate numerical expressions with verbal expressions. For example, the expression “middle-aged” is associated with the numerical expression “40 to 50 years old”. By associating the numerical expression and the verbal expression, for example, when searching for a document including a description of “age” of “40 to 50 years”, a document including “middle age” having the same meaning is simultaneously searched. be able to.

  As a method of associating numerical values with words in this way, a method of manually associating correspondence between words and numerical values can be considered. In Patent Document 1, a fuzzy theory membership function is used to associate numerical values used in quantitative expression with numerical values.

Patent No. 2764766

"Extracting attribute relationships from text" NL-164-4 pp.19-24 (2004)

  However, it is necessary to know the correspondence between words and numbers in the method of manually associating words and numbers. It is difficult to know in advance such correspondence for all quantitative expressions.

  Therefore, an object of the present invention is to provide a document analysis system and a dictionary creation system for automatically constructing a dictionary that associates words with numerical values for quantitative expressions.

  In the present invention, attribute values for the same thing are collected from the document, and quantitative expressions that are estimated are collected. By evaluating the range and frequency of the numerical values indicated by the numerical expressions, the numerical values corresponding to the verbal expressions are automatically obtained. Judgment is made to automatically build a dictionary that associates numerical values with numerical values for quantitative expressions.

  That is, the document analysis system according to the present invention includes an input unit for inputting a document file, a document part extraction unit for extracting a document part from the document file, and an attribute value candidate for the thing pointed to by the thing name and the thing name from the document part. The attribute attribute value candidate extraction unit to extract, the numeric expression processing unit to convert to numerical values when the attribute value candidate is a numeric expression, and a set of attribute value candidates for the same thing name Attribute value candidate aggregation part that aggregates a set of value candidates, and attribute evaluation by words from the aggregated attribute value candidate set, and evaluation of the relationship between the attribute value by the selected word and the attribute value by numerical expression Part.

  The dictionary creation system according to the present invention further includes a dictionary output unit that outputs attribute values based on words and numerical expressions corresponding thereto based on the evaluation result by the document analysis system.

  Quantitative expressions estimated as attribute values for the same thing are collected from the document, and the appropriateness of the numerical value indicated by the numerical expression is evaluated by a score, etc., so that the numerical value corresponding to the verbal expression is automatically judged. This makes it possible to automatically build a dictionary that associates words with numerical values for quantitative expressions.

The figure which shows the structural example of the numerical equivalent expression dictionary creation system by this invention. The figure which shows the structure of the computer which implement | achieves the system of this invention. Explanatory drawing of the process by a thing attribute value set candidate extraction part. Explanatory drawing of the process by an attribute value candidate total part. Explanatory drawing of the process by a correspondence evaluation part. Explanatory drawing of the process by a numerical expression process part. Explanatory drawing of the process by the conversion process part. Explanatory drawing of the process by a numerical value range process part. Explanatory drawing of the process by a numerical value range comparison process part. The figure explaining the flow of data. The figure which shows the other structural example of the numerical equivalent expression dictionary creation system by this invention. Explanatory drawing of the process by the correspondence evaluation part containing unit designation.

  DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below with reference to the drawings by way of system configuration examples of the present invention and processing operations on the system.

  As shown in FIG. 1, the dictionary creation system of the present invention includes a document file input unit 101, a document part extraction unit 102, an object attribute value set candidate extraction unit 103, an attribute value candidate totaling unit 104, a correspondence degree evaluation unit 105, a numerical value An expression processing unit 106, a conversion processing unit 107, a numerical value range processing unit 108, a numerical value range comparison unit 109, and a dictionary output unit 110 are provided.

  Further, as shown in FIG. 2, this system has a central processing unit 201 that has a central processing unit and processes information by a storage program method, a main storage device 202 composed of random access memory, a processing target document 2031 and processing results. An external storage device 203 for storing the dictionary 2032, a CD-ROM reading device 2041 or a DVD reading device 2042 used for inputting a document, an input device 204 such as a keyboard 2043, a CD-ROM for outputting information processing results such as a created dictionary It operates on a device having an output device 205 such as a writing device 2051, a DVD writing device 2052, and a display 2053. In addition, information can be input / output to / from an external information processing apparatus 207 via the network 206.

  The document file input unit 101 receives a document file input from the outside of the system in the form of a storage medium such as a DVD or a CD-ROM through the input device 204. Alternatively, a document file sent from the external information processing apparatus 207 via the network 206 is accepted. The received document file is stored in the external storage device 203. The document file may be an electronic document created by a word processor or the like, or may be a document file digitized using an optical character reader (OCR) or the like. In either case, the document is assumed to be character-coded. The type of character code is not particularly limited. A unique symbol for identification is given to the document file. Hereinafter, a unique symbol for identification is referred to as a document identifier.

  The document part extraction unit 102 extracts a document part from the input document file. A document part is a part considered as the contents of the document file, and data described in a natural language is a processing target of the present invention. Extracting a document part is a process of removing format information and metadata. The extracted document part is held in the main storage device.

  The thing attribute value set candidate extraction unit 103 has a function of exhaustively extracting attribute value candidate sets that explain the name of the thing and the thing from the extracted document portion. This function can be realized by correcting the invention disclosed in Non-Patent Document 1. Note that an object represents an object or an event. Objects include not only physical objects but also abstract objects.

  In Non-Patent Document 1, a set of a target, an attribute value, and an attribute name is extracted using a sentence pattern and a co-occurrence score. For example, if a sentence “Mt. Fuji is at altitude 3776 meters” is input, the sentence type “<object> <attribute name> is <attribute value>” is used, the object is “Mt. Fuji”, the attribute name is “elevation”, It is extracted that the attribute value is “3776 meters”. Using the frequency of co-occurrence data collected from a large document database, the likelihood that the target is “Mount Fuji” and the attribute name is “elevation”, the attribute name is “elevation”, and the attribute value is “3776 meters” Evaluate the likelihood that The function of the thing attribute value set candidate extraction unit 103 is based on Non-Patent Document 1 that extracts a triple set of a thing name, an attribute name, and an attribute value using a sentence pattern and selects an appropriate one using a co-occurrence score. Of the means described, this corresponds to a part in which triplets are extracted using sentence patterns so as to extract an object name and an attribute value.

  As shown in FIG. 3, the processing of the thing attribute value candidate extraction unit 103 includes a thing name extraction pattern dictionary 301, a morphological analysis process (S302), a noun phrase identification process (S303), and a thing name attribute value extraction process (S304). ) And output processing (S305). Note that the morphological analysis process (S302) is a well-known technique, and therefore detailed description thereof will be omitted below. Further, the noun phrase identification process (S303) is also a well-known technique known as chunking and the like, and thus detailed description thereof is omitted.

  In the object name extraction pattern dictionary 301, sentence patterns of object names and attribute values that can appear in a document are registered. The sentence pattern is, for example, “object name O of attribute value V”. According to Non-Patent Document 1, collection of sentence patterns can use a corpus having a dependency relationship, but extraction from a corpus having no dependency relationship may be used.

  In the event name attribute value detection process (S304), the event name is extracted using the event name extraction pattern dictionary 301. Only the noun phrases identified in the noun phrase identification process (S303) match the sentence pattern "attribute value expression V", "thing name O", and "arbitrary phrase". The thing name is the noun at the end of the phrase that matches the sentence pattern. Also, the pattern dictionary is applied recursively to the same sentence.

  In the output process (S305), the extracted thing name and attribute value are formed into a triple of “document identifier, thing name, attribute value” and output.

  For example, when the entity attribute value set candidate extraction unit 103 processes a sentence “sale for women aged 40 to 50” included in the document with the document identifier “DOC_ID_1”, the entity name extraction pattern “attribute value expression V The item name “female” and the attribute value expression “40 to 50 years old” are extracted from the “item name O”, and “DOC_ID_1, woman, 40 to 50 years old” is set as a triple of “document identifier, item name, attribute value”. Is output.

  Furthermore, if the document with the document identifier “DOC_ID_1” includes the sentence “middle-aged female as a new sales target”, the attribute name “female” is attributed to the attribute name extraction pattern “attribute value expression V property name O”. The value expression “middle age” is extracted, and “DOC_ID_1, female, middle age” is output as a triple of “document identifier, thing name, attribute value”.

  The attribute value candidate totalization unit 104 performs totalization processing for all “document identifiers, thing names, and attribute values” included in the document set created by the thing attribute value set candidate extraction unit 103. The flow of the counting process is shown in FIG.

  In the attribute value collection process (S401), data having the same combination of document identifier and thing name is collected, attribute values are extracted, and an attribute value set is created. The attribute value set of the thing name O_U extracted from the document identifier U is represented by SET_V (O_U). At the same time, an object name set SET_O having all the object names O_U as elements and a document identifier set SET_U having all document identifiers U as elements are created.

  For example, “DOC_ID_1, female, middle-aged”, “DOC_ID_1, female, 40 to 50 years old”, “DOC_ID_1, shoes, middle-aged”, “DOC_ID_1, shoes as triples of“ document identifier, thing name, attribute value ” , 20 to 23 cm ”,“ DOC_ID_2, female, young ”and“ DOC_ID_2, female, 150 to 160 cm ”are input. The SET_V (O_U) created from the document identifier “DOC_ID_1” and the thing name “female” is {40 to 50 years old, middle-aged}, the SET_V (O_U) created from the document identifier “DOC_ID_1” and the thing name “shoes” is {20 To 23 centimeters, middle-aged}. The SET_V (O_U) created from the document identifier “DOC_ID_2” and the thing name “female” is {150 to 160 cm, young}. SET_O becomes {women, shoes}. SET_U becomes {DOC_ID_1, DOC_ID_2}. In this description, a set including elements A and B is represented by {A, B}, and this notation is used hereinafter. When a set having a set as an element is described, a nested expression is used.

  In the attribute value set classification process (S402), the created attribute value set SET_V (O_U) is classified into two, an attribute value set SET_N (O_U) in numerical expression and a verbal attribute value set SET_W (O_U). . Judgment whether the attribute value is a numerical expression or a verbal expression is based on the fact that the character string in which the numerical value is expressed such as Arabic numerals or Chinese numerals is limited. Expressions outside the numerical value are judged as expressions by words.

  Through this process, SET_N (O_U) = {20 to 23 cm, 40 to 50 years old} and SET_W (O_U) = {middle age} are created from SET_V (O_U) created from the document identifier “DOC_ID_1”. From SET_V (O_U) created from the document identifier “DOC_ID_2”, SET_N (O_U) = {150 to 160 cm} and SET_W (O_U) = {young} are created.

  In the co-occurrence relationship creation process (S403), a set SET_C (O_U) including all combinations of elements of the set SET_N (O_U) and SET_W (O_U) created by the attribute value set classification process is created.

  By this processing, SET_C (O_U) = {{20 to 23 cm, middle age}, {40 to 50 years old, middle age}} is obtained from SET_N (O_U) and SET_W (O_U) created from the document identifier “DOC_ID_1”. Created. From SET_N (O_U) and SET_W (O_U) created from the document identifier “DOC_ID_2”, SET_C (O_U) = {{150 to 160 cm, young}} is created.

  In the set storage process (S404), the created sets SET_N (O_U), SET_W (O_U), and SET_C (O_U) are stored in the sets SET_N, SET_W, and SET_C for all the thing names.

  By this processing, from the data created from the document identifier “DOC_ID_1” and the document identifier “DOC_ID_2”, SET_N = {{20 to 23 cm, 40 to 50 years}}, {150 to 160 cm}}, SET_W = {{middle age }, {Young}}, SET_C = {{20 to 23 cm, middle age}, {40 to 50 years old, middle age}, {150 to 160 cm, young}}.

  FIG. 5 shows the processing of the correspondence evaluation unit 105 for the sets SET_N, SET_W, and SET_C created by the attribute value candidate totalization unit 104. The following processing will be described using the above example.

  One element of SET_W is extracted and set as SET_WTMP (S501). With this processing, {middle age} is extracted from SET_W = {{middle age}, {young}}, and set to SET_WTMP = {middle age}.

  If there is an element of SET_WTMP, one is taken out and 1 is added to the value of the hash table HASH_W using that element as a key. 1 is added to ALL_W (S502, S503).

  By this processing, KEY = middle age, “middle age” is registered in the hash table HASH_W, and 1 is added to the value. That is, the total number of occurrences of data with KEY = middle age is stored in HASH_W. Also add 1 to ALL_W. In other words, the number of times a value is assigned to KEY is stored in ALL_W.

  By performing the processing from S501 to S503 for all elements of SET_W, ALL_W becomes 2, and two types of keys “middle age” and “young” are registered in the hash table, and each value is 1. Become.

  The values of HASH_W are sorted in descending order of frequency, and the HASH_W key corresponding to the sort result order and the value obtained by dividing the value by ALL_W are paired and stored in the list WORD (S504).

  By this processing, (<middle age, 0.5>, <young, 0.5>) is stored in the list WORD. In this description, a list including elements A and B is represented by (A, B), and this notation is used hereinafter.

  One element of SET_N is extracted and set as SET_NTMP (S505). By this processing, SET_NMP = {{20 to 23 cm, 40 to 50 years}, {150 to 160 cm}} is extracted from {20 to 23 cm, 40 to 50 years}, and SET_NTMP = {20 to 23 cm, 40 To 50 years old}.

  If there is an element of SET_NTMP, one element is taken out and the element is processed by the numerical expression processing unit 106 (S506, S507). The contents of the processing in the numerical expression processing unit 106 will be described with reference to FIG. As a result of taking “20 to 23 centimeters” from {20 to 23 centimeters, 40 to 50 years old] by this processing and processing “20 to 23 centimeters” in numerical processing 106, the maximum value is 30, the minimum value is 20, and the unit As centimeters get.

  The numerical expression processing unit 106 converts a numeric string (including a range expression) included in the extracted numerical expression into a specific value (integer value or real value). Since the numeric string includes a range expression, the output is the maximum value and the minimum value of the numerical value included in the numerical expression. For example, a number string “100 to two hundred” outputs “maximum value: 200, minimum value: 100” (however, the number is not a letter but a value). The flow of processing is shown in FIG. In the following, the processing will be described by taking the numeric string “1000, 300” as an example.

  First, a numerical character string is acquired (S601). Next, it is inspected whether a comma is included (S602). If a comma is included, the process proceeds to S603, and if not, the process proceeds to S608. Since “12,300” includes a comma, the process proceeds to S603.

  In S603, it is inspected whether the three-digit scale format is satisfied. If it satisfies, the character string from which the comma is deleted is acquired (S605), and the process proceeds. If not, it is checked whether the numbers before and after the comma are serial numbers (S604). If it is not a sequential number, the number is divided by a comma (S606). In the case of sequential numbers, the process proceeds to S607. In the case of “1,000, three hundred”, since the three-digit scale format is not satisfied, the process proceeds to S604. Since the commas are consecutive numbers such as “2, three”, the process proceeds to S607.

  In the process of S607, variables A, B, C, and D are used. The first character before the comma is assigned to A, the first character after the comma is assigned to B, the first character before the comma to the second character before the comma is assigned to C, and the second character after the comma to the last character of the number string is assigned to D. A number string in which the values are concatenated in the order of C, A, and D and a number string in which the values are concatenated in the order of C, B, and D are created. In the subsequent processing, the created number string is set as a processing target. In the case of “1,000, three hundred”, C is 1,000, A is two, B is three, D is one hundred, and one thousand two hundred and one hundred three hundred are the subsequent processing targets.

  In next step S608, variables V, C1, C2, and F are used. The initial values are 0 for V, 1 for C1, 1 for C2, and 0 for F. Characters are extracted one by one in order from the right of the numerical character string (S609). Depending on the type of number, it will be processed as follows.

  It is inspected whether the numerical value is a Chinese numeral or an Arabic numeral from 0 to 9 (S610). When true, “numeric value” × C1 × C2 is added to V, and C1 is multiplied by 10 (S614).

  Whether the number is ten, hundred or thousand of Chinese numerals is inspected (S611). When true, if F is 1 in the determination of S615, C1 × C2 is added to V (S616). Further, a numerical value (10, 100, 1000) is substituted for C1, and 1 is substituted for F (S617).

  It is inspected whether the number is a Chinese number ten thousand, hundreds of thousands, trillions, kyo, or habit (S612). When true, if F is 1 in the determination of S618, C1 × C2 is added to V (S619). A numerical value (10000 or the like) is substituted for C2, 1 is substituted for C1, and 0 is substituted for F (S620).

  Whether or not there is a remainder in the number string is checked (S621), and if there is not, V is output and the process ends (S622), and if there is, the process is repeated from S609.

  When processing one thousand two hundred from S608 to S622, the characters are cut out in the order of “hundred”, “two”, and “thousand” in the process of S609, and the process proceeds to the following process. Since the first character is “100”, the determination in S611 is true, and the process proceeds to S615. Since the variable F is 0 with the initial value, the process proceeds to S617, and the value 100 as a hundred number is substituted into the variable C1, and 1 is substituted into the variable F. Subsequently, since the character is “two”, the determination in S610 is true, and the process proceeds to S614. The value of the variable V becomes 200 by 0 + 100 × 1 × 2. C1 becomes 1000 by 100 × 10. Finally, the character “1000” is processed. It becomes true by determination of S611, and progresses to S615. Since the variable F is 1, the result is true, and the process proceeds to S616, where the variable V is 200 + 1000 × 1 and becomes 1200. In S617, the value 1000 as a number of 1000 is substituted for C1, and 1 is substituted for the variable F. Next, it becomes false by determination of S621, the value 1200 of V is output (S622), and a process is complete | finished.

  The conversion processing unit 107 performs processing for converting a unit included in the numerical expression into a reference unit (hereinafter referred to as a reference unit). For example, when the reference unit of length is “meter”, 10 centimeter is converted to “0.1 meter”.

  As shown in FIG. 7, the conversion processing unit 107 includes a reference unit selection table 701, a unit structure analysis unit 702, a calculation unit 703, a prefix conversion table 704, and a unit conversion table 705.

  As shown in FIG. 7, the reference unit selection table 701 is a table having unit / classifiers, scales, and units taken as scales as item names. The scale and the units taken by the scale can be defined with reference to the Metrology Law and the Measurement Unit Ordinance, but scales and units not included in the law can also be defined. When an attribute value and unit / classifier are given, the corresponding reference unit is selected.

  In the case of a classifier (one “piece”, one “stand”, etc.) attached to the right of the number when counting the number of objects, the reference unit is “piece”. In addition, the percentage of content and the like is based on the reference unit.

  The unit structure analysis unit 702 analyzes the unit structure by a syntax analysis program based on a context free grammar. The parsing program is a well-known technique and will not be described here.

  The calculation unit 703 uses the unit structure analysis result, the prefix conversion table, and the unit conversion table to convert the extracted numerical value to a reference unit.

  As shown in the prefix conversion table 704 in FIG. 7, a value corresponding to N, which is N times the power of 10 is registered for the prefix.

  As shown in FIG. 7, the unit conversion table 705 registers conversion source units, reference units, and conversion formulas. In this formula, the numerical value extracted from the numerical expression is substituted for X and calculated. The data “30 as the maximum value, 20 as the minimum value, and centimeter as the unit” shown as an example above becomes “maximum value: 0.3, minimum value: 0.2, reference unit: meter”.

  Returning to FIG. 5, regarding the set of “minimum value, maximum value, reference unit” output as a result of numerical processing, the respective values are set to MIN, MAX, and UNIT. 1 is added to the value of the two-dimensional hash table HASH_N in which the pair <MIN, MAX> of MIN and MAX is the first key of hash and UNIT is the second key. Add 1 to ALL_N. Add UNIT to SET_R.

  By this processing, <0.3, 0.2> is registered as the first hash key in the hash table HASH_N, “Meter” is registered as the second hash key, and 1 is added to the value. That is, the total number of appearances of data with “maximum value: 0.3, minimum value: 0.2, reference unit: meter” is stored in HASH_N. Also add 1 to ALL_N. That is, the number of times that some numerical expression is extracted is stored in ALL_N. Add “meter” to SET_R.

  By performing the processing from S505 to S507 for all elements of SET_N, ALL_N becomes 3, and the hash table includes a set of three types of first hash key and second hash key “year, <50, 40>. ”,“ Meter, <1.6, 1.5> ”,“ Meter, <0.3, 0.2> ”are registered, and each value is 1. ALL_N becomes 3.

  SET_R, ALL_N, and HASH_N are input as the first, second, and third arguments to the numerical value range processing unit 108, and the hash table HashTblN that is the result of the processing is output (S508).

  The numerical value range processing unit 108 performs processing for calculating a score from the distribution of input numerical data. The processing flow is shown in FIG.

  As described above, the first argument is assigned to SET, the second argument is assigned to ALL, and the third argument is assigned to HASH (S801).

  Substituting the example used in the description, SET becomes {meter, age} and ALL becomes 3. HASH is a combination of the first hash key and the second hash key “age, <50, 40>”, “meter, <1.6, 1.5>”, “meter, <0.3, 0.2>” In step S801, data each having a value of 1 is registered.

  One value is extracted from the elements of the set SET and assigned to UNIT. This process is performed until all SET elements have been processed. Specifically, the element of the set SET is a unit (S802).

  In the example, first, “meter” is assigned to UNIT.

  All data with the UNIT value as the first hash key is extracted from the HASH and set in the hash. At this time, the hash key is the second hash key of HASH (S803).

  By this processing, data having <1.6, 1.5> and <0.3, 0.2> as hash keys and having a value of 1 is registered in the hash.

  For the set of the maximum value MAX and the minimum value MIN of the numerical data that is the hash key of hash, the largest value of the maximum values MAX is MMax, and the smallest value of the minimum values MIN is MMin (S804). Search whether the ratio of MMax and MMin is greater than 10. If larger, the process proceeds to S806, and if smaller, the process proceeds to S811. The processes after S806 are processes for dividing a range every 10 n powers, such as 1 to 10, 10 to 100, and counting the frequency in the range. The processes after S811 are processes for dividing the range at the same place, such as 10 to 20, 20 to 30, and counting the frequency in the range.

  In the example, MMax is 1.6 and MMin is 0.2. Since the ratio is smaller than 10, the process proceeds to S811.

If the determination in S805 is true, it is checked in S806 whether numerical data remains, and if it remains, only one set of numerical data is extracted (S807). For each of MAX and MIN, 10 to the power of int (log ₁₀ MAX) is assigned to MaxL, and 10 to the power of int (log ₁₀ MIN) is assigned to MinL. Here, int is a function that rounds down to the nearest whole number. While MaxL is divided by 10, while the value is equal to or smaller than MinL, the value of hash {<MAX, MIN>} is added to the value of hash HashTbl using MaxL as a hash key (S809, S810).

For example, when the numerical data set is MIN = 200 and MAX = 5000, MaxL is int (log ₁₀ 5000) is 3, so 10 to the third power = 1000, and MinL is int (log ₁₀ 200) is 2, so 10 2 Multiplier = 100. A value of hash {<5000,200>} is added to data having a set of “1000” of HashTbl and the value of UNIT as a hash key. Since the value obtained by dividing MaxL by 10 is 100 and not smaller than 100, data having a set of “100” and the value of UNIT as a hash key is also registered in HashTbl.

On the other hand, if the determination in S805 is false, after checking whether numerical data remains in S811, if it remains, only one set of numerical data values is taken out (S812). Substitute _{10 to the} power of int (log ₁₀ MIN) into MaxL and MaxO, and substitute the integer part of the value obtained by dividing MIN by MaxL into MaxE. Add MaxO to MaxE, and add hash {<MAX, MIN>} to the value of hash HashTbl (data type is an integer) with MaxE and UNIT as the hash key while the value is less than or equal to MAX Go (S811, S812).

Continuing with the above example, when the numeric data set is MIN = 1.5 and MAX = 1.6, MaxL is in the power of 10 = 1 because int (log ₁₀ 1.5) is 0, and MaxO Is 1 and MaxE is 1 because it is an integer part of 1.5 divided by 1. A value of hash {<1.6, 1.5>} is added to data having a set of “meter” and “1” in HashTbl as a hash key. When MaxO is added to MaxE, it becomes 2, and since it is larger than MAX, the next data is processed. The next data is that when MIN = 0.2 and MAX = 0.3, MaxL is int (log ₁₀ 0.2) 0, so 10 to the 0th power = 1, MaxO is 1, and MaxE is 0.2. It is 0 because it is an integer part of 1. A value of hash {<0.3, 0.2>} is added to data having a set of “meter” and “0” in HashTbl as a hash key. When MaxO is added to MaxE, it becomes 1, and since it is larger than MAX, the next data is to be processed. However, since there is no data, the next unit “year” is read from SET to UNIT, and the process proceeds to <50, 40>. This is the end of the description of the numerical value range processing 108.

  Returning to the description of the correspondence degree evaluation unit 105. The word W is extracted from the list WORD in descending order. Any number of words can be extracted. The frequency is regarded as an index indicating the appropriateness of attribute value expression.

  In the example, the words are “middle-aged” and “young”. Hereinafter, a case where “middle age” is assigned to W will be described as an example.

  All elements including the word W are extracted from SET_C, and a set created only from the numerical expression portion is set as SET_CTMP (S509, S510).

  By this processing, SET_CTMP is changed to SET_C = {20 to 23 cm, middle age}, {40 to 50 years old, middle age}, {150 to 160 cm, young}} to {{20 to 23 cm, middle age}, {40 to 50 years old, middle age}} is set.

  If there is an element of SET_CTMP, one element is taken out and the numerical expression processing unit 106 processes the element. Set each value to MIN, MAX, and UNIT for the “minimum value, maximum value, reference unit” set that is output as a result of numerical processing. 1 is added to the value of the two-dimensional hash table HASH_C in which the pair <MIN, MAX> of MIN and MAX is the first key of hash and UNIT is the second key. Add 1 to ALL_C. UNIT is added to SET_R (S511, S512). The numerical value range processing unit 108 processes SET_R, ALL_C, and HASH_C. The hash table of processing results is set as HashTblC (S513). Since these processes are the same as the processes already described, detailed description by example is omitted.

  As a result of the processing, HashTblN and HashTblC are obtained. In the example, <Meter, 1>, <Meter, 0>, <Year, 4> are registered as hash keys in HashTblN, and each value is 1. In HashTblC, <Meter, 0>, <Year, 4> are registered as hash keys. Each value is 1.

  HashTblN and HashTblC are processed by the numerical value range comparison unit 109 (S514).

  The flow of processing of the numerical value range comparison unit 109 is shown in FIG.

  A set of all keys of HashTblC is set as SET_F (S901). One element is extracted from SET_F and set as a KEY (S902). The unit included in the KEY is set to UNIT (S903).

  The sum of values including UNIT in the hash key of HashTblC is set in COUNT_C (S904). The sum of values including UNIT in the hash key of HashTblN is set in COUNT_N (S905).

  For KEY, normalize HashTblC value and HashTblN value by each COUNT, then divide. The value is set as a score S (S906).

  Outputs the KEY of the maximum score from the set of score S and KEY. It is possible to output up to score M as a set of score S and KEY. In this case, the score may be used as a reliability for the correctness of KEY.

  In the example, the <meter, 0> score is 1/2 ÷ 1/1 = 2 because the value of HashTblC is 1, the value of HashTblN is 1, the value of COUNT_C is 1, and the value of COUNT_N is 2. The score of <year, 4> is 1/1 = 1 because the value of HashTblC is 1, the value of HashTblN is 1, the value of COUNT_C is 1, and the value of COUNT_N is 1.

  The dictionary output unit 110 receives the result of the correspondence degree evaluation unit 105 and outputs a triple set of “word W, unit U, numerical value set <MIN, MAX>” as a dictionary.

  In the case of the example, a triplet of “middle age, age, <40, 50>” is registered in the dictionary. In this description, the description has been made with an example in which there are only two document files. However, it should be added that the number of documents is several hundred or more in actual operation.

  Through the processing described above, an attribute value “40 to 50” obtained from the sentence “A new brand that sells shoes for women aged 40 to 50 years is announced. This brand targets middle-aged women as a new sales target”. Collect the “age” and the item name “female”, the attribute value “middle age” and the item name “female”, such as “middle age” and “base unit: age, minimum value: 40, maximum value: 50” Correspondence can be acquired with a score. As a result, “middle age” and “40 to 50 years old” can be automatically associated with the numerical equivalent expression dictionary.

  The data flow of the above-described processing is summarized as shown in FIG. Read the document file from the input device, save the document file in the storage device, then repeatedly read the saved data and save the processing results, finally the quantitative expression using words and numerical expression using numerical values Output a dictionary of collected correspondences. In the dictionary, you may collect only one quantitative expression using numerical values corresponding to the quantitative expression by words, and the one with the highest score will be collected. May be collected.

  Note that the above processing is a method of automatically assigning correspondence between verbal expressions and numerical expressions. However, if sufficient data is not available, it is difficult to obtain sufficient accuracy with a completely automatic method. There is something wrong. In such a case, it can be considered that the correspondence evaluation unit 105 in FIG. 1 is changed to a unit designation correspondence evaluation unit 115 as shown in FIG.

  The flow of processing of the unit designation correspondence evaluation unit 115 at this time is shown in FIG. The processing is the same as that of the correspondence level evaluation unit 105 except that only the designated unit is processed. The processing from S1201 to S1205 shown in FIG. 12 is the same as the processing from S501 to S505 in FIG. 5, and the processing from S1208 to S1214 in FIG. 12 is the same as the processing from S508 to S514 in FIG. The difference is in the processing of S1206 and S1207, where the reference unit and the designated unit are checked for identity.

  As described above, it has been explained that the expression shown by words and the expression by numerical values can be automatically associated by the system shown in FIG.

DESCRIPTION OF SYMBOLS 101 Document file input part 102 Document part extraction part 103 Object attribute value set candidate extraction part 104 Attribute value candidate totaling part 105 Correspondence degree evaluation part 106 Numerical expression processing part 107 Conversion processing part 108 Numerical value range processing part 109 Numerical value range comparison part 110 Dictionary Output unit 115 Unit designation correspondence evaluation unit 201 Processing device 202 Main storage device 203 External storage device 204 Input device 205 Output device 301 Object name extraction pattern dictionary 701 Reference unit selection table 702 Unit structure analysis unit 703 Calculation unit 704 Prefix Conversion table 705 Unit conversion table

Claims (5)

An input unit for inputting a document file;
A document part extractor for extracting a document part from a document file;
After morphological analysis of the extracted document part, the noun phrase is identified, and then, by using the object name extraction pattern dictionary in which the sentence pattern of the attribute name and attribute value that can appear in the document part is registered, An object attribute value pair candidate extraction unit that extracts an attribute value candidate for the object pointed to by the object name and the object indicated by the object name with respect to the identified noun phrase;
If the attribute value candidate is a numerical expression, a number string is obtained from the attribute value candidate, and if the number string is separated by a comma every three digits, a number string is generated by removing the comma, If the numbers before and after are consecutive numbers, a number string represented by a number before the comma, a number up to two characters before the comma, and a number after the comma two characters, a number after the comma, and two characters after the comma Divide into a numeric string expressed by the previous number and the number after the comma, and if the number before and after the comma is not a sequential number, divide it into a numeric string before the comma and a numeric string after the comma The numerical value information is extracted in order from the end side of the numerical string obtained by executing the processing, and numerical values for the number of digits corresponding to the extracted numerical information are sequentially generated. numeric representation process of converting the acquired numeric string to a numeric And,
An attribute value candidate aggregation unit for creating a set of attribute value candidates for the same thing name from the attribute value candidates, and further creating a set of attribute values by words and a set of attribute values by numerical expression from the set;
Selecting one word in order from the set of attribute values by words , calculating a co-occurrence frequency of the attribute value by the selected words and the attribute value included in the set of attribute values by the numerical expression , A degree-of-correspondence evaluation unit that generates information indicating a correspondence relationship between the attribute value by the selected word and the attribute value included in the set of attribute values by the numerical expression based on the frequency ;
A document output unit that outputs a dictionary that collects correspondences between quantitative expressions using words and numerical expressions using numerical values based on information generated by the correspondence degree evaluation unit. Analysis system. 2. The document analysis system according to claim 1, wherein the numerical value is converted into a numerical value of a reference unit based on a unit included in the numerical expression processed by the numerical expression processing unit, a predetermined reference unit, and a unit conversion rule. A document analysis system comprising a conversion processing unit.   The document analysis system according to claim 1, wherein the correspondence evaluation unit sets only a numerical expression including a specified unit as a processing target. An input unit for inputting a document file;
A document part extractor for extracting a document part from a document file;
After morphological analysis of the extracted document part, the noun phrase is identified, and then, by using the object name extraction pattern dictionary in which the sentence pattern of the attribute name and attribute value that can appear in the document part is registered, An object attribute value pair candidate extraction unit that extracts an attribute value candidate for the object pointed to by the object name and the object indicated by the object name with respect to the identified noun phrase;
If the attribute value candidate is a numerical expression, a number string is obtained from the attribute value candidate, and if the number string is separated by a comma every three digits, a number string is generated by removing the comma, If the numbers before and after are consecutive numbers, a number string represented by a number before the comma, a number up to two characters before the comma, and a number after the comma two characters, a number after the comma, and two characters after the comma Divide into a numeric string expressed by the previous number and the number after the comma, and if the number before and after the comma is not a sequential number, divide it into a numeric string before the comma and a numeric string after the comma The numerical value information is extracted in order from the end side of the numerical string obtained by executing the processing, and numerical values for the number of digits corresponding to the extracted numerical information are sequentially generated. numeric representation process of converting the acquired numeric string to a numeric And,
An attribute value candidate aggregation unit for creating a set of attribute value candidates for the same thing name from the attribute value candidates, and further creating a set of attribute values by words and a set of attribute values by numerical expression from the set;
Selecting one word in order from the set of attribute values by words , calculating a co-occurrence frequency of the attribute value by the selected words and the attribute value included in the set of attribute values by the numerical expression , A degree-of-correspondence evaluation unit that generates information indicating a correspondence relationship between the attribute value by the selected word and the attribute value included in the set of attribute values by the numerical expression based on the frequency ;
A dictionary creation system comprising: a dictionary output unit for outputting, as a dictionary, attribute values by words and numerical expressions corresponding thereto based on information created in the correspondence evaluation unit. 5. The dictionary creation system according to claim 4 , wherein the numerical value is converted into a numerical value of a reference unit based on a unit included in the numerical expression processed by the numerical expression processing unit, a predetermined reference unit, and a unit conversion rule. A dictionary creation system comprising a conversion processing unit.

Images