JP5099498B2 - Data processing apparatus and data processing method - Google Patents

Description

  The present invention relates to a data processing apparatus and method for detecting an order between related data related to predetermined target data, and more particularly to a technique for detecting an order according to a predetermined rule or a rule generated based on machine learning. Involved.

  When searching for contact information of a company or an individual, searching on the Internet or searching a database stored on a local hard disk is routinely performed. Once such data is stored, it is not easily erased, and even if a company moves, the previous address is often output as a search result.

  In addition to information about this address, when related data related to certain target data, such as company name change, company personnel information, personal work information, product model number information, is updated, which is newer There are many situations where it is necessary to resolve whether the data is related.

  By the way, as shown in Non-Patent Documents 1 and 2, research for extracting the address of a company from document data such as a web page, and research for extracting information on personnel within a company from a single document are conventionally known. . However, by using the company name and personal name as input, the company's address change information and person's affiliation change information are fully utilized in advanced natural language processing technology including technology that automatically estimates the document date and supervised machine learning techniques. There is no prior art that comprehensively handles and retrieves multiple documents on the web.

Satoshi Sato, Address Search Using the World Wide Web, Journal of Information Processing Society of Japan, Vol.42, No.1, pp.59-67, 2001 Sekine, Information extraction from text-Extracting specific information from documents-Information processing, Vol.40, No.4, pp.370-373, 1999

  The present invention was created in view of the above-described problems of the prior art, and an object of the present invention is to provide a technique for accurately detecting before and after two related data related to target data.

The present invention can also provide the following data processing apparatus.
That is, the invention described in claim 1 is a data processing device that detects the order of the first related data and the second related data that are related to each other with respect to the related data related to the predetermined target data. Related data extracting means for respectively extracting the first related data and the second related data from the same or different files stored in the storage means, and the first related data and the second related data from a network or local storage means The related data co-occurrence file extracting means for extracting the related data co-occurrence file co-occurring, and the relationship between the first related data and the second related data from the related data co-occurrence file with reference to a predetermined relational data relation rule And a related data inter-relationship detecting means for detecting before and after and an output means for outputting a detection result.

  According to the second aspect of the present invention, the related data relation rule is information relating to a character string that is included or not included at least between the first related data and the second related data. The related data relation detecting means extracts a character string between the first related data and the second related data in the related data co-occurrence file, and collates with the related data relation rule. And

  According to the invention described in claim 3, the relation rule between related data uses a plurality of teacher files in which two related data for teachers whose context is known in advance co-occurs, A learning result obtained by machine learning using at least one of the appearance position of two teacher-related data, a character string that is included or not included simultaneously, or tag information that is included simultaneously, , Extracting the feature from the related data co-occurrence file, using the first related data and the second related data as input, and referring to the learning result between the first related data and the second related data It is characterized by calculating before and after.

  According to a fourth aspect of the present invention, there is provided the data processing device according to any one of the first to third aspects, wherein the related data extracting means co-occurs with the target data from a network or a local storage means. The first related data and the second related data are extracted, respectively.

  According to the fifth aspect of the present invention, in the data processing device that detects the order of the first related data and the second related data that are related to each other with respect to the related data related to the predetermined target data, the data is stored on the network or locally. Related data extracting means for extracting the first related data and the second related data from the same or different files stored in the means, and the singular or plural that co-occurs with the first related data from the file containing the first related data A co-occurrence data extracting means for extracting one or more second co-occurrence data that co-occurs with the second related data from a file including the second related data; Referring to the predetermined co-occurrence data relational rules regarding the front and back between the first co-occurrence data and the second co-occurrence data, the front and rear between the first co-occurrence data and the second co-occurrence data are detected. A data processing apparatus comprising: a co-occurrence data relation detecting means; and an output means for outputting the detection results as the first and second related data co-occurring with the first and second related data. To do.

  According to the invention described in claim 6, the co-occurrence data co-occurrence for extracting the co-occurrence data co-occurrence file in which the first co-occurrence data and the second co-occurrence data co-occur from a network or local storage means. You may provide the processing apparatus provided with the file extraction means.

  According to the seventh aspect of the present invention, in the character string, the relation rule between the co-occurrence data is included or not included at least between the first co-occurrence data and the second co-occurrence data. The co-occurrence data interrelationship detection means extracts a character string between the first co-occurrence data and the second co-occurrence data in the co-occurrence data co-occurrence file, and the co-occurrence data It is characterized by collating with data relation rules.

  According to the eighth aspect of the present invention, the co-occurrence data relation rule includes a plurality of teacher files each including two pieces of teacher-related data whose context is known in advance and a plurality of character strings. And a learning result obtained by machine learning using one or more character strings included together with the teacher-related data in the teacher file as features, wherein the co-occurrence data relationship detection means includes the first co-occurrence data and The second co-occurrence data is input as a feature, and before and after the first co-occurrence data and the second co-occurrence data are calculated with reference to the learning result.

  According to the ninth aspect of the present invention, in the configuration in which the target data to be processed by the data processing device is the address and the related data is the destination, the old and new between the first related data and the second related data. It is characterized by detecting a relationship.

The present invention can also provide the following data processing method.
The invention according to claim 10 is a computer data processing method for detecting the order of the first related data and the second related data which are related to each other with respect to the related data related to the predetermined target data. The related data extraction step for extracting the first related data and the second related data from the same or different files stored on the network or the local storage means, respectively, the related data co-occurrence file extracting means is on the network or local A related data co-occurrence file extracting step for extracting a related data co-occurrence file in which the first related data and the second related data co-occur from the storage means; Before and after the first related data and the second related data, with reference to the predetermined relation data relation rules. Related data between relationship detection step of, output means, and having an output step of outputting the detection result.

  The order determined by the present invention may be any order, such as new and old, before and after, evaluation, and importance between related data. However, it is not necessary to use the relational rules for those that are uniquely determined by the laws of nature, such as the magnitude of numerical values. Not subject to. In other words, the present invention is intended for new and old addresses due to address changes, before and after manually arranged data, evaluations obtained from questionnaire results, importance determined by creators, etc. It is determined by work or mental action.

  The invention according to claim 11 is information relating to a character string, wherein the relation rule between related data is included or not included at least between the first related data and the second related data. The related data relationship detecting means extracts a character string between the first related data and the second related data in the related data co-occurrence file, and collates with the related data relationship rule. Features.

The invention according to claim 12 uses a plurality of teacher files in which two related data for teachers whose related relations are known in advance are co-occurred in the relation rule between related data, In this configuration, at least one of an appearance position of teacher-related data, a character string that is included or not included at the same time, or tag information that is included at the same time is used as a learning result as a feature. For the machine learning, a known machine learning module that performs a supervised machine learning process using a support vector machine or a maximum entropy method is used.
The related data inter-relationship detecting means extracts the feature from the related data co-occurrence file, inputs the first related data and the second related data, and refers to the learning result to refer to the first related data. Before and after the data and the second related data is calculated.

  According to a thirteenth aspect of the present invention, in the related data extracting step, the related data extracting means extracts the first related data and the second related data co-occurring with the target data from the network or local storage means, respectively. It is characterized by.

  The invention according to claim 14 is a computer data processing method for detecting the order of the first related data and the second related data succeeding each other with respect to related data related to the predetermined target data, the related data extracting means Includes a related data extraction step for extracting the first related data and the second related data from the same or different files stored in the storage means on the network or locally, and the co-occurrence data extracting means includes the first related data. Extracting one or a plurality of first co-occurrence data co-occurring with the first related data from the file and the first or plurality of first co-occurring data co-occurring with the second related data from the file containing the second related data A co-occurrence data extracting step for extracting two co-occurrence data; A co-occurrence data inter-relationship detecting step for detecting the front and back between the first co-occurrence data and the second co-occurrence data with reference to a predetermined co-occurrence data relation rule, and an output means, An output step is provided for outputting the data before and after the first related data and the second related data.

  According to a fifteenth aspect of the present invention, in the above data processing method, after the co-occurrence data extraction step, the co-occurrence data co-occurrence file extraction means is the first co-occurrence data from a network or local storage means. And a co-occurrence data co-occurrence file extracting step for extracting a co-occurrence data co-occurrence file in which the second co-occurrence data co-occurs.

  The invention according to claim 16 is the data processing method as described above, wherein a co-occurrence data relation rule is included or not included at least between the first co-occurrence data and the second co-occurrence data. Information relating to a character string, wherein the co-occurrence data relationship detection means extracts a character string between the first co-occurrence data and the second co-occurrence data in the co-occurrence data co-occurrence file, It is collated with the co-occurrence data relation rule.

The invention according to claim 17 uses a plurality of teacher files in which each of the co-occurrence data relation rules includes two pieces of teacher-related data whose contexts are known in advance and a plurality of character strings. In this configuration, one or a plurality of character strings included together with the teacher-related data in the teacher file is used as a learning result by machine learning. The machine learning can be performed before the related data relationship detection step using a support vector machine or a known machine learning module that performs supervised machine learning processing of the maximum entropy method.
Then, the co-occurrence data relationship detecting means inputs the first co-occurrence data and the second co-occurrence data as features, and refers to the learning result between the first co-occurrence data and the second co-occurrence data. It is characterized by calculating before and after.

  In the invention according to claim 18, in the data processing method described above, in the configuration in which the target data is the address and the related data is the destination, the old and new relationship is established between the first related data and the second related data. It is characterized by detecting.

The present invention has the following effects by providing the above configuration.
In other words, according to the invention described in claim 1 or 10, the order between two related data related to the target data can be determined with high accuracy based on the relational rules between related data. Processing that was judged from the context before and after can be automated.

Since the order targeted by the present invention is determined by human actions and mental actions as described above, it is difficult to become familiar with computer processing.
In contrast, the present invention first extracts two related data related to the target data, and further extracts a related data co-occurrence file in which they co-occur. According to this method, it is possible to extract related data optimum for order detection for a large amount of data, and to detect the order with high accuracy from the related data.

  Since the invention according to claim 2 or 11 uses that a character string between two related data is included or not included as a relation rule between related data, it can be more easily performed by comparing character strings of a computer. The order can be detected.

  According to the third or twelfth aspect of the present invention, the order between related data is learned from teacher data using machine learning, and the result is used as a relation rule between related data, which contributes to more accurate detection. .

  According to the invention described in claim 4 or 13, the relevance between the related data and the target data is more reliable by extracting the data co-occurring with the target data instead of extracting only the target data in the above. In addition, the number of target data to be processed is suppressed by limiting to co-occurring data. As a result, it is possible to increase the order detection accuracy and speed up the processing.

  According to the invention described in claim 5 or 14, the order can be detected not by directly detecting the order from each related data but by the co-occurrence data co-occurring therewith. As a result, even if it is difficult to determine the order due to the nature of the related data, it is possible to detect the order with high accuracy by comparing the co-occurrence data.

  According to the invention described in claim 6 or 15, the relationship between the co-occurrence data can be accurately grasped by using the file in which the co-occurrence data co-occurs with each related data.

  According to the invention of claim 7 or 16, since it is used that the character string between the two co-occurrence data is included or not included as the co-occurrence data relation rule, the character string of the computer The order can be easily detected by comparison.

  According to the invention described in claim 8 or 17, the order between the co-occurrence data is learned from the teacher data by using machine learning, and the result is used as a relation rule between the co-occurrence data, so that the detection can be performed with higher accuracy. Contribute.

  According to the invention described in claim 9 or 18, the address data is used as the target data, the destination is used as the related data, and is often changed, and it is difficult to determine which is new only by comparing the related data. The present invention can be applied to this.

Hereinafter, embodiments of the present invention will be described based on examples shown in the drawings. The embodiment is not limited to the following.
Example 1
FIG. 1 is a block diagram of a data processing apparatus (hereinafter referred to as this apparatus) according to the present invention. The present invention can be easily realized by a known personal computer, and each step of the present invention is executed and processed by a CPU (10) that controls arithmetic processing, machine learning, text processing, and the like. As is well known, the CPU (10) operates in cooperation with a memory (not shown), and in addition to input means such as a keyboard and mouse (11), a monitor (12) for displaying output results, an external storage such as a hard disk. A device (13) is provided.
In addition, a network adapter (14) connected to a network such as the Internet is provided as data acquisition input means for acquiring text data, files, and the like.

The CPU (10) includes an input unit (101), a related data extraction unit (102), a related data co-occurrence file extraction unit (103), a related data relationship detection unit (104), and an output unit (105). It has been.
And the program described by the well-known programming language operates CPU (10) and the hardware linked with it, and the function of each part (101)-(105) demonstrated below is implement | achieved.

Hereinafter, each processing of the present invention according to claims 1 to 4 will be described in detail with reference to a processing flowchart shown in FIG.
First, the input unit (101) receives the first content file (20) and the second content file (21) from the external storage device (3) or a server such as the Internet or a LAN (Local Area Network) via the network adapter (40). ) Is acquired and taken into the CPU (10).
Each content file (20) (21) may be on the same storage device or server, or may be stored separately.

Examples of content files (20) and (21) are described in a database in which data (name, company name, address, and telephone number) is assigned to a plurality of items such as an address book, or in HTML (HyperTextMarkup Language). Even a web page, such as a source text of a created web page or a web page of a patent gazette, can include text including data in which items and contents are accurately associated with each other.
For the sake of clarity, the following description will be continued with the target data as “company name” and the related data as “address”. Thus, as described in claim 9 of the present invention, when the target data is the address and the related data is the destination, it can be used to detect, for example, the new / old relationship of the destination as its context.

  The address is the name, company name, etc. of the other party to be written on a letter or certificate, and the address is the address location. For example, “ΔΔ Corporation” is the address, and its address “1-2-3 Chuo-ku, Tokyo Station” is the destination.

The related data extraction unit (102) extracts related data related to the target data (22) defined in advance from each content file (20) (21) which is a different file. (Related data extraction step: S10)
For the sake of explanation, they are called the first related data (24) and the second related data (25), but their order is unknown, and the present invention detects the order between the two related data related to the target data (22). Is.

  For example, assuming that the company has moved, consider "old address" (order is before) and "new address" (after order). In this case, the first content file has a web page in which the old address is described, and the second content file has a web page in which the new address is described, from which the old address and the new address are the first related data, the second It will be extracted as related data. Although the details will be described later, it is not necessary here that the target data (22) itself appears in each content file (20) (21), and even if the company name is not described on the page itself, it is manually performed beforehand. The present invention is also applied to the case where the relatedness is selected and the content file is input thereon.

Each content file (20) (21) includes one piece of data, and the simplest configuration is to extract each related data (24) (25) that is surely related to the target data (22) in advance. In general, the content files (20) and (21) contain a plurality of data. For example, data that is not related to the detection of the order such as a telephone number, a person in charge name, and product information is included together with the old address.
In the present invention, since the order may be determined by comparing a large amount of related data, it is not necessary to select whether it is really necessary as related data at this stage.

However, in order to increase the processing speed and obtain only the necessary results, it is possible to select and extract in advance. For example, if you want to extract only the address, the CPU (10) performs known text processing, and if you extract the text that ends with a prefecture name or city name and ends with numbers, you can easily extract only the address. You can also.
Furthermore, by extracting only the character string following “Headquarters”, one related data that is expected to be equivalent in content may be specified and extracted from a page on which a plurality of addresses are described.

  Using the extracted first related data (24) and second related data (25), the related data co-occurrence file extraction unit (103) uses the related data co-occurrence file ( 231) is extracted. (Related data co-occurrence file extraction step: S11)

The content data (23) may be stored either in the external storage device (13) or on a server such as the Internet. However, the larger the amount of data, the more reliably the co-occurring data can be extracted. Therefore, the latter is preferable.
If there are multiple co-occurring files, all of them may be extracted and proceed to the next process, or the file creation date / time is the latest, or the ratio of the two related data is high. You may choose one with the smallest data capacity.

  Related data cooccurrence file extraction unit (103) is not limited to the case to be determined whether to search where previously, the first known search engine site, send each associated data, web pages they co-occur May be extracted as a related data co-occurrence file (231).

Next, the related data relation detecting unit (104) refers to the related data relation rule (130) stored in the external storage device, and detects the order between the related data. (Related data relationship detection step: S12)
The present invention proposes two methods, a rule-based method as in the invention described in claim 2 and the like, and a method by machine learning as in the invention described in claim 3 and the like.

First, the rule-based method will be described.
In the case of the rule base, a rule is manually defined in advance and a determination is made according to the rule. However, as in the present invention, the order between related data is automatically determined by referring to the content data (23). If the data (23) is enormous, it is virtually impossible. The present invention can detect the order with high accuracy even in such a case.

Examples of the related data co-occurrence file (231) according to the claims of the present invention are shown in FIGS. The related data co-occurrence file (231) is an example that can be used in common in each embodiment described later.
FIG. 3 (A) shows an example (231a) of the extracted web page. In various texts, “△△ will move to the following” (22a) is displayed, and “ New address: XXXXXX ”(25a),“ current address: ●●●●● ”(24a).
FIG. 3B is another display example (231b), and after the display of “△△ Office relocation notice” (22b), “Old address: ●●●●●” (24b), “New Address: XXXXXX ”(25b).

As is apparent, the target data of the present invention is ΔΔ, the first related data is the old address ●●●●●, and the second related data is the new address xxxxxx.
Actually, there are many notation methods in the case of an address change, but the following related data relation rules (130) including these are prepared. The related data relationship rule (130) of the present embodiment is an example of a related data relationship rule that can be commonly used in a rule-based method such as claims 2 and 7.

  In the related data relationship detection unit (104), in the case of FIG. 3 (A), the second related data (25a) is mentioned above, the first related data (24a) is mentioned later, and the character string included between them. Since the character string “current address” included in the relation rule between related data (130) can be found from the second relation data xxxxxxx (25a) after applying the rule of number 1, the first relation Data ●●●●● (24a) is detected as previous.

In the case of FIG. 3B, the first related data (24b) is mentioned above, the second related data (25b) is mentioned later, and the character string included between them is changed to the related data relation rule (130). Since the included character string “new address” can be found, the rule of number 6 is applied, and the first related data ●●●●● (24b) is the previous, second related data xxxxxxx (25b) Detected later.
Although only the character string included in the related data relation rule (130) is defined, a character string that is not included may be defined.

Next, in FIG. 4, using the table on the web page, “old address” and “new address” are described as item names, and the first related data (24c) and the second related data (25c) are displayed in the lower column. Are listed.
In such a case, the relation rule between related data (130) can be obtained by using a tag in HTML. For example, when a table frame tag is interposed between related data, the related data on the left side can be the front and the related data on the right side can be the back. In addition, the related data in the lower column or the left column of the “old address” character string may be defined as the front, and the related data in the lower column or the left column of the “new address” character string may be defined as the rear.

  In addition, when the character size of the first related data (24d) and the second related data (25d) is different as shown in FIG. An order may be detected from a tag designating a character size by providing a rule with a larger one after.

In the present invention, since a plurality of related data co-occurrence files (231) may be extracted, there may be a case where a plurality of conditions match the related data inter-relationship rules (130). In such a case, simply, related data having a large number determined as “before” by majority vote may be set before and related data having a large number determined as “after” may be set after.
Further, if the weights are defined in the relation rule between related data (130) as shown in Table 2, and numbers 2, 4 and 5 are extracted, for example, the probability that the above-mentioned related data is before is 0. 4. The accuracy afterwards may be determined as 0.8 + 0.4 = 1.2 (the opposite is true for later-related data) and after the accuracy is high.

  In the present invention, even if the relation rule (130) between related data is used as described above, it is not only determined according to the rule, but also based on a large amount of data, it is detected including which is likely to be earlier or later. It has a feature in that it can.

The detection result is output from the output unit (105). (Output step: S13)
The output in the present invention may be any one of display from the monitor (12), recording to the external storage device (13), output to an external server via the network adapter (14), and the like. The data processing device (1) of the present invention may be installed in a data search device and used so as to change the display order of search results in accordance with the order detected by the device (1).

  In FIG. 2, two different content files (20) and (21) are input in this embodiment, but the same content file may be input to the related data extraction unit (102) to extract two related data. Good. In this case, since related data co-occurs in the content file in the first place, this may also be used by the related data relationship detection unit (204) together with other content data (23).

  As an embodiment of the invention described in claim 4 or the like, the related data extraction unit (102) may extract related data co-occurring with the target data. As described above, it is not necessary when it is known that the first content file (20) or the like is related to the target data in advance, but when the content file is extracted from the Internet or the like, the target data (22) is It is necessary to extract the appearing files and use them as content files.

  In this case, all the character strings in the content file in which the target data (22) appears may be used as the related data (24) (25), but more preferably a predetermined number of characters before and after the target data (22) appears. It is good also considering the character string in this as related data (24) (25). Thereby, a character string in the vicinity that is generally considered to be highly related can be used as related data.

The following advanced methods can also be applied to the extraction of related data and co-occurrence data in the present invention.
Taking co-occurrence data as an example, a method for extracting co-occurrence data including a large number of word groups A (a word group refers to one or more words) constituting related data will be described.

(1) Explanation of basic method (TF / IDF method)
(Equation 1)
score (D) = Σ (tf (w, D) * log (N / df (w)))
Add by w ∈W
W is a set of related data, tf (w, D) is the number of occurrences of w in the content data, df (w) is the number of documents in which W appears in all documents, N is the total number of documents , Document data with high score (D) is output as co-occurrence data. By doing in this way, when many common words and phrases are extracted as related data, it can prevent that many meaningless co-occurrence data are extracted.
Similarly, when extracting related data, significant related data can be extracted.

(2) Explanation of Okapi weighting by Robertson et al.
This method is described in Non-Patent Document 3.

Murata Maki, Ma Ao, Uchimoto Kiyotaka, Osaku Hiromi, Uchiyama Masao, Isahara Hitoshi “Information Retrieval Using Location and Field Information” Natural Language Processing (Journal of the Language Processing Society) April 2000, Volume 7, Issue 2 , p.141 to p.160 It is known that the number 2 in Non-Patent Document 13 has good performance. The product of tf term and idf term before taking the product by Σ becomes Okapi's weighting method, and this value is used for the word weight.

If it is an expression of Okapi (Equation 2)
score (D) = Σ (tf (w, D) / (tf (w, D) + length / delta) * log (N / df (w)))
Add by w ∈W

length is the length of data D, delta is the average length of data, and the length of data is the number of bytes of data, the number of words included in the data, or the like.

Further, the following information search can be performed.
(Okapi reference)
The Okapi equation and SMART equation disclosed in Non-Patent Documents 4 and 5 can also be used. As a more advanced information retrieval method, these Okapi SMART formulas may be used instead of just formulas using tf · idf.

S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford Okapi at TREC-3, TREC-3, 1994 Amit Singhal AT & T at TREC-6, TREC-6, 1997

 In these methods, more accurate information retrieval can be performed using not only tf · idf but also the length of content data.

  In this method of extracting co-occurrence data including more word groups A, Rocchio's formula (Non-patent Document 6) can be used.

J. J. Rocchio, Relevance feedback in information retrieval, The SMART retrieval System, Edited by G. Salton, Prentice Hall, Inc., page 313-323, 1971

This method can be used instead of log (N / df (w))
(Equation 3)
{E (t) + k_af * (RatioC (t)-RatioD (t))} * log (N / df (w))
use.

E (t) = 1 (target data)
= 0 (otherwise)
RatioC (t) is the occurrence rate of t in the related data group B
RatioD (t) is the appearance rate of t in the content data group C
The score (D) is obtained by replacing log (N / df (w)) with the above equation, and the larger the value, the more the co-occurrence data including the word group A is extracted.

 A set W of words w to be added when Σ of score (D) is added is both the original target data and the word group A. However, the original target data and the word group A should not overlap.

  Another method is to add Σ of score (D). The set W of words w is only the word group A. However, the original target data and the word group A should not overlap.

Here, a complicated method is used with the roccio formula. However, as the sum of the number of occurrences of words in word group A is larger, it may be extracted as co-occurrence data including more word groups A. Also, the larger the difference in the appearance of the word group A, the more co-occurrence data including the word group A may be extracted.
The co-occurrence data including the word group A can be extracted by the above method.

(Example 2)
Following Example 1 using the rule base, a method using machine learning is described as Example 2. The following examples relate to the technique of claim 3 of the present invention.
A more detailed configuration of the related data relationship detection unit (104) in the present embodiment is shown in FIG. Here, a teacher data input unit (1041), a solution-feature pair extraction unit (1042), a machine learning processing unit (1043), and a relationship determination unit (1044) are provided.

  In the machine learning method, a learning process in a known machine learning module and a solution estimation process using the learning process are integrated. In the implementation of the present invention, the learning process is not necessarily required, and the external storage device (13) may be provided only with the relation rule (130) between related data formed as a result of machine learning. In that case, all that is necessary for the related data relationship detection unit (104) is the relationship determination unit (1044) that determines the order according to each machine learning method.

  Various methods of machine learning are known, but each method will be briefly described here. Prepare a large set of problem-solution pairs so that you can learn, learn what kind of solution the problem will be, and use the learning results to guess the solution even for new problems (See, for example, Non-Patent Document 7 to Non-Patent Document 9 below).

Maki Murata, Language processing based on machine learning, Faculty of Science and Engineering, Ryukoku University. Invited lecture. 2004. http://www2.nict.go.jp/x/x161/member/murata/ps/kougi-ml-siryou-new2 .pdf Japanese-English translation of tense aspect modality using support vector machine, Maki Murata, Ma Aoi, Kiyotaka Uchimoto, Hitoshi Isahara, IEICE Language Understanding and Communication Study Group NLC2000-78, 2001. CRL in the SENSEVAL2J dictionary task, Masaki Murata, Masao Uchiyama, Kiyotaka Uchimoto, Mao Ai, Hitoshi Isahara, IEICE NLC2001-40, 2001.

In order to convey the problem situation to the machine, what kind of problem is required, features (elements constituting the problem with information used for analysis) are required. The problem is expressed by the feature. For example, in the problem of estimating the tense of Japanese sentence ending expressions,
Problem: "He speaks." --- Solution "Current"
Is given as an example, “He speaks.” “Speaks.” “Speaks.” “Su.” “.”.

  In other words, the machine learning method prepares many sets of feature set-solution pairs, learns with it, learns what kind of feature set the solution will be, and uses the learning results. This is a method of extracting a set of features from a new problem and inferring a solution in the case of the feature.

As shown in FIG. 6, the CPU (10) includes a solution-feature pair extraction unit (1042) and a machine learning processing unit (1043) as a pre-stage processed by the relationship determination unit (1044). Here, the machine learning process obtains a classification result (solution) as to how the distributed text data is classified as shown in FIG.
As a machine learning technique in the machine learning processing unit (1043), for example, a technique such as a k-nearest neighbor method, a simple Bayes method, a decision list method, a maximum entropy method, or a support vector machine method is used.

  The k-nearest neighbor method is a method for obtaining a classification destination (solution) by using the k most similar cases instead of the most similar case, and by majority decision of the k cases. k is a predetermined integer number, and generally an odd number between 1 and 9 is used.

  The Simple Bayes method is a method of estimating the probability of each classification based on Bayes' theorem and determining the classification having the highest probability value as a classification destination.

  In the Simple Bayes method, the probability of outputting the classification a in the context b is given by the following formula 4.

Here, the context b is a set of features f _j (∈F, 1 ≦ j ≦ k) set in advance. p (b) is the appearance probability of context b. Here, it is independent of the classification a and is not calculated because it is a constant. P (a) (where P is a tilde at the top of p) and P (f _i | a) are the probabilities estimated from the teacher data, respectively, and the probability f of class a, and the feature f for class a means the probability of having _i . When the value obtained by performing maximum likelihood estimation as P (f _i | a) is used, the value often becomes zero, and it may be difficult to determine the classification destination because the value of Equation 5 is zero. Therefore, smoothing is performed. Here, the smoothing using the following Equation 6 is used.

Here, freq (f _i , a) means the number of cases having the feature f _i and the classification a, and freq (a) means the number of cases having the classification a.

  The decision list method uses features and combinations of classification destinations as rules, stores them in the list in a predetermined priority order, and when input to be detected is given, from the highest priority in the list This is a method in which input data is compared with the feature of the rule, and the classification destination of the rule having the same feature is set as the classification destination of the input.

In the decision list method, the probability value of each classification is obtained using only one of the features f _j (εF, 1 ≦ j ≦ k) set in advance as a context. The probability of outputting classification a in a context b is given by

(Equation 7)
p (a | b) = p (a | fmax)

However, fmax is given by the following equation (8).

P (a _i | f _j ) (where P is a tilde at the top of p) is the rate of appearance of classification a _i when feature f _j is in the context.

In the maximum entropy method, assuming that a set of preset features f _j (1 ≦ j ≦ k) is F, a number 10 that represents entropy is maximized while satisfying a predetermined conditional expression (Equation 9) below. This is a method of obtaining a probability distribution p (a, b) for the classification and obtaining a classification having the largest probability value among the probabilities of the respective classifications determined according to the probability distribution.

However, A and B mean a set of classification and context, and g _j (a, b) is a function that is 1 if the context b has a feature f _j and the classification is a, and 0 otherwise means. P (a _i | f _j ) (where P is a tilde at the top of p) means the rate of appearance of (a, b) in the known data.

  Equation 9 is to obtain the expected value of the frequency of the output and feature pair by multiplying the probability p and the function g which means the appearance of the output and feature pair. With the constraint that the expected values calculated based on the probability distribution obtained on the left side are equal, entropy maximization (smoothing of the probability distribution) is performed to obtain the probability distribution of the output and the context. Details of the maximum entropy method are described in Non-Patent Document 10 below.

Eric Sven Ristad, Maximum Entropy Modeling for Natural Language, (ACL / EACL Tutorial Program, Madrid, 1997

  The support vector machine method is a method of classifying data composed of two classifications by dividing a space by a hyperplane. FIG. 8 shows the concept of margin maximization in the support vector machine method. In FIG. 8, a white circle means a positive example, a black circle means a negative example, a solid line means a hyperplane that divides the space, and a broken line means a surface that represents the boundary of the margin area. FIG. 8A is a conceptual diagram when the interval between the positive example and the negative example is small (small margin), and FIG. 8B is a conceptual diagram when the interval between the positive example and the negative example is wide (large margin). is there.

  At this time, if the two classifications consist of positive and negative examples, the larger the interval (margin) between the positive and negative examples in the learning data, the less likely it is to make an incorrect classification with open data. As shown in FIG. 8B, a hyperplane that maximizes this margin is obtained, and classification is performed using it.

  Basically, it is as described above, but usually, an extension of the method that a small number of cases may be included in the inner area of the margin in the training data, or an extension that makes the linear part of the hyperplane nonlinear ( The one with the introduction of the kernel function is used.

  This extended method is equivalent to classification using the following discriminant function (Equation 11), and the two classes can be discriminated depending on whether the output value of the discriminant function is positive or negative.

Where x is the context (set of features) to be identified, and x _i and y _j (i = 1, ..., l, y _j ∈ {1, -1}) are the context of the learning data and the classification destination Means the function sgn
sgn (x) = 1 (x ≧ 0)
-1 (otherwise)
Further, each α _i is for the case where the number 12 is maximized under the constraints of the numbers 13 and 14.

  The function K is called a kernel function, and various functions are used. In this embodiment, the following polynomial is used.

(Equation 15)
K (x, y) = (x ・ y + 1) d

C and d are constants set experimentally. For example, C was fixed at 1 throughout all treatments. Moreover, d is trying two kinds of 1 and 2. Here, x _i satisfying α _i > 0 is called a support vector, and the portion taking the sum of Expression 8 is usually calculated using only this case. That is, only actual cases called support vectors are used for actual analysis.

  Details of the extended support vector machine method are described in Non-Patent Document 11 and Non-Patent Document 12 below.

Nello Cristianini and John Shawe-Taylor, An Introduction to Support Vector Machines and other kernel-based learningmethods, (Cambridge University Press, 2000) Taku Kudoh, Tinysvm: Support Vectormachines, (http://chasen.org/~taku/software/TinySVM/, 2002)

  The support vector machine method handles data with two classifications. Therefore, when dealing with cases where the number of classifications is 3 or more, usually, a method such as the pair-wise method or the one-VS rest method is used in combination.

  The pairwise method generates all pairs (n (n-1) / 2) of two different classification destinations in the case of data having n classifications, and the binary classifier determines which is better for each pair. In other words, it is a method of obtaining a classification destination by a majority decision of classification destinations based on n (n-1) / 2 binary classifications, which is obtained by a support vector machine method processing module.

  For example, when there are three classification destinations, a, b, and c, the one-VS rest method generates three sets of classification destination a and other, classification destination b and other, classification destination c and other, The learning process is performed on the set of the support vector machine method. Then, in the estimation process based on the learning result, the learning results of the three sets of support vector machines are used. See how the three support vector machines are estimated as candidates to be estimated. Of the three support vector machines, it is the non-other classification target and the separation plane of the support vector machine. This is a method for obtaining a classification destination of a thing farthest from the object. For example, when a candidate is farthest from the separation plane in the support vector machine created by the learning process of “classification destination a and others”, the candidate classification destination is estimated as a.

  As described above, there are various machine learning techniques, but any of the present invention can be used for the related data relationship detection unit (104). That is, when relations between related data, for example, two related data are successively input, a solution of 1 is obtained if the order is correct, 0 if the order is not correct, and the probability thereof can be obtained as a solution.

In learning, the teacher data input unit (1041) inputs teacher data prepared in advance. The teacher data may be stored in an external storage device or the like. The teacher data includes two related data that are known in advance, and the solution is, for example, 1 if the included order is correct, and 0 if the included order and the correct answer are opposite. Good. The method of determining this solution can be appropriately designed according to the machine learning method and the required result.
In addition, the method of obtaining a solution differs as follows depending on any of the machine learning methods described above.

  For example, in the embodiment of the present invention, when the machine learning processing unit (1043) uses the k-nearest neighbor method as a machine learning method, the machine learning processing unit (1043) is input by the teacher data input unit (1041). Define the similarity between cases based on the ratio of overlapping features (the ratio of how many of the same features) in the set of features extracted from the teacher data, and learn the similarities and cases defined above Information is stored in the relation rule (130) between related data.

  The relationship determination unit (1044) then determines whether the data extracted by the solution-feature pair extraction unit (1042) from the related data co-occurrence file (231) is the correctness of the context defined in the related data relationship rule (130). The k features are selected from the cases of the related data relation rule (130) in descending order of the probability that the data is correct, and the majority of the selected k features is determined. The classification destination of whether or not it is correct is estimated as a solution.

  That is, in the relationship determination unit (1044), the degree of what kind of solution (classification destination) is likely to be obtained for each piece of extracted data is determined by the number of votes of majority vote with the selected k features, The number of votes acquired by the classification “correct (the order of related data in the related data co-occurrence file is in the correct order)”. If the number of votes is less than a majority, the order of appearance and the reverse are correct.

  When the simple Bayes method is used as the machine learning method, the machine learning processing unit (1043) uses a combination of a solution of the case and a set of features as learning result information for the case of the teacher data. Store in the relation rule (130).

  Then, when the related data co-occurrence file extraction unit (103) extracts the related data co-occurrence file (231), the relationship determination unit (1044) determines the solution of the learning result information of the related data inter-relationship rule (130). Based on the pair with the feature set, the probability of being classified into each category related to whether the appearance order is correct or not is calculated for the feature set obtained by the solution-feature pair extraction unit (1042) based on Bayes' theorem Then, the classification having the largest probability value is estimated as the classification (solution) of the feature for the data.

  In other words, in the relationship determination unit (1044), the probability of becoming a certain solution in the case of a set of features about the extracted data is set as the probability of being classified into each category, here the probability of being classified as “appearance order is correct”. And

  When the decision list method is used as the machine learning method, the machine learning processing unit (1043) sets a list of rules of features and classification destinations in a predetermined priority order for the example of the teacher data, and the related data relationship rule (130). Then, when the related data co-occurrence file extraction unit (103) extracts the related data co-occurrence file (231), the relationship determination unit (1044) has a higher priority in the list of the relationship rules between related data (130). In order, the feature of the extracted expression pair candidate and the rule feature are compared, and the classification destination of the rule having the same feature is estimated as the candidate classification destination (solution).

  In other words, the relationship determination unit (1044) determines the degree of likelihood of being a solution in the case of a set of features of the extracted data by using a predetermined priority order or a numerical value or scale corresponding thereto, in this case, “appearance order is correct. ”In the list of probabilities of classification.

  When the maximum entropy method is used as the machine learning method, the machine learning processing unit (1043) specifies a class that can be a solution from the example of the teacher data, satisfies the predetermined conditional expression, and indicates the entropy. A probability distribution consisting of a set of features when maximizing and a binomial classification that can be a solution is obtained and stored in the relation rule (130) between related data. When the related data co-occurrence file extraction unit (103) extracts the related data co-occurrence file (231), the relationship determination unit (1044) uses the probability distribution of the relationship rule between related data (130). Then, the probability of a classification that can be a solution for the set of features of the extracted file is obtained, a classification that can be a solution having the largest probability value is specified, and the specified classification is estimated as a candidate solution. That is, in the relationship determination unit (1044), the probability of becoming a certain solution in the case of the set of features of the extracted data is set as the probability of being classified into each category, in this case, the probability of being classified as “appearance order is correct”. And

When the support vector machine method is used as the machine learning method, the machine learning processing unit (1043) identifies a class that can be a solution from the example of the teacher data, divides the class into a positive example and a negative example, In a space whose dimension is a set of case features according to a predetermined execution function using a function, the interval between the positive example and the negative example of the case is maximized, and the positive example and the negative example are divided by a hyperplane. A plane is obtained and stored in the relation rule (130) between related data.
The related data relationship rule (130) of the present embodiment is an example of a related data relationship rule that can be commonly used in the method using machine learning according to claim 3 or the like.

  When the related data co-occurrence file extraction unit (103) extracts the related data co-occurrence file (231), the relationship determination unit (1044) uses the hyperplane of the relationship rule between related data (130), Identify whether the feature set for the extracted data is on the positive side or the negative side in the space divided by the hyperplane, and determine the classification determined based on the identified result as the candidate solution. Estimated.

  That is, in the relationship determination unit (1044), the degree of the likelihood of becoming a solution in the case of the set of features of the extracted data is determined by the distance from the separation plane to the space of the positive example (data in which the appearance order is correct). Magnitude. More specifically, when the data in which the order of appearance is correct is a positive example and the data that is not reputation information is a negative example, the data located in the space on the positive example side with respect to the separation plane is determined as “data in which the order of appearance is correct”. Then, the distance from the separation plane of the case is set to a degree in which the appearance order of the data is correct.

Furthermore, in the present invention, a method using a known neural network or a method using multiple regression analysis can be used as a machine learning method.
For example, if there are two types of classification to be obtained, multiple regression analysis can be used. The method of executing multiple regression analysis on a computer is detailed in Non-Patent Document 13.

"Time series analysis and forecasting with Excel", Chapter 3, Ohm

  In the case of multiple regression analysis, as many explanatory variables x as the number of features are prepared, and the presence or absence of the features is represented by 1,0 as the value of the explanatory variable x. The objective variable (explained variable) may be obtained with a value of 1 for a certain classification and a value of 0 for another classification.

As described above, the present invention uses a machine learning module having any known machine learning method to generate a relational rule (130) between related data, and then the relation determination unit (1044) It is accurately determined whether or not the order of appearance is correct.
Whether or not the appearance order is correct may be output as either “appearance order is correct” or “appearance order and correct order are opposite” by the machine learning method as described above. In some cases, the “correct probability” is output. The “probability of appearance order” may be output together with the probabilities in descending order. Further, it may be output together with a format for indicating the probability, for example, a character color, a character size, or a mark indicating the probability.

In the present embodiment, a predetermined number can be extracted in order from the one with the highest probability or the highest one in either the rule-based method or the method using machine learning. It is also possible to set a certain threshold value and extract a value exceeding the threshold value. Only a predetermined number may be extracted from those having a predetermined probability or higher and a higher probability. In this way, the criteria for extracting data in this way are arbitrary.
The threshold value and the predetermined number may be fixed in advance in the apparatus, or may be changed by the user.

  In the present invention, the appearance positions of two related data can be used as features. The appearance position here includes not only an absolute position but also a relative position of two related data. The absolute appearance position is, for example, the number of characters, the line, the column, etc., where the related data in the file starts. The relative position includes, for example, the number of characters indicating the number of characters before (after), as well as which is before and after.

As a feature, a character string included simultaneously with related data can be used. For example, whether or not a character string such as “new address” and “old address” as shown in the rule base is included or what is included can be used as a feature.
Moreover, it can also be made into the feature that it is not contained conversely. In other words, the fact that the character string “change” or “transfer” is not included suggests that the related data co-occurrence file may not indicate the context between related data in the first place, and the probability is calculated accordingly. It can be used as a feature to do.

  The character string included at the same time may be a partial character string of the related data. For example, if part of the teacher-related data includes an old building name and a new building name, it is possible to detect that the new building name is later data by using those character strings as features. Contribute to. It can be detected in the same way when the housing display is changed due to a merger of municipalities.

Tag information may be used as a feature. Similar to the configuration described above, a tag of a border line of a table of related data, a tag for setting a font, or the like can be used as a feature.
These can be used not only alone but also in combination. For example, in the table as shown in FIG. 4, a character string that is simultaneously included as “old address” and “new address”, tag information that each related data is arranged immediately below each, and first related data (24c) The relative positional relationship that the second related data (25c) is immediately after it can be used as the feature.

  In order to efficiently extract characteristic related data in the related data extraction unit (102) of the present invention, a technique for extracting a specific expression may be used. In other words, the CPU (10) of the present invention includes a determination target noun extraction unit (not shown), and extracts specific expressions from the first content file (20) and the second content file (21), respectively. This will be briefly described below.

(1) A method using machine learning to extract proper expressions
There is a method of extracting a specific expression using machine learning (see, for example, Non-Patent Document 14 below).

Masayuki Asahara, Yuji Matsumoto, Utilization of Redundant Morphological Analysis in Japanese Named Expression Extraction Information Processing Society of Japan Natural Language Processing Study Group NL153-7 2002

First, for example, the sentence “Japan's prime minister is Mr. Koizumi” is divided into each character, and the correct characters such as B-LOCATION and I-LOCATION are assigned to the divided characters as follows: Set the correct answer. The first column below is each divided character, and the correct tag of each character is the second column.

Day B-LOCATION
I-LOCATION
O
Neck O
Phase O
Is O
Small B-PERSON
Izumi I-PERSON
O
N
At O
O
. O

In the above, B-??? is a tag that means the beginning of the type of proper expression below the hyphen. For example, B-LOCATION means the beginning of a unique expression called place name, and B-PERSON means the beginning of a unique expression called person name. Also, I-??? is a tag that means other than the beginning of the type of proper expression below the hyphen, and O is other than these. Therefore, for example, the character “day” is a character that corresponds to the beginning of the unique name “place name”, and the character “book” is the unique name “place name”.

  In this way, the correct answer of each character is set, learned from such data, this correct answer is estimated with new data, and from this correct answer tag, the beginning of each proper expression and how far it is. Is recognized and the proper expression is estimated.

When learning from the correct data set for each character, the system uses various information in the form of features. For example,
Day B-LOCATION
Part of
Japan-B Noun-B
Such information is used. Japan-B means the beginning of the word Japan, and noun-B means the beginning of the noun. For the recognition of words and parts of speech, for example, the morphological analysis by Chasen described above is used. The above-mentioned Chasen can also estimate the part of speech of each word, so if you enter "go to school" you get the following results.

School Gakkou School Noun-General
To He To Particle-Case particle-General
Go Iku Go Verb-Independence
EOS
In this way, each line is divided so that one word is included, and reading and part-of-speech information are given to each word.

  For example, in the non-patent document 14 described above, as features, the characters constituting the input sentence itself (for example, “small”), character type (for example, hiragana, katakana, etc.), part of speech information, tag Information (for example, “B-PERSON” etc.) is used.

 Learning using these features. Investigate what features appear in the characters that estimate the tag and the surrounding characters, learn what features are likely to appear when the features appear, and use the learning results to create new data Perform tag estimation. For machine learning, for example, a support vector machine is used.

  In addition to the above-described method, there are various methods for extracting the proper expression. For example, there is a method of extracting a specific expression using a maximum entropy model and a rewrite rule (see Non-Patent Document 15).

Uchimoto Kiyotaka, Ma Aoi, Murata Maki, Osaku Hiromi, Uchiyama Masao, Isahara Hitoshi, Entity Expression Extraction Based on Maximum Entropy Model and Rewriting Rules, Journal of the Language Processing Society, Vol. 7, No. 2, 2000

Also, for example, the following non-patent document 16 uses a support vector machine to
It describes a technique for extracting expressions.

Hiroyasu Yamada, Taku Kudo, Yuji Matsumoto, Japanese Named Expression Extraction Using SupportVector Machine, Transactions of Information Processing Society of Japan, Vol.43, No.1 ", 2002

(2) Method using created rules
There is also a method of extracting a specific expression by manually creating a rule.
For example,
A noun + “san” means a person
Name as noun + "Prime Minister"
Noun + “corporation” means company name + “town” means place name
For example, a noun + “city” is a place name.

  A specific expression is extracted by the above method, and, for example, a person name or a company name can be extracted from the extracted expressions by the solution feature pair extraction unit (1042).

  By extracting only the specific expression as related data in this way, it is possible to prevent the extraction of related data that does not need to detect the context, and the order of the present invention is detected particularly for related data that is important for the target data. be able to.

(Example 3)
The present invention is not limited to the above-described method of detecting the front and back of the related data, but can also provide a method of detecting the front and rear from the co-occurrence data co-occurring with the related data. Examples of the present invention described in claims 5 to 8 will be described below.
FIG. 9 is a configuration diagram of the data processing apparatus (1 ′) according to the present embodiment. The same components as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  The CPU (10) includes an input unit (106), a related data extraction unit (107), a co-occurrence data extraction unit (108), a co-occurrence data relationship detection unit (109), and an output unit (105). . The processing shown in FIG. 10 is executed with this configuration.

  First, the input unit (106) is connected to the first content file (20) and the second content file (21) from the external storage device (3) or a server such as the Internet or a LAN (Local Area Network) via the network adapter (40). ) Is acquired and taken into the CPU (10).

  In the related data extracting unit (107) as the related data extracting means according to claim 5, each content file (20) (21) which is a file different from the related data related to the target data (22) defined in advance. Extract from (Related Data Extraction Step: S10) This process is the same as in the first embodiment. The content file may be the same file.

  In the co-occurrence data extraction unit (108), which is a co-occurrence data extraction means, one or a plurality of co-occurrence with the first related data (24) from the first content file (20) or the content data (23). From the first co-occurrence data (30), the second content file (21) and the content data (23) which are character strings, the first or plural character strings which co-occur with the second related data (25). 2 Co-occurrence data (31) is extracted. (Co-occurrence data extraction step: S20)

As another embodiment, as described in claim 6 and the like, a co-occurrence data co-occurrence file extraction step (not shown) is provided immediately after the co-occurrence data extraction step (S20). A co-occurrence data co-occurrence file in which two co-occurrence data co-occur may be extracted.
From the extracted co-occurrence data and co-occurrence data co-occurrence file, the co-occurrence data inter-relationship detection unit (109), which is a co-occurrence data inter-relationship detection means, refers to the co-occurrence data inter-relationship rule (131) and The order of the co-occurrence data (30) and the second co-occurrence data (31) is detected. (Co-occurrence data relationship detection step: S21)

  In this embodiment, the co-occurrence data relationship detection is performed instead of the related data relationship detection step (S12). However, the co-occurrence data may be processed in the same manner as the case of the related data. The co-occurrence data relationship rule (131) is not different from the related data relationship rule (130).

In the co-occurrence data relationship detection step (S21), the co-occurrence data relationship rule (131) created based on the rule base as described in claim 7 or the like may be used, or in claim 8 or the like. A co-occurrence data relation rule (131) created by machine learning as described above may be used.
The rule base creation method and the machine learning method are the same as in the above embodiment.

Further, the output unit (110) outputs the before and after related data in accordance with the determined co-occurrence data before and after. (Output step: S22)
That is, when it is detected that the first co-occurrence data is later and the second co-occurrence data is before, the first related data is output as the next and the second related data is output as the previous.

  The above processing will be described using a specific example. The first content file (20e) as shown in FIG. 11A includes target data (22e) and first related data (24e). The second content file (21e) includes the same target data (22e) and second related data (25e). These are extracted in the related data extraction step (S10).

  Next, from the two files (231e) and (231f) in the content data (23), the first co-occurrence data (30e) and the second related data (25e) co-occur with the first related data (24e), respectively. The second co-occurrence data (31e) that occurs is extracted in the co-occurrence data extraction step (S20).

  As shown in the figure, the two related data only have different building names, ●●●●● building and XXXXXX building, and the rule-based example 1 and machine learning were used. Even in Example 2, the context of both may not be determined. At that time, the method using the co-occurrence data according to the present invention is applied.

That is, the date of October 1, 2005 for the ●●●●● building co-occurs with the date of April 1, 2008 for the xxx building, and the co-occurrence data ( 30e) (31e) is compared to determine the front and rear of the building and the xxx building.
The example given here is a simple example. If it is determined that the relationship between co-occurrence data is determined before and after the date in the co-occurrence data relationship rule, the co-occurrence data relationship detection step (S21). , The second co-occurrence data (31e) on April 1, 2008 is detected later.

As a result, in the output step (S22), the second related data (25e) in which the co-occurrence data is determined to be later is output, and the first related data (24e) is output to the front.
The content data (23) from which the co-occurrence data is extracted can be obtained from the co-occurrence data with extremely high accuracy by using data in which the completed information of the building is regularly described, such as the illustrated “building completion information”. It is possible to detect before and after related data. However, since the present invention can be detected from a large number of content data (23) based on a large number of rule bases or by machine learning, it is not so routine and there are many old dates that co-occur. It can be detected from a lot of ambiguous data such as many new ones.

  Another embodiment is shown in FIGS. First, as shown in FIG. 13, the first related data (24g) and the second related data (25g) related to the target data ΔΔ (22g) are extracted from one content file (20g). (S10) Thus, the related data may be extracted from the same content file (20g).

  Unlike the example shown in FIG. 3A, even when a new address and an old address co-occur, a graphic is used to represent “new” and “old” instead of characters. It is not possible to detect the context (new / old relationship) between the two.

  Therefore, the first co-occurrence data (30g) and the second co-occurrence data (31g) that co-occur with the first related data (24g) and the second related data (25g) are extracted. As shown in FIG. 14, in each part of one content file (20) (20h) (20i), a character string appearing within a predetermined number of characters (for example, within 50 characters before and after) is co-occurred from each related data. It can be data. In the present example, the first co-occurrence data is “XYZ Building” (30 g) co-occurring with the old address “1-2-3 Chuo-ku Ekimae-mae” (24 g), and the new address “Chuo-ku Tokyo Station” “ABC Building” (31 g) co-occurring with “9-8-7” (25 g) is the second co-occurrence data.

  Further, a co-occurrence data co-occurrence file (231g) in which these two co-occurrence data (30g) (31g) co-occur is extracted. In the file (231g), “kara” is included between the character string “XYZ building” and the character string “ABC building”, and “ABC building” is a new address even if the rule base is used as described above. Can be determined.

  Co-occurrence data is used in this way because related data may not always be optimal for detecting the order. For example, when the related data to be detected before and after is an address, it is difficult to detect the front and back by looking at only the address unless the address display is changed. Even when the relation rule (130) between related data is used as a machine learning result, there are cases where it can be detected with high accuracy by learning from the position, etc., but the position is only characteristic around the appearance position, and only the address data after that. In the case of related data, selection may not be possible with sufficient accuracy.

  In addition to the example shown above, when the phone co-occurs with the address and uses them as co-occurrence data (30) (31), the notation method of the telephone area code, for example, has been changed. It is conceivable that the facts before and after the relevant data can be detected from the facts. That is, when the first co-occurrence data (30) is the telephone number “(0424) 12-1234” and the second co-occurrence data (31) is the telephone number “(042) 321-1234” In the co-occurrence data relation rule (131), if the learning result that the area code is newer (after) is stored, the first related data (24) and the second related data are stored. Even if (25) is compared, if it is not clear before and after, it is possible to detect which is before or after.

  Further, co-occurrence data may be used as data co-occurring with each related data. That is, if the content file is a company name / zip code / address and the primary co-occurrence data corresponding to the relevant data (address) of the target data (company name) is a zip code, it is further separated from the zip code. When the secondary co-occurrence data co-occurring in the content file is a post office name, the post office name can be easily detected. In such a case, secondary and tertiary co-occurrence data can also be used.

(Application to other languages)
The present invention can be applied to languages other than Japanese. For example, the content file or the content data may be an English text file, and the target data, related data, co-occurrence data, etc. may all be English words or clauses (a set of two or more words).
Words are easy to extract in a language such as English, but proper nouns can be extracted accurately by performing morphological analysis and obtaining part-of-speech information. As a technique for performing English morphological analysis, there is a technique disclosed in Non-Patent Document 17.

Eric Brill, Transformation-BasedError-Driven Learning and Natural Language Processing: A Case Study in Part-of-Speech Tagging, Computational Linguistics, Vol. 21, No. 4, p.543-565,1995.

  Also, a related data extraction step (S10), a related data co-occurrence file extraction step (S11), a related data relationship detection step (S12), a co-occurrence data (S20), a co-occurrence data relationship detection step (S21), In each process of the present invention, such as the origin data co-occurrence file extraction step, execution processing can be performed after Japanese and English are translated.

For example, when related data “1-2-3 Ekimae, Chuo-ku, Tokyo” is included for the target data “Triangle Corporation”, they are translated into “△△ Corporation”, “Tokyo Chuo-ku Station 1 -2-3 ".
Such translation can be simply replaced by using a translated word dictionary, that is, a dictionary in which translated words are written in pairs, such as car-car, or word / text can be translated by known translation software. .

Finally, an example of an algorithm for acquiring address change information and an experimental example thereof are shown.
First, using a predetermined search engine, an AND search is performed on the company name “Celestar Communications Co., Ltd.” and “Head Office”. A part of the search result is shown below.

以降省略

Omitted

  From these search results, when there is no word “company” between the company name and the word “head office”, the expression after “head office” is extracted as a partial expression of the address (excluding symbols and the like). The date expression is extracted by the following regular expression.

  Then, the number of bytes between the company name and the word “head office” is calculated. When the number of bytes, address partial expression, date expression, and output data of the original search engine are output separated by spaces, the result is as follows.

  The following input 1 and input 2 data are created from the output of Table 5. In other words, data for AND search of “company name” and “partial expression of address” is created for the type of partial expression of address.

  AND search for inputs 1 and 2 with a search engine. At this time, the outputs of the input engines 1 and 2 were as follows.

以下省略

Omitted

以下省略

Omitted

  Get complete address information from search engine results. For this reason, if there is no word “company” or “laboratory” between the company name and the address partial expression, the expression after the address partial expression must not include punctuation marks, abbreviations, parenthesis expressions, and blank expressions. Things are extracted as address expressions, including partial expressions of addresses. The date expression is extracted by the following regular expression.

  It calculates the number of bytes between the company name and the partial expression of the address, and outputs the number of bytes, address expression, date expression, and output data of the original search engine separated by a space.

  Addresses in the output of Table 5 are complemented with the expressions of Tables 10 and 11, and further output in the order of date.

Next, an experiment was also conducted on a method for acquiring company name change information.
1. Perform an AND search on the search engine with the words “company name” and “change”. The results were as follows.

以降省略

Omitted

2.From the sentence including company name and change
[A] matches the pattern of ..... "[B]" .....
Extract [A] and [B].
[A] is the original company name, and [B] is the new company name.
Hereafter, [A], [B], and the extracted original sentence are output separated by spaces.

  As described above, it was found that a lot of company name change information can be extracted by this method. Furthermore, by preparing an existing company name dictionary and extracting only those whose [A] is in the existing company name dictionary, it is considered that the company name change information can be acquired with higher performance.

1 is an overall configuration diagram of a data processing apparatus of the present invention. It is a process flowchart of the data processing method of this invention. Related data co-occurrence file example Related data co-occurrence file example Related data co-occurrence file example It is a block diagram of the related data relationship detection part in the 3rd Example of this invention. It is explanatory drawing explaining the process of machine learning. It is explanatory drawing explaining the process of machine learning (SVM). It is a whole block diagram of the data processor in 3rd Example of this invention. It is a processing flowchart of the data processing method in 3rd Example of this invention. Related data co-occurrence file example Example of co-occurrence data co-occurrence file Related data co-occurrence file example Example of co-occurrence data co-occurrence file Example of co-occurrence data co-occurrence file

Explanation of symbols

1 Data processing device 10 CPU
11 Keyboard / Mouse 12 Monitor 13 Hard Disk 14 Network Adapter 101 Input Unit 102 Related Data Extraction Unit 103 Related Data Co-occurrence File Extraction Unit 104 Related Data Relationship Detection Unit 105 Output Unit 130 Related Data Relationship Rule Database

Claims (18)

In the data processing apparatus for detecting the order of the first related data and the second related data that are related to each other with respect to the related data related to the predetermined target data,
Related data extracting means for extracting the first related data and the second related data from the same or different files stored in the storage means on the network or locally, respectively;
Related data co-occurrence file extracting means for extracting a related data co-occurrence file in which the first related data and the second related data co-occur from a network or local storage means;
A related data relationship detecting means for detecting before and after between the first related data and the second related data with reference to a predetermined related data relationship rule from the related data co-occurrence file;
An output means for outputting the detection result. The related data relationship rule is information relating to a character string that is included or not included at least between the first related data and the second related data,
2. The relational data relation detecting means extracts a character string between the first relational data and the second relational data in the relational data co-occurrence file and collates with the relational relation relation rule. The data processing apparatus described in 1. The relation rule between related data uses a plurality of teacher files in which two related data for teachers whose context is known in advance co-occurs, and the appearance positions of the two related data for teachers in the teacher file or simultaneously A learning result obtained by machine learning using at least one of a character string included or not included, or tag information included at the same time as a feature,
The related data relation detecting means extracts the feature from the related data co-occurrence file, and inputs the first related data and the second related data, and refers to the learning result to refer to the first related data. The data processing apparatus according to claim 1, wherein before and after the second related data is calculated. The data processing device,
The data processing apparatus according to any one of claims 1 to 3, wherein the related data extracting unit extracts first related data and second related data that co-occur with the target data from a network or a local storage unit. In the data processing apparatus for detecting the order of the first related data and the second related data that are related to each other with respect to the related data related to the predetermined target data,
Related data extracting means for extracting the first related data and the second related data from the same or different files stored in the storage means on the network or locally, respectively;
One or more first co-occurrence data co-occurring with the first related data is extracted from the file including the first related data, and the second related data is co-existing with the file including the second related data. Co-occurrence data extracting means for extracting one or a plurality of second co-occurrence data to occur,
Co-occurrence data for detecting the front and rear between the first co-occurrence data and the second co-occurrence data with reference to a predetermined co-occurrence data relation rule regarding the front and rear between the first co-occurrence data and the second co-occurrence data Inter-relationship detection means;
A data processing apparatus comprising: output means for outputting the detection results as before and after the first related data and the second related data co-occurring therewith. In the data processing device,
The co-occurrence data co-occurrence file extracting means for extracting the co-occurrence data co-occurrence file in which the first co-occurrence data and the second co-occurrence data co-occur from a network or local storage means. Data processing equipment. The co-occurrence data relation rule is information relating to a character string that is included or not included at least between the first co-occurrence data and the second co-occurrence data,
The co-occurrence data relationship detecting means extracts a character string between the first co-occurrence data and the second co-occurrence data in the co-occurrence data co-occurrence file, The data processing device according to claim 6. The co-occurrence data relation rule uses a plurality of teacher files each including two pieces of teacher-related data whose context is known in advance and a plurality of character strings, and the teacher relations in the teacher file A learning result obtained by machine learning using one or more character strings included with data as features,
The co-occurrence data relationship detecting means inputs the first co-occurrence data and the second co-occurrence data as features, and refers to the learning result between the first co-occurrence data and the second co-occurrence data. The data processing apparatus of Claim 5. In the configuration in which the target data to be processed by the data processing device is the address and the related data is the destination.
The data processing apparatus according to claim 1, wherein a new / old relationship is detected between the first related data and the second related data. A computer data processing method for detecting a sequence of first related data and second related data, which are related to predetermined target data,
A related data extracting step in which the related data extracting means extracts the first related data and the second related data from the same or different files stored in a network or a local storage means,
A related data co-occurrence file extracting means for extracting a related data co-occurrence file in which the first related data and the second related data co-occur from a network or local storage means;
A related data relationship detecting means for detecting a relation between the first related data and the second related data with reference to a predetermined related data relationship rule from the related data co-occurrence file;
A data processing method, wherein the output means includes an output step of outputting the detection result. The related data relationship rule is information relating to a character string that is included or not included at least between the first related data and the second related data,
11. The related data relationship detecting means extracts a character string between the first related data and the second related data in the related data co-occurrence file and collates with the related data relationship rule. The data processing method described in 1. The relation rule between related data uses a plurality of teacher files in which two related data for teachers whose context is known in advance co-occurs, and the appearance positions of the two related data for teachers in the teacher file or simultaneously A learning result obtained by machine learning using at least one of a character string included or not included, or tag information included at the same time as a feature,
The related data relation detecting means extracts the feature from the related data co-occurrence file, and inputs the first related data and the second related data, and refers to the learning result to refer to the first related data. The data processing method according to claim 10, further comprising: calculating before and after the second related data. Said data processing method comprising:
13. The related data extracting unit extracts first related data and second related data co-occurring with the target data from a network or a local storage unit in the related data extracting step, respectively. Data processing method. A computer data processing method for detecting a sequence of first related data and second related data, which are related to predetermined target data,
A related data extracting step in which the related data extracting means extracts the first related data and the second related data from the same or different files stored in a network or a local storage means,
The co-occurrence data extracting means extracts one or a plurality of first co-occurrence data that co-occurs with the first related data from the file including the first related data, and from the file including the second related data. A co-occurrence data extraction step of extracting one or a plurality of second co-occurrence data co-occurring with the second related data;
The co-occurrence data relation detecting means refers to a predetermined co-occurrence data relation rule regarding the front and back between the first co-occurrence data and the second co-occurrence data, and the first co-occurrence data and the second co-occurrence data A co-occurrence data inter-relationship detection step for detecting before and after
A data processing method, characterized in that the output means has an output step of outputting the detection results as before and after the first related data and the second related data co-occurring therewith. In the data processing method, after the co-occurrence data extraction step,
Co-occurrence data co-occurrence file extraction means extracts a co-occurrence data co-occurrence file in which the first co-occurrence data and the second co-occurrence data co-occur from a network or local storage means The data processing method according to claim 14, further comprising an extraction step. The co-occurrence data relation rule is information relating to a character string that is included or not included at least between the first co-occurrence data and the second co-occurrence data,
The co-occurrence data relationship detecting means extracts a character string between the first co-occurrence data and the second co-occurrence data in the co-occurrence data co-occurrence file, The data processing method according to claim 15, wherein collation is performed. The co-occurrence data relation rule uses a plurality of teacher files each including two pieces of teacher-related data whose context is known in advance and a plurality of character strings, and the teacher relations in the teacher file A learning result obtained by machine learning using one or more character strings included with data as features,
The co-occurrence data relationship detecting means inputs the first co-occurrence data and the second co-occurrence data as features, and refers to the learning result between the first co-occurrence data and the second co-occurrence data. The data processing method according to claim 14 or 15, wherein the data before and after is calculated. In the data processing method, the target data is a destination and the related data is a destination.
18. The data processing method according to claim 10, wherein a new / old relationship is detected between the first related data and the second related data.

Images