JP6782644B2 - Information processing systems, information processing methods, and computer programs - Google Patents

Description

The present invention relates to data processing techniques, particularly to information processing systems, information processing methods, and computer programs.

In order to efficiently search a large number of documents that match the keyword, morphological analysis of the documents of the search population may be executed in advance, and index data for search may be created.

JP-A-2002-297589

For morphological analysis, it is necessary to prepare a dictionary for analysis in which words and the like are recorded in advance. This dictionary is required to have few word omissions, that is, completeness. In particular, when the search target is a technical document such as a patent document, many technical terms may be described, and it is not easy to manually guarantee the completeness of the dictionary.

The present invention has been made in view of the above problems, and a main object of the present invention is to provide a technique for supporting the realization of a suitable dictionary for natural language processing such as morphological analysis.

In order to solve the above problems, the information processing system of a certain aspect of the present invention is a dictionary used in natural language processing by a computer, and a dictionary in which a subject word, which is a word whose meaning is established by itself, is stored. A dictionary storage unit to store, a phrase extraction unit that extracts a plurality of phrases from the character string of the first document, and the remaining character string excluding at least the main word stored in the dictionary as a phrase, and a phrase extraction unit When the same character string exists at the beginning of a predetermined number or more of the plurality of phrases, the same character string is extracted as the subject word by the subject word extraction unit and the subject word extraction unit. It is equipped with a dictionary update unit that stores the subject word in the dictionary.

Another aspect of the present invention is an information processing method. In this method, an information processing device that stores a dictionary in which a subject word, which is a word whose meaning is established by itself, is stored in a dictionary used in natural language processing by a computer, is used from a character string of the first document. A phrase extraction step that extracts a plurality of phrases using at least the remaining character string excluding the main word stored in the dictionary as a phrase, and the beginning part of a predetermined number or more of the phrases extracted in the phrase extraction step. If the same character string exists in, the subject word extraction step of extracting the same character string as the subject word and the step of storing the subject word extracted in the subject word extraction step in the dictionary are executed.

It should be noted that any combination of the above components and the conversion of the expression of the present invention between a program, a recording medium in which the program is stored, and the like are also effective as aspects of the present invention.

According to the present invention, it is possible to support the realization of a suitable dictionary for natural language processing such as morphological analysis.

It is a figure which shows the structure of the information processing system of an Example. It is a block diagram which shows the functional structure of the dictionary making apparatus of FIG. It is a figure which shows the character string of a phrase schematically. 4 (a), 4 (b), and 4 (c) are diagrams showing an example of phrase division. It is a figure which shows the example of a plurality of 2nd phrases output from a phrase division part. It is a figure which shows the example of the subject word dictionary. It is a figure which shows the example of the attendant word dictionary. It is a block diagram which shows the functional structure of the natural language processing apparatus of FIG. It is a figure which shows the example of a candidate entry. It is a figure which shows the example of the character string which is the word extraction processing target. It is a flowchart which shows the operation of a dictionary making apparatus. It is a flowchart which shows the operation of a natural language processing apparatus.

First, the outline of the embodiment will be described. The information processing system of the embodiment supports natural language processing, such as morphological analysis, on patent documents written in various languages. Dictionaries used in natural language processing by computers are required to have complete words, that is, completeness, but many technical terms are described in patent documents, and a dictionary created manually is sufficient. It is difficult to provide comprehensiveness. Therefore, in the embodiment, we propose a first technique for automatically creating a highly comprehensive dictionary by eliminating human involvement in the process of creating a dictionary for natural language processing based on a large number of patent documents. Further, in the embodiment, as a natural language processing using a dictionary automatically created by the first technique, a second technique for extracting words from patent documents with high accuracy is also proposed.

FIG. 1 shows the configuration of the information processing system 10 of the embodiment. The information processing system 10 includes a patent document DB 12, a dictionary creation device 14, and a natural language processing device 16. Each device of FIG. 1 is connected via a communication network 18 including LAN, WAN, and the Internet.

The patent document DB 12 is a database server that holds data of a plurality of patent documents, for example, data of a large number of patent documents whose applications have been published. The dictionary creation device 14 is an information processing device that creates a dictionary (hereinafter, also referred to as a “morpheme dictionary”) used in morphological analysis processing by the natural language processing device 16, and may be, for example, a PC / server.

The natural language processing device 16 uses the morphological dictionary created by the dictionary creating device 14 to execute morphological analysis processing on the patent document data held in the patent document DB 12. Specifically, the natural language processing device 16 extracts words from each patent document data, and transmits the extracted word data including a list of the extracted words from each patent document to a predetermined external device (not shown).

The device to which the extracted word data is transmitted may be a device that creates index data for patent search based on the extracted word data of each patent document. Further, the device may be a device that provides a patent search service to a user terminal (not shown) using the index data. The natural language processing device 16 may further include an index data creation function and / or a patent search service providing function, and may be implemented as, for example, a patent search device.

There is no limit to the physical number of each device shown in FIG. For example, the functions of the dictionary creating device 14 of the embodiment may be distributed and executed by a plurality of devices, and similarly, the functions of the dictionary creating device 14 may be distributed and executed by a plurality of devices. Further, both the function of the dictionary creating device 14 and the function of the natural language processing device 16 of the embodiment may be collectively provided by one information processing device.

FIG. 2 is a block diagram showing a functional configuration of the dictionary creation device 14 of FIG. The dictionary creation device 14 includes a control unit 20, a storage unit 22, and a communication unit 24. The control unit 20 executes various data processes for creating a morpheme dictionary. The storage unit 22 is a storage area for data updated or referenced by the control unit 20. The communication unit 24 communicates with the external device according to a predetermined communication protocol. The control unit 20 exchanges data with the patent document DB 12 and the natural language processing device 16 via the communication unit 24.

Each block shown in the block diagram of the present specification can be realized by an element or a mechanical device such as a CPU or a memory of a computer in terms of hardware, and can be realized by a computer program or the like in terms of software. , Draws a functional block realized by their cooperation. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by combining hardware and software. For example, a computer program including a module corresponding to each block in the control unit 20 may be installed in the storage of the dictionary creation device 14 via a predetermined recording medium or network. Then, the CPU of the dictionary creating device 14 may appropriately read a computer program from the storage to the main memory and execute the computer program to exert a function corresponding to each block in the control unit 20.

The storage unit 22 includes a dictionary storage unit 26. The dictionary storage unit 26 stores a subject word dictionary and an incidental word dictionary as two types of morpheme dictionaries. The subject word dictionary stores words whose meanings are established by themselves (hereinafter, also referred to as "subject words"). The subject word is typically an independent word (noun, verb, adjective, etc.) and includes, for example, a character string such as "dining room", "patent", or "nation". The subject word in the examples is limited to two or more characters.

The incidental word dictionary stores words used in association with the subject word (hereinafter, also referred to as "accompanying words"). Accompanying words are typically particles (particles, auxiliary verbs). For example, incidental words include "o", "ha", "desu", "department" of "development department", "kai" of "advisory panel", and "no" of "your". The contingent word may be a word whose phrase is established only when it is combined with the subject word. Further, the incidental word may be a word whose meaning is established (in other words, the meaning is determined) only when it is combined with the subject word.

The control unit 20 includes a phrase reading unit 28, a phrase dividing unit 30, a subject word extraction unit 32, a dictionary update unit 34, an accompanying word extraction unit 36, and a dictionary output unit 38. The phrase reading unit 28 and the phrase dividing unit 30 function as a phrase extraction unit in cooperation with each other, and the remaining character string excluding at least the subject word stored in the subject word dictionary from the character string described in the patent document. Is identified as a phrase and multiple phrases are extracted.

The phrase reading unit 28 reads the character strings described in each patent document from the plurality of patent documents stored in the patent document DB 12. The phrase reading unit 28 extracts a character string from a processing target position in the character string (for example, the first character in the initial state) to a predetermined delimiter (punctuation mark, reading point, period, comma, etc.) as the first phrase. And the process of advancing the processing target position by the amount of the extracted phrase are repeated.

The phrase dividing unit 30 refers to the subject word dictionary and the incidental word dictionary, and extracts a plurality of second phrases obtained by dividing the first phrase from the first phrase extracted by the phrase reading unit 28. The phrase dividing unit 30 identifies at least the remaining character string excluding the subject word stored in the subject word dictionary as the second phrase, and extracts a plurality of second phrases. The phrase division unit 30 of the embodiment outputs both the phrase indicating the subject word stored in the subject word dictionary and the remaining character string excluding the subject word as the second phrase.

The process of the phrase dividing unit 30 will be described with reference to FIG. One box in FIG. 3 represents one character. X is a certain character, b is each character of the subject word stored in the subject word dictionary, and C is a character of the incidental word stored in the incidental word dictionary. (1) The phrase dividing unit 30 holds the next search position set Ps (initially, the head position is 0). Ps is a set of information indicating a position for determining whether or not it is a subject word. The element of Ps (that is, information indicating the position) may be information indicating the number of bytes in the entire document, or information indicating the distance from the beginning of the first phrase (character unit, byte unit, etc.). In FIG. 3, Ps = {P0, P1}.

(2) The phrase dividing unit 30 searches for the next word from the minimum position of Ps (P0 in FIG. 3). In FIG. 3, the subject word "bbbb" is found. (3) The phrase dividing unit 30 outputs the character string (“XX” in FIG. 3) from P0 to the found subject word as a phrase, and outputs the found subject word “bbbb” as a phrase. In the example of FIG. 3, "XX" corresponds to the character string excluding the subject word from the first phrase.

When one or more subject words are found in the above (2), the following (4) and (5) are executed sequentially or in parallel for all the subject words. (4) The phrase dividing unit 30 inserts Ps into one subject word (in FIG. 3, the position immediately after the end of “bbbb” (in the example of FIG. 3, the position of the character “C”, that is, P2)). ) When the adjunct word immediately after the subject word in (4) above is the position immediately after the end of the adjunct word (the letter "X" in the example of FIG. 3), that is, P3) is put in Ps. (6) When the subject word is not found in (2) above, the phrase is output from the processing position (for example, P0) at that time to the end of the character string. (7) The above steps (2) to (6) are repeated until Ps becomes empty. For example, when the processing of P0 is completed, PS = {P1, P2, P3}, and in the next (2), the next word is searched from P1.

4 (a), 4 (b), and 4 (c) show examples of phrase division. Here, it is assumed that "Japan" and "promotion" are recorded in the subject word dictionary, and "word" and "kai" are recorded in the accompanying word dictionary. FIG. 4A shows the first phrase which is the original data before division. The phrase division unit 30 recognizes "Japan" and "promotion" as the subject words because they exist in the subject word dictionary, and outputs them as the second phrase. Further, the phrase dividing unit 30 searches from the position of the "word", and since the start of the subject word immediately after is "shake", the character string from "word" to before "shake", that is, "word" is searched. Output as the second phrase. Since "word" is an incidental word, the next subject word is searched from both the position of "word" and the position of "shake". Similarly, since "kai" is an incidental word, the next subject word is searched for from both the position of "kai" and the position of "day" (Fig. 4 (b)).

Since there is no subject word registered in the dictionary in the character string after "kai", the second phrase is output from the search position to the end of the character string. FIG. 4C shows five second phrases output from the phrase dividing unit 30. When a plurality of subject words are detected from the same position, the phrase dividing unit 30 executes the above processing for all the detected words. In this case, a plurality of second phrases including the second phrase that partially overlaps are extracted.

Returning to FIG. 2, when the same character string exists at the beginning of a predetermined number or more of the plurality of second phrases output from the phrase dividing unit 30, the subject word extraction unit 32 has the same character. Extract the column as a candidate for the subject word. Specifically, the subject word extraction unit 32 arranges a plurality of second phrases output from the phrase division unit 30 in the order of the character code. The subject word extraction unit 32 outputs the first character string as a candidate for the subject word when the first character string is common over a plurality of second phrases. In other words, when the same character code value is continuous for two or more characters from the beginning of each of the plurality of second phrases, the character string indicated by the same character code value is used as a candidate for the subject word.

FIG. 5 shows an example of a plurality of second phrases output from the phrase dividing unit 30. In the example of FIG. 5, all of "Japan", "Nihon University", and "Japanese cuisine" repeatedly appear at the beginning of phrases exceeding a predetermined threshold value (five in the example). Therefore, the subject word extraction unit 32 extracts "Japan", "Nihon University", and "Japanese cuisine" as candidates for the subject word. The above threshold value may be set to an appropriate value based on the knowledge of the system developer, an experiment using the information processing system 10, and the like. For example, when the completeness of the subject word is strongly required, the threshold value may be reduced, and when the completeness and a certain degree of accuracy are required, the threshold value may be increased.

The phrase division unit 30 or the subject word extraction unit 32 may permanently store information indicating a plurality of second phrases (here, referred to as an “extracted phrase list”) in the storage unit 22. In the extracted phrase list, a second phrase extracted over a plurality of dictionary creation timings and a plurality of patent documents may be accumulated. The subject word extraction unit 32 adds the second phrase output from the phrase division unit 30 to the extraction phrase list, and is the same as the first portion of the predetermined number or more of the second phrases recorded in the extraction phrase list. If the character string of is present, the same character string may be extracted as a candidate for the subject word. According to this aspect, the subject word can be extracted more accurately based on a plurality of dictionary creation timings and a second phrase extracted over a plurality of patent documents.

Further, the phrase dividing unit 30 may suppress the extraction of the subject word stored in the subject word dictionary as the second phrase from the first phrase. Further, the subject word extraction unit 32 extracts the second phrase indicating the subject word stored in the subject word dictionary from the target of the subject word extraction process among the plurality of second phrases output from the phrase division unit 30. It may be excluded.

Returning to FIG. 2, the dictionary update unit 34 stores the subject word extracted by the subject word extraction unit 32 in the subject word dictionary of the dictionary storage unit 26. The dictionary update unit 34 determines this candidate as the subject word unless a part including the beginning of one subject word candidate extracted by the subject word extraction unit 32 is established as another subject word. Register in the subject language dictionary. When both the first character string and the second character string including the first character string are extracted as the subject word, the dictionary update unit 34 stores the first character string in the subject word dictionary. , Suppresses storing the second character string in the subject language dictionary. In other words, the dictionary update unit 34 uses the subject word dictionary for candidates composed of a combination of a plurality of subject words (hereinafter, also referred to as "compound words") among the candidates for the subject words extracted by the subject word extraction unit 32. Suppress storing in. As described above, in order to improve the comprehensiveness of the search, relatively short subject words are preferentially stored in the subject word dictionary of the embodiment.

For example, when both "Japan" and "Japanese food" are extracted as candidates for the subject word, the dictionary update unit 34 stores "Japan" in the subject language dictionary while storing "Japanese food" in the subject language dictionary. Discard it without storing it, in other words, ignore it. It can be expected that "cooking" will be registered separately in the subject language dictionary. Further, when "Japanese food" is already stored in the subject word dictionary and "Japan" is newly extracted as a candidate for the subject word, the dictionary update unit 34 stores "Japan" in the subject word dictionary and " "Japanese food" may be deleted from the subject dictionary. In this case, regarding the number of appearances described later, the number of appearances of "Japanese cuisine" may be taken over as the number of appearances of "Japan".

FIG. 6 shows an example of a subject word dictionary. In the subject word dictionary of the embodiment, a plurality of subject words and the number of times each subject word appears in a plurality of documents are recorded in association with each other. The dictionary update unit 34 newly stores candidates that are not stored in the subject word dictionary (that is, candidates for a new subject word) among the candidates for the subject word in the subject word dictionary, and sets 1 for the number of occurrences thereof. Further, the dictionary update unit 34 suppresses the duplication of storing the candidates stored in the subject word dictionary (that is, the subject words detected in the past) among the candidates of the subject word in the subject word dictionary. Increment the number of occurrences of the subject word. As a modification, instead of recording the number of appearances itself, an index value of the frequency of appearance based on the number of appearances may be recorded.

Returning to FIG. 2, the accompanying word extraction unit 36 has the number of characters of the remaining character strings excluding the subject words stored in the subject word dictionary from the character strings read from a plurality of patent documents, which are equal to or less than a predetermined threshold value. Extract the character string as a candidate for an accompanying word. In other words, the incidental word extraction unit 36 selects a character string whose length is equal to or less than a predetermined threshold among the character strings sandwiched between the two subject words in the character strings read from a plurality of patent documents as candidates for the incidental word. Extract as. If the length of the sub-character string after the last subject word in the character string is equal to or less than a predetermined threshold value, the contingent word extraction unit 36 extracts the sub-character string as a candidate for the contingent word. The threshold value in the examples is "2". The character string read from the patent document may be the first phrase read by the phrase reading unit 28.

The incidental word extraction unit 36 determines that, among the extracted candidates for the incidental word, a character string whose number of extractions as a candidate is equal to or higher than a predetermined order is regarded as an auxiliary word. Specifically, the contingent word extraction unit 36 holds a table in which a plurality of contingent word candidates are associated with the number of times each candidate is extracted, and is associated with the candidate each time the contingent word candidate is extracted. Increment the number of extractions. The number of extractions can be said to be a value indicating the number of occurrences or the frequency of occurrence of the accompanying word candidate over a plurality of documents. The incidental word extraction unit 36 determines that the candidate with the highest number of extractions is the incidental word. The timing of determining the incidental word may be the time when the subject word extraction process or the subject word dictionary update process for one or more patent documents is completed.

In this way, by providing the threshold value for the length of the incidental word (two characters, etc.) and the threshold value for determining the incidental word (upper 5%, etc.), the comprehensiveness of the incidental word extraction can be enhanced. These threshold values may be set to appropriate values based on the knowledge of the system developer, experiments using the information processing system 10, and the like. For example, when the completeness of an adjunct word is strongly requested, the adjoint word judgment threshold value may be increased (for example, the top 10%), and when a certain degree of accuracy is required along with the completeness, the adjunct word judgment threshold value is small (for example, the top 3). %) May be used.

The dictionary update unit 34 stores the candidate character string determined to be an incident word by the incident word extraction unit 36 in the incident word dictionary. FIG. 7 shows an example of an accompanying word dictionary. In the incidental word dictionary of the embodiment, a plurality of incidental words and the number of times each incidental word appears in a plurality of documents are recorded in association with each other. The dictionary update unit 34 newly stores the candidates of the incidental words that are not stored in the incidental word dictionary (that is, the candidates of the new incidental words) in the incidental word dictionary, and sets the number of occurrences to 1. On the other hand, the dictionary update unit 34 suppresses duplication of storage of candidates stored in the incident word dictionary (that is, incident words detected in the past) among the candidates for the incident word in the incident word dictionary. The number of occurrences of the accompanying word is incremented.

The dictionary output unit 38 outputs at least the updated dictionary data to a predetermined external device when at least one of the subject word dictionary and the incidental word dictionary is updated by the dictionary update unit 34. In the embodiment, the latest updated dictionary data is transmitted to the natural language processing device 16, but as a modification, the latest subject word dictionary and incidental word dictionary may be output to another type of server or storage.

FIG. 8 is a block diagram showing a functional configuration of the natural language processing device 16 of FIG. The natural language processing device 16 includes a control unit 20, a storage unit 22, a control unit 40 corresponding to the communication unit 24, a storage unit 42, and a communication unit 44 of the dictionary creation device 14. The control unit 40 exchanges data with the patent document DB 12 and the dictionary creation device 14 via the communication unit 44.

The storage unit 42 includes a dictionary storage unit 46 and an analysis result storage unit 48. The dictionary storage unit 46 corresponds to the dictionary storage unit 26 of the dictionary creation device 14, and holds the main word dictionary and the incidental word dictionary created by the dictionary creation device 14. The analysis result storage unit 48 holds data indicating the result of natural language processing by the control unit 40. Specifically, it holds extracted word data for each patent document, including a list of words extracted from each of a plurality of patent documents.

The control unit 40 includes a document reading unit 50, a score setting unit 52, a word extraction unit 54, and an analysis result providing unit 56. The document reading unit 50 reads the character string described in the document from the document (patent document stored in the patent document DB 12 in the embodiment) to be processed in natural language. For example, a character string may be read in sentence units using a punctuation mark, a period, or the like as a delimiter, or a character string may be read in phrase units as in the phrase reading unit 28.

The score setting unit 52 identifies the subject word starting from each character in the character string read by the document reading unit 50 with reference to the subject word dictionary of the dictionary storage unit 46. The score setting unit 52 assigns a score according to the attribute of each subject word to each subject word starting from each character according to the subject word dictionary, the accompanying word dictionary, and a predetermined score assigning standard. The scoring criteria of the examples are aimed at extracting words that are natural to humans, and are a combination of the following three criteria. As a modification, any one or two criteria may be used.

(Criteria 1) Of the subject words starting with each character in the character string, the score of the subject word with a relatively small number of concomitant words is higher than the score of the subject word with a relatively large number of concomitant words. Enlarge. The score may be adjusted to increase as the number of accompanying words included is smaller. This is because the subject word containing many incidental words is likely to be a compound word by nature, and it is considered that it is not preferable to extract it as a word.

(Criteria 2) Among the subject words starting with each character in the character string, the score of the subject word having a relatively large number of characters is made higher than the score of the subject word having a relatively small number of characters. The subject word with a large number of characters may be adjusted so that the score becomes larger. This is because a subject word having a large number of characters is considered to be a characteristic word in a document and is considered to be an appropriate extraction target.

(Criteria 3) Of the subject words starting with each character in the character string, the score of the subject word associated with the subject word dictionary with a relatively large number of occurrences is higher than the score of the subject word with a relatively small number of occurrences. Enlarge. The score may be adjusted so that the subject word that appears more frequently has a higher score. This is because the subject words that appear frequently are those used in many documents and are considered to be appropriate as extraction targets.

The score setting unit 52 of the embodiment creates one candidate entry for each character in the character string read by the document reading unit 50. FIG. 9 shows an example of candidate entries. The position "e0" in the figure indicates a processing target position (in other words, a current processing position) in the read character string. “E0 + n” (n is a positive integer) in the figure indicates the distance (for example, the number of characters) from the processing target position. The score setting unit 52 identifies a subject word starting with each character (referred to as “word candidate” here), and sets the number of characters of each word candidate in the candidate entry. Further, the score setting unit 52 sets the score calculated for each word candidate in the candidate entry.

The method of calculating the score will be described. The score setting unit 52 reduces the score element (P) of the word candidate as the number of accompanying words included in each word candidate increases according to the above criterion 1. Further, the score setting unit 52 further reduces the score element (P) when the first character of the word candidate corresponds to the accompanying word. The formula for calculating the score element (P) is shown below.

“W” in Equation 1 is a character string of word candidates. The "L" in Equations 1 and 2 is a set of all concomitant words. “C” in Equation 1 is a function for acquiring the number of occurrences of the characters a and b as an incidental word from the incidental word dictionary. Therefore, "C (a)" and "C (b)" are the number of occurrences of the letters a and b as incidental words, respectively. “P0” in the equation 2 is a special penalty value when the first character (W0) of the word candidate is the accompanying word (L), and is set in the range of 0 <P0 <1. Β in Equation 1 is a constant. The fraction in Equation 1 is evaluated for each incidental word in the word candidate, and P becomes smaller as the number of incidental words included increases, and P becomes smaller if the first character is an incidental word. Further, the larger the number of occurrences (C (a)) associated with the incidental word in the word candidate, in other words, the smaller the P becomes as the frequently used incidental word is included. The score element (P) is a value that reflects a penalty based on the inclusion of ancillary words.

Next, the score setting unit 52 determines the score (Q) of the word candidate based on the word candidate P (W) obtained in Equation 1, the number of characters of the word candidate (r), and the number of appearances of the word candidate (t). To do. The formula for calculating the score (Q) is shown below.

In the function f of Equation 3, the three parameters, that is, the word candidate P (W), the number of word candidate characters (r), the number of appearances of the word candidate (t), and the score of the word candidate (Q) are positively correlated. It is a function that determines the score (Q) of a word candidate. According to Equation 3, the score setting unit 52 determines that the larger the P (W) of the word candidate, the larger the number of characters (r) of the word candidate, and the larger the number of appearances (t) of the word candidate, the more the word. The candidate score (Q) is calculated to be large. For example, the function f in Equation 3 may be a function that multiplies the word candidate P (W), the number of character candidates (r), and the number of appearances of the word candidate (t). That is, it may be "Q = P (W) × r × t".

The word extraction unit 54 refers to the score of each word candidate set by the score setting unit 52, and refers to the score given to the first word candidate starting from the processing target position in the character string and the first word candidate. The first word candidate is extracted as a word according to the magnitude relationship with the score given to the second word candidate starting from the second character in the following. The word extraction unit 54 of the embodiment also uses the score given to the third word candidate starting from the character immediately after the first word candidate as a comparison target of the magnitude relationship.

Specifically, the word extraction unit 54 identifies the score given to the word candidate (referred to as “current candidate” here) starting from the processing target position in the character string as the “current candidate score”. Further, the word extraction unit 54 identifies the score given to the word candidate (referred to as "next candidate" here) starting from the character immediately after the current candidate in the character string as the "next candidate score". Further, the score given to the word candidate (referred to as "intermediate candidate" here) starting from the second character or later in the current candidate is identified as the "intermediate candidate score" (ν).

Since the current candidate score and the next candidate score are calculated by the score setting unit 52, the word extraction unit 54 newly calculates the intermediate candidate score ν. The formula for calculating the intermediate candidate score ν is shown below.

E0 in the equation 4 is a candidate entry (that is, a current candidate) of the processing target position, and | e0 | is the number of characters of the current candidate. T is all candidate entries after the current candidate. S decomposes T into a plurality of words from (processing target position +1) to (processing target position + | e0 | -1) as the word start position (i), and adds up the scores of each word. The function S in Equation 4 can be said to be a function that provides an intermediate candidate score when the word at the processing target position (that is, e0) is not extracted.

When the current candidate score is 0, the word extraction unit 54 recognizes that the characters at the processing target position do not constitute the subject word, discards one character at the processing target position, and advances the processing target position by one. When the current candidate score is 0, the character at the processing target position may be recognized as a word (for example, an incidental word), and one character at the processing target position may be extracted as a word.

If the current candidate score is greater than 0, the word extraction unit 54 calculates ν according to Equation 4. When ν ≦ current candidate score or ν ≦ next candidate score is established, the word extraction unit 54 extracts a word starting from the processing target position (that is, the current candidate) and advances the processing target position by the number of characters of the current candidate. On the other hand, when ν> the current candidate score and ν> the next candidate score are satisfied, the word extraction unit 54 discards one character at the processing target position and advances the processing target position by one. This is because the score is higher when the word starting from the character in the middle of the candidate is extracted.

FIG. 10 shows an example of a character string to be processed for word extraction. The figure corresponds to FIG. 9, and the processing target position is indicated by e0. The word extraction unit 54 identifies the score “15” of the candidate entry of e0 in FIG. 9 as the current candidate score, and the score “11” of the candidate entry of e0 + 5 (because the number of characters of the word candidate of e0 is 5) in FIG. Is identified as the next candidate score. Then, in FIG. 10, the total value “18” of the score “8” of the candidate entry of e0 + 1 in FIG. 9 and the score “10” of the candidate entry of e0 + 3 in FIG. 9 is identified as the intermediate candidate score (ν). In the example of FIG. 10, since ν> the current candidate score and ν> the next candidate score are established, the word extraction unit 54 discards the character “new” at the processing target position and advances the processing target position by one.

Actually, the word extraction unit 54 includes the intermediate candidate starting from the third character of the current candidate, the total value of the scores of the words starting immediately after the intermediate candidate, and the intermediate candidate starting from the fourth character of the current candidate. The total value of the scores of the words starting immediately after the intermediate candidate is also calculated. Then, as shown in Equation 4, the maximum value among the plurality of types of total values is adopted as the value of ν.

The word extraction unit 54 associates a plurality of words extracted from a character string read from a certain patent document with the patent document, and records them in the extracted word data of the analysis result storage unit 48. The analysis result providing unit 56 is the extracted word data stored in the analysis result storage unit 48, that is, the data obtained by decomposing the character string of each patent document into words is predetermined periodically or as required. It is transmitted to an external device (for example, a patent document search device).

The operation of the information processing system 10 with the above configuration will be described.
FIG. 11 is a flowchart showing the operation of the dictionary creation device 14. The dictionary creation device 14 executes the dictionary creation process shown in FIG. 11 when it receives an instruction from the user and / or periodically at a predetermined timing. At the start of the dictionary creation process, both the main word dictionary and the auxiliary word dictionary of the dictionary storage unit 26 are empty. That is, in the initial state of the dictionary creation process, a dictionary in which the subject word and the incidental word are predetermined is unnecessary.

The phrase reading unit 28 of the dictionary creating device 14 acquires a plurality of patent documents from the patent document DB 12 and reads the first phrase from each patent document (S10). The phrase dividing unit 30 divides the character string indicated by the first phrase into a character string indicating the subject word stored in the subject word dictionary of the dictionary storage unit 26 and a character string excluding the subject word. A plurality of second phrases are extracted (S12). If the subject word is not detected in the first phrase, such as when the subject word dictionary is empty, the phrase division unit 30 outputs the first phrase as it is as the second phrase.

When the character strings starting from the beginning (that is, at least a part of the character strings of the second phrase) are common in the predetermined number or more of the second phrases (Y in S14), the subject word extraction unit 32 uses the first character strings. It is extracted as a candidate for the subject word (S16). The dictionary update unit 34 is a candidate that excludes compound words from the candidates for the subject word extracted by the subject word extraction unit 32, and is a candidate that indicates a new subject word that is not stored in the subject word dictionary. Store in the dictionary (S18). At that time, the dictionary update unit 34 also updates the number of occurrences associated with each subject word. If there is no common character string starting from the beginning in the second phrase of a predetermined number or more (N in S14), the processes of S16 and S18 are skipped.

The incidental word extraction unit 36 refers to the subject word dictionary, and when there is a character string of two or less characters (that is, a character string that does not constitute the subject word) between the two subject words in the first phrase (Y in S20). ), The character string is extracted as a candidate for an accompanying word, and the number of extractions of each candidate is added (S22). The contingent word extraction unit 36 determines that the candidate having a relatively high number of extractions among the candidates for the plurality of contingent words is regarded as the contingent word. The dictionary update unit 34 stores the candidate character string determined to be an incident word by the incident word extraction unit 36 in the incident word dictionary (S24). If there is no character string of 2 characters or less between the two subject words in the first phrase (N in S20), the processing of S22 and S24 is skipped.

If the patent document in which the character string reading is not completed remains (N in S26), the process returns to S10, and if the character string reading from all the patent documents is completed (Y in S26), the dictionary output unit 38 stores the dictionary. The subject word dictionary and the accompanying word dictionary stored in the unit 26 are transmitted to the natural language processing device 16 (S28). The accompanying word extraction process may be executed each time the subject word extraction process of each of the plurality of patent documents is completed, or may be executed after the subject word extraction process of all the patent documents is completed. Further, the dictionary output unit 38 may upload the subject word dictionary and the incidental word dictionary to a predetermined server, and the natural language processing device 16 may download the subject word dictionary and the incidental word dictionary from the server.

FIG. 12 is a flowchart showing the operation of the natural language processing device 16. The natural language processing device 16 executes the natural language processing shown in FIG. 12 when it receives an instruction from the user and / or periodically at a predetermined timing. The natural language processing in the embodiment is a process of extracting a word (particularly the subject word) from the character string described in the patent document.

When it is detected that the update timing of the search index data in the external device (referred to as the patent search device here) (not shown) has been reached (Y in S30), the document reading unit 50 of the natural language processing device 16 causes the patent document DB 12 A plurality of patent documents are obtained from the above, and the character strings of the respective patent documents are read (S32). The score setting unit 52 identifies the subject word starting from each character in the read character string. The score setting unit 52 derives the score of each subject word starting from each character according to a predetermined score giving standard, and creates a plurality of candidate entries corresponding to the plurality of characters (S34).

The word extraction unit 54 compares the scores of each candidate entry by magnitude, extracts words from the character string of the patent document based on the result, and updates the extracted word data of the analysis result storage unit 48 (S36). If the patent document in which the word extraction process is not completed remains (N in S38), the process returns to S32. When the word extraction process from all the patent documents is completed (Y in S38), the analysis result providing unit 56 transmits the extracted word data of the analysis result storage unit 48 to the patent search device (S40). The patent search device updates the index data for patent document search by using, for example, the extracted word data provided by the natural language processing device 16. If it is not the update timing of the search index data in the patent search device (N in S30), the subsequent processing is skipped and the flow of this figure ends.

According to the information processing system 10 of the embodiment, it is possible to automatically extract the subject word and the incidental word and automatically create the subject word dictionary and the incidental word dictionary even if there is no predetermined data for the subject word and the incidental word. it can. In addition, by automatically extracting the first character string that is common to multiple phrases as the subject word and automatically extracting the short character string between the subject words as the incidental word, a highly comprehensive subject word dictionary and incidental word dictionary can be obtained. Can be created. For example, an erroneously spelled character string is usually excluded from the dictionary, but in the information processing system 10 of the embodiment, it can be extracted as a subject word or an incidental word if it is actually described in a document, and is a highly comprehensive morpheme dictionary. Can be provided. In addition, when a new document is added (for example, when a new patent publication is issued), the subject word dictionary and the incidental word dictionary can be updated quickly and efficiently.

Further, according to the information processing system 10 of the embodiment, in the process of extracting a word from a character string of a document, a score is given to each subject word starting from each character of the character string. Then, according to the magnitude relationship between the score of the first subject word starting from the processing target position and the score of the second subject word starting from the second character onward in the first subject word, the first subject word Decide whether or not to extract. This makes it easier to cut out words from appropriate positions. Further, in the embodiment, the word can be extracted at a more appropriate position by including the score of the third subject word starting from the character immediately after the first subject word in the comparison target.

The present invention has been described above based on examples. This embodiment is an example, and it is understood by those skilled in the art that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. ..

The first modification will be described. In the first modification, it is permitted to register a compound word composed of a combination of a plurality of subject words in a dictionary. Specifically, the dictionary update unit 34 of the dictionary creation device 14 has a first character string (for example, "Japan") and a second character string including the first character string (for example, "Japanese cuisine"). When extracted as a candidate for the subject word, both the first character string and the second character string may be stored in the subject word dictionary. As a result, compound words are also recorded in the subject language dictionary.

In the character string to be processed by the natural language processing device 16, there may be a plurality of subject words (that is, current candidate e0) starting from a specific processing target position, such as "Japan" and "Japanese food". The score setting unit 52 of the natural language processing device 16 calculates a score for each of one or more subject words (for example, "Japan" and "Japanese food") starting from each character in the character string.

The formula for calculating the intermediate candidate score ν used by the word extraction unit 54 of the natural language processing device 16 is shown below.

D0 in Equation 5 is a set of one or more current candidates e0 existing at the beginning of T. The word extraction unit 54 executes the same calculation as in Equation 4 of the embodiment for each of the one or more current candidates to obtain the maximum value of the function S, and further, the maximum value of the function S across one or more current candidates. Let the value be the intermediate candidate score ν. For example, suppose that there are two current candidates (current candidate 1, current candidate 2) and two next candidates (next candidate 1, next candidate 2). When ν> current candidate 1 score, ν> next candidate 1 score, ν> current candidate 2 score, ν> next candidate 2 score are established, the word extraction unit 54 discards one character at the processing target position and processes the target. You may advance the position by one. If the above conditions are not satisfied and the current candidate 1 score ≥ the current candidate 2 score, the current candidate 1 is extracted as a word, and if the current candidate 1 score <current candidate 2 score, the current candidate 2 is a word. It may be extracted as.

According to the first modification, it is possible to correspond to the extraction of a word as a compound word. In the first modification, the subject word that is not a compound word is registered in the subject word dictionary as in the embodiment, and the completeness of the dictionary is not deteriorated.

A second modification will be described. The phrase reading unit 28 of the dictionary creation device 14 identifies the character code classification (for example, Latin characters, CJK characters, Arabic characters, Russian characters, etc.) corresponding to the character code value of the character string read from the patent document as the target language. You may. For example, when the character code of the patent document is Unicode, the punctuation reading unit 28 may identify the character code classification of the character string read from the patent document by referring to the Unicode code table. As one aspect of the second modification, the phrase reading unit 28 reads a continuous character string of the target language as a phrase, while ignoring characters (character code values) that do not correspond to the target language, meanings such as blanks. You may replace it with a character that you do not have and read it.

As another aspect of the second modification, the phrase reading unit 28 of the dictionary creation device 14 may allow a plurality of character code divisions. For example, the phrase reading unit 28 may read a character string spanning a plurality of character code divisions as the first phrase. In this case, the phrase reading unit 28 belongs to the character string belonging to the first division and the second division when the character code division changes from the first division to the second division in the middle of the character string of the patent document. The character string may be extracted as a different first phrase. Alternatively, the phrase dividing unit 30 divides the first phrase extracted by the phrase reading unit 28 based on the known subject word, and further, the character code division is changed from the first division in the middle of the first phrase. When changing to the second division, the character string belonging to the first division and the character string belonging to the second division may be extracted as different second phrases.

A third modification will be described. Although not mentioned in the above embodiment, the search history of the patent document may be fed back to the dictionary. Specifically, in the subject word dictionary of the dictionary creating device 14, an index value (hereinafter, referred to as “search frequency value”) indicating the frequency used for the search is recorded in association with each subject word. The dictionary creation device 14 periodically acquires data indicating a keyword or key phrase used in the patent search (hereinafter referred to as “search history”) from the patent search device (not shown), and searches the storage unit 22. It also has a search history acquisition unit that stores it in the history storage unit. The dictionary update unit 34 of the dictionary creation device 14 refers to the search history stored in the search history storage unit and the subject word dictionary to identify the subject words included in the search history, and searches for each subject word in the search history. Identify the number of times. The dictionary update unit 34 derives the search frequency value of each subject word (the number of searches itself may be used) based on the number of searches of each subject word, and stores it in the subject word dictionary.

In the subject word dictionary, the search frequency value may be stored together with the number of occurrences of each subject word in the patent document of the patent document DB 12, and the search frequency value may be stored instead of the number of occurrences. The score setting unit 52 of the natural language processing device 16 may use the search frequency value instead of the number of occurrences of the examples when determining the score of the word starting from each character in the read character string. The search frequency value may be used with the number of occurrences of the examples. For example, the score setting unit 52 may adjust the score of a word having a relatively high search frequency value to be higher than the score of a word having a relatively low search frequency value.

A fourth modification will be described. Although not mentioned in the above embodiment, the frequency of use of each word in the wording of the patent document may be fed back to the dictionary. Specifically, the dictionary creating device 14 acquires a wording history indicating the number of times or frequency of use of each word in the wording of the patent document from the natural language processing device 16, and the wording history storage unit of the storage unit 22. It also has a wording history acquisition unit to be stored in. The dictionary update unit 34 of the dictionary creation device 14 refers to the wording history stored in the wording history storage unit to specify the number of times or frequency of use of each subject word in the wording history. The dictionary update unit 34 stores the number of times or frequency of use of each subject word in the subject word dictionary. When determining the score of a word starting from each character in the read character string, the score setting unit 52 of the natural language processing device 16 replaces the number of occurrences at the time of creating the dictionary described in the embodiment with the word conversion. The number of uses or frequency of use may be used. For example, the score setting unit 52 may adjust the score of a word having a relatively high frequency of use to be higher than the score of a word having a relatively low frequency of use.

A fifth modification will be described. In the above embodiment, it is assumed that the subject word dictionary and the incidental word dictionary are empty in the initial state of dictionary creation, that is, the subject word and the incidental word are not registered in advance. As a modification, an existing dictionary (for example, a subject word dictionary and an incidental word dictionary manually created in advance) may be stored in the dictionary storage unit 26 of the dictionary creation device 14 in the initial state of dictionary creation. A manually created dictionary has high accuracy and accuracy of recorded words, but low comprehensiveness. The dictionary update unit 34 of the dictionary creation device 14 adds the subject word automatically extracted by the subject word extraction unit 32 to the existing subject word dictionary, and adds the incidental word automatically extracted by the incidental word extraction unit 36 to the existing incidental word. Add to the dictionary. As a result, a dictionary excellent in both accuracy and completeness can be created, and the accuracy and completeness of natural language processing in the natural language processing apparatus 16 can be further improved.

A sixth modification will be described. In the above embodiment, a technique for supporting morphological analysis was proposed, but here, other usage examples are given. The techniques described in the examples can be applied to (1) tag cloud. For example, it is a dictionary created by the dictionary creation device 14 based on a large number of documents, and a plurality of words are referred to in the document by referring to a dictionary in which words (subject words and / or incidental words) are associated with the number of occurrences. It is possible to generate an image expressed in a mode (shape, pattern, color, etc.) according to the number of appearances.

In addition, the techniques described in the examples can be applied to (2) creating a report on word variation. For example, a dictionary created by the dictionary creation device 14 based on a large number of documents in a specific field, with reference to a dictionary in which a word (subject word and / or ancillary word) is associated with the number of occurrences, and a word in a certain period of time. By graphing the fluctuation of the number of appearances of the above, it is possible to generate information indicating the trend or change in the specific field. In addition, the technique described in the examples can be applied to (3) creation of a Japanese dictionary. For example, the dictionary creation device 14 can automatically generate a subject language dictionary and an incidental word dictionary of each national language based on a document written in each national language.

A seventh modification will be described. In the above embodiment, the subject word dictionary and the incidental word dictionary are created based on the patent documents, but the subject word dictionary and the incidental word dictionary can be created based on the documents and document data of various types and various fields. Of course. When the natural language processing device 16 executes natural language processing on a document in a specific field, the dictionary creating device 14 extracts a subject word and an incidental word from the document in the specific field, thereby performing the document in the specific field. You may create a subject dictionary and an adjunct dictionary suitable for natural language processing.

Any combination of the embodiments and modifications described above is also useful as an embodiment of the present invention. The new embodiments resulting from the combination have the effects of the combined embodiments and variants. It is also understood by those skilled in the art that the functions to be fulfilled by each of the constituent elements described in the claims are realized by a single component or a combination thereof shown in the embodiments and modifications. ..

10 Information processing system, 14 Dictionary creation device, 16 Natural language processing device, 26 Dictionary storage unit, 32 Subject word extraction unit, 34 Dictionary update unit, 36 Accompanying word extraction unit, 46 Dictionary storage unit, 52 Score setting unit, 54 words Extraction unit.

Claims (10)

A dictionary storage unit that stores a dictionary that stores the main word, which is a dictionary used in natural language processing by a computer and whose meaning is established by itself.
A phrase extraction unit that extracts a plurality of phrases from the character string of the first document, using at least the remaining character string excluding the main word stored in the dictionary as a phrase.
When the same character string exists at the beginning of a predetermined number or more of the plurality of phrases extracted by the phrase extraction unit, the subject word extraction unit that extracts the same character string as the subject word and
A dictionary update unit that stores the subject words extracted by the subject word extraction unit in the dictionary, and
An information processing system characterized by being equipped with. When both the first character string and the second character string including the first character string are extracted as the subject words, the dictionary update unit stores the first character string in the dictionary. The information processing system according to claim 1, wherein the storage of the second character string in the dictionary is suppressed. With an additional word extraction section,
The dictionary storage unit stores a subject word dictionary, which is a dictionary in which the subject word is stored, and an incidental word dictionary, which is a dictionary in which incidental words, which are words used in association with the subject word, are stored.
The incidental word extraction unit extracts, as an incidental word, a character string having a predetermined number of characters or less from the remaining character strings excluding the subject word stored in the subject word dictionary from the character string of the first document.
The information processing system according to claim 1 or 2, wherein the dictionary update unit stores the incidental words extracted by the incidental word extraction unit in the dictionary. The incidental word extraction unit extracts a character string having a length of a predetermined value or less from the remaining character strings as a candidate for an incidental word, and determines that a character string whose number of extractions as a candidate is equal to or higher than a predetermined order is an accessory word. The information processing system according to claim 3, wherein the information processing system is characterized by the above. It also has a score setting section and a word extraction section.
The dictionary storage unit stores a subject word dictionary, which is a dictionary in which the subject word is stored, and an incidental word dictionary, which is a dictionary in which incidental words, which are words used in association with the subject word, are stored.
The score setting unit identifies a subject word starting with each character in the character string read from the second document to be processed in natural language by referring to the subject word dictionary, and among the subject words starting with each character. , A subject word with a relatively small number of concomitant words is given a larger score than a subject word with a relatively large number of concomitant words.
The word extraction unit starts from the score given to the first subject word starting from the processing target position in the character string read from the second document, and the second and subsequent characters in the first subject word. The information processing system according to claim 1, wherein the first subject word is extracted as a word according to a magnitude relationship with a score given to the second subject word. It also has a score setting section and a word extraction section.
The score setting unit identifies the subject word starting with each character in the character string read from the second document to be processed in natural language by referring to the dictionary, and the number of characters among the subject words starting with each character. Gives a higher score to a subject word with a relatively large number of characters than a subject word with a relatively small number of characters.
The word extraction unit starts from the score given to the first subject word starting from the processing target position in the character string read from the second document, and the second and subsequent characters in the first subject word. The information processing system according to claim 1, wherein the first subject word is extracted as a word according to a magnitude relationship with a score given to the second subject word. It also has a score setting section and a word extraction section.
The dictionary update unit records the number of occurrences of each subject word in the first document in the dictionary.
The score setting unit identifies a subject word starting with each character in a character string read from a second document to be processed in natural language by referring to the dictionary, and among the subject words starting with each character, the above-mentioned A subject word with a relatively large number of occurrences is given a higher score than the subject word with a relatively small number of appearances.
The word extraction unit starts from the score given to the first subject word starting from the processing target position in the character string read from the second document, and the second and subsequent characters in the first subject word. The information processing system according to claim 1, wherein the first subject word is extracted as a word according to a magnitude relationship with a score given to the second subject word. The word extraction unit starts from the score given to the first subject word starting from the processing target position in the character string read from the second document and the second and subsequent characters in the first subject word. The first subject word is a word according to the magnitude relationship between the score given to the second subject word and the score given to the third subject word starting from the character immediately after the first subject word. The information processing system according to any one of claims 5 to 7, wherein the information processing system is extracted as. An information processing device that stores a dictionary that stores a subject word, which is a dictionary used in natural language processing by a computer and whose meaning is established by itself.
A phrase extraction step of extracting a plurality of phrases from the character string of the first document, using at least the remaining character string excluding the subject word stored in the dictionary as a phrase.
When the same character string exists at the beginning of a predetermined number or more of the plurality of phrases extracted in the phrase extraction step, the subject word extraction step for extracting the same character string as the subject word and
A step of storing the subject word extracted in the subject word extraction step in the dictionary, and
An information processing method characterized by executing. An information processing device that stores a dictionary that stores a subject word that is a word whose meaning is established by itself, which is a dictionary used in natural language processing by a computer.
A phrase extraction function that extracts a plurality of phrases from the character string of the first document by using at least the remaining character string excluding the main word stored in the dictionary as a phrase.
When the same character string exists at the beginning of a predetermined number or more of the plurality of phrases extracted by the phrase extraction function, the subject word extraction function for extracting the same character string as the subject word and
A function to store the subject words extracted by the subject word extraction function in the dictionary, and
A computer program to realize.

Images