WO2011070980A1 - Dispositif de création de dictionnaires - Google Patents

Dispositif de création de dictionnaires Download PDF

Info

Publication number
WO2011070980A1
WO2011070980A1 PCT/JP2010/071696 JP2010071696W WO2011070980A1 WO 2011070980 A1 WO2011070980 A1 WO 2011070980A1 JP 2010071696 W JP2010071696 W JP 2010071696W WO 2011070980 A1 WO2011070980 A1 WO 2011070980A1
Authority
WO
WIPO (PCT)
Prior art keywords
word
input
words
output
cluster
Prior art date
Application number
PCT/JP2010/071696
Other languages
English (en)
Japanese (ja)
Inventor
弘紀 水口
大 久寿居
幸貴 楠村
Original Assignee
日本電気株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電気株式会社 filed Critical 日本電気株式会社
Priority to JP2011545194A priority Critical patent/JP5708495B2/ja
Priority to US13/515,135 priority patent/US20120303359A1/en
Publication of WO2011070980A1 publication Critical patent/WO2011070980A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/237—Lexical tools
    • G06F40/242—Dictionaries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G06F16/353—Clustering; Classification into predefined classes

Definitions

  • the present invention relates to a dictionary creation device, a word collection method, and a recording medium.
  • a dictionary creation technique is known in which a small number of similar words are input to create a dictionary that collects a large number of similar words from literature data, Web pages, and the like.
  • the dictionary is a set of the same kind of words having a common superordinate concept.
  • Non-Patent Document 1 An example of the dictionary creation method described above is described in Non-Patent Document 1. The outline of this dictionary creation method is shown below.
  • the first input word is referred to as a seed word.
  • Web pages including seed words are collected using a Web search engine.
  • a pattern for separating the seed word from other words is created from the collected Web pages.
  • a word is extracted from the Web page and added to the seed word. Note that the process from inputting a seed word until the word is extracted is called a turn.
  • Web pages are further collected using the seed word to which the word is added. After repeating this several turns, the extracted word is output as a set (dictionary) of words of the same type as the seed word.
  • a word newly added to the seed word may be a different type of word from the seed word.
  • words such as ramen shop names and udon shop names that are published in the same literature and have similar patterns are newly seeded. For example, it is added to a word.
  • different types of words are added to the seed word one after another from the different types of words, and many types of words different from the seed words are collected, which may deteriorate the accuracy of the dictionary.
  • the reliability of the word extracted in each turn is obtained, and only words having a certain reliability or higher are added to the seed word and adopted in the next turn. Yes.
  • the reliability for example, a statistic based on the number of appearances of the pattern, a statistic based on the number of words detected from the pattern, or the like is used.
  • the number of Web pages that can be extracted based on word patterns is adopted as the reliability, and the number of Web pages that can be extracted is less than a predetermined number, and thus differs by not adding to the seed word. Prevents the collection of different types of words.
  • the present invention has been made in view of the above circumstances, and a dictionary creation device, a word collection method, and a recording medium capable of suitably outputting to a user what kinds of different words are collected
  • the purpose is to provide.
  • a dictionary creation device provides: Accepts an input of a word, outputs a word related to the input word input from the document data, and thereafter adds the output word to the input word until a predetermined condition is reached, and adds the word related to the input word to the document
  • An input / output process recording means for recording information indicating an input / output process between an input word and an output word output by the input word in a dictionary multiplication process of collecting words by repeating output from data;
  • Cluster classification means for classifying words collected in the dictionary multiplication process into clusters based on information recorded in the input / output process recording means; For each cluster classified by the cluster classification means based on the information recorded in the input / output process recording means, whether or not the words in the cluster are the same type of words as the input word that received the input first Homogenous discrimination means for discriminating Associating the words collected in the dictionary multiplication process, the cluster to which the word belongs, and information indicating whether or not the word constituting the cluster
  • the word collection method is: Accepts an input of a word, outputs a word related to the input word input from the document data, and thereafter adds the output word to the input word until a predetermined condition is reached, and adds the word related to the input word to the document
  • An input / output process recording step for recording information indicating an input / output process between an input word and an output word output by the input word in the dictionary multiplication process in which words are collected by repeating output from data;
  • a cluster classification step of classifying the words collected in the dictionary multiplication process into clusters, For each cluster classified by the cluster classification step based on the information recorded in the input / output process recording step, whether or not the word in the cluster is the same type of word as the first input word received Homogeneous determination step for determining Associating the words collected in the dictionary multiplication process, the cluster to which the word belongs, and information indicating whether or not the word constituting the cluster is the same type of word as
  • the recording medium is Computer Accepts an input of a word, outputs a word related to the input word input from the document data, and thereafter adds the output word to the input word until a predetermined condition is reached, and adds the word related to the input word to the document
  • An input / output process recording means for recording information indicating an input / output process between an input word and an output word output by the input word in a dictionary multiplication process of collecting words by repeating output from data;
  • Cluster classification means for classifying the words collected in the dictionary multiplication process into clusters based on information recorded in the input / output process recording means; For each cluster classified by the cluster classification means based on the information recorded in the input / output process recording means, whether or not the words in the cluster are the same type of words as the input word that received the input first Homogeneous discrimination means for discriminating Associating the words collected in the dictionary multiplication process, the cluster to which the word belongs, and information indicating whether or not the word constituting the cluster is the same type of word as the
  • the words collected in the dictionary construction are clustered, and it is determined for each cluster whether or not the words are of the same type as the first input word. Therefore, it is possible to suitably output to the user what kinds of different words are collected.
  • 10A and 10B are diagrams illustrating a configuration example of information stored in the word group storage unit. It is a flowchart for demonstrating operation
  • the dictionary creating apparatus 100 includes an input unit 101, a dictionary multiplication unit 102, a clustering unit 103, a type determination unit 104, an output unit 105, a document storage unit 106, and a collection process storage unit. 107 and a collected word storage unit 108.
  • the input unit 101 includes a keyboard and a mouse.
  • the user inputs a word (seed word) as a sample for creating a dictionary (a set of similar words) via the input unit 101.
  • the dictionary multiplication unit 102 uses a conventional method as described in Non-Patent Document 1 to perform dictionary multiplication processing for collecting words of the same type as the seed word from the document stored in the document storage unit 106. . Further, the dictionary multiplication unit 102 records in the collection process storage unit 107 information indicating what process the word was collected in this dictionary multiplication process. Details of the dictionary multiplication process performed by the dictionary multiplication unit 102 will be described later.
  • the clustering unit 103 classifies (clusters) the words collected by the dictionary multiplying unit 102 into a plurality of clusters based on information stored in the collection process storage unit 107. Details of the processing performed by the clustering unit 103 will be described later.
  • the type discriminating unit 104 inputs the cluster and the words included in the cluster, refers to the information stored in the collection process storage unit 107, and the words constituting the cluster are the same type of words as the seed words It is determined whether or not. Details of the processing performed by the type determination unit 104 will be described later.
  • the output unit 105 outputs various information. For example, the output unit 105 outputs (displays) the words collected by the dictionary multiplication process with information indicating whether the words are heterogeneous or the same as the seed word for each classified cluster.
  • the document storage unit 106 stores data defining each document that is a target of word collection by the dictionary multiplication unit 102. Each document data is given an ID (document ID).
  • the collection process storage unit 107 stores information indicating what input / output process the word was collected in the dictionary multiplication process. Specifically, as shown in FIG. 2, the collection process storage unit 107 generates the number of turns of the turn, the input word input in the turn, and the input word for each turn in the dictionary multiplication process. The output words output according to the pattern thus recorded are recorded in association with each other. For example, it can be seen from the top entry of FIG. 2 that “Restaurant X” is extracted by the pattern created from “Restaurant S” in the first turn of the dictionary multiplication process.
  • each collected word is stored in association with a cluster ID indicating which cluster each word is classified into. . Also, in each cluster, whether the words constituting the cluster are the same type of word as the seed word (the seed word itself is also the same type when included in the cluster), or is a different type of word Information indicating whether or not. For example, it can be seen from FIG. 3 that “Restaurant A” and “Restaurant B” are classified into cluster 1, and that cluster 1 is composed of words of the same type as seed words. Similarly, “Udon C” and “Udon D” are classified into Cluster 2, and it can be seen that Cluster 2 is composed of different types of words from the seed words.
  • the dictionary creation device 100 Next, an operation of processing performed by the dictionary creation device 100 will be described.
  • the user operates the input unit 101 to input one or more words (seed words) that serve as samples for creating a dictionary (a set of similar words). Then, it instructs to create a dictionary based on the input seed word.
  • the dictionary creating apparatus 100 performs a dictionary creating process shown in FIG.
  • the dictionary breeding unit 102 When the dictionary creation process is started, first, the dictionary breeding unit 102 performs a dictionary breeding process using a conventional method, and collects words related to the input seed word (step S100).
  • step S100 Details of the dictionary multiplication process (step S100) will be described with reference to the flowchart of FIG.
  • the dictionary multiplication unit 102 registers the seed word input by the user in the collected word storage unit 108 (step S101). Then, the dictionary multiplication unit 102 increments a counter i (initial value 0) indicating the number of turns by 1 (step S102).
  • the dictionary multiplication unit 102 randomly selects a predetermined number of words from the words stored in the collected word storage unit 108 (step S103). Then, the dictionary multiplication unit 102 detects a document containing the selected seed word from the documents stored in the document storage unit 106 (step S104). Here, only a document including all the selected seed words may be detected, or a document including a predetermined number of seed words among the selected seed words may be detected.
  • the dictionary multiplication unit 102 identifies the position where the seed word selected in step S103 appears in the detected document, and creates a pattern that separates the seed word from other parts (step S105). For example, a predetermined number of character strings before and after a portion where a seed word appears in the document may be adopted as a pattern.
  • the dictionary multiplication unit 102 extracts words that match the created pattern from the document stored in the document storage unit 106 (step S106). Then, the dictionary multiplication unit 102 adds the extracted word to the collected word storage unit 108 (step S107).
  • the dictionary multiplication unit 102 extracts in step S106 using information indicating the number of turns this time (that is, the value of the counter i), each word (input word) selected in step S103, and a pattern created from the input word.
  • the collected words (output words) are associated with each other and stored in the collection process storage unit 107 (step S108).
  • the dictionary multiplication unit 102 determines whether or not a predetermined termination condition for terminating the dictionary multiplication is satisfied (step S109).
  • a termination condition for example, any condition such as whether the number of words stored in the collected word storage unit 108 has reached a predetermined number or the number of turns has reached a predetermined number can be employed. .
  • an end condition that repeatedly collects words for at least two turns.
  • step S109 If it is determined that the end condition is not satisfied (step S109; No), the dictionary multiplication unit 102 repeats steps S102 to S108, and continues to collect words from the seed word to which a new word has been added. If it is determined that the end condition is satisfied (step S109; Yes), the dictionary multiplying unit 102 ends the dictionary multiplying process and moves the process to the clustering unit 103.
  • the clustering unit 103 performs a clustering process for classifying the words collected by the dictionary multiplication process into clusters (step S200).
  • FIG. 6 is a flowchart showing details of the clustering process (step S200).
  • the clustering unit 103 first selects two words for which the degree of cohesion between words has not yet been calculated from the collected word storage unit 108 (step S201).
  • the clustering unit 103 calculates the degree of cohesion between the two selected words based on the information stored in the collection process storage unit 107 (step S202).
  • the degree of cohesion between words is an index whose value increases as words that input common words or words that output common words in the dictionary multiplication process described above. For example, the ratio of the words that are input to two words from the common word among the words that are input to each of the two words, and the word that outputs two words that are common to the two words that are output from each of the two words Can be calculated as the degree of cohesion between two words.
  • the cohesion degree between two words a and b is Sim (a, b)
  • Sim_in (a, b) is a value indicating the ratio of words input from a common word among the words input to the words a and b.
  • Sim_in (a, b) (number of common words input to both words a and b) / ((number of words input to word a) + (number of words input to word b)) ).
  • Sim_out (a, b) is a value indicating the ratio of words that output a common word among the words output by the two words a and b.
  • Sim_out (a, b) (number of common words from both words a and b) / ((number of words output by word a) + (number of words output by word b)) Can be sought.
  • the clustering unit 103 determines whether or not the cohesion degree has been calculated for all pairs of seed words stored in the collected word storage unit 108 (step S203).
  • step S203 When the cohesion degree is not calculated for all pairs of seed words (step S203; No), the clustering unit 103 selects two seed words for which the cohesion degree has not been calculated and calculates the cohesion degree (step S201, Step S202) is repeated.
  • the clustering unit 103 uses the calculated cohesion degree as a similarity, and publicly known methods such as the shortest distance method, the longest distance method, and the group average method Clustering is performed using the clustering method, and the seed words stored in the collected word storage unit 108 are classified into a plurality of clusters (step S204). Then, the clustering unit 103 records the clustered result (step S205). Specifically, the clustering unit 103 assigns a cluster ID to the words stored in the collected word storage unit 108 so that the result of classification into clusters is reflected. This completes the clustering process.
  • the degree of cohesion between the collected words is calculated by the clustering process, and the collected words are classified into a plurality of clusters based on the calculated degree of cohesion.
  • FIG. 7 is a graph showing the input / output relationship between words in turn 1 to turn 3 of the dictionary multiplication process when the information shown in FIG. 2 is stored in the collection process storage unit 107.
  • each word is represented by a node and connected by an arc (arrow) from the input word to the output word.
  • arc arrow
  • clustering using a known clustering method is performed with the degree of cohesion between these words as the similarity. For example, two clusters of cluster 1 ⁇ restaurant A, restaurant B ⁇ and cluster 2 ⁇ udon C, udon D ⁇ are formed from this degree of cohesion and stored in the collected word storage unit 108 as shown in FIG. A cluster ID is assigned to each existing word.
  • the type determination unit 104 performs the same type determination process for determining whether or not the cluster classified by the clustering process is composed of words of the same type as the first input word (seed word). Perform (step S300).
  • FIG. 8 is a flowchart showing details of the homogeneity discrimination processing (step S300).
  • the type discriminating unit 104 selects one cluster that has not been subjected to homogenous discrimination from the collected word storage unit 108 and a word included in the cluster (step S301). .
  • the type determination unit 104 refers to the collection process storage unit 107 to determine whether or not the word in the selected cluster is the same type of word as the first input word (seed word) ( Step S302). This determination may be made based on the proximity of each word in the cluster to the seed word. Specifically, the type determination unit 104 calculates the number of turns required to output each word in the cluster from the seed word and the number of turns required for each word in the cluster to output the seed word. Based on the calculated number of turns, it may be determined whether the type is the same or different.
  • the type determination unit 104 records the determination result in the collected word storage unit 108 (step S303).
  • the type discriminating unit 104 discriminates whether or not the above-described homogenous discrimination has been performed on all the clusters stored in the collected word storage unit 108 (step S304).
  • step S304 If there is a cluster that has not been subjected to homogenous discrimination (step S304; No), the type discriminating unit 104 repeats the process of selecting the cluster and performing homogenous discrimination (steps S301 to S303).
  • step S304 If there is no cluster that has not been subjected to the same type determination (step S304; Yes), the same type determination process ends.
  • the word “Restaurant A” in the cluster 1 is output from the seed word “Restaurant S” in the shortest turn by the route “Restaurant S ⁇ Restaurant A”.
  • “Restaurant A” outputs the seed word “Restaurant T” in the shortest turn through the route “Restaurant A ⁇ Restaurant T”. Therefore, the reciprocal number 1 of the shortest number of turns 1 is set as a value representing the proximity of the “restaurant A” to the seed word.
  • the word “Restaurant B” in the cluster 1 is output from the seed word “Restaurant S” in the shortest turn by the route “Restaurant S ⁇ Restaurant B”.
  • “Restaurant B” outputs the seed word “Restaurant T” in the shortest turn by the route “Restaurant B ⁇ Restaurant T”. Therefore, the reciprocal number 1 of the shortest number of turns 1 is set as a value representing the proximity to the seed word of “Restaurant B”. Therefore, the closeness to the seed word in the entire cluster 1 is 1 taking the average of the closeness of “Restaurant A” and “Restaurant B”. Since this value is equal to or greater than the threshold value 0.6, the cluster 1 is determined to be the same type, and the result is stored in the collected word storage unit 108.
  • the word “Udon C” in cluster 2 is the seed word “Restaurant S” or “Restaurant” in the shortest two turns by a route such as “Restaurant S ⁇ Restaurant Z ⁇ Udon C” or “Restaurant T ⁇ Restaurant W ⁇ Udon C”. "T”. Therefore, the reciprocal number 0.5 of the shortest number of turns 2 is set as a value representing the proximity to the seed word of “Udon C”.
  • the word “Udon D” in the cluster 2 is a seed word “Restaurant S” in the shortest two turns by a route such as “Restaurant S ⁇ Restaurant Z ⁇ Udon D” or “Restaurant T ⁇ Restaurant W ⁇ Udon D”. Alternatively, it is output from “Restaurant T”. Therefore, the reciprocal number 0.5 of the shortest number of turns 2 is set as a value representing the proximity to the seed word of “Udon D”. Therefore, the proximity to the seed word in the entire cluster 2 is 0.5, which is an average of the proximity of the udon C and the udon D. Since this value is equal to or less than the threshold value 0.6, the cluster 2 is determined to be different and the result is stored in the collected word storage unit 108.
  • the output unit 105 refers to the collected word storage unit 108, associates the information with the collected words classified into clusters and discriminated as being the same or different from the seed word.
  • To output (display) step S400).
  • the output unit 105 outputs “Cluster 1 ⁇ Restaurant A, Restaurant B ⁇ : Same kind, Cluster 2 ⁇ Udon C, Udon D ⁇ : Different kind”, and the like. This completes the dictionary creation process.
  • each word collected by the dictionary multiplication process is classified into a cluster. Then, for each cluster, whether or not it is composed of the same type of word as the seed word is determined and output. Accordingly, it is possible to suitably output to the user what kinds of different words are collected.
  • a dictionary creation device 200 As shown in FIG. 9, a dictionary creation device 200 according to the second embodiment includes a word selection unit 201, a re-execution unit 202, and a word group storage unit 203 added to the dictionary creation device 100 according to the first embodiment. It is a configuration.
  • symbol is attached
  • the collected words are stored in association with group names that are identification information of the groups to which the words belong. .
  • the word selection unit 201 refers to the word group storage unit 203, selects one uncollected group, and selects a predetermined number of words from the selected group. Then, the word selection unit 201 instructs the dictionary multiplication unit 102 to execute a dictionary multiplication process using the selected word as a seed word.
  • the re-execution unit 202 adds the group name to the words collected, classified into clusters, and determined to be the same type or different from the seed words, and adds them to the word group storage unit 203. Then, when there is a group that has not yet been collected, the re-execution unit 202 instructs the word selection unit 201 to select a word from the group.
  • the other units are the first implementation. Since processing similar to that of the embodiment is performed, description thereof is omitted here. However, the seed word that is used as the starting point of word collection by the dictionary multiplication unit 102 is a word selected by the word selection unit 201.
  • a plurality of words are registered as a group 1 in the word group storage unit 203 in advance. Further, it is assumed that this group 1 is a collection incomplete group described later. It is assumed that no group other than group 1 is registered at this time.
  • the dictionary creating apparatus 200 performs a dictionary creating process shown in FIG.
  • the word selection unit 201 refers to the word group storage unit 203 and selects a predetermined number of words from among the words included in the uncollected group (that is, group 1). Is selected as a seed word (step S50).
  • the dictionary multiplication unit 102 performs a dictionary multiplication process in the same manner as in the first embodiment, and collects the same type of words as the seed words (step S100). However, here, the word selected in step S50 is used as a seed word.
  • the clustering unit 103 performs clustering processing as in the first embodiment, and classifies the words collected by the dictionary multiplication processing into clusters (step S200).
  • the type determination unit 104 performs the same type determination process as in the first embodiment, and determines whether or not the cluster includes words of the same type as the seed word (step S300).
  • the re-execution unit 202 performs word group update processing for registering the words constituting the cluster in the word group storage unit 203 for each cluster for which it is determined whether the seed word is the same or different from the seed word (grouping). Step S330).
  • Fig. 12 shows the details of the word group update process.
  • the re-execution unit 202 selects one unprocessed cluster from the clusters clustered in step S200 described above (step S331).
  • the re-execution unit 202 refers to the result of the same type determination process in step S300, and determines whether or not the selected cluster is composed of words of the same type as the seed word (step S332).
  • step S332 If it is the same type as the seed word (step S332; Yes), the re-execution unit 202 assigns the same group name as the seed word and registers the word in the selected cluster in the word group storage unit 203 (step S333). Then, the process proceeds to step S337.
  • the re-execution unit 202 refers to the word group storage unit 203 and is already stored in the word group storage unit 203 among the words in the selected cluster. It is determined whether or not there is a word (existing word) (step S334).
  • step S334 When it is determined that there is an existing word (step S334; Yes), the re-execution unit 202 attaches the same group name as the group name attached to the existing word, and converts the words in the selected cluster to the word group. Register in the storage unit 203 (step S335). Then, the process proceeds to step S337.
  • step S334 When it is determined that there is no existing word (step S334; No), the re-execution unit 202 assigns the newly issued group name and registers the word in the selected cluster in the word group storage unit 203 (step). S336). Then, the process proceeds to step S337.
  • step S337 the re-execution unit 202 determines whether or not the processing for registering the words in the cluster in the word group storage unit 203 has been performed for all the clustered clusters.
  • step S337 If there is a cluster that has not yet been registered in the word group storage unit 203 (step S337; No), the re-execution unit 202 selects an unprocessed cluster, and selects a word in the cluster as the word group storage unit 203. A series of processes (step S331 to step S336) registered in the above are repeated.
  • step S337 When the process of registering words in the word group storage unit 203 is performed in all clusters (step S337; Yes), the word group update process ends.
  • the re-execution unit 202 determines whether or not there is a group for which word collection has not yet been completed (hereinafter referred to as an incomplete collection group) (step S360). For example, a group that satisfies any of the following conditions a) to d) may be determined as a collection incomplete group.
  • step S360 When there is an incomplete collection group (step S360; Yes), the re-execution unit 202 instructs the word selection unit 201 to select a seed word from one of the collection incomplete groups. Then, the words are collected from the seed words, clustered, determined whether the seed words are the same or different, and the grouping process is repeated (steps S50 to S330).
  • step S360 If there is no collection incomplete group (step S360; No), the output unit 105 outputs the collected words. However, a group name to which the word belongs is acquired from the word group storage unit 203 in addition to the cluster to which the word belongs and information indicating whether the cluster is the same type of seed word. These pieces of information are output (displayed) in association with the collected words. This completes the dictionary creation process.
  • the dictionary creation process is started in this state, first, the words “Restaurant S” and “Restaurant T” in the group 1 are selected (step S50). Subsequently, a dictionary multiplication process is executed using the “restaurant S” and “restaurant T” as seed words, and words are collected (step S100). The collected words are clustered based on the degree of cohesion (step S200), and for each cluster, it is determined whether or not the seed words “restaurant S” and “restaurant T” are the same type (step S300). . Here, it is assumed that the following clusters 1 to 5 are created.
  • Cluster 1 (same type): “Restaurant A” “Restaurant B” ⁇ Cluster 2 (different): “Udon C” “Udon D”
  • Cluster 3 (same type): “Restaurant X” “Restaurant Z” “Restaurant W”
  • Cluster 4 (same type): “Restaurant S” “Restaurant T”
  • Cluster 5 (different type): “Udon G” “Udon H”
  • a word group update process is performed in which words in the cluster are grouped and registered in the word group storage unit 203 (step S330).
  • the words in these clusters are registered in the word group storage unit 203 as the same group 1 word as the seed word. Is performed (step S333).
  • the cluster 2 and the cluster 5 are different words from the seed word, and the words in these clusters are not yet stored in the word group storage unit 203. Therefore, the words in the cluster 2 and the cluster 5 are registered in the word group storage unit 203 with the new group names of the group 2 and the group 3, respectively (step S336).
  • the words in the clusters 1 to 5 are registered in the word group storage unit 203 with a group name.
  • one of the groups (that is, group 2 or group 3) is selected, and word collection using the words in the selected group as a new seed word is performed. A series of processes to be performed is repeated.
  • the word selection unit 201 of the dictionary creation device 200 of the second embodiment is replaced with a second word selection unit 301, as shown in FIG.
  • an interword cohesion degree storage unit 302 is newly added.
  • symbol is attached
  • the detailed description of the same components as those of the first embodiment and the second embodiment is the same as that of the first embodiment and the second embodiment, and the detailed description thereof is omitted.
  • the second word selection unit 301 refers to the word group storage unit 203, selects one uncollected group, and selects a plurality of words from the words included in the selected group. At this time, the second word selection unit 301 refers to the inter-word cohesion degree storage unit 302 and preferentially selects words satisfying a predetermined degree of cohesion.
  • the predetermined condition is, for example, a condition such that “75% of the words in the group are selected in descending order of cohesion, and the remaining 25% are selected in descending order of cohesion”. Selecting only words with a high degree of cohesion collects only frequently occurring words, so the accuracy of collecting similar words to seed words increases, but the number of collected words decreases and the collection efficiency decreases. Getting worse. Therefore, when it is desired to perform word collection that emphasizes collection efficiency over collection accuracy, it is desirable to employ the above conditions. In addition, when it is desired to perform word collection that places importance on collection accuracy over collection efficiency, it is desirable to adopt conditions such as “select words in a group in descending order of cohesion”. It is assumed that condition information defining such word selection conditions is stored in advance in the storage unit of the dictionary creation device 300.
  • the inter-word cohesion degree storage unit 302 stores the inter-word cohesion degree calculated by the clustering unit 103. Specifically, as shown in FIG. 14, the inter-word cohesion degree storage unit 302 stores two words and the cohesion degree between the two words in association with each other. For example, from the top entry in FIG. 14, the cohesion degree between “Restaurant S” and “Restaurant T” is 0.9.
  • the user operates the input unit 101 to instruct to create a dictionary.
  • the dictionary creation device 300 performs the dictionary creation process shown in FIG. 11 as in the second embodiment.
  • the second word selection unit 301 refers to the word group storage unit 203 to select one uncollected group, refers to the inter-word cohesion degree storage unit 302, and selects a group based on a predetermined condition. A predetermined number (four) of the words in the group are selected as seed words (step S50).
  • the second word selection unit 301 first selects two words having the highest degree of cohesion between words among the words in the group. Next, the second word selection unit 301 selects one word having the highest degree of cohesion with each of the two words. Then, the second word selection unit 301 selects each of these three words and one word having a low degree of cohesion.
  • the dictionary multiplying unit 102 performs a dictionary multiplying process for collecting the same kind of words using the four words selected by the second word selecting unit 301 as seed words (step S100).
  • the clustering unit 103 clusters the collected words (step S200).
  • the clustering unit 103 records the words calculated for clustering and the cohesion degree between the words in the inter-word cohesion degree storage unit 302.
  • the type determining unit 104 determines, for each cluster, whether or not the cluster is composed of words of the same type as the seed word (step S300).
  • the re-execution unit 202 groups the collected words (step S330). If there is an uncollected group (step S360; Yes), the process of selecting a seed word from the uncollected group and collecting the words is repeated. If there is no uncollected group (step S360; No) The process ends.
  • the words in the group are not selected at random, but are selected in consideration of the degree of cohesion between the words. Therefore, it is possible to collect words corresponding to various scenes.
  • a word is extracted from a document stored in the document storage unit 106.
  • the present invention is not limited to this.
  • a word is extracted from a Web page on the Internet using an Internet search engine. May be.
  • FIG. 15 is a block diagram showing an example of a physical configuration when the dictionary creation devices 100, 200, and 300 according to the embodiments of the present invention are mounted on a computer.
  • the dictionary creation devices 100, 200, and 300 according to the embodiments of the present invention can be realized by a hardware configuration similar to a general computer device.
  • the dictionary creation devices 100, 200, and 300 include a control unit 21, a main storage unit 22, an external storage unit 23, an operation unit 24, a display unit 25, and an input / output unit 26.
  • the main storage unit 22, the external storage unit 23, the operation unit 24, the display unit 25, and the input / output unit 26 are all connected to the control unit 21 via the internal bus 20.
  • the control unit 21 includes a CPU (Central Processing Unit) and the like, and executes the dictionary creation process in each of the above-described embodiments according to the control program 30 stored in the external storage unit 23.
  • CPU Central Processing Unit
  • the main storage unit 22 includes a RAM (Random-Access Memory) or the like, loads a control program 30 stored in the external storage unit 23, and is used as a work area of the control unit 21.
  • RAM Random-Access Memory
  • the external storage unit 23 includes a non-volatile memory such as a flash memory, a hard disk, a DVD-RAM (Digital Versatile Disc Random-Access Memory), and a DVD-RW (Digital Versatile Disc Disc Rewritable).
  • a control program 30 to be executed is stored in advance. Further, the external storage unit 23 supplies the data stored in the control program 30 to the control unit 21 according to the instruction of the control unit 21 and stores the data supplied from the control unit 21. Further, the external storage unit 23 physically stores the document storage unit 106, the collection process storage unit 107, the collection word storage unit 108, the word group storage unit 203, and the inter-word cohesion degree storage unit 302 in each of the above-described embodiments. Realize.
  • the operation unit 24 includes a pointing device such as a keyboard and a mouse, and an interface device that connects the keyboard and the pointing device to the internal bus 20.
  • a seed word and an instruction to start dictionary creation processing are supplied to the control unit 21 via the operation unit 24.
  • the display unit 25 includes a CRT (Cathode Ray Tube) or an LCD (Liquid Crystal Display), and displays various information. For example, the display unit 25 displays each collected word with information on whether it is the same or different from the seed word for each cluster.
  • CTR Cathode Ray Tube
  • LCD Liquid Crystal Display
  • the input / output unit 26 is composed of a wireless transceiver, a wireless modem or a network termination device, and a serial interface or a LAN (Local Area Network) interface connected thereto. For example, words may be collected from web pages on the Internet via the input / output unit 26.
  • the processing of the second word selection unit 301 is performed by the control program 30 using the control unit 21, the main storage unit 22, the external storage unit 23, the operation unit 24, the display unit 25, the input / output unit 26, and the like as resources. Run by.
  • the central part that performs processing of the dictionary creation devices 100, 200, and 300 including the control unit 21, the main storage unit 22, the external storage unit 23, the operation unit 24, the input / output unit 26, the internal bus 20, and the like is as follows. It can be realized using a normal computer system regardless of a dedicated system. For example, a computer program for executing the above operation is stored and distributed on a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.), and the computer program is installed in the computer.
  • the dictionary creation devices 100, 200, and 300 that perform the above-described processing may be configured.
  • the dictionary creation devices 100, 200, and 300 may be configured by storing the computer program in a storage device included in a server device on a communication network such as the Internet and downloading it by a normal computer system.
  • the functions of the dictionary creation devices 100, 200, and 300 are realized by sharing an OS (operating system) and an application program, or by cooperation between the OS and the application program, only the application program portion is stored in a recording medium or the like. You may store in a memory
  • the computer program may be posted on a bulletin board (BBS, Bulletin Board System) on a communication network, and the computer program may be distributed via the network.
  • BSS bulletin Board System
  • the computer program may be started and executed in the same manner as other application programs under the control of the OS, so that the above-described processing may be executed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Machine Translation (AREA)

Abstract

L'invention concerne un dispositif caractérisé en ce que, lors de la collecte de mots par un traitement d'expansion de dictionnaire, une unité (102) d'expansion de dictionnaire mémorise des informations indiquant par quel type de processus d'entrée et de sortie un mot a été collecté dans une unité (107) de stockage du processus de collecte. Une unité (103) de regroupement classifie le mot qui a été collecté par le traitement d'expansion du dictionnaire dans des groupes sur la base d'informations enregistrées dans l'unité (107) de stockage du processus de collecte. Ensuite, une unité de détermination du type (104) détermine si un mot constitutif d'un groupe est du même type qu'un mot d'amorçage ou d'un type différent, pour chaque groupe dans lequel le mot a été classifié sur la base des informations enregistrées dans l'unité (107) de stockage du processus de collecte. De plus, une unité (105) de sortie associe des informations indiquant le mot collecté, le groupe auquel appartient le mot et si le groupe est du même type que le mot d'amorçage ou d'un type différent, et en effectue l'affichage.
PCT/JP2010/071696 2009-12-11 2010-12-03 Dispositif de création de dictionnaires WO2011070980A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2011545194A JP5708495B2 (ja) 2009-12-11 2010-12-03 辞書作成装置、単語収集方法、及び、プログラム
US13/515,135 US20120303359A1 (en) 2009-12-11 2010-12-03 Dictionary creation device, word gathering method and recording medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2009282304 2009-12-11
JP2009-282304 2009-12-11

Publications (1)

Publication Number Publication Date
WO2011070980A1 true WO2011070980A1 (fr) 2011-06-16

Family

ID=44145525

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2010/071696 WO2011070980A1 (fr) 2009-12-11 2010-12-03 Dispositif de création de dictionnaires

Country Status (3)

Country Link
US (1) US20120303359A1 (fr)
JP (1) JP5708495B2 (fr)
WO (1) WO2011070980A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7384354B2 (ja) 2020-02-04 2023-11-21 本田技研工業株式会社 情報処理装置、情報処理方法およびプログラム

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9304703B1 (en) * 2015-04-15 2016-04-05 Symbolic Io Corporation Method and apparatus for dense hyper IO digital retention
US9817728B2 (en) 2013-02-01 2017-11-14 Symbolic Io Corporation Fast system state cloning
US10133636B2 (en) 2013-03-12 2018-11-20 Formulus Black Corporation Data storage and retrieval mediation system and methods for using same
US9628108B2 (en) 2013-02-01 2017-04-18 Symbolic Io Corporation Method and apparatus for dense hyper IO digital retention
US10061514B2 (en) 2015-04-15 2018-08-28 Formulus Black Corporation Method and apparatus for dense hyper IO digital retention
US20170083013A1 (en) * 2015-09-23 2017-03-23 International Business Machines Corporation Conversion of a procedural process model to a hybrid process model
CN106649563B (zh) * 2016-11-10 2022-02-25 新华三技术有限公司 一种网站分类字典的构建方法及装置
WO2019126072A1 (fr) 2017-12-18 2019-06-27 Formulus Black Corporation Systèmes, dispositifs et procédés informatiques à base de mémoire vive (ram)
US11163952B2 (en) * 2018-07-11 2021-11-02 International Business Machines Corporation Linked data seeded multi-lingual lexicon extraction
US10725853B2 (en) 2019-01-02 2020-07-28 Formulus Black Corporation Systems and methods for memory failure prevention, management, and mitigation

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007207218A (ja) * 2006-01-06 2007-08-16 Sony Corp 情報処理装置および方法、並びにプログラム

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1159661C (zh) * 1999-04-08 2004-07-28 肯特里奇数字实验公司 用于中文的标记和命名实体识别的系统
US20020032564A1 (en) * 2000-04-19 2002-03-14 Farzad Ehsani Phrase-based dialogue modeling with particular application to creating a recognition grammar for a voice-controlled user interface
WO2000073936A1 (fr) * 1999-05-28 2000-12-07 Sehda, Inc. Modelisation de dialogue a base de locution convenant particulierement pour la creation de grammaires de reconnaissance destinees a des interfaces utilisateurs a commande vocale
GB2362238A (en) * 2000-05-12 2001-11-14 Applied Psychology Res Ltd Automatic text classification
US20020099730A1 (en) * 2000-05-12 2002-07-25 Applied Psychology Research Limited Automatic text classification system
US6941513B2 (en) * 2000-06-15 2005-09-06 Cognisphere, Inc. System and method for text structuring and text generation
US6892189B2 (en) * 2001-01-26 2005-05-10 Inxight Software, Inc. Method for learning and combining global and local regularities for information extraction and classification
US6970881B1 (en) * 2001-05-07 2005-11-29 Intelligenxia, Inc. Concept-based method and system for dynamically analyzing unstructured information
JP2003242176A (ja) * 2001-12-13 2003-08-29 Sony Corp 情報処理装置および方法、記録媒体、並びにプログラム
US7454430B1 (en) * 2004-06-18 2008-11-18 Glenbrook Networks System and method for facts extraction and domain knowledge repository creation from unstructured and semi-structured documents
US7624102B2 (en) * 2005-01-28 2009-11-24 Microsoft Corporation System and method for grouping by attribute
WO2006083939A2 (fr) * 2005-01-31 2006-08-10 4Info, Inc. Systeme et procede de hierarchisation de reponses de recherche
JP5073494B2 (ja) * 2005-05-09 2012-11-14 株式会社ＭｅｔａＭｏＪｉ 文書処理装置および文書処理方法
US8200695B2 (en) * 2006-04-13 2012-06-12 Lg Electronics Inc. Database for uploading, storing, and retrieving similar documents
US7822701B2 (en) * 2006-06-30 2010-10-26 Battelle Memorial Institute Lexicon generation methods, lexicon generation devices, and lexicon generation articles of manufacture
US8196039B2 (en) * 2006-07-07 2012-06-05 International Business Machines Corporation Relevant term extraction and classification for Wiki content
CN101136020A (zh) * 2006-08-31 2008-03-05 国际商业机器公司 自动扩展参考数据的系统和方法
JP5283208B2 (ja) * 2007-08-21 2013-09-04 国立大学法人 東京大学 情報検索システム及び方法及びプログラム並びに情報検索サービス提供方法

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007207218A (ja) * 2006-01-06 2007-08-16 Sony Corp 情報処理装置および方法、並びにプログラム

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
"Kenkyu Hyoka Kagakuron no Tameno Kagaku Keiryogaku Nyumon, 1st edition", 30 March 2004, MARUZEN CO., LTD., article YUKO FUJIGAKI ET AL., pages: 67 - 72 *
HIDEKI KAWAI ET AL.: "Cost-effective Search Strategy for Bootstrapping Lexicon Acquisition", TRANSACTIONS OF INFORMATION PROCESSING SOCIETY OF JAPAN, vol. 1, no. 1, 15 November 2008 (2008-11-15), pages 36 - 48 *
HIROAKI OSHIMA ET AL.: "Seikaigo Pair Zenzo ni yoru Kanrengo Shutoku no Tameno Ryohoko Kobun Pattern Hakken", THE 1ST FORUM ON DATA ENGINEERING AND INFORMATION MANAGEMENT -DEIM FORUM- RONBUNSHU, 9 May 2009 (2009-05-09), Retrieved from the Internet <URL:http://db-event.jpn.org/deim2009/proceedings/files/B9-l.pdf> [retrieved on 20101228] *
HIROKI MIZUGUCHI ET AL.: "Web Chishiki o Riyo shita Bootstrap ni yoru Jisho Zoshoku Shuho", THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS, PROCEEDINGS OF THE 18TH DATA ENGINEERING WORKSHOP, 1 June 2007 (2007-06-01), Retrieved from the Internet <URL:http://www.ieice.org/iss/de/DEWS/DEWS2007/pdf/e8-5.pdf> [retrieved on 20101228] *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7384354B2 (ja) 2020-02-04 2023-11-21 本田技研工業株式会社 情報処理装置、情報処理方法およびプログラム

Also Published As

Publication number Publication date
US20120303359A1 (en) 2012-11-29
JP5708495B2 (ja) 2015-04-30
JPWO2011070980A1 (ja) 2013-04-22

Similar Documents

Publication Publication Date Title
JP5708495B2 (ja) 辞書作成装置、単語収集方法、及び、プログラム
US20030120644A1 (en) Method, apparatus, and computer program product for locating data in large datasets
US20140298321A1 (en) Installation control method and installation control apparatus
WO2017067156A1 (fr) Procédé de détermination de liste de listes de lecture, dispositif, dispositif électronique et support de stockage
CA3059929C (fr) Methode de recherche textuelle, appareil et support de stockage non transitoire lisible par ordinateur
JP5588811B2 (ja) データ分析支援システム及び方法
CN113543117B (zh) 携号转网用户的预测方法、装置及计算设备
JP2013246826A (ja) 対象の異構造インスタンスの属性値整列システム、対象の異構造インスタンスの属性値整列方法、及びプログラム
CN111259189B (zh) 一种音乐分类方法及装置
CN109241360B (zh) 组合字符串的匹配方法及装置和电子设备
JP5761029B2 (ja) 辞書作成装置、単語収集方法、及び、プログラム
JP5980520B2 (ja) 効率的にクエリを処理する方法及び装置
JP6705764B2 (ja) 生成装置、生成方法、及び生成プログラム
JP2007034878A (ja) 情報処理方法、情報処理装置および情報処理プログラム
JP5716966B2 (ja) データ分析装置、データ分析方法及びプログラム
JP5325131B2 (ja) パターン抽出装置、パターン抽出方法及びプログラム
CN110705889A (zh) 一种企业筛选方法、装置、设备及存储介质
JP5292247B2 (ja) コンテンツタグ収集方法、コンテンツタグ収集用プログラム、コンテンツタグ収集システム及びコンテンツ検索システム
JP6008067B2 (ja) テキスト処理システム、テキスト処理方法およびテキスト処理プログラム
CN109446408A (zh) 检索相似数据的方法、装置、设备及计算机可读存储介质
JP6190341B2 (ja) データ生成装置、データ生成方法、及びプログラム
JP5824429B2 (ja) スパムアカウントスコア算出装置、スパムアカウントスコア算出方法、及びプログラム
JP5494066B2 (ja) 検索装置、検索方法および検索プログラム
JPWO2016013209A1 (ja) 文集合抽出システム、方法およびプログラム
KR20190139227A (ko) K-부정합 검색을 위한 필터를 생성하는 시스템 및 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10835901

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2011545194

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 13515135

Country of ref document: US

122 Ep: pct application non-entry in european phase

Ref document number: 10835901

Country of ref document: EP

Kind code of ref document: A1