KR102196508B1 - Method and system for constructing named entity dictionary of using unsupervised learning - Google Patents

Abstract

The present invention provides a method and system for constructing an entity name dictionary through unsupervised learning, comprising: collecting document data for each category of collecting document data on the Internet and extracting a category of the document data; Formal usage pattern for each category derived by analyzing the sentence structure through the natural language processing process for document data, and analyzing the usage pattern for each category of the entity name based on the meaning of each category of the registered entity name registered in the entity name dictionary DB An entity name usage pattern registration step of registering a usage pattern DB for each category; And determining the meaning of the unregistered entity name by applying a formal usage pattern registered in the category-specific usage pattern DB according to a corresponding category to a usage pattern in which the unregistered entity name is used in document data in which an unregistered entity name exists, and the It is a method for constructing an entity name dictionary through unsupervised learning and a system for the same, comprising the step of registering an entity name dictionary in the entity name dictionary DB as a new entity name by matching an unregistered entity name to the meaning, and According to the present invention, by grasping the meaning of the entity name for each category through comprehensive analysis of the document data in which the category is classified, it is possible to increase the reliability of recognition of the newly emerged entity name, and furthermore effectively recognize the meaning change of the entity name.

Description

{Method and system for constructing named entity dictionary of using unsupervised learning}

The present invention relates to a method and system for constructing an entity name dictionary through unsupervised learning, and more specifically, by determining a usage pattern in which the entity name is used for each category for entity names existing on big data produced in large volumes every day. A method of constructing a database for the use pattern of individual entity names, and when an unregistered entity name occurs, the meaning of the unregistered entity name is identified by analyzing the usage pattern by considering the category of the document in which the unregistered entity name exists, and registered in the entity name dictionary. And a system for this is disclosed.

With the proliferation of the digital economy, a big data environment in which a lot of information and data are produced is coming around us. Big data refers to large-scale data that includes not only numerical data but also text and image data in its size, and its generation period is short compared to data generated in the past analog environment.

In particular, due to the rapid development and explosive spread of digital communication technologies such as the Internet, networks, and mobile devices, digital communication devices are deeply involved in people's general life, and data, which are footprints people leave everywhere, are increasing exponentially. Accordingly, a large number of new individual names, which are newly used names due to the progress of people's individuality, society, and culture, are occurring.

As such new entity names generally spread widely in a short period of time, but their use cycle is short, identifying and reflecting newly emerging entity names as quickly as possible is one of the key issues in digital communication technology.Therefore, various technologies related to upgrading the entity name dictionary Are being presented.

FIG. 1 shows a configuration diagram of a method for constructing an entity name dictionary according to the prior art (refer to Korean Patent Publication No. 10-2011-0068137). Referring to FIG. 1, a system 20 for constructing an entity name dictionary according to the prior art. Looking at, in the conventional technology generally introduced, the web document collector 21 collects web documents including various entity names distributed on the Internet 10, and each of the corresponding documents through the information extractor 23 After grasping and extracting the meaning of the sentences, the new entity name included therein is extracted by the entity name extractor 25, and the meaning is matched to be registered in the entity name dictionary DB 29. In order to promote appropriate use of additionally discovered entity names, the entity names that can be used in each category may be classified and registered through the category determiner 27.

In such a conventional technique, the meaning of an entity name is recognized only with contextual information, and most of it is a method of recognizing entity names through corpus-based learning, and the addition of new words is exponentially due to the rapid expansion of data. It is difficult to grasp and reflect the exact meaning of a new entity name in a situation where the number of changes and the number of changes in the existing word meaning is increasing.

For example, when document data including an entity name as shown in FIG. 2 exists, in FIG. 2(a), the mayor means'mayor' in "Mayor Won-soon Park", but in FIG. 2(b), " In the cloud market", the market means'market', so it may be possible to identify the meaning of the entity name "Park Won-soon" or "Cloud" to some extent in the overall document content, but if such a pattern exists in a short sentence There may be an error in recognizing the entity name "Park Won-soon" as the name of'market', and also an error in recognizing the entity name "Cloud" as the person name of'mayor'. Such a problem is not a fragmentary problem only in the language of Hangul, but a problem that can occur in all languages of the world, such as English, Japanese, and French, where homophones exist.

Therefore, a method of constructing an entity name dictionary that can increase the reliability of recognition of newly emerged entity names by extracting comprehensive information on the grammatical meaning and notational meaning of entity names through analysis of big data is determined. In addition, it is necessary to introduce a method for constructing an entity name dictionary that can recognize the transformation of the entity name or the meaning of a word.

(Patent Document 1) KR10-2011-0068137 A (object name dictionary construction system and construction method, 2011.06.22)

The present invention seeks to solve the problems of the prior art as described above, and extracts comprehensive information on the grammatical and notational meanings of entity names through analysis of big data, and determines the meaning of the entity name. We would like to suggest a method of constructing an entity name dictionary that can increase the reliability of recognition for and further recognize the transformation of the entity name or the meaning of a word.

In order to achieve the above technical problem, a method for constructing an entity name dictionary through unsupervised learning according to the present invention includes the steps of collecting document data for each category of collecting document data on the Internet and extracting a category of the document data; Formal usage pattern for each category derived by analyzing the sentence structure through the natural language processing process for document data, and analyzing the usage pattern for each category of the entity name based on the meaning of each category of the registered entity name registered in the entity name dictionary DB An entity name usage pattern registration step of registering a usage pattern DB for each category; And determining the meaning of the unregistered entity name by applying the official usage pattern registered in the category-specific usage pattern DB according to the category to the unregistered entity name usage pattern in which the unregistered entity name is used in document data in which the unregistered entity name exists. And, it may include the entity name dictionary registration step of registering the entity name dictionary DB as a new entity name by matching the unregistered entity name to the meaning.

Preferably, in the registering of the entity name use pattern, the sentence structure is analyzed through a natural language processing process for the document data, and the meaning registered in the entity name dictionary DB for the registered entity name included in the analyzed sentence structure is Extracting one or more usage pattern candidates in which the registered entity name is used for each category from the document data as a basis; Calculating a usage frequency of the usage pattern candidate on one or more document data corresponding to the category of the usage pattern candidate; Selecting a formal usage pattern for each category from one or more usage pattern candidates based on the usage frequency; And matching the official usage pattern to the category and registering it in a usage pattern DB for each category.

More preferably, the step of extracting the use pattern candidate may include analyzing a morpheme through a natural language processing process for the document data, and extracting a word classified as a noun from among the analyzed morphemes; Extracting a meaning of a registered entity name among the extracted words from the entity name dictionary DB; And extracting one or more usage pattern candidates that are combined with one or more of the nouns based on the meaning of the registered entity name and used for each category of the registered entity name.

The calculating of the frequency of use includes calculating the frequency of use by counting the number of use of the use pattern candidates for one or more document data corresponding to the category of the use pattern candidate, and selecting the formal use pattern. In the step, a usage pattern candidate having the highest value among the usage frequencies of the usage pattern candidates may be selected as a formal usage pattern for each category.

In addition, the pre-registering of the entity name may include classifying document data in which an unregistered entity name exists among document data as analysis target data and extracting a category of the analysis target data; Analyzing a sentence structure through a natural language processing process for the analysis target data to extract one or more unregistered entity name usage patterns used in the analysis target data; Extracting a formal usage pattern corresponding to the unregistered entity name usage pattern from the formal usage patterns corresponding to the category of the analysis target data from the category-specific usage pattern DB; Determining the meaning of the unregistered entity name by applying the extracted formal usage pattern to the unregistered entity name usage pattern; And registering the unregistered entity name with the meaning in the entity name dictionary DB as a new entity name.

Preferably, the extracting the use pattern of the unregistered entity name includes: analyzing a morpheme through a natural language processing process for the analysis target data, and extracting a word classified as a noun among the analyzed morphemes; And extracting one or more unregistered entity name usage patterns that are combined with one or more of the nouns and used for each category of the unregistered entity name.

Further, in the step of collecting document data for each category, document data classified in categories on the Internet may be parsed through an entity name collecting bot, and a category may be extracted from the document data.

Here, the document data may include one or more of news, academic literature, or patent literature classified by category.

In addition, the system for constructing an entity name dictionary through unsupervised learning according to the present invention includes: a document data collection unit that collects document data on the Internet and extracts a category of the document data; A usage pattern analysis unit that derives a formal usage pattern for each category in which an entity name is used from document data based on the meaning of each category of the registered entity name registered in the entity name dictionary DB, and registers it in a usage pattern DB for each category; And determining the meaning of the unregistered entity name by applying the official usage pattern registered in the category-specific usage pattern DB according to the category to the unregistered entity name usage pattern in which the unregistered entity name is used in document data in which the unregistered entity name exists, It may include an entity name analysis unit that registers in the entity name dictionary DB.

Preferably, the usage pattern analysis unit comprises: a natural language processing unit for analyzing morphemes through a natural language processing process for document data; An entity name usage pattern extracting unit that extracts one or more usage pattern candidates in which the entity name is used from document data based on the meaning of each category of the registered entity name registered in the entity name dictionary DB; And a formal usage pattern generator that selects a formal usage pattern for each category from one or more usage pattern candidates based on the usage frequency and registers it in a usage pattern DB for each category.

In addition, the entity name analyzer includes: an unregistered entity name extracting unit that recognizes document data including an unregistered entity name from the document data and extracts it as analysis target data; A natural language processing unit for analyzing a morpheme through a natural language processing process for the data to be analyzed; An unregistered entity name analysis unit that determines the meaning of the unregistered entity name by applying a formal usage pattern registered in the category-specific usage pattern DB according to a corresponding category to an unregistered entity name usage pattern in which the unregistered entity name is used in the analysis target data; And a new entity name registration unit for registering the unregistered entity name in the entity name dictionary DB as a new entity name in correspondence with the meaning.

Further, the document data collection unit may include a document data collection unit equipped with an entity name collecting bot (Bot) for collecting document data including an entity name among document data classified into categories on the Internet; And a document data classification unit that recognizes a category of the collected document data and classifies the document data by corresponding category.

Preferably, in the usage pattern DB, formal usage patterns of entity names are classified and stored by category, and the entity name dictionary DB may be classified and stored by entity names and their meanings.

According to the present invention, by grasping the meaning of the entity name by category through comprehensive analysis of the document data in which the category is classified, it is possible to increase the reliability of recognition of the newly emerged entity name, and further effectively recognize the change in the meaning of the entity name. .

In particular, by judging the meaning of a new entity name or an entity name whose meaning has changed based on the usage pattern of the entity name by category, it is possible to accurately grasp the meaning of the entity name existing in the same sentence structure or the meaning of the existing entity name. You will be able to.

Furthermore, it is possible to extract a usage pattern to grasp the meaning of an entity name even in unstructured data that is not general, and also, the existing entity name recognition engine by determining the meaning of the entity name based on the usage pattern extracted for each category without using an entity name recognizer. It is not dependent on the performance of and can overcome the recognition errors that occur in the entity name recognition engine due to a unilateral analysis.

1 shows a configuration diagram of an entity name dictionary construction method according to the prior art,
2 shows an embodiment of a recognition error for an entity name existing in document data according to the prior art,
3 shows a schematic configuration diagram of an entity name dictionary construction system through unsupervised learning according to the present invention,
4 is a block diagram of an embodiment of a document data collection unit in the system for constructing an entity name dictionary through unsupervised learning according to the present invention,
5 shows a configuration diagram of an embodiment of a usage pattern analysis unit in the system for constructing an entity name dictionary through unsupervised learning according to the present invention,
6 shows a configuration diagram of an embodiment of an entity name analysis unit in the entity name dictionary construction system through unsupervised learning according to the present invention,
7 is a schematic flowchart of a method for constructing an entity name dictionary through unsupervised learning according to the present invention,
FIG. 8 is a schematic flowchart illustrating a method for constructing an entity name dictionary through unsupervised learning according to the present invention;
9 shows an embodiment of generating a use pattern of entity names for each category of FIG. 8,
10 is a diagram illustrating various embodiments in which a usage pattern of entity names for each category of FIG. 8 is generated,
11 is a schematic flowchart of registering a new entity name for each category in the entity name dictionary in the entity name dictionary construction method through unsupervised learning according to the present invention,
12 shows an embodiment of registering a new entity name for each category of FIG. 11 in the entity name dictionary.

In order to explain the present invention, operational advantages of the present invention, and objects achieved by the implementation of the present invention, the following will illustrate a preferred embodiment of the present invention and look at it with reference.

First, terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention, and expressions in the singular may include a plurality of expressions unless the context clearly indicates otherwise. In addition, in the present application, terms such as "comprise" or "have" are intended to designate the existence of features, numbers, steps, actions, components, parts, or a combination thereof described in the specification, but one or more other It is to be understood that it does not preclude the presence or addition of features, numbers, steps, actions, components, parts, or combinations thereof.

In describing the present invention, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

The present invention discloses a method of constructing an entity name dictionary by recognizing the meaning of a newly emerged new entity name or a modified meaning of an existing entity name through a comprehensive analysis of document data existing after a corresponding category is classified on the Internet.

In particular, in the present invention, a use pattern for each category used in document data is created in consideration of the meaning of each category of a registered entity name registered in the entity name dictionary DB, and a new type of entity name having a new or modified meaning in the category is used. To understand the meaning of, the following will first look at the entity name dictionary construction system through unsupervised learning according to the present invention, and then look at the entity name dictionary construction method through unsupervised learning according to the present invention using the same. .

3 shows a schematic configuration diagram of an entity name dictionary construction system through unsupervised learning according to the present invention.

The entity name dictionary construction system 100 according to the present invention may be schematically configured to include a document data collection unit 110, a usage pattern analysis unit 130, and an entity name analysis unit 150.

The document data collection unit 110 collects document data including an entity name among various document data generated in real time on the Internet 10. 3 shows the document data collection unit 110 as a separate configuration, but depending on the situation, the document data collection unit is a part of the usage pattern analysis unit 130 or the entity name analysis unit 150 and is used when necessary. The pattern analysis unit 130 or the entity name analysis unit 150 may be configured to collect corresponding document data by itself.

The usage pattern analysis unit 130 derives a formal usage pattern for each category in which an entity name is used from document data based on the meaning of each category of the registered entity name registered in the entity name dictionary DB, and registers it in the usage pattern DB 200 for each category. do. Here, in the usage pattern DB 200, the formal usage pattern of the entity name is classified and stored by category. In FIG. 3, the usage pattern DB 200 is shown to exist outside the entity name dictionary construction system 100. Accordingly, it may be included as a part of the usage pattern analysis unit 130.

In addition, the entity name analysis unit 150 analyzes the usage pattern in which the unregistered entity name is used in the document data in which the unregistered entity name exists as a formal usage pattern registered in the usage pattern DB 200 for each category according to the corresponding category, and the unregistered entity The meaning of the name is determined and registered in the entity name dictionary DB (300). Here, in the entity name dictionary DB 300, registered entity names and their meanings may be classified and stored by category, and in FIG. 3, the entity name dictionary DB 300 exists outside the entity name dictionary building system 100. Although shown as shown, it may be included as a part of the entity name analyzer 150 as needed, similar to the usage pattern DB 200.

Looking at each configuration of the entity name dictionary building system 100 according to the present invention in more detail through an embodiment, first, FIG. 4 is a document data collection unit 110 in the entity name dictionary building system through unsupervised learning according to the present invention. ), the document data collection unit 110 is a document equipped with an entity name collection bot (Bot) that collects document data including an entity name among document data classified into categories on the Internet. A data collection unit 111 and a document data classification unit 115 for recognizing a category of the collected document data and classifying document data for each corresponding category may be included.

The document data collected here is document data in which the corresponding category is classified in advance, and may include news, academic literature, and patent literature. For example, in the case of news, politics, economy, society, life/culture, IT/science, etc. The article's category is classified as, and in the case of a patent document, the category is classified by the relevant technical field through IPC (International Patent Classification), which is an international patent classification. Furthermore, the present invention is not limited thereto, and information data classified in advance may be included. For example, it may include information data that exists after categories are classified on an Internet blog or a cafe, or information data for each category exchanged between individuals on a social network.

Furthermore, the document data collection unit 110 collects document data for each category periodically or when necessary, and the document data classification unit 115 prepares the document data 11 for each category by classifying the collected document data by corresponding category. Although it may be done, preferably, it is possible to collect document data of a specific category by interlocking with the usage pattern analysis unit 130 and extract the category of the collected document data. In addition, the unregistered object may be interlocked with the entity name analysis unit 150 It is also possible to collect document data with names and extract categories of collected document data.

Next, FIG. 5 shows a configuration diagram of an embodiment of the usage pattern analysis unit 130 in the system for constructing an entity name dictionary through unsupervised learning according to the present invention. The usage pattern analysis unit 130 includes a natural language processing unit 131 ), an entity name use pattern extraction unit 135, and a formal use pattern generation unit 137.

The natural language processing unit 131 analyzes a morpheme through a natural language processing process for document data. In particular, the morpheme is analyzed based on the morpheme dictionary 350 for a sentence including the registered entity name. The entity name usage pattern extraction unit 135 extracts one or more usage pattern candidates in which the entity name is used from document data based on the meaning of each category of the registered entity name registered in the entity name dictionary DB 300, and a formal usage pattern generation unit Reference numeral 137 selects a formal usage pattern for each category from among one or more usage pattern candidates based on each usage frequency of one or more usage pattern candidates extracted by the entity name usage pattern extraction unit 135, and uses a usage pattern DB for each category (200). ), and the usage pattern DB 200 may be classified by corresponding category to store formal usage patterns. Details of the operations of the entity name usage pattern extraction unit 135 and the formal usage pattern generation unit 137 will be described later in the method of constructing an entity name dictionary through unsupervised learning according to the present invention.

Through the usage pattern analysis unit 130, the usage pattern DB 200 accumulates formal usage patterns for each category for the registered entity name based on the meaning of each category, and the entity name analysis unit 150 stores a usage pattern analysis unit. By applying the official usage pattern for each category accumulated in the usage pattern DB 200 through 130, the meaning of the unregistered entity name can be determined more accurately and with high reliability. It shows a configuration diagram of an embodiment of the entity name analysis unit in the entity name dictionary building system through learning.

The entity name analysis unit 150 may include an unregistered entity name extraction unit 151, a natural language processing unit 153, an unregistered entity name analysis unit 155, and a new entity name registration unit 157.

The unregistered entity name extracting unit 151 recognizes document data including the unregistered entity name from the document data and extracts it as analysis target data. For this purpose, the unregistered entity name extraction unit 151 itself includes the unregistered entity name on the Internet. Although a configuration capable of parsing document data may be provided, preferably, the unregistered entity name extraction unit 151 collects document data including the unregistered entity name in connection with the document data collection unit 110, and An unregistered entity name included in the document data may be recognized based on the entity name dictionary DB 300 to extract data to be analyzed.

The natural language processing unit 153 analyzes morphemes through a natural language processing process for the analysis target data extracted by the unregistered entity name extraction unit 151, and analyzes the morphemes based on the morpheme dictionary 350 for sentences containing unregistered entity names. can do.

The unregistered entity name analysis unit 155 analyzes the usage pattern in which the unregistered entity name is used in the analysis target data as a formal usage pattern registered in the category-specific usage pattern DB 200 according to the corresponding category to determine the meaning of the unregistered entity name. It is determined that the operation of the unregistered entity name analysis unit 155 will be described in more detail later in the method of constructing an entity name dictionary through unsupervised learning according to the present invention.

The new entity name registration unit 157 registers a new entity name in the entity name dictionary DB 300 based on the meaning of the non-registered entity name determined by the non-registered entity name analysis unit 155. At this time, the entity name dictionary DB 300 may be classified by category and a new entity name may be stored.

The present invention also proposes a method of constructing a faster and more reliable entity name dictionary using the entity name dictionary construction system through unsupervised learning according to the present invention. Hereinafter, an entity through unsupervised learning according to the present invention Let's look at how to build a name dictionary.

Since the method of constructing an entity name dictionary through unsupervised learning according to the present invention is presented based on the entity name dictionary construction system through unsupervised learning according to the present invention, as described above, the entity name dictionary through unsupervised learning according to the present invention. It will be described with reference to the embodiment of the construction system together.

7 is a schematic flowchart of a method for constructing an entity name dictionary through unsupervised learning according to the present invention.

The method of constructing an entity name dictionary through unsupervised learning according to the present invention is a process of constructing a use pattern DB 200 for each category by creating a formal use pattern for each category for the entity name included in the document data. It can be classified into a process of constructing the category-specific entity name dictionary DB 300 by determining the meaning of the unregistered entity name by applying the official usage pattern of the category-specific usage pattern DB 200 to the included unregistered entity name usage pattern.

First, looking at the process of building the use pattern DB 200 for each category, the document data in which the categories existing in the Internet 10 are classified is collected (S110), and the category of the collected document data is extracted (S120) to The meaning of the registered entity name included in the document data is determined for each category based on the category to which the data belongs, and the entity name usage pattern for each category used in the document data is analyzed (S210). Then, based on the analyzed result, the official use pattern for each category in which the registered entity name is used is extracted (S270), and the generated official use pattern is registered in the use pattern DB 200 for each category (S290).

For a more detailed look at the process of constructing the use pattern DB 200 for each category, refer to FIG. 8, which is a flowchart of generating a use pattern for each category in the method for constructing an entity name dictionary through unsupervised learning according to the present invention. .

First, the document data collection unit 110 collects document data in which categories existing on the Internet 10 are classified (S110a). In this case, the document data classified into the category includes the registered entity name registered in the entity name dictionary DB 300 Included. The category of the collected document data is extracted (S120a), and the extracted category and document data are provided to the usage pattern analysis unit 130 (S130a). Since the collected document data has already been categorized, the document data collection unit 110 may skip the process of extracting the category (S120a) and immediately provide the document data to the usage pattern analysis unit 130.

The usage pattern analysis unit 130 analyzes the sentence structure through the natural language processing process (S220) for the provided document data, and the meaning registered in the entity name dictionary DB 300 for the registered entity name included in the analyzed sentence structure On the basis of, one or more usage pattern candidates used for each category of the registered entity name are extracted from the document data (S230).

For a more detailed description, referring to the embodiment of generating the use pattern of entity names for each category shown in FIG. 9 together, news on the Internet is classified into a corresponding category, and in FIG. 9, news in which the category is society (A1). Represents document data. In FIG. 9, it is assumed that the entity name "Park Won-soon" as news document data including the entity name "Park Won-soon" is a registered entity name already registered in the entity name dictionary DB 300.

The natural language processing unit 131 of the usage pattern analysis unit 130 analyzes the morpheme through the natural language processing process for the sentence (B1) containing the "Park Won-soon" entity name. At this time, the morpheme is analyzed based on the morpheme dictionary 350 Among the analyzed morphemes, only words classified as nouns are extracted. As a result of the analysis of the sentence B1, the noun words Park Won-soon, Mayor, Lotte World, establishment, decision, and overturn are extracted.

In addition, the entity name usage pattern extraction unit 135 of the usage pattern analysis unit 130 extracts the registered entity name "Park Won-soon", grasps the meaning of "Park Won-soon" in the social category of the entity name dictionary DB 300, and registers Recognize that the entity name "Park Won-soon" is the name of a person in the social category.

In addition, the entity name usage pattern extraction unit 135 of the usage pattern analysis unit 130 combines the usage pattern in which the registered entity name is used in the sentence B1 with a noun as a person name that is the meaning of the social category of the registered entity name "Park Won-soon". As shown in FIG. 9, as shown in FIG. 9, a 2 gram pattern of "person name + market" as a first usage pattern candidate, a 3 gram pattern of "person name + market + Lotte World" as a second usage pattern candidate, and a third usage pattern. As candidates, four usage pattern candidates are extracted with a 4 gram pattern of "person name + market + lotte world + construction" and a 5 gram pattern of "person name + market + lotte world + construction + overturn" as the fourth usage pattern candidate.

Referring back to FIG. 8 and looking at the subsequent process, when a usage pattern candidate from the document data for the registered entity name is extracted (S230), the usage frequency is calculated from the document data from which the usage pattern candidate is calculated for the extracted usage pattern candidate. However, more preferably, it is possible to improve accuracy by calculating the frequency of use in which each usage pattern candidate is used in a plurality of document data belonging to a corresponding category, and thus through the document data collection unit 110 as shown in FIG. Receives a plurality of document data belonging to the corresponding category (S240), calculates the frequency of use of each use pattern candidate in these document data (S250), and a formal use pattern candidate among the use pattern candidates based on the calculated use frequency Select (S280). Then, the selected formal use pattern candidate is registered in the use pattern DB 200 for each category (S290).

Looking at the process of selecting and registering the official use pattern candidates with continued reference to FIG. 9, the official use pattern generation unit 137 of the use pattern analysis unit 130 is a first use pattern that is the extracted four use pattern candidates. The frequency of use is calculated for each of the candidate, the second usage pattern candidate, the third usage pattern candidate, and the fourth usage pattern candidate. At this time, the usage frequency may be calculated based only on the document data of B1, but it is used to increase accuracy. The formal usage pattern generation unit 137 of the pattern analysis unit 130 may calculate a usage frequency for each usage pattern candidate from various document data of a corresponding category in conjunction with the document data collection unit 110.

In FIG. 9, the number of usage frequency calculation results for each usage pattern candidate, the first usage pattern candidate, the second usage pattern candidate, the third usage pattern candidate, and the fourth usage pattern candidate are used on document data of a corresponding category, respectively. It was calculated 10 times, 3 times, 2 times and 1 time, and the official use pattern generation unit 137 of the use pattern analysis unit 130 selects the first use pattern candidate having the highest use frequency among each use pattern candidate. The official usage pattern is selected, and "person name + market" in the social category is registered in the usage pattern DB 200 for each category as the formal usage pattern.

Through the process according to the present invention described above, a use pattern DB for each category can be constructed by creating a formal use pattern for each category for the entity name included in the document data, and as various examples of the use pattern for each category, Referring to FIG. 10, despite the usage pattern consisting of "individual name + market" in both (a) and (b) of FIG. 10, in FIG. 10 (a), as shown in FIG. 9, society (A1 ) In the category, "object name + market" can be registered as a formal usage pattern of "person name + market" through sentence B1, and in Fig. 10(b), "object name" through sentence B2 in IT (A2) category +Market" may be registered as a formal usage pattern of "Service Name + Market".

As described above, in the present invention, by determining the usage pattern in which the entity name is used for each category, it is possible to determine the exact usage pattern for the entity name of the homophone or the entity name existing in a similar sentence structure.

Next, referring to the process of building the entity name dictionary DB 300 back to FIG. 7, the document data in which categories are classified on the Internet 10 is collected (S110). For the construction of the use pattern DB 200 previously Although the previously collected document data may be used, the document data may be newly collected again, and in this case, document data in which an unregistered entity name exists is collected.

In addition, the usage pattern in which the unregistered object name is used in the document data in which the unregistered object name exists is analyzed (S310), and the official usage pattern belonging to the corresponding category is extracted from the constructed usage pattern DB 200 for each category, and the unregistered object name is used. By applying the formal usage pattern corresponding to the pattern (S340), a more accurate meaning of the unregistered entity name can be grasped, and a new entity name is created (S380) by matching the meaning of each category of the identified unregistered entity name to the entity name dictionary DB Register at 300 (S390).

In order to examine the process of building the entity name dictionary DB 300 for each category in more detail, a flow chart for registering new entity names for each category in the entity name dictionary in the entity name dictionary building method through unsupervised learning according to the present invention. See 11.

First, the document data collection unit 110 collects document data in which categories existing in the Internet 10 are classified (S110b). In the process of collecting the document data, the document data collection unit 110 uses the use pattern DB for each category ( 200) may be previously collected document data, but preferably separately collected document data. In particular, the document data collection unit 110 is an unregistered entity that is not registered in the entity name dictionary DB 300 Document data containing names can also be selectively collected. Alternatively, the document data collection unit 110 does not determine whether the entity name is registered, and the entity name analysis unit 150 converts the document data including the unregistered entity name from the document data collected by the document data collection unit 110 as analysis target data. It can also be classified and selectively extracted.

In addition, the category of the collected document data is extracted (S120b), and the extracted category and document data are provided to the entity name analysis unit 150 (S130b), where the collected document data is already classified, so the document data collection unit In step 110, the process of extracting the category (S120b) may be omitted and the document data may be immediately provided to the entity name analyzer 150.

The entity name analysis unit 150 classifies the document data including the unregistered entity name as analysis target data and analyzes the sentence structure through the natural language processing process (S320) for the analysis target data. Looking at an embodiment of registering a new entity name for each category in the entity name dictionary as shown in FIG. 12, as news document data of a social category including “Nam-Sik Heo”, which is an unregistered entity name, the entity name analysis unit 150 The unregistered entity name extracting unit 151 of recognizes document data including the unregistered entity name based on the entity name dictionary DB 300 and extracts it as analysis target data. In addition, the natural language processing unit 153 of the entity name analysis unit 150 analyzes the morpheme through the natural language processing process for the sentence (B2) containing the unregistered entity name “Namsik Heo”, at this time, based on the morpheme dictionary 350 Is analyzed to extract only words classified as nouns among the analyzed morphemes. As a result of analyzing the sentence B2, the noun words Heo Nam-sik, the mayor, the third year, pledge, fulfillment rate, and 80 are extracted.

In this way, the analysis of the sentence including the unregistered entity name is performed, and the meaning of the unregistered entity name is identified. Returning to FIG. 11 again, the pattern of using the unregistered entity name in which the unregistered entity name is used in the sentence structure analyzed through the natural language processing process Analysis (S330) to extract one or more unregistered entity name usage patterns, and apply the official usage pattern registered in the category-specific usage pattern DB (200) to unregistered entity name usage patterns (S360) to correspond to the unregistered entity name The meaning of in is determined (S370). To continue this process with reference to FIG. 12 again, the non-registered entity name analysis unit 155 of the entity name analysis unit 150 retrieves the non-registered entity name from the B2 sentence classified into the social category of the non-registered entity name “Nam-Sik Heo”. The unregistered entity name usage pattern is extracted by combining it with a noun. As shown in FIG. 12, the first unregistered entity name usage pattern is "Heo Nam-sik + market", and the second unregistered entity name usage pattern is 3 gram pattern " Namsik Heo + Mayor + 3rd year", Namsik Heo + Mayor + 3rd year + Commitment" as a 4 gram pattern for the use of unregistered entity names, and Namsik Heo + Mayor + 3rd year as a 5 gram pattern for the 4th unregistered entity An unregistered entity name usage pattern may be extracted by "+commitment + fulfillment rate", and the like, in FIG. 12, it will be described as extracting four unregistered entity name usage patterns.

In addition, the unregistered entity name analysis unit 155 of the entity name analysis unit 150 extracts the official usage pattern belonging to the category from the category-specific usage pattern DB 200 and applies it to the unregistered entity name usage pattern. As a formal usage pattern that can be matched on the category-specific usage pattern DB 200 among the unregistered entity name usage patterns shown in FIG. 9, the formal usage pattern of "person name + market" created through the embodiment of FIG. 9 may be applied. , As the formal usage pattern of "person name + market" can be applied to the first unregistered entity name usage pattern, by applying the formal usage pattern of "person name + market" to the first unregistered entity name usage pattern, "Nam-Sik Heo" becomes "person name. It can be judged as meaning

When the meaning of the unregistered entity name is determined through such a process, the new entity name registration unit 157 of the entity name analyzer 150 refers to “Nam-sik Heo” and “Nam-sik Heo” with reference to FIGS. 11 and 12 again. A new entity name is created in correspondence with the meaning of "person name" (S380), and this is registered in the entity name dictionary DB 300 (S390).

As described above, in the present invention, the meaning of the entity name for each category is identified through comprehensive analysis of the document data in which the category is classified, thereby increasing the reliability of recognition of the newly emerged entity name, and further, it is possible to effectively recognize the transformation of the meaning of the entity name.

In particular, by judging the meaning of a new entity name or an entity name whose meaning has changed based on the usage pattern used for each category, it is possible to accurately grasp the meaning of the entity name of the homophone or the meaning of the existing entity name.

Furthermore, it is possible to extract a usage pattern to grasp the meaning of an entity name even in unstructured data that is not general, and also, the existing entity name recognition engine by determining the meaning of the entity name based on the usage pattern extracted for each category without using an entity name recognizer. It is not dependent on the performance of and can overcome the recognition errors that occur in the entity name recognition engine due to a unilateral analysis.

The above description is only illustrative of the technical idea of the present invention, and those of ordinary skill in the technical field to which the present invention pertains will be able to make various modifications and variations without departing from the essential characteristics of the present invention. Accordingly, the embodiments described in the present invention are not intended to limit the technical spirit of the present invention, but to explain the technical spirit, and the technical spirit of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

10: Internet,
100: entity name dictionary construction system,
110: document data collection unit,
111: document data collection unit, 115: document data classification unit,
130: usage pattern analysis unit,
131: natural language processing unit, 135: entity name usage pattern analysis unit,
137: formal use pattern generation unit,
150: entity name analysis unit,
151: unregistered entity name extraction unit, 153: natural language processing unit,
155: unregistered entity name analysis unit, 157: new entity name registration unit,
200: use pattern DB,
300: object name dictionary DB,
350: morpheme dictionary.

Claims (13)

Collecting document data for each category of collecting document data on the Internet and extracting a category of the document data;
Analyzes sentence structure through natural language processing of document data, and detects one or more usage pattern candidates in which the registered entity name is used for each category in the document data based on the meaning of each category of the registered entity name registered in the entity name dictionary DB And registering an official use pattern derived based on the use frequency of the detected use pattern candidate in a use pattern DB for each category; And
Extracts one or more unregistered entity name usage patterns in which the unregistered entity name is used from document data in which the unregistered entity name exists among document data, and officially uses registered in the category-specific usage pattern DB according to the category in the unregistered entity name usage pattern Non-supervised learning comprising the step of registering an entity name dictionary DB as a new entity name by determining the meaning of the unregistered entity name by applying a pattern and matching the unregistered entity name to the meaning Method of building an entity name dictionary through The method of claim 1,
The entity name usage pattern registration step,
The sentence structure is analyzed through the natural language processing process for the document data, and the registered entity name in the document data by category based on the meaning registered in the entity name dictionary DB for the registered entity name included in the analyzed sentence structure. Extracting one or more usage pattern candidates to be used;
Calculating a usage frequency of the usage pattern candidate on one or more document data corresponding to the category of the usage pattern candidate;
Selecting a formal usage pattern for each category from one or more usage pattern candidates based on the usage frequency; And
And registering the formal usage pattern with the category and registering it in a usage pattern DB for each category. The method of claim 2,
The step of extracting the use pattern candidate,
Analyzing a morpheme through a natural language processing process for the document data, and extracting a word classified as a noun from among the analyzed morphemes;
Extracting a meaning of a registered entity name among the extracted words from the entity name dictionary DB; And
And extracting one or more usage pattern candidates that are combined with one or more of the nouns based on the meaning of the registered entity name and used for each category of the registered entity name. The method of claim 2,
The step of calculating the use frequency,
The usage frequency is calculated by counting the number of usage pattern candidates used for one or more document data corresponding to the category of the usage pattern candidate,
The step of selecting the formal use pattern,
An entity name dictionary construction method through unsupervised learning, characterized in that selecting a usage pattern candidate having the highest value among the usage frequencies of the usage pattern candidates as formal usage patterns for each category. The method of claim 1,
The entity name pre-registration step,
Classifying document data in which an unregistered entity name exists among the document data as analysis target data and extracting a category of the analysis target data;
Analyzing a sentence structure through a natural language processing process for the analysis target data to extract one or more unregistered entity name usage patterns used in the analysis target data;
Extracting a formal usage pattern corresponding to the unregistered entity name usage pattern from the formal usage patterns corresponding to the category of the analysis target data from the category-specific usage pattern DB;
Determining the meaning of the unregistered entity name by applying the extracted formal usage pattern to the unregistered entity name usage pattern; And
And registering the entity name dictionary DB as a new entity name by matching the unregistered entity name to the meaning. The method of claim 5,
The step of extracting the use pattern of the unregistered entity name,
Analyzing a morpheme through a natural language processing process for the data to be analyzed, and extracting a word classified as a noun from among the analyzed morphemes; And
And extracting one or more unregistered entity name usage patterns combined with one or more of the nouns and used for each category of the unregistered entity name. The method of claim 1,
The step of collecting document data by category,
A method of constructing an entity name dictionary through unsupervised learning, characterized in that, through an entity name collecting bot (Bot), the document data classified in categories on the Internet is parsed and the category is extracted from the document data. The method of claim 7,
The document data,
A method of constructing an entity name dictionary through unsupervised learning, characterized in that the category includes one or more of classified news, academic literature, or patent literature. A document data collection unit collecting document data on the Internet and extracting a category of the document data;
Analyzes sentence structure through natural language processing of document data, and detects one or more usage pattern candidates in which the registered entity name is used for each category in the document data based on the meaning of each category of the registered entity name registered in the entity name dictionary DB And a usage pattern analysis unit that registers a formal usage pattern derived based on a usage frequency of the detected usage pattern candidate in a usage pattern DB for each category; And
Extracts one or more unregistered entity name usage patterns in which the unregistered entity name is used from document data in which the unregistered entity name exists among document data, and officially uses registered in the category-specific usage pattern DB according to the category in the unregistered entity name usage pattern Through unsupervised learning, characterized in that it comprises an entity name analysis unit that determines the meaning of the unregistered entity name by applying a pattern, and registers the entity name dictionary DB as a new entity name by matching the unregistered entity name to the meaning. Entity name dictionary construction system. The method of claim 9,
The usage pattern analysis unit,
A natural language processing unit that analyzes a morpheme through a natural language processing process for document data;
An entity name usage pattern extracting unit that extracts one or more usage pattern candidates in which the entity name is used from document data based on the meaning of each category of the registered entity name registered in the entity name dictionary DB; And
An entity through unsupervised learning, comprising: a formal usage pattern generator for selecting a formal usage pattern for each category from one or more usage pattern candidates based on the usage frequency of the usage pattern candidate and registering it in a usage pattern DB for each category People dictionary building system. The method of claim 9,
The entity name analysis unit,
An unregistered entity name extracting unit for recognizing document data including an unregistered entity name from the document data and extracting it as analysis target data;
A natural language processing unit for analyzing a morpheme through a natural language processing process for the data to be analyzed;
An unregistered entity name analysis unit that determines the meaning of the unregistered entity name by applying a formal usage pattern registered in the category-specific usage pattern DB according to a corresponding category to an unregistered entity name usage pattern in which the unregistered entity name is used in the analysis target data; And
And a new entity name registration unit for registering the unregistered entity name in the entity name dictionary DB as a new entity name by matching the meaning of the unregistered entity name. The method of claim 9,
The document data collection unit,
A document data collection unit equipped with an entity name collecting bot (Bot) for collecting document data including an entity name among document data categorized on the Internet; And
An entity name dictionary construction system through unsupervised learning, comprising: a document data classification unit that recognizes a category of collected document data and classifies document data by corresponding category. The method of claim 9,
The use pattern DB,
Formal usage patterns of registered entity names are classified and saved by category,
The entity name dictionary DB,
An entity name dictionary construction system through unsupervised learning, characterized in that the registered entity name and its meaning are classified and stored by category.

Images