KR101931714B1 - System and method for extracting named entity using similar document recommand device - Google Patents

Abstract

The present invention provides an entity name recognition system for extracting an entity name from a document using a similar document recommendation apparatus, the system comprising: an input unit for receiving an analysis target document for which object name extraction is required; a similar document recommendation And an object name extracting unit for extracting an object name from the duplicated character string by comparing the document to be analyzed and a similar document to detect a duplicate character string, wherein the similar document recommendation apparatus comprises: A document analysis unit for separating the included words into morphemes, a document vectorizing unit for re-expressing the document to be analyzed or a plurality of reference documents as vectors using the morpheme analyzed by the document analyzing unit, the result of re- And a vector of the document to be analyzed and a vector of a plurality of reference documents, Among the reference document it shall include a comparison of the number of similar documents to extract and analyze the target document.

Description

TECHNICAL FIELD The present invention relates to a system and a method for recognizing an object name from a document using a similar document recommendation apparatus,

The present invention relates to a method for extracting an object name using a language processing technique.

Information on products and services provided by companies is information that is created from the perspective of the enterprise. However, recently, various social network services (SNS) have been exploited to generate customer-oriented experience information. Therefore, companies are rapidly and accurately analyzing information that is mass-produced through social network services and listen to the voice of customers reflected in the information.

Named Entity Recognizing is known as a technique for accurately analyzing customer-centric experience information. An object name is a word or a set of letters and numbers, such as a name of a person, an organ, a technique, and a product name such as a serial number. A technique for recognizing an entity name is disclosed in Korean Patent Laid-Open Publication No. 10-2009-0124980 entitled " Object Name Dictionary Construction System and Construction Method ". However, since the conventional art is a method of recognizing an entity name based on statistics after machine learning through a natural language processing process of document data, a considerable amount of time and effort are required for the learning process and the performance tuning process.

The inventors of the present invention have conducted research and efforts to solve the above problems, and have completed the present invention.

The inventors of the present invention propose a system for effectively extracting an object name using a similar document recommendation apparatus.

On the other hand, other unspecified purposes of the present invention will be further considered within the scope of the following detailed description and easily deduced from the effects thereof.

According to an aspect of the present invention, there is provided an information processing apparatus comprising: an input unit receiving an analysis target document that requires extraction of an entity name;

A similar document recommendation device for extracting a similar document similar to the analysis target document; And

And an object name extracting unit for extracting the object name from the duplicated character string by comparing the document to be analyzed and the similar document,

The similar document recommendation apparatus

A document analyzer for separating words included in the analysis target document or the plurality of reference documents into morphemes;

A document vectorization unit for re-expressing a document to be analyzed or a plurality of reference documents as a vector using a morpheme analyzed by the document analysis unit;

A database for storing a result of re-rendering a plurality of reference documents as a vector; And

And a comparator for comparing the vector of the document to be analyzed and the vectors of the plurality of reference documents and extracting the document to be analyzed and the similar document from among the plurality of reference documents, The object recognition system of the present invention includes:

In a preferred embodiment, the document analyzer comprises: an analysis module for stemming the document;

A tagging module for tagging parts of words included in a document; And

Nouns, verbs, adjectives. It is preferable to include a filter module for removing part of speech other than a substantial morpheme including adverbs.

In a preferred embodiment, the document vectorization unit may include a substitution module for digitizing the morpheme input from the document analysis apparatus using a word embedding technique into a vector; And

And an expression module for expressing the entire document as a vector using the vector numerical value of the morpheme input from the substitution unit.

A second aspect of the present invention provides a similar document recommendation apparatus comprising: analyzing a plurality of reference documents and re-expressing them as a vector;

The similar document recommendation apparatus analyzes the document to be analyzed and re-expresses the document as a vector;

The similar document recommendation apparatus compares the vector of the document to be analyzed with the vector of the reference document stored in the database to extract a similar document similar to the document to be analyzed among the plurality of reference documents;

Detecting an overlapping character string by comparing the document to be analyzed with a similar document by the object name recognition system;

The object name recognition system includes a step of determining a range of object names in a duplicated character string and extracting the object name, wherein the object name recognition method extracts the object name from the document using the similar document recommendation apparatus .

The present invention can efficiently extract the entity name using the similar document recommendation apparatus. Therefore, there is no need of a separate pre-establishment procedure, which saves time and money.

In addition, the present invention can easily cope with a change in language such as the ambiguity, abbreviated expression, and coined word of a word, which is different for each domain, by comparing the similar document derived by using the similar document recommendation apparatus with the analysis subject document. That is, it is possible to flexibly cope with the change of language compared with the technique of extracting the object name by defining the dictionary and the rule in advance.

In addition, since similar documents that can be compared using the similar document recommendation apparatus are selected, it is not necessary to classify similar documents for each domain and category, so that it is efficient in terms of time and cost.

Further, the present invention has an advantage that data can be quickly processed in real time because a similar document is searched in a database built in advance.

On the other hand, even if the effects are not explicitly mentioned here, the effect described in the following specification, which is expected by the technical features of the present invention, and its potential effects are treated as described in the specification of the present invention.

1 is a view showing an embodiment in which the entity name recognition system of the present invention cooperates with a similar document recommendation system through a network.
2 is a diagram showing an embodiment in which the entity name recognition system of the present invention includes a similar document recommendation apparatus.
3 is a diagram showing a preferred embodiment of the similar document recommendation apparatus of the present invention.
4 is a diagram showing a preferred embodiment of the object name extracting unit of the present invention.
5 is a diagram illustrating a preferred embodiment for determining entity names in the present invention.
6 and 7 are views showing a preferred embodiment of the character index structure.
It is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may obscure the subject matter of the present invention.

1 is a view showing an embodiment in which the entity name recognition system of the present invention cooperates with a similar document recommendation system through a network.

As can be seen from FIG. 1, the entity name recognition system 100 of the present invention can coordinate with the similar document recommendation system 200 through a network. That is, when the analysis target document requesting the entity name recognition is obtained from the user terminal 10, the analysis target document is sent to the similar document recommendation system 200 existing in the outside and the document similar to the analysis target document is recommended. For this purpose, the object name recognition system 100 may include a module for wired or communication with the similar document recommendation system 200.

Meanwhile, in another embodiment, the entity name recognition system 100 of the present invention can be designed to include a similar document recommendation system. When the similar document recommendation system is included in the object name recognition system, it is referred to as a similar document recommendation apparatus in order to distinguish it from the outside through the network. Hereinafter, an example in which the object name recognition system includes a similar document recommendation apparatus will be described.

2 is a diagram showing an embodiment in which the entity name recognition system of the present invention includes a similar document recommendation apparatus.

As shown in FIG. 2, the entity name recognition system 100 includes an input unit 110, a similar document recommendation apparatus 120, and an entity name extraction unit 130.

The input unit 110 receives the analysis target document from which the entity name extraction is requested from the user terminal 10. [

The similar document recommendation apparatus 120 receives the analysis target document from the input unit 110 and extracts a similar document similar to the analysis target document. The extracted similar document is transmitted to the object name extracting unit 130.

The object name extracting unit 130 compares the document to be analyzed and a similar document to detect duplicate characters, and extracts the object names from the duplicated characters.

3 is a diagram showing a preferred embodiment of the similar document recommendation apparatus of the present invention.

3, the similar document recommendation apparatus 120 includes a document analysis unit 121, a document vectorization unit 123, a database 125, and a comparison unit 127. As shown in FIG.

The document analyzer 121 separates words included in the analysis target document or the plurality of reference documents into morphemes. To this end, in a preferred embodiment, the document analyzer comprises an analysis module for stemming the document, a tagging module for tagging the part of speech contained in the document, and a noun, verb, adjective. And a filter module for removing part of speech other than the substantial morpheme including adverbs.

The document vectorizing unit 123 re-expresses the analysis target document or the plurality of reference documents as a vector by using the morpheme analyzed by the document analysis unit. To this end, in a preferred embodiment, the document vectorization unit includes a substitution module for digitizing the morpheme input from the document analysis apparatus using a word embedding technique, and a substitution module for substituting the entire document As a vector.

The database 125 stores the result of re-rendering the plurality of reference documents as a vector.

The comparing unit 127 compares the vector of the document to be analyzed and the vectors of the plurality of reference documents to extract the document to be analyzed and the similar document from among the plurality of reference documents.

4 is a diagram showing a preferred embodiment of the object name extracting unit of the present invention.

4, the object name extraction unit 130 may include a string extension module 131, a string search module 133, a string index module 135, and an object name determination module 137.

The character string extension module 131 sequentially specifies a character string from the beginning of the document to be analyzed. The string is expanded by one character from the start position. The maximum string length can be specified in the condition. The string can contain spaces.

For example, assume that the first sentence of the document to be analyzed has the following sentence.

Example: 'Five years of civil war'

The string expansion module 131 expands the character sequence one character at a time starting from the starting character '5' of the above example sentence. An extended character string is '5 years', and an extended character string is '5 years'. The string expansion module 131 expands the string until the string does not appear in the similar document. That is, if the document to be analyzed and the similar document have the same string, the extension is continued. If the same string does not exist, the expansion is stopped. For example, if the similar document contains the string '5', it extends the string. If the next string, '5 years', is in the similar document, then extend the string again. However, if the similar document does not contain the string '5 years', it stops expansion.

If the expansion is interrupted, start the string expansion again from the first character of the next word, 'my'. String expansion can be aborted based on sentence structure, punctuation marks such as periods. In addition, it is possible to set the maximum extent of the extension in morpheme units, rather than expanding to the next word.

The string search module 133 searches whether the character string determined by the string string module 131 is within the similar document. For example, if the string determined by the string expansion module 131 is '5', the character string '5' is searched for in the similar document. If it exists, it searches the similar document for the string '5 years' extended by the string extension module 131. The string search module 133 may provide a search result after removing a space if a space is included in the string.

The string index module 135 stores a string existing in the similar document through the string search module 133 among the strings determined by the string extension module 131. [ That is, a character string common to the character string determined in the analysis target document and the character existing in the similar document is stored. The string index module 135 stores each character contained in the string in a hierarchical tree structure.

For example, it is assumed that the common strings found through the string extension module 131 and the string search module 133 are '5 years', 'civil war', 'Syria', 'in Syria', and 'Syrian human rights observatory'. The string '5 years' appeared only once in one similar document, the string 'Syria' appeared 21 times in six similar documents, the string 'in Syria' appeared three times in three similar documents, Human Rights Observatory "is assumed to have appeared twice in three similar documents. Suppose that the string '5 years' is 1, 'Syria' is 3, 'Syria is 2', and 'Syrian Human Rights Observatory' appears once in the document to be analyzed. In this case, the string index module 135 can store the string index structure in the form of a hierarchical tree structure as shown in FIG. That is, the string index module 135 arranges the strings common to the analysis target document and the similar document in a hierarchical tree structure, and displays the frequency numbers in the corresponding strings. The frequency can be expressed by the following structure.

[String, TF / DF / QTF]

TF is the frequency of occurrence of the string in the similar document, DF is the number of similar documents in which the string appears, and QTF is the frequency of occurrence of the string in the document to be analyzed.

The object name determination module 137 processes the result of the color date generated by the string index module 135 through a decision function to finally determine the object name. The decision function computes the object name weight for each character string stored in the index result generated by the string index module 135. The weight may be a frequency in the document to be analyzed. However, in other embodiments, the weights may be the number of similar documents, the number of similar documents, and the length of the character string in addition to the frequency of the document to be analyzed.

The decision function in the preferred embodiment is as follows.

According to the above decision function, the weight is calculated by adding the resultant value of the defined formula using the occurrence frequency (QTF) of the character string in the document to be analyzed, the occurrence frequency (TF) in the similar document and the number of similar documents (DF) And the log value of the depth of the string. The decision function increases as the number of occurrences in the similar document and the number of occurrences of similar documents decreases. In addition, the longer the length of the string, the higher the weight.

The result of calculating the weight by the above method is shown in FIG. The weight is the number after the string.

The object name determination module 137 calculates a weight on the string as shown in FIG. 7 and extracts the object name candidates in descending order of weight. For example, "Syria + Human Rights Observatory", "Syria +", "Syria" and "5 years" can be proposed as candidates for individual names. In the post-processing process, the candidates for the object name may include a string with no branch such as '5 years' and a string with a branch such as 'Syria + Human Rights Observatory'. If the last string in the '+' sign that indicates separation matches the search result, then the last string can be omitted. Meanwhile, the entity name determined by the entity name determination module 137 may be provided to the user terminal through the output unit. The results provided may include similar documents received from the similar document recommendation device in addition to the entity name. Weights of entity names can also be provided and can be sorted in descending order according to weights.

5 is a diagram illustrating a preferred embodiment for determining entity names in the present invention.

When the analysis target document is received from the user terminal, the similar document recommendation device receives it and recommends similar documents. To this end, the similar document recommendation apparatus analyzes a plurality of reference documents and re-expresses them as a vector, analyzes the document to be analyzed and re-expresses it as a vector, compares the vector of the document to be analyzed and the vector of the reference document stored in the database , And extracting similar documents similar to the analysis target document from among the plurality of reference documents.

The entity name recognition system obtains a similar document (S1100), and detects the duplicated character string by comparing the document to be analyzed and the similar document. The object name recognition system designates the initial character string (S1200). And searches for a specified character string (S1300).

If the designated character string exists, the character string is stored in the character string index (S1400), and the character string is expanded by one character (S1410).

If the designated character string does not exist, it is determined whether the character string can be moved to the next word (S1500). If it is movable, the character string is moved to the next word (S1510). If the character is not movable, the object name is finally determined (S1600).

For reference, the method of the present invention can be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, magneto-optical media such as floptical disks, A hard disk drive, a flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, as well as machine accords such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The scope of protection of the present invention is not limited to the description and the expression of the embodiments explicitly described in the foregoing. It is again to be understood that the present invention is not limited by the modifications or substitutions that are obvious to those skilled in the art.

Claims (4)

An input unit for receiving an analysis target document for which object name extraction is required;
A similar document recommendation device for extracting a similar document similar to the analysis target document; And
And an object name extracting unit for extracting the object name from the duplicated character string by comparing the document to be analyzed and the similar document,
The similar document recommendation apparatus
A document analyzer for separating words included in the analysis target document or the plurality of reference documents into morphemes;
A document vectorization unit for re-expressing a document to be analyzed or a plurality of reference documents as a vector using a morpheme analyzed by the document analysis unit;
A database for storing a result of re-rendering a plurality of reference documents as a vector; And
And a comparison unit that compares a vector of the document to be analyzed with a vector of a plurality of reference documents to extract a document to be analyzed and a similar document from among a plurality of reference documents,
The object-
An object name is determined based on a frequency of appearance of a character string in a document to be analyzed, an appearance frequency of the character string in the similar document, a number of similar documents in which the character string appears, and a length of the character string, Wherein the similar document recommendation apparatus is characterized in that the appearance frequency of the character string in the document is inversely proportional to the appearance frequency of the character string in the similar document and the number of similar documents in which the character string appears, An object name recognition system for extracting object names. The method according to claim 1,
The document analysis unit
An analysis module for stemming documents;
A tagging module for tagging parts of words included in a document; And
Nouns, verbs, adjectives. And a filter module for removing parts of speech other than the substantial morpheme including adverbs. A system for recognizing an object name for extracting an object name from a document using a similar document recommendation apparatus. The method according to claim 1,
The document vectoring unit
A substitution module for digitizing the morpheme inputted from the document analyzing device into a vector by using a word embedding technique; And
And an expression module for expressing the entire document as a vector by using a vector numerical value of a morpheme input from the substitution unit, wherein the object name is extracted from the document using the similar document recommendation apparatus. The similar document recommendation apparatus analyzes a plurality of reference documents and re-expresses them as a vector;
The similar document recommendation apparatus analyzes the document to be analyzed and re-expresses the document as a vector;
The similar document recommendation apparatus compares the vector of the document to be analyzed with the vector of the reference document stored in the database to extract a similar document similar to the document to be analyzed among the plurality of reference documents;
Detecting an overlapping character string by comparing the document to be analyzed with a similar document by the object name recognition system;
Wherein the object name recognition system determines a range of object names in a duplicate string and extracts an object name,
The extracting of the entity name may include extracting an entity name by a weight using an appearance frequency of a character string in the document to be analyzed, a frequency of appearance of the character string in the similar document, a number of similar documents in which the character string appears, Wherein the weight is inversely proportional to an occurrence frequency of a character string in the analysis target document, a frequency of occurrence in which the character string appears in the similar document, a length of the character string, and a number of similar documents in which the character string appears. A method for recognizing an entity name from a document using a similar document recommendation apparatus.

Images