KR102540947B1 - A document information processing system through automatic contruction of thesaurus and a document information processing method - Google Patents

Abstract

A document information processing system according to an embodiment of the present invention is a document information processing system which implements a process for processing documents for a search database. The document information processing system includes: a processing module which processes the target document to obtain processing information consisting of at least one sentence; and a thesaurus module which automatically generates a thesaurus for a reference word, which is a word extracted from the processing information. The thesaurus module can automatically generate a thesaurus by calculating the degree of similarity between words using unsupervised learning.

Description

Document information processing system and document information processing method through automatic construction of thesaurus dictionary

The present invention relates to a document information processing system and a document information processing method equipped with automatic construction of a thesaurus.

With the transition to the digital age, many analog parts are increasingly being converted into digital. This change of the times is changing the environment of households, businesses, and public institutions. In particular, recently, the change in the office is the most extreme. All documents in the office are digitized according to the paperless policy, and internal payment and management are maintained as digital documents.

Here, in order to effectively manage and search digital documents, it is important to construct an effective database. Accordingly, a process of processing documents, extracting keywords and summaries of documents, and arranging them is required. For this purpose, processes of pre-processing documents are required. In addition, effective definition and management of synonyms are very important in order to effectively classify, compare, and judge documents. However, in the past, since thesaurus was obtained from an external server, there was a problem of not effectively reflecting the clear intention and purpose of the document information processing system.

SUMMARY OF THE INVENTION The present invention is intended to solve the above problems, and to provide a document information processing system and document information processing method through automatic construction of a thesaurus in order to construct an efficient search database.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems described above, and other technical problems may exist.

A document information processing system according to an embodiment of the present invention implements a process of processing a document for a search database, wherein a target document is processed to obtain processing information consisting of at least one sentence. processing module; and a thesaurus module for automatically generating a thesaurus for reference words, which are words extracted from the processed information, wherein the thesaurus module calculates a degree of similarity between words using unsupervised learning to create a thesaurus. can be automatically generated.

In addition, the thesaurus module extracts the processed information by using at least one of a method of dividing into sentence units, a method of replacing Chinese characters with Korean characters, a method of removing individual grapheme, and a method of removing spaces before and after sentences. A preprocessor for preprocessing, a word similarity calculation unit for calculating a degree of similarity between words with respect to the reference word generated by preprocessing from the processed information, and the reference word based on the degree of similarity between words calculated by the word similarity calculation unit It may be provided with a dictionary construction unit that adds a thesaurus.

In addition, the word similarity calculation unit may convert the reference word into a vector form so that a degree of similarity between words is reflected.

The thesaurus module further includes a word similarity learning unit generating a word similarity model for converting the reference word into Word2Vec in consideration of a degree of similarity between words, wherein the word similarity learning unit performs unsupervised learning on the word A similarity model can be created.

In addition, the dictionary builder may load all learned words and generate the thesaurus based on a degree of similarity with the reference word.

In addition, the dictionary builder may generate the thesaurus based on information similar to the reference word by decoding the entire list of words learned in an encoded state after loading.

In addition, a summary extraction module for extracting a summary sentence from the processing information; and a keyword extraction module for extracting important keywords from the processing information.

In addition, the summary extraction module includes a sentence importance determination unit for calculating the importance of each sentence from the processed information, summary sentence extraction for extracting a summary from the processed information in consideration of the importance of the sentence, and the importance of the processed information on the target document. The method may further include a sentence model generation unit that generates an importance determination model for determining importance, and the sentence model generation unit may generate the importance determination model by machine learning.

In addition, the summary extraction module calculates summary accuracy of the importance judgment models using a pre-verification sentence, and selects a sentence model selection unit from among a plurality of importance judgment models based on the summary accuracy. The pre-verification sentences are further provided, and the sentence model selection unit may select the pre-verification sentences according to the document format.

A document information processing method according to an embodiment of the present invention is a document information processing method implemented by a document information processing system, comprising the steps of processing a target document by a processing module to obtain processing information consisting of at least one sentence. phosphorus acquisition step; and automatically generating a thesaurus for the reference word, which is a word extracted from the processed information, wherein the thesaurus module calculates a degree of similarity between words using unsupervised learning, and automatically generates the thesaurus. It can be.

The document information processing system and document information processing method through automatic construction of a thesaurus according to the present invention can effectively build a thesaurus.

In addition, document management efficiency can be maximized.

In addition, the management efficiency of the database can be maximized.

In addition, convenience of use can be maximized.

However, the effects of the present invention are not limited to the above-mentioned effects, and effects not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings.

1 is a relationship diagram of a document information processing system according to an embodiment of the present invention
Figure 2 is a block diagram of a document information processing system according to an embodiment of the present invention
3 is a diagram for explaining a process implemented by a thesaurus module included in a document information processing system according to an embodiment of the present invention.
4 is a diagram for explaining a process implemented by a summary extraction module provided in a document information processing system according to an embodiment of the present invention.
5 is a diagram for explaining a process implemented by a keyword extraction module provided in a document information processing system according to an embodiment of the present invention.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the presented embodiments, and those skilled in the art who understand the spirit of the present invention may add, change, delete, etc. other elements within the scope of the same spirit, through other degenerative inventions or the present invention. Other embodiments included within the scope of the inventive idea can be easily proposed, but it will also be said to be included within the scope of the inventive concept.

1 is a relationship diagram of a document information processing system according to an embodiment of the present invention, and FIG. 2 is a configuration diagram of a document information processing system according to an embodiment of the present invention.

Referring to FIGS. 1 and 2 , the document information processing system 100 according to an embodiment of the present invention may collect document data (hereinafter referred to as a document) from an external server 200 .

For example, the external server 200 may be a public institution server, a corporate server, and/or a personal server.

However, it is not limited thereto, and the type of the external server 200 can be variously modified at a level obvious to those skilled in the art.

As an example, the external server 200 may refer to a server that collects documents and processes data to calculate processing information.

The server referred to in the present invention may include other configurations for performing the server environment of the server. The server may include any type of device.

As an example, the server may be a digital device, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone, which includes a processor and a memory and arithmetic capability.

As an example, the server may be a web server. However, it is not limited to this, and the type of server can be variously changed at a level obvious to those skilled in the art.

Documents may be electronically created or managed documents within an institution.

For example, the document may include internal payment documents, report documents in which business progress is accumulated, and the like.

However, it is not limited to this, and the type of the document can be variously modified at a level obvious to those skilled in the art.

The document information processing system 100 may be connected to the external server 200 through a network to enable information communication.

Electronic documents may be documents containing data such as various texts, and may be in various formats such as various administrative documents, reports, theses, and evaluations, and may include odt, pdf, ppt, pptx, xls, xlsx, doc, docx, hwp, and hwpx. etc., and may be stored in a database.

The online network referred to in the present invention may be a core network integrated with a wired public network, a wireless mobile communication network, or a mobile Internet, etc. , HTTPS (Hyper Text Transfer Protocol Secure), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), etc. It comprehensively means a data communication network capable of transmitting and receiving data in various forms without being limited thereto.

In the document information processing system 100 according to an embodiment of the present invention, in the document information processing system 100 implementing a process of processing documents for a search database, a target document is processed to consist of at least one sentence. It may include a processing module 110 that acquires processed information, a summary extraction module 120 that extracts a summary from the processed information, and a keyword extraction module 130 that extracts important keywords from the processed information.

In addition, a thesaurus module 140 may be further included to automatically generate a thesaurus for reference words that are words extracted from the processing information.

The processing module 110 may be connected to the external server 200 for information communication and request and receive processing information.

For example, the processing module 110 may be a communication module.

As an example, the communication module may include a cellular module, a WiFi module, a Bluetooth module, a GNSS module, an NFC module, an RF module, a 5G module, an LTE module, a NB-IOT module, and/or a LoRa module.

However, it is not limited to this, and the module included in the communication module can be variously modified at a level obvious to those skilled in the art.

Processing information may be information composed of several sentences.

Here, several sentences may be summaries of one document or may be partially extracted sentences.

The thesaurus module 140 pre-processes the processed information using at least one of a method of separating into sentence units, a method of replacing Chinese characters with Korean characters, a method of removing single grapheme, and a method of removing spaces before and after sentences. A pre-processing unit 141 that calculates the degree of similarity between words with respect to the reference word generated by pre-processing from the processed information A dictionary builder 145 may be provided to add the reference word to a thesaurus based on similarity.

In addition, the thesaurus module 140 includes a tag unit 142 that attaches a tag to the reference word and a word similarity learning unit that generates a word similarity model that converts the reference word into Word2Vec in consideration of similarity between words ( 144) may be further provided.

The pre-processing unit 141 may separate each document into sentence units.

The pre-processing unit 141 may separate processed information into sentence units and replace foreign characters such as Chinese characters, English characters, and Japanese characters with Korean characters.

In addition, the pre-processing unit 141 may remove individual grapheme such as 'ㅇ' and 'ㅁ' from processed information, and may pre-process processed information by removing spaces before and after sentences.

However, it is not limited to this, and if there are other necessary pretreatment conditions, they may be added.

The tag unit 142 may generate a reference word by separating morphemes from preprocessed information.

Also, the tag unit 142 may attach an object name tag to the reference word.

The tag unit 142 may remove sentences having a length of 3000 or more.

This may be because most of the sentences with a length of 3000 or more are in the form of sentences such as tables and images.

For example, the tag unit 142 may select and use appropriate ones such as Nori, Mecab, khaiii, and Okt as morpheme analyzers.

The tag unit 142 may integer-encode the reference word to which the morpheme tag and/or the entity name tag are attached.

Here, the integer index used for encoding may be previously stored in the tag unit 142 .

The word similarity calculation unit 143 may convert the encoded reference word into a vector form so that the degree of similarity between words is reflected.

For example, the word similarity calculation unit 143 may generate a reference word as word2Vec.

The algorithm generated by word2Vec utilizes an existing known technology, and a detailed description thereof may be omitted.

Here, the word similarity calculation unit 143 may designate the number of bits as at least 3 or more.

Here, the number of bits may be designated as a fixed value for additional learning.

The word similarity learning unit 144 may generate the word similarity model through unsupervised learning.

The word similarity learning unit 144 may load a previously generated word similarity model, input a tagged reference word, and perform re-learning.

Learning parameters may be set identically to existing models.

An algorithm for unsupervised learning uses a known technique, and a detailed description thereof may be omitted.

The dictionary builder 145 may load all learned words and generate the thesaurus based on the degree of similarity with the reference word.

The dictionary builder 145 may generate the thesaurus based on information similar to the reference word by decoding the entire list of words learned in an encoded state after loading.

The dictionary construction unit 145 may load the entire learned word list in an integer encoding state.

Past reference words were learned in an integer-encoded state, and integer encoding of the learned reference words may be stored in the tag unit 142 .

The dictionary construction unit 145 may load all integer encodings of words stored in the tag unit 142 and then decode them.

The dictionary builder 145 may decode a word list expressed in integer encoding.

The dictionary constructing unit 145 may exclude morphemes tagged with designated unused morphemes from reference words.

For example, based on the Nori morpheme analyzer, the list of unused morphemes is 'UNKNOWN', 'JKB', 'JX', 'JKO', 'JKS', 'JKQ', 'JKG', 'JKC', 'JKV', 'JC ', 'EC', 'EF', 'EP', 'ETN', 'ETM', 'XPN', 'XSN', 'XSV', 'XSA', 'MAG', 'MAJ', 'IC', 'VX', 'VCP', 'VCN', 'NR', 'NP', 'NNBC', 'NNB', 'SF', 'SE', 'SSO', 'SSC', 'SC', 'SY' ', 'SL', or 'SN'. However, without being limited thereto, the list of disused morphemes may be modified at a level obvious to those skilled in the art according to the morpheme analyzer adopted.

The dictionary constructing unit 145 may exclude morphemes tagged with a designated unusable entity name.

In addition, the dictionary construction unit 145 may exclude single-letter morphemes.

However, it is not limited to this, and may be added if there are other necessary disuse conditions.

The dictionary builder 145 may extract word vectors most similar to the reference word.

For example, the dictionary builder 145 may extract 10 word vectors most similar to the reference word using word2Vec.

Here, the dictionary builder 145 may exclude words having a cosine similarity of less than 0.5 from the list of similar words.

In addition, the dictionary builder 145 may exclude words corresponding to disused morphemes from the list of similar words.

Here, the number of extracted word vectors and the criterion for cosine similarity may be adjusted according to the result.

The dictionary builder 145 may newly update the thesaurus by adjusting words similar to the reference word into one group.

The summary extraction module 120 may include a sentence importance determiner 121 that calculates the importance of each sentence from the processed information and a summary sentence extractor 122 that extracts the summary from the processed information in consideration of the importance of the sentence. .

In addition, the summary extraction module 120 determines the summary accuracy of the importance judgment models by using the sentence model generation unit 123 for generating an importance judgment model for determining the importance of the processed information on the target document and a pre-verification sentence. The sentence model selection unit 124 for calculating and selecting one of the plurality of importance judgment models based on the summary accuracy and the sentence feedback unit 125 for determining whether to update the pre-verified sentence are further provided. can be provided

In addition, the summary extraction module 120 includes a sentence normalization unit 126 that normalizes the sentence importance determined by the sentence importance determination unit 121 based on a common criterion and sentences that rank the importance of each sentence based on the importance tag. A ranking unit 127 may be further provided.

The sentence model generation unit 123 may generate an importance judgment model through machine learning.

There may be a plurality of importance judgment models.

For example, the importance judgment model may be 'SBERT' or 'TextRank'.

However, it is not limited thereto, and the type of the importance judgment model can be variously modified at a level obvious to those skilled in the art.

Machine learning can be a concept that includes deep learning.

Here, deep learning may be performed using a back propagation algorithm, which is an algorithm for updating weights of a neural network using labeled data of an output layer, but is not limited thereto.

Here, since the deep neural network and the back propagation algorithm are conventionally known, a detailed description thereof may be omitted.

In order for the sentence model generation unit 123 to create an importance determination model, a number of documents, processed information obtained by processing the documents, and data on importance ranks for each sentence in the processed information may be utilized.

The sentence model selection unit 124 may select an importance determination model to be used by the sentence importance determination unit 121 by using the pre-verification sentence.

The pre-verification sentence consists of a whole document and a summary of the full text of the document as one group, and may be composed of a plurality of groups.

Here, the pre-verification sentences may be classified according to document types.

For example, a pre-verification sentence when the document is a thesis and a pre-verification sentence when the document is a report may be different from each other.

In addition, the pre-verification sentence may consist of processed information consisting of excerpts from the full text of the document and a summary within the processed information as one group or as a plurality of groups.

The sentence model selector 124 may extract sentences having a predetermined criterion of importance from the pre-verification sentences through the importance judgment model, and compare the extracted sentences with summary sentences in the pre-verification sentences to calculate 'summarization accuracy'. .

The sentence model selector 124 may select an importance judgment model having the highest 'summarization accuracy'.

The sentence model selection unit 124 may select the pre-verification sentence according to the document format.

The sentence importance determining unit 121 may determine the importance of sentences in processed information by using the importance determining model selected by the sentence model selection unit 124 .

Here, the sentence importance determiner 121 may attach an importance tag for each sentence.

The sentence normalization unit 126 may normalize the sentence importance determined by the sentence importance determining unit 121 based on a common criterion.

The importance judgment model expresses the importance of a sentence or the semantic similarity with the entire document as a number, but the expression standard and range may be different.

This may cause errors in ranking sentences and extracting summaries later.

In order to prevent this, the sentence normalization unit 126 normalizes the importance values based on a predetermined criterion so that comparable importance values can be obtained even when different importance judgment models are used.

The sentence ranking unit 127 may determine the order of normalized sentences.

The summary sentence extraction unit 122 may extract a summary sentence by extracting sentences within a predetermined number in order of high ranking.

That is, the summary sentence extraction unit 122 may extract a number of sentences desired by the user in order of sentences of high importance, and determine them as summary sentences containing key contents.

The sentence feedback unit 125 may determine the appropriateness of the pre-verified sentence and the appropriateness of data that is a basis for generating the importance judgment model.

The sentence feedback unit 125 may request update of the pre-verification sentence when summary accuracy of the importance judgment model calculated using the pre-verification sentence is equal to or greater than a predetermined first criterion.

Specifically, the sentence feedback unit 125 may calculate summary accuracy for each importance judgment model using the full text of all pre-verified sentences or some extracted sentences.

Here, when the accuracy of at least one summary is greater than or equal to a first predetermined criterion, the sentence feedback unit 125 may request update of the pre-verification sentence.

For example, the first predetermined criterion may be 95%.

However, it is not limited thereto, and specific numerical values of the predetermined first judgment criterion may be variously modified at a level obvious to those skilled in the art.

Since there is a possibility of overfitting the similarity judgment, too high summary accuracy may not be an accurate importance judgment.

The sentence feedback unit 125 may request to check learning data of the importance determination model when summary accuracy of the importance determination model calculated using the pre-verification sentence is less than a predetermined second determination criterion.

Specifically, the sentence feedback unit 125 may calculate summary accuracy for each importance judgment model using the full text of all pre-verified sentences or some extracted sentences.

Here, when the accuracy of at least one summary is less than the predetermined second criterion, the sentence feedback unit 125 transmits a signal requesting inspection of the learning data of the importance judgment model to the user terminal of the document information processing system 100. can

Here, the terminal may refer to a device capable of processing information processing operations as a kind of computing device.

For example, the terminal may include a desktop computer, a laptop computer, a smart phone, a mobile terminal including a personal digital assistant (PDA), a portable multimedia player (PMP), a portable terminal, and/or a smart TV.

However, it is not limited thereto, and the type of terminal can be variously modified at a level obvious to those skilled in the art.

If the summary accuracy is too low, it is very likely that the document data that is the basis for generating the importance judgment model are inaccurate.

Accordingly, the sentence feedback unit 125 may improve the summary accuracy of the importance judgment model by feeding back the sentence feedback.

In addition, the sentence importance judged by the sentence importance determiner 121 is again transferred to the sentence model generator 123 and used as data for machine learning of the importance decision model.

For example, the predetermined second criterion may be 60%.

However, it is not limited thereto, and specific numerical values of the predetermined second criterion can be variously modified at a level obvious to those skilled in the art.

The keyword extraction module 130 includes a sentence similarity determiner 131 that calculates a similarity with the target document for each word of the processed information and a keyword extractor 132 that extracts keywords from the processed information in consideration of the importance of words. can be provided.

In addition, the keyword extraction module 130 includes a keyword model generation unit 133 for generating a sentence similarity determination model for determining the similarity of the processed information on the target document, and selection accuracy of the sentence similarity determination models using a pre-verified sentence. A keyword model selection unit 134 that selects one sentence similarity judgment model from among a plurality of sentence similarity judgment models based on the selection accuracy and a word feedback unit 135 that determines whether to update the pre-verified sentence by calculating ) may be further provided.

In addition, the keyword extraction module 130 includes a keyword normalization unit 136 that normalizes the similarity of words determined by the sentence similarity determination unit 131 based on a common criterion and a ranking of importance for each sentence based on an importance tag. A keyword ranking unit 137 may be further provided.

The keyword model generation unit 133 may generate a sentence similarity judgment model through machine learning.

There may be a plurality of sentence similarity judgment models.

For example, the sentence similarity judgment model may be 'SBERT' or 'TextRank'.

However, it is not limited thereto, and the type of the sentence similarity judgment model can be variously modified at a level obvious to those skilled in the art.

Machine learning can be a concept that includes deep learning.

Here, deep learning may be performed using a back propagation algorithm, which is an algorithm for updating weights of a neural network using labeled data of an output layer, but is not limited thereto.

Here, since the deep neural network and the back propagation algorithm are conventionally known, a detailed description thereof may be omitted.

In order for the keyword model generation unit 133 to generate a sentence similarity judgment model, a number of documents, processed information obtained by processing documents, and data on importance ranks for each sentence in the processed information may be utilized.

The keyword model selection unit 134 may select a similarity judgment model to be used by the sentence similarity determination unit 131 using the pre-verification sentence.

The pre-verification sentence is composed of a whole document and keywords of the document as one group, and may be composed of a plurality of groups.

Here, the pre-verification sentences may be classified according to sentence types.

For example, a pre-verification sentence when the document is a thesis and a pre-verification sentence when the document is a report may be different from each other.

In addition, the pre-verification sentence may consist of processed information consisting of excerpts from the full text of the document and a summary within the processed information as one group or as a plurality of groups.

The keyword model selector 134 may extract words having a similarity of a predetermined criterion from a pre-verification sentence through a sentence similarity judgment model, and compare the extracted words with keywords in the pre-verification sentence to calculate 'selection accuracy'. there is.

The keyword model selector 134 may select a sentence similarity judgment model having the highest 'selection accuracy'.

The keyword model selection unit 134 may select the pre-verification sentence according to the document format.

The sentence similarity determining unit 131 may use the sentence similarity determining model selected by the sentence model selection unit 124 to determine how similar a keyword in processed information is to a document.

Here, the word importance determiner may attach a similarity tag to each word in the sentence.

Here, the thesaurus described above may be used to determine how similar a keyword in processed information is to a document.

For example, the degree of similarity between the keyword and the document may be calculated based on the frequency of occurrence of words similar to the keyword in the processed information in the document.

The keyword normalization unit 136 may normalize the similarities of the words determined by the word importance determining unit based on a common criterion.

The sentence similarity judgment model expresses the sentence similarity, which is the similarity of words to the entire document, as a number, but the expression standard and range may be different.

This may cause errors when extracting keywords later.

In order to prevent this, the keyword normalization unit 136 normalizes the importance value based on a predetermined criterion so that a comparable sentence similarity value can be obtained even when different sentence similarity judgment models are used.

The keyword extraction unit 132 selects words within a predetermined number in order of highest sentence similarity ranking and selects them as keywords.

That is, the keyword extraction unit 132 may extract a number of words desired by the user in the order of words having a high sentence similarity and select them as keywords.

The word feedback unit 135 may determine the appropriateness of the pre-verified sentence and the appropriateness of data that is a basis for generating a sentence similarity determination model.

The word feedback unit 135 may request update of the pre-verification sentence when the selection accuracy of the sentence similarity judgment model calculated using the pre-verification sentence is equal to or higher than a predetermined third criterion.

Specifically, the word feedback unit 135 may calculate selection accuracy for each sentence similarity judgment model using the full text of all pre-verified sentences or partially extracted sentences.

Here, when the selection accuracy of at least one or more is equal to or higher than the predetermined third criterion, the word feedback unit 135 may request renewal of the pre-verification sentence.

For example, the predetermined third criterion may be 95%.

However, it is not limited thereto, and specific numerical values of the predetermined third criterion can be variously modified at a level obvious to those skilled in the art.

Since there is a possibility of overfitting the sentence similarity judgment, too high summary accuracy may not be an accurate importance judgment.

When the selection accuracy of the sentence similarity determination model calculated using the pre-verified sentence is less than a predetermined fourth criterion, the word feedback unit 135 may request inspection of learning data of the sentence similarity determination model.

Specifically, the word feedback unit 135 may calculate selection accuracy for each sentence similarity judgment model using the full text of all pre-verified sentences or partially extracted sentences.

Here, when at least one sorting accuracy is less than the predetermined fourth criterion, the word feedback unit 135 transmits a signal requesting inspection of the learning data of the sentence similarity judgment model to the user terminal of the document information processing system 100. can do.

Here, the terminal may refer to a device capable of processing information processing operations as a kind of computing device.

For example, the terminal may include a desktop computer, a laptop computer, a smart phone, a mobile terminal including a personal digital assistant (PDA), a portable multimedia player (PMP), a portable terminal, and/or a smart TV.

However, it is not limited thereto, and the type of terminal can be variously modified at a level obvious to those skilled in the art.

If the selection accuracy is too low, it is very likely that the document data, which is the basis for generating the sentence similarity judgment model, are inaccurate.

Accordingly, the word feedback unit 135 may improve the summary accuracy of the importance judgment model by feeding back the words.

In addition, the sentence similarity of the words determined by the word importance determining unit is transmitted to the sentence model generating unit 123 again, and may be used as data for machine learning of the importance determining model.

For example, the predetermined fourth criterion may be 60%.

However, it is not limited thereto, and specific numerical values of the predetermined fourth criterion can be variously modified at a level obvious to those skilled in the art.

The pre-verification sentences may be divided into pre-verification sentences for sentences and pre-verification sentences for keywords.

Here, the pre-verified sentence for the sentence is used in the summary extraction module 120, and the pre-verified sentence for the keyword can be used in the keyword extraction module 130.

3 is a diagram for explaining a process implemented by a thesaurus module provided in a document information processing system according to an embodiment of the present invention, and FIG. 4 is provided in a document information processing system according to an embodiment of the present invention. 5 is a diagram for explaining a process implemented by a keyword extraction module included in a document information processing system according to an embodiment of the present invention.

3, in the document information processing method implemented by a document information processing system according to an embodiment of the present invention,

A document information processing method according to an embodiment of the present invention includes an acquisition step in which a target document is processed by a processing module and processed information consisting of at least one sentence is acquired, and a summary is extracted from the processed information. It may include an extraction step, a keyword extraction step in which important keywords are extracted from the processed information, and a thesaurus automatically generated for reference words that are words extracted from the processed information.

3 is a flowchart of a thesaurus construction step implemented by the thesaurus module, and a detailed description thereof may be omitted to the extent that it overlaps with the above description.

4 is a flowchart of a summary extraction step implemented by the summary extraction module, and a detailed description thereof may be omitted to the extent that it overlaps with the above description.

5 is a flowchart of a keyword extraction step implemented by a keyword extraction module, and a detailed description thereof may be omitted to the extent that it overlaps with the above description.

In the accompanying drawings, in order to more clearly express the technical idea of the present invention, components that are not related to or detached from the technical idea of the present invention are briefly expressed or omitted.

In the above, the configuration and characteristics of the present invention have been described based on the embodiments according to the present invention, but the present invention is not limited thereto, and various changes or modifications can be made within the spirit and scope of the present invention. It is apparent to those skilled in the art, and therefore such changes or modifications are intended to fall within the scope of the appended claims.

110: processing module 120: summary extraction module
130: keyword extraction module 140: thesaurus module

Claims (10)

For a search database, in a document information processing system implementing a process of processing documents,
A processing module for obtaining processing information consisting of at least one sentence by processing the target document;
a thesaurus module automatically generating a thesaurus for reference words that are words extracted from the processing information;
a summary extraction module extracting a summary sentence from the processing information; and
A keyword extraction module for extracting important keywords from the processing information;
The thesaurus module,
A pre-processing unit that pre-processes the processed information by using a method of separating into sentence units, a method of replacing Chinese characters with Korean characters, a method of removing individual grapheme, and a method of removing spaces before and after sentences. A tag unit that separates morphemes to generate reference words and attach tags thereto, a word similarity calculation unit that calculates a degree of similarity between words with respect to the reference words, and a thesaurus of the reference words based on the degree of similarity between the reference words A word similarity learning unit for generating a word similarity model for converting the reference word into Word2Vec in consideration of a degree of similarity between words and a dictionary construction unit added to the word similarity;
The word similarity calculation unit converts the reference word into a vector form so that the degree of similarity between words is reflected,
The word similarity learning unit,
Creating the word similarity model through unsupervised learning;
Characterized in that the dictionary construction unit excludes morphemes tagged with designated unused morphemes from among the reference words.
Document information processing system.
delete delete delete According to claim 1,
The pre-construction unit,
Loading all learned words and generating the thesaurus based on the degree of similarity with the reference word.
Document information processing system.
According to claim 5,
The pre-construction unit,
Loading and then decoding the entire list of words learned in an encoded state to generate the thesaurus based on information similar to the reference word,
Document information processing system.
delete According to claim 1,
The summary extraction module,
A sentence importance determination unit for calculating the importance of each sentence from the processed information, extracting a summary for extracting a summary from the processed information in consideration of the importance of the sentence, and generating an importance judgment model for determining the importance of the processed information on the target document Further comprising a sentence model generation unit,
The sentence model generation unit,
Generating the importance judgment model by machine learning,
Document information processing system.
According to claim 8,
The summary extraction module,
Further comprising a sentence model selection unit for calculating summary accuracy of the importance judgment models using a pre-verification sentence and selecting one importance judgment model from among a plurality of the importance judgment models based on the summary accuracy;
The pre-verification sentence,
Classified according to document type,
The sentence model selection unit,
Selecting the pre-verification sentence according to the document format,
Document information processing system.
In the document information processing method implemented by the document information processing system,
an acquisition step in which a target document is processed by a processing module to obtain processing information consisting of at least one sentence;
automatically generating a thesaurus for reference words that are words extracted from the processed information;
extracting a summary from the processing information; and
Extracting important keywords from the processing information;
The step of automatically generating the thesaurus,
The processing information is pre-processed using a method of separating in sentence units, a method of replacing Chinese characters with Korean characters, a method of removing individual grapheme, and a method of removing blanks before and after sentences, and morpheme separation to create reference words and tag is attached, the reference word is converted into a vector form to calculate the degree of similarity between words, and the reference word is added to the thesaurus based on the degree of similarity between the reference words, but the unused morpheme tag specified among the reference words is used. A morpheme with is not added to the thesaurus,
The degree of similarity between the words is calculated through a word similarity model that converts the reference word into Word2Vec,
The word similarity model is generated through unsupervised learning,
Document information processing method.

Images