KR102269737B1 - Information Classification Method Based on Deep-Learning And Apparatus Thereof - Google Patents

Abstract

The following embodiment relates to a deep learning-based information classification method. A deep learning-based information classification method according to an embodiment comprises: collecting web pages related to an entity; Corresponding to each of the web pages, the corresponding web page features are extracted according to the importance of at least one word included in the corresponding web page in the set of web pages, and the normalized frequency of the web page including the corresponding word among the web pages. to do; determining, based on characteristics of the web pages, frequencies indicative of the degree to which the web pages are associated with an entity; classifying the web pages into relevance data and non-relevance data for an entity based on the frequencies of the web pages; and generating a training data set using the classification result and features.

Description

Information Classification Method Based on Deep-Learning And Apparatus Thereof

The following embodiment relates to a deep learning-based information classification method and an apparatus for performing the same.

With the fourth industrial revolution, artificial intelligence-based data mining techniques that extract valuable knowledge from big data are playing an important role for technological innovation and corporate productivity improvement.

All information about an entity such as a movie or a hotel exists on the web, and in order to build knowledge of an entity without human intervention, it is necessary to collect numerous web pages and reveal the relationship between the web page and the entity.

Entity has problems such as multiple entities with similar names, ambiguous meanings, few entities in web pages, and entity changes over time. Also, in the real world, web pages are not just static data, but complex dynamic data that changes fluidly over time.

Traditional classification techniques such as Support Vector Machines (SVM) and ensemble models have difficulties in developing distinctive features that distinguish entities by domain experts. On the other hand, deep learning techniques do not need to develop features because they extract features from the model itself, and they solve complex nonlinear problems well, but deep learning techniques have overfitting problems due to the complexity of the model.

Methods for building knowledge of entities by using AI-based data mining techniques to solve these problems have been proposed.

In an embodiment, it is to provide a deep learning-based information classification method for automatically generating and learning a learning data set for building knowledge of an entity, and an apparatus for performing the same.

collecting web pages related to the entity; Corresponding to each of the web pages, the corresponding web page according to the importance of at least one word included in the corresponding web page in the set of web pages, and the normalized frequency of web pages including the corresponding word among the web pages extracting features; determining, based on characteristics of the web pages, frequencies indicative of the degree to which the web pages are associated with the entity; classifying the web pages into relevance data and non-relevance data for the entity based on the frequencies of the web pages; and generating a training data set by using the classification result and the features, a deep learning-based information classification method may be provided.

The collecting of the web pages may include: generating at least one query that can be combined using attributes of the entity; and collecting the web pages for the at least one query through a web search engine.

The classifying into relevance data and non-relevance data with the entity includes classifying web pages having a frequency greater than or equal to a standard as the relevance data by using the frequency distribution of the web pages, and a web having a frequency lower than the standard. Pages may include classifying the irrelevant data.

The extracting of the features of the corresponding web page may include: calculating a relevance to the entity by substituting words included in the corresponding web page into the following equation; and extracting words having a high calculated value among the words for which the relevance is calculated as the feature according to a predetermined criterion.

Formula:

-score(w,t) is the relevance dl of the entity of the word t in the web page w, α is the parameter for weighted average of the first and second terms, and tf-id(t) is the relevance dl of the word t in the web page group W The word t is a statistical number indicating the degree to which the word t contained in the web page w is significant, freq(w) is the frequency of the web page w containing the word t, and ∑freq(w') is the number of elements contained in the document family W The sum of the frequencies of web pages w', the left term is the importance of the word t in the document group W, and the right term is the normalized frequency value of the web page w containing the word t-

learning an information classification model based on the training data set; classifying the web pages based on the learned information classification model; and providing the information classification model and the training data set.

Classifying the web pages based on the learned information classification model may include classifying the entity and relevance for the web pages.

An apparatus for deep learning-based information classification, comprising: one or more processors; Memory; and one or more programs stored in the memory and configured to be executed by the one or more processors, the programs comprising: collecting web pages related to an entity; Corresponding to each of the web pages, the corresponding web page according to the importance of at least one word included in the corresponding web page in the set of web pages, and the normalized frequency of web pages including the corresponding word among the web pages extracting features; determining, based on characteristics of the web pages, frequencies indicative of the degree to which the web pages are associated with the entity; classifying the web pages into relevance data and non-relevance data for the entity based on the frequencies of the web pages; and generating a training data set using the classification result and the features.

In an embodiment, it is possible to provide a deep learning-based information classification method for automatically generating and learning a learning data set for building knowledge of an entity, and an apparatus for performing the same.

1 is a flowchart of a deep learning-based information classification method according to an embodiment.
2 is an example of a method for generating all combinable queries, according to one embodiment.
3 is a conceptual diagram for explaining division into relevance or non-relevance according to frequency according to an embodiment.
4 is an automatic learning data generation algorithm for explaining a method for automatically generating a training data set according to an embodiment.
5 illustrates a modeled information classification model according to an embodiment.
6 is a block diagram of an apparatus for performing a deep learning-based information classification method according to an embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed herein are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiment according to the concept of the present invention These may be embodied in various forms and are not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one element from another element, for example, without departing from the scope of rights according to the concept of the present invention, a first element may be named as a second element, Similarly, the second component may also be referred to as the first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is mentioned that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Expressions describing the relationship between elements, for example, “between” and “between” or “directly adjacent to”, etc., should be interpreted similarly.

The terminology used herein is used only to describe specific embodiments, and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

1 is a flowchart of a deep learning-based information classification method according to an embodiment. In an embodiment, it may operate through an apparatus for performing an information classification method.

In step 110, the device collects web pages related to the entity.

In embodiments, it may receive, collect, or crawl web pages for entities. A web page may be collected by a query (search word) generated by utilizing the attributes of the entity.

A web page may refer to an expression of a certain intention, idea, or thought in writing or a symbol. For example, a web page may mean any form that includes information about an entity online.

FIG. 2 is for explaining an example of a method of generating all combinable queries by utilizing attributes on entities in the embodiment.

In an embodiment, upon receiving a web page, all combinable queries may be generated utilizing attributes for entity e. For example, it is assumed that entities and properties can be expressed in three ways as shown in Table 1. An attribute of an entity according to an embodiment may be expressed in more numbers.

In addition, queries generated by utilizing the attributes of the above entities can be represented as shown in Table 2.

Attributes and queries are generated as shown in Tables 1 and 2, and web pages are collected through a web search engine for each query, but top k web pages with high relevance to the query may be collected. For example, web pages collected through all the queries q1 to q7 may mean main web pages of the corresponding entity e.

이라고 할 때, 모든 가능한 조합은 e={

}로 표현할 수 있다.Thus, the properties of entity e are

, all possible combinations are e={

} can be expressed as

In step 120, the device classifies the collected web pages based on the frequency.

In an embodiment, the web page may be classified into relevance data and non-relevance data based on a frequency distribution of the web pages.

When the frequency of the web page is high, it can be determined that the relevance to the entity is high, and when the frequency of the web page is low, it can be determined that the relevance with the entity is low. When classifying a web page, for example, the highest n frequency and n lowest frequency web pages may be classified into relevance data and non-relevance data according to a predetermined criterion.

FIG. 3 is a conceptual diagram for explaining a method of classifying a web page as data related to an entity or data not related to an entity, according to an embodiment.

Referring to the graph of FIG. 3 , when each entity is sorted in descending order by frequency, documents with high frequency may be classified as having high relevance to the entity, and documents with low frequency may be classified as having low relevance to the entity.

In an embodiment, documents with high frequency are divided into relevance data and documents with low frequency are divided into irrelevant data based on relevance or non-relevance.

In step 130, the device extracts at least one feature of the entity from each of the classified data.

In order to extract the feature, the relevance to the entity may be calculated for each of the relevance data and the non-relevance data, and the feature may be extracted based on the calculated result.

In an embodiment, the association with an entity may be calculated based on Equation (1).

Here, score(w,t) means the relevance to the entity of the word t in the web page w, α is a parameter for calculating the weighted average of the first and second terms, and tf-idf(t) is the document group W means a statistical value indicating the degree of importance of the word t included in the web page w in , freq(w) is the frequency in the document group W of the web page w containing the word t, and ∑freq(w') is the document It is the sum of the frequencies of web pages w' included in group W. tf-idf() means an important word for classifying the web page w when the word t appears in the web page w and does not occur frequently in the document group W. freq() means the importance of a document that occurs frequently through the frequency feature of a web page w containing the word t. In Equation 1, the left term means the importance of the word t in the document group W, and the right term means the normalized frequency value of the web page w including the word t.

In an embodiment, based on a value calculated from the relevance data, a predetermined number of higher words may be extracted as words having high relevance to an entity and used as relevance characteristics. Similarly, higher words may be extracted from irrelevant data and used as irrelevant features. Relevant features and non-relevant features may be arranged in a descending order of a calculated value to constitute one feature.

In step 140, the device generates a training data set using the features.

In embodiments, a training data set may be generated using relevant and non-relevant features extracted from the relevant and non-relevant data.

4 is an example of an algorithm for automatically generating training data for explaining a method for automatically generating a training data set in an embodiment.

In an embodiment, as described above, all combinable queries may be generated by utilizing the _{attributes (a 1} to a _{n ) of the entity e.} Through the generated queries, higher-order documents that are highly related to entity e can be collected. Based on the frequency distribution of the collected documents, documents with high frequency may be classified as relevance data, and documents with low frequency may be classified as irrelevant data.

For words in the classified data, a relevance to an entity may be calculated based on Equation 1 above, and features may be extracted from the calculated relevance and non-relevance data. Through the extracted features, it is possible to generate a training data set representing relevant data and non-relevant data.

Relevance and non-relevance data, features extracted from the data, and the generated training data set can be represented as follows.

Relevance data: w1={t1, t2, t3, t4}, w2={t1, t3, t4, t5}

Unrelevant data: w3={t2, t3, t4, t5}, w5={t3, t4, t5, t6}

Top 2 measure words (word, measure) of relevance data: (t1, score(w1, t1)), (t2, score(w1, t2))

Top 2 measure words (words, measures) in unrelated data: (t5, score(w3, t5)), (t6, score(w4, t6))

Features in descending order of measurements: {t1=score(w1, t1), t2=score(w1, t2), t5=score(w3, t5), t6=score(w4, t6)}

Relevance data expressed as extracted features: w1=<relevance, t1=score(w1, t1), t2=score(w1, t2), t3=0, t4=0 >, w2=<relevance, t1=score(w1) , t1), t3=0, t4=0, t5=score(w3, t5)>

Unrelevant data expressed as extracted features: w3=<unrelevant, t2=score(w1, t2), t3=0, t4=0, t5=score(w3, t5)>, w4=<unrelevant, t3= 0, t4=0, t5=score(w3, t5), t6=score(w4, t6)>

Training dataset: w1=<relevance, t1=score(w1, t1), t2=score(w1,t2), t3=0, t4=0 >, w2=<relevance, t1=score(w1, t1), t3=0, t4=0, t5=score(w3, t5)>, w3=<relevant, t2=score(w1, t2), t3=0, t4=0, t5=score(w3, t5)> , w4=<relevance, t3=0, t4=0, t5=score(w3, t5), t6=score(w4, t6)>

A deep learning-based information classifier on which knowledge of entities trained with a training dataset is built: D

Test web page: w5 =<t5, t6, t7, t8>

Test web pages expressed as extracted features: w5=<undecided, t5=score(w3, t5), t6=score(w4, t6), t7=0, t8=0>

Results of Information Classifier D for Test Web Pages: Irrelevance

w1, w2, w3, w4, w5 represent web pages, t1, t2, t3, t4, t5, t6, t7, t8 represent words, score(w, t) is Equation 1 in the web page w represents the relevance of the word t to the entity, and D may mean a deep learning model.

In step 150, the device learns an information classification model based on the training data set.

In an embodiment, a deep learning-based information classification model may be trained through the generated training data set. For example, a training dataset is trained on a deep learning model such as a Feed-forward Neural Network (FNN), a Convolutional Neural Network (CNN), or a Recurrent Neural Network (RNN). As a learning result, a model in which knowledge of an entity that classifies an entity is built can be modeled.

5 illustrates a modeled information classification model according to an embodiment.

According to the embodiment of FIG. 5 , a deep learning model including an LSTM cell may be trained based on training data sets. x ₁ to x _{n -1} may be n words included in a feature extracted from a web page.

Output values of the LSTM layer of h ₀ to h _{n −1} may be summed and input to the softmax layer to be output as normalized values. For example, the output value of the softmax layer may include a probability of being related to an entity, a probability of not being related to an entity, and the like.

In step 160, the device classifies the web pages based on the learned information classification model.

In embodiments, it is possible to classify associations with entities for web pages based on the learned model. For example, a web page can be expressed as an extracted feature, and it can be evaluated through a learned information classification model.

In an embodiment, in order to classify the relevance to the entity, a learned information classification model and data on which the learning of the entity is completed may be provided.

6 is a block diagram of an apparatus for performing a deep learning-based information classification method according to an embodiment.

The device 600 according to the embodiment may be configured to include a memory 610 and a processor 620 , and a deep learning-based information classification method stored in the memory 610 may be executed through the processor 620 . The deep learning-based information classification method according to the embodiment is based on the method described with reference to FIGS. 1 to 5 .

The device 600 may provide web pages highly relevant to the entity by receiving (or obtaining, collecting, crawling, and processing the web page for the entity).

The device 600 may be implemented as a personal computer (PC), a data server, or a portable device. For example, a portable device may be a laptop computer, a mobile phone, a smart phone, a tablet PC, a mobile internet device (MID), a personal digital assistant (PDA), an EDA ( enterprise digital assistant), digital still camera, digital video camera, PMP (portable multimedia player), PND (personal navigation device or portable navigation device), handheld game console, It may be implemented as an e-book or a smart device. The smart device may be implemented as a smart watch, a smart band, or a smart ring.

The device 600 collects web pages related to an entity.

In embodiments, it may receive, collect, or crawl web pages for entities. A web page may be collected by a query (search word) generated by utilizing the attributes of the entity.

A web page may refer to an expression of a certain intention, idea, or thought in writing or a symbol. For example, a web page may mean any form that includes information about an entity online.

The device 600 classifies the collected web pages based on the frequency.

In an embodiment, the web page may be classified into relevance data and non-relevance data based on a frequency distribution of the web pages.

When the frequency of the web page is high, it can be determined that the relevance to the entity is high, and when the frequency of the web page is low, it can be determined that the relevance with the entity is low. When classifying a web page, for example, the highest n frequency and n lowest frequency web pages may be classified into relevance data and non-relevance data according to a predetermined criterion.

The apparatus 600 extracts at least one feature of an entity from each of the classified data.

In order to extract the feature, the relevance to the entity may be calculated for each of the relevance data and the non-relevance data, and the feature may be extracted based on the calculated result.

In an embodiment, the association with the entity may be calculated based on Equation 1 above.

In an embodiment, based on a value calculated from the relevance data, a predetermined number of higher words may be extracted as words having high relevance to an entity and used as relevance characteristics. Similarly, higher words may be extracted from irrelevant data and used as irrelevant features. Relevant features and non-relevant features may be arranged in a descending order of a calculated value to constitute one feature.

The device 600 generates a training data set using the features.

In embodiments, a training data set may be generated using relevant and non-relevant features extracted from the relevant and non-relevant data.

The device 600 trains an information classification model based on the training data set.

In an embodiment, a deep learning-based information classification model may be trained through the generated training data set. For example, a training dataset is trained on a deep learning model such as a Feed-forward Neural Network (FNN), a Convolutional Neural Network (CNN), or a Recurrent Neural Network (RNN). As a learning result, a model in which knowledge of an entity that classifies an entity is built can be modeled.

The device 600 classifies the web pages based on the learned information classification model.

In embodiments, it is possible to classify associations with entities for web pages based on the learned model. For example, a web page can be expressed as an extracted feature, and it can be evaluated through a learned information classification model.

In an embodiment, in order to classify the relevance to the entity, a learned information classification model and data on which the learning of the entity is completed may be provided.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more of these, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be permanently or temporarily embody in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (13)

collecting web pages related to the entity;
In response to each of the web pages,
Corresponding web page features using the importance of at least one word included in the web page in the set of web pages, and normalized frequencies of web pages including the corresponding word among the web pages - the characteristics are relevance characteristics and unrelated features;
determining, based on characteristics of the web pages, frequencies of the web pages;
classifying the web pages into relevance data and non-relevance data for the entity based on the frequencies of the web pages; and
generating a training data set using the classification result and the features;
containing,
Deep learning-based information classification method.
According to claim 1,
The step of collecting the web pages is,
generating at least one combinable query by utilizing attributes of the entity; and
collecting the web pages for the at least one query through a web search engine;
containing,
Deep learning-based information classification method.
According to claim 1,
The step of classifying into relevance data and non-relevance data with the entity comprises:
classifying web pages having a frequency greater than or equal to a standard as the relevance data by using a frequency distribution of the web pages, and classifying web pages having a frequency lower than the standard as the non-relevance data;
containing,
Deep learning-based information classification method.
According to claim 1,
The step of extracting the features of the corresponding web page,
calculating relevance to the entity by substituting the words included in the corresponding web page into the following equation; and
extracting words having a high calculated value among the words for which the relevance is calculated as the features according to a predetermined criterion
containing,
Deep learning-based information classification method.
Formula:

-score(w,t) is the relevance dl for the entity of the word t in the web page w, α is the parameter for calculating the weighted average of the first and second terms, and tf-id(t) in the web page group W The word t is a statistical number indicating the degree to which the word t contained in the web page w is important, freq(w) is the frequency of the web page w containing the word t, and ∑freq(w') is the number of elements contained in the document family W The sum of the frequencies of web pages w', the left term is the importance of the word t in the document group W, and the right term is the normalized frequency value of the web page w containing the word t-
According to claim 1,
learning an information classification model based on the training data set;
classifying the web pages based on the learned information classification model; and
providing the information classification model and the training data set;
further comprising,
Deep learning-based information classification method.
6. The method of claim 5,
Classifying the web pages based on the learned information classification model comprises:
classifying the entity and relevance for the web pages;
containing,
Deep learning-based information classification method.
A computer program stored in a medium for executing the method of any one of claims 1 to 6 in combination with hardware.
In the apparatus for classification of deep learning-based information,
one or more processors;
Memory; and
one or more programs stored in the memory and configured to be executed by the one or more processors;
The program is
collecting web pages related to the entity;
In response to each of the web pages,
Corresponding web page features using the importance of at least one word included in the web page in the set of web pages, and normalized frequencies of web pages including the corresponding word among the web pages - the characteristics are relevance characteristics and unrelated features;
determining, based on characteristics of the web pages, frequencies of the web pages;
classifying the web pages into relevance data and non-relevance data for the entity based on the frequencies of the web pages; and
generating a training data set using the classification result and the features;
containing,
Device.
9. The method of claim 8,
The step of collecting the web pages is,
generating at least one combinable query by utilizing attributes of the entity; and
collecting the web pages for the at least one query through a web search engine;
to do,
Device.
9. The method of claim 8,
The step of classifying into relevance data and non-relevance data with the entity comprises:
classifying web pages having a frequency greater than or equal to a standard as the relevance data by using a frequency distribution of the web pages, and classifying web pages having a frequency lower than the standard as the non-relevance data;
to do,
Device.
9. The method of claim 8,
The step of extracting the features of the corresponding web page,
calculating relevance to the entity by substituting the words included in the corresponding web page into the following equation; and
extracting words having a high calculated value among the words for which the relevance is calculated as the features according to a predetermined criterion
to do,
Device.
Formula:

-score(w,t) is the relevance dl for the entity of the word t in the web page w, α is the parameter for calculating the weighted average of the first and second terms, and tf-id(t) in the web page group W The word t is a statistical number indicating the degree to which the word t contained in the web page w is important, freq(w) is the frequency of the web page w containing the word t, and ∑freq(w') is the number of elements contained in the document family W The sum of the frequencies of web pages w', the left term is the importance of the word t in the document group W, and the right term is the normalized frequency value of the web page w containing the word t-
9. The method of claim 8,
learning an information classification model based on the training data set;
classifying the web pages based on the learned information classification model; and
providing the information classification model and the training data set;
to do more,
Device.
13. The method of claim 12,
Classifying the web pages based on the learned information classification model comprises:
classifying the entity and relevance for the web pages;
to do,
Device.

Images