KR20230022040A - Apparatus and method for identification and categorization of request message - Google Patents

Abstract

Disclosed are a method for classification of a request message and a device for performing the same. The device for classification of request message according to an embodiment comprises: a data set acquisition unit which acquires a first data set including a plurality of message data each labeled with a first label indicating whether or not a predefined request is included and a second data set including a plurality of message data each labeled with a second label representing a predefined type of request; a data preprocessor unit which performs preprocessing on each of the first data set and the second data set to generate a first training data set including a plurality of first training data from the first data set and generate a second training data set including a plurality of second training data from the second data set; a feature extraction unit which extracts features for each of the plurality of first learning data and the plurality of second learning data based on a probabilistic language model; and a model learning unit which trains a request message identification model to predict whether the predefined request for unlabeled message data is included using characteristics of each of the plurality of first learning data, and trains a request message classification model to predict the type of the predefined request for the unlabeled message data using characteristics of each of the plurality of second learning data. Accordingly, the present invention can accurately classify a plurality of sentences.

Description

Request message classification apparatus and method {APPARATUS AND METHOD FOR IDENTIFICATION AND CATEGORIZATION OF REQUEST MESSAGE}

The disclosed embodiments relate to techniques for classifying request messages.

Recently, the application of SNS (Social Network Service) for disaster response and management tasks has been studied in various ways. In particular, social media platforms such as Twitter, an American social network service, include various types of information useful for systematic response to emergency and disaster situations.

However, these social media platforms not only have a vast amount of information, but also various people with different languages and language habits write messages, identifying and requesting messages including requests for disaster among various types of messages It is necessary to classify and quickly respond to disaster situations.

Accordingly, when a disaster occurs, the need to quickly and accurately identify and classify messages uploaded through SNS is increasing.

Republic of Korea Patent Publication No. 10-2001-0064269 (published on July 9, 2001)

The disclosed embodiments are intended to provide technical means for constructing learning data necessary for identifying and classifying request messages.

According to an embodiment, an apparatus for classifying a message request message includes a first data set including a plurality of message data labeled with a first label indicating whether each predefined request is included, and a first data set each indicating the type of the predefined request. A data set acquisition unit acquiring a second data set including a plurality of message data labeled with 2 labels, performing preprocessing on each of the first data set and the second data set to obtain a plurality of data sets from the first data set. A data preprocessing unit for generating a first training data set including first training data and generating a second training data set including a plurality of second training data from the second data set, based on a probabilistic language model The pre-definition of unlabeled message data by using a feature extractor extracting a feature for each of the plurality of first training data and the plurality of second training data and a feature for each of the plurality of first training data. For learning a request message identification model for predicting whether or not a requested request is included, and for predicting the type of the predefined request for the unlabeled message data using a feature of each of the plurality of second training data and a model learning unit for learning a request message classification model.

The data pre-processing unit generates the first training data set through at least one of removing and converting some of the characters included in each of the plurality of message data included in the first data set, and The second learning data may be generated through at least one of removing and converting some of the characters included in each of the plurality of included message data.

The probabilistic language model generates a plurality of strings for each of the plurality of first training data by dividing each of the plurality of first training data into one or more preset string units, and at least a part of the plurality of strings A word set including at least one word generated by merging is generated, and a feature vector for each of the plurality of first training data is based on an appearance frequency of each of the at least one word included in the word set. can be extracted.

The probabilistic language model assigns a first weight to each of at least one word included in the word set using TF-IDF (Term Frequency-Inverse Document Frequency) to obtain a characteristic for each of the plurality of first training data vectors can be extracted.

The probabilistic language model may extract a feature vector for each of the plurality of first training data by assigning a second weight based on a preset rule to a preset word among at least one word included in the word set. there is.

The model learning unit may train the request message classification model such that a similarity between at least two pieces of data having a similar type of the predefined request among the plurality of second training data becomes smaller.

The method may further include a performance evaluation unit that evaluates performance of the request message identification model and the request message classification model based on at least one of a micro-average measurement and a macro-average measurement.

According to an embodiment, a method of classifying a message request message is a method performed in a computing device having one or more processors and a memory storing one or more programs executed by the one or more processors, each of which predefined requests. A first data set including a plurality of message data labeled with a first label indicating whether to include a first data set and a second data set including a plurality of message data each labeled with a second label indicating a type of predefined request obtaining, generating a first training data set including a plurality of first training data from the first data set by performing preprocessing on each of the first data set and the second data set; generating a second training data set including a plurality of second training data from the set; extracting a feature for each of the plurality of first training data and the plurality of second training data based on a probabilistic language model; and learning a request message identification model for predicting whether or not the predefined request for unlabeled message data is included using features of each of the plurality of first training data, and learning the plurality of second training data. and training a request message classification model to predict the type of the predefined request for the unlabeled message data using the feature for each.

The generating may include generating the first training data set through at least one of removing and converting some of characters included in each of a plurality of message data included in the first data set, and generating the second data set. The second learning data may be generated through at least one of removing and converting some of the characters included in each of the plurality of message data included in .

The probabilistic language model generates a plurality of strings for each of the plurality of first training data by dividing each of the plurality of first training data into one or more predetermined string units, and at least some of the plurality of strings A word set including at least one word generated by merging is generated, and a feature vector for each of the plurality of first training data is based on an appearance frequency of each of the at least one word included in the word set. can be extracted.

The probabilistic language model assigns a first weight to each of at least one word included in the word set using term frequency-inverse document frequency (TF-IDF) to obtain a characteristic for each of the plurality of first training data vectors can be extracted.

The probabilistic language model may extract a feature vector for each of the plurality of first training data by assigning a second weight based on a preset rule to a preset word among at least one word included in the word set. there is.

In the training, the request message classification model may be trained so that a similarity between at least two pieces of data having a similar type of the predefined request among the plurality of second training data becomes smaller.

The method may further include a performance evaluation unit that evaluates performance of the request message identification model and the request message classification model based on at least one of a micro-average measurement and a macro-average measurement.

According to the disclosed embodiments, it is possible to identify whether a predefined request is included in message data of a very short length uploaded on SNS, such as a tweet of Twitter, and classify the message according to the type of the predefined request. .

According to the disclosed embodiments, it is possible to accurately classify a plurality of sentences that can be classified with similar meanings with respect to sentences made of short words.

1 is a block diagram for explaining a request message classification apparatus according to an embodiment
2 is an exemplary diagram of pseudocode for explaining a method of constructing learning data according to an exemplary embodiment;
3 is a block diagram for explaining a probabilistic language model for feature vector extraction according to an exemplary embodiment;
4 is a flowchart for explaining a method of classifying a request message according to an embodiment
5 is a block diagram for illustrating and describing a computing environment including a computing device according to an exemplary embodiment;

Hereinafter, specific embodiments will be described with reference to the drawings. The detailed descriptions that follow are provided to provide a comprehensive understanding of the methods, devices and/or systems described herein. However, this is only an example and disclosed embodiments are not limited thereto.

In describing the embodiments, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the disclosed embodiments, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the disclosed embodiments, which may vary according to intentions or customs of users or operators. Therefore, the definition should be made based on the contents throughout this specification. Terminology used in the detailed description is only for describing the embodiments and should in no way be limiting. Unless expressly used otherwise, singular forms of expression include plural forms. In this description, expressions such as "comprising" or "comprising" are intended to indicate any characteristic, number, step, operation, element, portion or combination thereof, one or more other than those described. It should not be construed to exclude the existence or possibility of any other feature, number, step, operation, element, part or combination thereof.

1 is a block diagram illustrating an apparatus for classifying a request message according to an exemplary embodiment.

The request message classification apparatus 100 according to an embodiment identifies whether a predefined request is included in any message data, and if the predefined request is included, a message is sent based on the type of the identified request. classify the data

Message data refers to postings of very short length that are uploaded on social media, for example, tweets on Twitter.

As shown, the request message classification apparatus 100 according to an embodiment includes a learning data set acquisition unit 110, a data pre-processing unit 120, a feature extraction unit 130, and a model learning unit 140. .

The data set acquisition unit 110 acquires a first data set labeled with a first label indicating whether a predefined request is included and a second data set labeled with a second label indicating the type of the predefined request.

According to an embodiment, the learning data set acquisition unit 110 identifies whether a message input by a user includes a predefined request, and clearly classifies the type of request included in the message according to the type of the predefined request. To do this, a data set labeled with a label indicating whether or not a predefined request is included in the request message classification apparatus 100 and a data center labeled with a label indicating the type of the predefined request may be obtained.

Here, according to Cambridge and Oxford dictionaries, request means to ask for something or to suggest that someone do something in a polite and social way. However, as people use creative and irregular language while composing messages and express themselves in unstructured, non-grammatical and concise ways, it is necessary to extend the semantics of requests to provide assistance in effectively identifying them.

Thus, a predefined request can be defined by extending 'Request' defined in the Cambridge Dictionary or the Oxford Dictionary, where a predefined request is, for example, 'I am injured and I need a first aid kit (I am injured). Declarative requests such as “have an injury, need first aid box”, interrogative requests such as “could you bring first aid box?”, and earnest requests. Imperative requests can include three basic requests, and an earnest request can include, for example, a command such as 'Get me first aid box', a command such as 'Bring me a first aid box' It can be extended to include two sub-requests of a dictionary meaning request such as 'me first aid box, please'.

Kinds of predefined requests are categories for classifying predefined requests, for example, food requests, medical requests, shelter requests, clothes requests, money requests, resource requests, etc. may contain requests.

According to an embodiment, in detail, the data set acquisition unit 110 may acquire a first data set labeled with a first label indicating whether a predefined request is included. That is, the first data set may be composed of two classes: with request and without request.

Also, according to an embodiment, the data set acquisition unit 110 may acquire a second data set labeled with a second label indicating a predefined request type. That is, the second data set may consist of, for example, six classes of resources: food, medical care, shelter, clothing, and money.

The data preprocessor 120 performs preprocessing on each of the first data set and the second data set to generate a first training data set from the first data set and second training data from the second data set.

According to an embodiment, the data pre-processing unit 120 identifies characters included in one or more data included in each of the first data set and the second data set, and removes or converts some of the identified characters through at least one of training data can be generated.

According to an embodiment, the data pre-processing unit 120 may generate learning data by removing or converting some of the characters identified from one or more pieces of data according to a preset sequence.

2 is an exemplary diagram of pseudocode for explaining a method of generating learning data according to an exemplary embodiment.

Specifically, FIG. 2 shows data processing algorithm 1 performed by the data pre-processing unit 120 .

Referring to FIG. 2, Algorithm 1 (200) identifies characters included in one or more data based on input/output data information and parameter information (210) for performing Algorithm 1 and a preset sequence, and identifies some of the identified characters. It includes an execution function list 220 including an execution sequence for a plurality of functions for removing and converting, a tag generalization function 230, a character misspelling function 240 and a lemma extraction function 250.

According to an embodiment, the data pre-processing unit 120 may generate learning data by removing or converting some of the characters identified from one or more pieces of data according to a preset sequence.

Meanwhile, according to an embodiment, the preset order may be performed in the same order as the function execution order 220 of Algorithm 1.

First, the data preprocessing unit 120 may remove NON-ASCII characters included in the above data through the RemovingNonASCIICharacters(tw) function included in the data execution function list 220.

Thereafter, the data pre-processing unit 120 removes all languages other than English included in one data from which NON-ASCII characters are removed through the RemoningNonEnglishTweet (tw _NOASCII ) function included in the execution function list 220, so that all languages other than English are removed. It can reduce training time wasted due to language.

Thereafter, the data pre-processing unit 120 converts English included in one or more data from which all other languages have been removed to lowercase letters through LowerCaseConversion (tw _Eng ) included in the execution function list 220 to obtain a word based on uppercase letters. fragmentation can be prevented.

Thereafter, the data pre-processing unit 120 may remove unnecessary information by removing stopwords among English letters converted to lowercase through RemovingPunction (tw _LC ) included in the execution function list 220 .

Thereafter, the data preprocessing unit 120 may remove punctuation marks from one or more data through RemovingStopWods (tw _NoP ) included in the execution function list 220 .

Thereafter, the data preprocessing unit 120 may remove zero or single word length characters from one or more data through RemovingTweetsHavingNullSinglWordLength (tw _NoSW ) included in the execution function list 220 .

Thereafter, the data preprocessing unit 120 may generalize a tag among one or more pieces of data through the tag generalization function 230, GeneralizationOfTags(tw _L>p ), and preserve tag information.

The tag generalization function 230 may perform a function of generalizing various tags including hashtags, numbers, mentions, retweets, and hyperlinks. For example, the tag generalization function 230 converts a 'number' in one or more pieces of data to NUM, 'www.example123.com' to URL, 'RT @user-name' to RT, '@ user-name' can be replaced with MENT. For example, the tag generalization function 230 may generalize data “He is going to school @akram, www.example.com” to “He is going to school MENT URL”.

Thereafter, the data pre-processing unit 120 may correct misspellings included in English characters through the character misspelling correction function 240, SpellCorrection(tw _GoT ).

Thereafter, the data pre-processing unit 120 may convert the English letters of which tags are generalized through at least one of lemmatization and stemming.

Thereafter, the data preprocessing unit 120 may convert English letters to basic English letters through SpellCorrection(tw _GoT ), which is the lemma extraction function 250 .

Specifically, the lemma extraction function 250 may convert English letters from which stop words are removed based on a dictionary to basic English letters. For example, “He writes a letter” and “He wrote a letter” can be converted to “He writes a letter”. Stem extraction can remove or convert suffixes based on rules centering on the root.

According to an embodiment, a predetermined sequence for identifying characters included in one or more pieces of data included in each of the first data set and the second data set, and removing and converting some of the identified characters is performed based on the above-described sequence. It may be, but is not limited thereto.

The feature extractor 130 extracts features of training data included in each of the first training data set and the second training data set based on the probabilistic language model.

According to an embodiment, the probabilistic language model cuts each of one or more pieces of data included in the first training data set into string units including characters of N (Number, natural number) or less, and merges parts of the one or more cut pieces of data. A word set including at least one word may be generated by merging, and a feature vector for the first training data set may be extracted based on the word set.

Further, according to an embodiment, the probabilistic language model assigns a first weight to each of at least one character included in the word set using TF-IDF (Term Frequency-Inverse Document Frequency) for the word set to obtain a first value. A feature vector for training data can be extracted.

In addition, according to an embodiment, the probabilistic language model extracts a feature vector for the first training data by assigning a second weight based on a preset rule for a specific character among at least one character included in the word set. can

3 is a structural diagram illustrating a probabilistic language model according to an exemplary embodiment.

Specifically, FIG. 3 is a structural diagram for explaining a probabilistic language model including a modified N-gram and a rule-based structure.

According to an embodiment, the feature extractor 120 may extract features of training data included in each of the first training data set and the second training data set based on a probabilistic language model. In this case, the probabilistic language model may include a modified N-gram and a rule-based structure.

Referring to FIG. 3, the probabilistic language model includes a modified N-gram structure 310 (hereinafter referred to as 'TF structure') for calculating TF values, and a modified N-gram structure for calculating TF-IDF values ( 320) (hereinafter 'TF-IDF structure') and a rule-based feature structure 350.

According to an embodiment, the feature extractor 120 transforms N-grams, cuts each of one or more pieces of data included in the training data set into character string units including N (Number, natural numbers) or less characters, and cuts A word set including at least one word may be generated by merging one or more pieces of data. At this time, it is assumed that N is 3.

Specifically, the word set generated by the feature extractor 120 may be generated through modified N-grams. That is, the modified N-gram is cut into string units including one character (N=1), cut into string units including two characters (N=2), and string units including three characters. By cutting (N=3), Unigrams, bigrams, and trigrams are obtained, and a word set can be generated by merging Unigrams and bigrams with bigrams and trigrams, respectively.

The TF structure 310 includes a structure for extracting a feature vector for assigning a first weight to each of at least one character included in a word set based on a term frequency through a modified N-gram.

The TF-IDF structure 320 generates a feature vector for assigning a first weight to each of at least one character included in a word set based on a term frequency-inverse document frequency through a modified N-gram. Contains structure to extract.

The rule-based feature structure 330 includes a structure for extracting a feature vector for training data by assigning a second weight based on a preset rule for a special character among at least one character included in a word set. The Ruled-based Feature included in the rule-based feature structure 330 is preset and means rule-based feature extraction. Here, the preset rules are as shown in Table 1, but are not limited thereto.

One \b Represents the word boundary 2 . indicates any character except a newline 3 * Represents zero or more occurrences of the preceding 4 ? Represents zero or one occurrence of the preceding 5 | Refers to OR operator 6 \s Used to match whitespace characters i.e., tab, return etc. 7 \w Refers to word character i.e., a-z, A-Z, _ and 0-9.

The model learning unit 140 uses a request message identification model for predicting whether or not a predefined request for unlabeled data is included using the first training data set and unlabeled data using the second training data set. Train a request message classification model to predict the type of predefined request for

According to an embodiment, the model learning unit 140 may train a request message classification model such that a similarity between two or more pieces of data included in the second training data set having similar types of at least predefined requests becomes smaller.

For example, the cosine similarity of the two sentences "He loves me" and "He like me" through a general N-gram is 0.6. The model learning unit 140 A probabilistic language model including modified N-grams may be trained to reduce the similarity between “He loves me” and “He like me”. According to an embodiment, the similarity between “He love me” and “He like me” through a probabilistic language model is reduced to 0.4 and 0.333 when uni-grams+bi-grams and uni+bi+trigrams are used, respectively. can do.

Meanwhile, according to an embodiment, the request message classification apparatus 100 evaluates the performance of the request message identification model and the request message classification model based on at least one of micro-average measurement and macro-average measurement. It may further include a performance evaluation unit that does.

Table 2 contains a list of seven classifiers used for message identification and classification.

S# Classifier Model One Decision tree (DT) 2 Gradient Boosting (GB) 3 Logistic Regression (LR) 4 Multilayer Perceptron (MLP) 5 Naive Bayes (NB) 6 Support Vector Machine (SVM) 7 Random Forest (RForest)

According to an embodiment, the performance evaluation unit may evaluate performance of the request message identification model and the request message classification model based on at least one of a micro-average measurement and a macro-average measurement.

A method of aggregating measurements of micro-average instances and considering them as average values per document, where each one or more data points are weighted equally, and since micro-averages are calculated according to the number of occurrences, classes with a high number of occurrences may become more important. there is.

The precision of the micro-average can be calculated through Equation 1.

: 분류가 옳은 검출(True Positive)

: Detection with correct classification (True Positive)

: 분류가 틀린 검출(False Positive)

: Detection of incorrect classification (False Positive)

The recall of the micro-average can be calculated through Equation 2.

: 분류가 옳은 검출

: detection of correct classification

: 분류가 틀린 검출

: Detection of misclassification

: 분류되어야 하는 분류가 검출되지 않음(False Negative)

: The classification that should be classified is not detected (False Negative)

The micro-average F-score can be calculated through Equation 3.

: 마이크로 평균의 F-score

: F-score of micro-average

: 마이크로 평균의 예측값

: predicted value of micro-average

: 마이크로 평균의 재현율

: Recall of the micro-average

또는

이며

는 0이 되고, 완벽한 시스템의 경우,

,

이 되어

가 1 이 될 수 있다. At this time,

or

is

becomes 0, and for a perfect system,

,

become

can be 1.

: 분류가 옳은 검출

: detection of correct classification

: 분류가 틀린 검출

: Detection of misclassification

The recall of the macro average can be calculated through Equation 5.

: 분류가 옳은 검출

: detection of correct classification

: 분류가 틀린 검출

: Detection of misclassification

: 분류되어야 하는 분류가 검출되지 않음

: Classification that should be classified is not detected

The F-score of the macro average can be calculated through Equation 6.

: 매크로 평균의 F-score

: F-score of macro average

: 매크로 평균의 예측값

: Predicted value of the macro average

: 매크로 평균의 재현율

: Recall of the macro average

또는

이며

는 0이 되고, 완벽한 시스템의 경우,

,

이 되어

가 1 이 될 수 있다.At this time,

or

is

becomes 0, and for a perfect system,

,

become

can be 1.

), 재현율(

), F-score(

), 매크로 평균 예측값(

), 재현율(

), F-score(

)에 대한 성능 평가 결과는 다음의 표 3과 같다. The micro-average prediction value (

), recall (

), F-score(

), macro average predicted value (

), recall (

), F-score(

The performance evaluation results for ) are shown in Table 3 below.

Classifier Feature Set

A DT TF(Uni+Bi-grams)+Rules 76.66 76.53 76.66 76.35 76.42 76.66 76.66 GB TF(Uni+Bi-grams)+Rules 79.43 79.4 79.43 79.06 79.43 79.18 79.43 LR TF(Uni+Bi-grams)+Rules 82.38 82.33 82.38 82.11 82.38 82.2 82.38 MLP TF(Uni+Bi+Tri-grams)+Rules 81.03 80.91 81.03 80.86 81.03 80.88 81.03 NB TF(Uni+Bi+Tri-grams)+Rules 81.35 82.28 81.35 80.53 81.35 80.82 81.35 RF Uni-grams (TFs) 78.01 77.87 78.01 77.8 78.01 77.83 78.01 SVM _ 54.21 - 54.21 50 54.21 - 54.21

4 is a flowchart illustrating a method of classifying a request message according to an exemplary embodiment.

The method shown in FIG. 4 may be performed, for example, by the above-described request message classification apparatus 100.

In step 410, the request message classification apparatus 100 performs a first data set labeled with a first label indicating whether a predefined request is included or not and a second data set labeled with a second label indicating the type of predefined request. Acquire

In step 420, the request message classification apparatus 100 performs pre-processing on each of the first data set and the second data set to generate a first training data set from the first data set and a second training data set from the second data set. generate data

In step 430, the request message classification apparatus 100 extracts features of training data included in each of the first training data set and the second training data set based on the probabilistic language model.

In step 440, the request message classification apparatus 100 uses a request message identification model for predicting whether or not a predefined request for unlabeled data is included using the first training data set and the second training data set Train a request message classification model to predict the types of predefined requests for unlabeled data.

In the illustrated flowchart, the method is divided into a plurality of steps, but at least some of the steps are performed in reverse order, combined with other steps, performed together, omitted, divided into detailed steps, or not shown. One or more steps may be added and performed.

5 is a block diagram illustrating and describing a computing environment 10 including a computing device according to an exemplary embodiment. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components other than those described below.

The illustrated computing environment 10 includes a computing device 12 . In one embodiment, computing device 12 may be request message classification device 100 .

Computing device 12 includes at least one processor 14 , a computer readable storage medium 16 and a communication bus 18 . Processor 14 may cause computing device 12 to operate according to the above-mentioned example embodiments. For example, processor 14 may execute one or more programs stored on computer readable storage medium 16 . The one or more programs may include one or more computer-executable instructions, which when executed by processor 14 are configured to cause computing device 12 to perform operations in accordance with an illustrative embodiment. It can be.

Computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. Program 20 stored on computer readable storage medium 16 includes a set of instructions executable by processor 14 . In one embodiment, computer readable storage medium 16 includes memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other forms of storage media that can be accessed by computing device 12 and store desired information, or any suitable combination thereof.

Communications bus 18 interconnects various other components of computing device 12, including processor 14 and computer-readable storage medium 16.

Computing device 12 may also include one or more input/output interfaces 22 and one or more network communication interfaces 26 that provide interfaces for one or more input/output devices 24 . An input/output interface 22 and a network communication interface 26 are connected to the communication bus 18 . Input/output device 24 may be coupled to other components of computing device 12 via input/output interface 22 . Exemplary input/output devices 24 include a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices, and/or a photographing device. input devices, and/or output devices such as display devices, printers, speakers, and/or network cards. The exemplary input/output device 24 may be included inside the computing device 12 as a component constituting the computing device 12, or may be connected to the computing device 12 as a separate device distinct from the computing device 12. may be

Meanwhile, embodiments of the present invention may include a program for performing the methods described in this specification on a computer, and a computer readable recording medium including the program. The computer readable recording medium may include program instructions, local data files, local data structures, etc. alone or in combination. The media may be specially designed and configured for the present invention, or may be commonly available in the field of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and specially configured to store and execute program instructions such as ROM, RAM, and flash memory. Hardware devices are included. Examples of the program may include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter.

Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications are possible to the above-described embodiments without departing from the scope of the present invention. . Therefore, the scope of the present invention should not be limited to the described embodiments and should not be defined, and should be defined by not only the claims to be described later, but also those equivalent to these claims.

10: Computing environment
12: computing device
14: Processor
16: computer readable storage medium
18: communication bus
20: program
22: I/O interface
24: I/O device
26: network communication interface
100: request message classification device
110: data set acquisition unit
120: data pre-processing unit
130: feature vector extraction unit
140: model learning unit

Claims (14)

A first data set including a plurality of message data each labeled with a first label indicating whether or not a predefined request is included, and a plurality of message data each labeled with a second label indicating a type of predefined request a data set acquiring unit acquiring a second data set;
Preprocessing is performed on each of the first data set and the second data set to generate a first training data set including a plurality of first training data from the first data set, and a plurality of training data sets from the second data set. a data pre-processing unit generating a second training data set including second training data;
a feature extraction unit extracting features for each of the plurality of first training data and the plurality of second training data based on a probabilistic language model; and
A request message identification model for predicting whether or not the predefined request for unlabeled message data is included is learned using the feature of each of the plurality of first training data, and each of the plurality of second training data and a model learning unit that trains a request message classification model for predicting a type of the predefined request for the unlabeled message data using a feature for the message classification.
The method of claim 1,
The data pre-processing unit,
generating the first training data set through at least one of removing and converting some of the characters included in each of a plurality of message data included in the first data set;
and generating the second learning data through at least one of removing and converting some of the characters included in each of the plurality of message data included in the second data set.
The method of claim 1,
The probabilistic language model generates a plurality of strings for each of the plurality of first training data by dividing each of the plurality of first training data into one or more preset string units, and at least a part of the plurality of strings A word set including at least one word generated by merging is generated, and a feature vector for each of the plurality of first training data is based on an appearance frequency of each of the at least one word included in the word set. A message classifier for extracting .
The method of claim 3,
The probabilistic language model assigns a first weight to each of at least one word included in the word set using TF-IDF (Term Frequency-Inverse Document Frequency) to obtain a characteristic for each of the plurality of first training data A message classification device that extracts vectors.
The method of claim 3,
The probabilistic language model extracts a feature vector for each of the plurality of first training data by assigning a second weight based on a preset rule to a preset word among at least one word included in the word set. message classifier.
The method of claim 1,
The model learning unit,
and learning the request message classification model so that a similarity between two or more pieces of data having similar types of at least the predefined request among the plurality of second training data becomes small.
The method of claim 1,
and a performance evaluation unit that evaluates performance of the request message identification model and the request message classification model based on at least one of a micro average measurement and a macro average measurement.
one or more processors; and
A method performed in a computing device having a memory storing one or more programs executed by the one or more processors,
A first data set including a plurality of message data each labeled with a first label indicating whether or not a predefined request is included, and a plurality of message data each labeled with a second label indicating a type of predefined request obtaining a second data set;
Preprocessing is performed on each of the first data set and the second data set to generate a first training data set including a plurality of first training data from the first data set, and a plurality of training data sets from the second data set. generating a second training data set including second training data;
extracting a feature for each of the plurality of first training data and the plurality of second training data based on a probabilistic language model; and
A request message identification model for predicting whether or not the predefined request for unlabeled message data is included is learned using the feature of each of the plurality of first training data, and each of the plurality of second training data and training a request message classification model to predict the type of the predefined request for the unlabeled message data using a feature for the message.
The method of claim 8,
The generating step is
generating the first training data set through at least one of removing and converting some of the characters included in each of a plurality of message data included in the first data set;
The message classification method of generating the second training data through at least one of removing and converting some of the characters included in each of the plurality of message data included in the second data set.
The method of claim 8,
The probabilistic language model generates a plurality of strings for each of the plurality of first training data by dividing each of the plurality of first training data into one or more predetermined string units, and at least some of the plurality of strings A word set including at least one word generated by merging is generated, and a feature vector for each of the plurality of first training data is based on an appearance frequency of each of the at least one word included in the word set. A message classification method for extracting .
The method of claim 10,
The probabilistic language model assigns a first weight to each of at least one word included in the word set using TF-IDF (Term Frequency-Inverse Document Frequency) to obtain a characteristic for each of the plurality of first training data A message classification method that extracts vectors.
The method of claim 11,
The probabilistic language model extracts a feature vector for each of the plurality of first training data by assigning a second weight based on a preset rule to a preset word among at least one word included in the word set. How to classify messages.
The method of claim 8,
The learning step is
and learning the request message classification model so that a similarity between two or more pieces of data having similar types of at least the predefined request among the plurality of second training data becomes small.
The method of claim 8,
and a performance evaluation unit that evaluates performance of the request message identification model and the request message classification model based on at least one of a micro average measurement and a macro average measurement.

Images