KR102280490B1 - Training data construction method for automatically generating training data for artificial intelligence model for counseling intention classification - Google Patents

Abstract

Disclosed is a training data construction method for an electronic device for an artificial intelligence (AI) model, which classifies counseling intentions. The training data construction method comprises: a step of extracting a plurality of keywords from one or more basic sentences registered for one of a plurality of pre-saved counseling intentions; a step of identifying a similar keyword whose similarity to one or more of the extracted plurality of keywords is equal to a critical value or higher; a step of collecting a plurality of sentences from a database based on the plurality of keywords and similar keywords; and a step of selecting one or more sentences whose similarity to the basic sentence is equal to the critical value or higher among the collected plurality of sentences. The present invention aims to provide the training data construction method for an electronic device for an AI model which classifies counseling intentions, which is able to reduce labor cost and data accumulation time.

Description

{ TRAINING DATA CONSTRUCTION METHOD FOR AUTOMATICALLY GENERATING TRAINING DATA FOR ARTIFICIAL INTELLIGENCE MODEL FOR COUNSELING INTENTION CLASSIFICATION }

The present disclosure relates to an electronic device for constructing training data, and more particularly, to an electronic device for constructing training data for an artificial intelligence model for classifying counseling intentions.

Due to the increase of non-face-to-face work due to the global spread and prolonged COVID-19, the demand for customer consultation through chatbots, website bulletin boards, or phone consultations is continuously increasing.

In particular, in the case of private companies or public institutions that aim to provide customer-oriented customized management or services, storing and analyzing customer voices through various channels such as web, app, phone, and e-mail is the first improvement in achieving this goal. It is a task.

However, there is a problem that a lot of human resources are required to analyze and classify the customer's voice as unstructured data expressed in voice or text, not in a clear form of data expressed in numbers.

In order to solve this problem, many private companies have introduced artificial intelligence engines that perform text-based analysis. However, in order for the AI engine to analyze accurate data, it is necessary to manually generate a large amount of training data and classification is required.

In the end, in order to automatically extract and classify accurate customer consultation intentions, it is more important to secure a large amount of training data classified by various consultation intentions in order for the engine to learn and reason rather than to introduce a text analysis-based artificial intelligence engine itself.

However, in the existing method of constructing counseling intention training data through keyword or sentence registration, each sentence for each counseling intention is considered and registered within the human knowledge range, or sentences for each counseling intention within the collected source data are written one by one. Since it is a method of constructing training data by selection, it has the disadvantage that the total amount of data for training the artificial intelligence model is absolutely insufficient, or data used in various expressions cannot be secured.

In this regard, automatic training to enable the artificial intelligence system to automatically classify customer intentions accurately in place of the manual selection of humans in the large amount of documents collected through a large number of online postings, chatbots, internal civil complaint boards, and phone transcripts, etc. There is a need for research that can build an accurate counseling intention classification engine by providing a data construction system and securing a large amount of training data within a short time with minimal human intervention.

Registered Patent Publication No. 10-21685040000 (Apparatus and method for analyzing consistency between sentences in text documents)

The present disclosure provides an electronic device capable of automatically generating a large amount of training data without artificial effort in constructing training data used for training an artificial intelligence model for classifying counseling intentions and a training data building method.

Objects of the present disclosure are not limited to the above-mentioned purposes, and other objects and advantages of the present disclosure that are not mentioned may be understood by the following description, and will be more clearly understood by examples of the present disclosure. Moreover, it will be readily apparent that the objects and advantages of the present disclosure may be realized by the means and combinations thereof indicated in the claims.

According to an embodiment of the present disclosure, there is provided a method of constructing training data of an electronic device for an artificial intelligence model for classifying consultation intentions, at least one basic registered consultation intention among a plurality of pre-stored consultation intentions. extracting a plurality of keywords from a sentence; identifying a first similar keyword having a similarity to at least one of the extracted plurality of keywords equal to or greater than a threshold; based on the plurality of keywords and the first similar keyword, from a database The method may include collecting a plurality of sentences, and selecting at least one first sentence having a similarity to the basic sentence or more from among the plurality of collected sentences. The artificial intelligence model may be trained based on the basic sentence and the selected first sentence.

The collecting of the plurality of sentences may include generating a search word by combining keywords of a certain ratio or more among the plurality of keywords and keywords including the first similar keyword, and performing a search on the generated search word to obtain the plurality of sentences. sentences can be obtained.

The method of constructing training data of the electronic device may include storing the selected first sentence as training data matching the counseling intention in which the basic sentence is registered.

In addition, the method for constructing training data of the electronic device may include: generating a group including keywords having a similarity level equal to or greater than a threshold value among a plurality of keywords included in the plurality of collected sentences; at least one of the plurality of collected sentences generating a plurality of reconstructed sentences by replacing the keywords included in the group with other keywords in the group within the sentences, at least one second of the plurality of reconstructed sentences having a similarity to the basic sentence equal to or greater than a threshold value The method may further include selecting a sentence. In this case, the artificial intelligence model may be trained based on the basic sentence, the selected first sentence, and the selected second sentence.

On the other hand, the method for constructing training data of the electronic device includes the steps of dividing and storing the sentences including the basic sentence and the selected first sentence into training data and verification data, and storing the sentences corresponding to the training data, based on the sentences corresponding to the training data, It may include training an artificial intelligence model to identify the consultation intention in which the basic sentence is registered, and verifying the artificial intelligence model by inputting a sentence corresponding to the verification data into the artificial intelligence model.

Here, the verifying includes inputting a sentence corresponding to the verification data into the artificial intelligence model, identifying whether the artificial intelligence model outputs the consultation intention in which the basic sentence is registered, and the artificial intelligence model When the basic sentence outputs the registered counseling intention, it may be determined that the verification of the artificial intelligence model is successful.

In addition, the method for constructing training data of the electronic device includes the steps of: extracting a plurality of keywords from the verification sentence when verification of the artificial intelligence model through at least one verification sentence included in the verification data fails; identifying a second similar keyword having a similarity to at least one of the plurality of extracted keywords equal to or greater than a threshold; collecting a plurality of sentences from a database based on the plurality of keywords and the second similar keyword; The method may include selecting at least one third sentence having a similarity to the basic sentence or more from among the plurality of sentences. In this case, the artificial intelligence model may be trained based on the verification sentence and the selected third sentence.

The training data construction method according to the present disclosure has an advantage in that a large amount of training data can be secured only by registering some sentences for each consultation intention.

In particular, in the training data construction method according to the present disclosure, there is no need for a person to manually classify the consultation intention or to construct a sentence for learning the consultation intention by a large number of personnel to construct the learning data, thereby reducing labor costs and Data accumulation time can be dramatically reduced.

1 is a diagram for schematically explaining training data construction and an artificial intelligence model of an electronic device according to the present disclosure;
2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure;
3 is a flowchart illustrating a method of constructing training data of an electronic device according to an embodiment of the present disclosure;
4 is a flowchart illustrating an operation in which an electronic device additionally generates and selects a sentence according to an embodiment of the present disclosure;
5 is a block diagram illustrating a functional configuration of an electronic device according to an embodiment of the present disclosure.

Before describing the present disclosure in detail, a description will be given of the description of the present specification and drawings.

First, terms used in the present specification and claims have been selected in consideration of functions in various embodiments of the present disclosure. However, these terms may vary depending on the intention of a person skilled in the art, legal or technical interpretation, and emergence of new technology. Also, some terms are arbitrarily selected by the applicant. These terms may be interpreted in the meanings defined in this specification, and if there is no specific term definition, it may be interpreted based on the general content of the present specification and common technical knowledge in the art.

Also, the same reference numerals or reference numerals in each drawing attached to this specification indicate parts or components that perform substantially the same functions. For convenience of description and understanding, the same reference numerals or reference numerals are used in different embodiments. That is, even though all components having the same reference number are illustrated in a plurality of drawings, the plurality of drawings do not mean one embodiment.

In addition, in this specification and claims, terms including an ordinal number such as “first” and “second” may be used to distinguish between elements. This ordinal number is used to distinguish the same or similar elements from each other, and the meaning of the term should not be construed as limited due to the use of the ordinal number. As an example, the use order or arrangement order of the components combined with the ordinal number should not be limited by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and is not intended to It is to be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In an embodiment of the present disclosure, terms such as “module”, “unit”, “part”, etc. are terms used to refer to a component that performs at least one function or operation, and these components are hardware or software. It may be implemented or implemented as a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, and are integrated into at least one processor, except when each needs to be implemented in individual specific hardware. can be implemented as

In addition, in an embodiment of the present disclosure, when a part is connected to another part, this includes not only direct connection but also indirect connection through another medium. In addition, the meaning that a certain part includes a certain component means that other components may be further included, rather than excluding other components, unless otherwise stated.

1 is a diagram for schematically explaining training data construction and an artificial intelligence model of an electronic device according to the present disclosure.

Referring to FIG. 1 , the electronic device 100 may build training data for training the artificial intelligence model 10 .

The artificial intelligence model 10 may be utilized in at least one system that provides a customer consultation service, and the present system may be a system including the electronic device 100 .

The system may provide a customer consultation service by means such as a chatbot, a website bulletin board, a phone consultation, or other online consultation, and in this case, the customer consultation service may be provided through at least one user terminal, but is not limited thereto.

When at least one text is input, the artificial intelligence model 10 may output at least one consultation intention matching the input text among a plurality of preset consultation intentions.

The text may be text received from the customer, or a message received from the customer (terminal) through at least one web page or application.

In addition, the text may be a text obtained by converting a customer's voice received during customer consultation.

The plurality of counseling intentions may correspond to various intentions such as complaints, inquiries, disability reports, compensation requests, praise, contract/cancellation, and proposals.

For example, when a text such as “Why are you doing this work?” is input, the artificial intelligence model 10 may identify the consultation intention as 'dissatisfaction'.

The artificial intelligence model 10 may include at least one network simulating a human neural network, and may be trained based on deep learning while the weights between nodes constituting each layer are updated.

The artificial intelligence model 10 may be trained based on training data including a plurality of texts (sentences, words, etc.) matching different consultation intentions, and matching the input text among the plurality of consultation intentions. It can be implemented as a classifier model for identifying counseling intentions.

The electronic device 100 may create a training environment for the artificial intelligence model 10 by generating a plurality of training data using only a small number of basic sentences registered as training data for each counseling intention.

Hereinafter, a configuration of the electronic device 100 and a method of constructing training data will be described in more detail with reference to the drawings.

2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the electronic device 100 includes a memory 110 and a processor 120 .

The electronic device 100 may be implemented as a server (system) composed of at least one computer. In addition, the electronic device 100 may correspond to various terminal devices such as a desktop PC, a notebook PC, a kiosk, a smart phone, and a tablet PC.

The memory 110 is a configuration for storing an operating system (OS) for controlling overall operations of the components of the electronic device 100 and at least one instruction or data related to the components of the electronic device 100 . .

The memory 110 may include non-volatile memory such as ROM and flash memory, and may include volatile memory such as DRAM. Also, the memory 110 may include a hard disk, a solid state drive (SSD), or the like.

The memory 110 may include information on a plurality of consultation intentions, information on basic sentences registered for each consultation intention, the aforementioned artificial intelligence model 10 , and the like. However, in the case of the artificial intelligence model 10 , at least one external device capable of training the artificial intelligence model 10 by receiving training data from the electronic device 100 rather than the memory 110 of the electronic device 100 . may be stored in

The processor 120 is a configuration for overall controlling the electronic device 100 . Specifically, the processor 120 may perform operations according to various embodiments of the present disclosure by executing at least one instruction stored in the memory 110 while being connected to the memory 110 .

The processor 120 may include one or more processors. In this case, the one or more processors may include a general-purpose processor such as a CPU, an AP, a digital signal processor (DSP), etc., a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or the like, or an artificial intelligence-only processor such as an NPU. The AI-only processor may be designed with a hardware structure specialized for training or use of a specific AI model.

Meanwhile, the electronic device 100 may further include a communication unit, a user input unit, etc. in addition to the memory 110 and the processor 120 .

The communication unit may include circuits and modules for performing communication with at least one external device in various wired/wireless communication methods.

The user input unit is configured to receive various commands or information from the user, such as receiving a basic sentence for each consultation intention from the user, receiving at least one text input, or receiving a request for training an artificial intelligence model. The user input unit may be implemented as a button, a touch sensor, a microphone, a camera, and various other sensors.

When the electronic device 100 is implemented as a server, the electronic device 100 is connected to at least one external device capable of receiving a user input (eg, various terminal devices, input devices such as a keyboard/mouse, etc.) and a communication unit. to receive user input.

3 is a flowchart illustrating a method of constructing training data of an electronic device according to an embodiment of the present disclosure.

FIG. 3 assumes a situation in which at least one basic sentence is registered for each consultation intention on the electronic device 100 .

The basic sentence is a sentence that can be a meaningful clue for determining the consultation intention, and may be registered for each consultation intention according to a user input or preset by a developer.

For example, with respect to the consultation intention 'dissatisfaction', basic sentences such as 'How the hell are you doing?', 'I'm getting annoyed', and 'The service is messed up' are displayed on the memory 110 of the electronic device 100. can be registered.

Referring to FIG. 3 , the electronic device 100 may extract a plurality of keywords from at least one basic sentence registered for one consultation intention among a plurality of pre-stored consultation intentions ( S310 ).

Specifically, the electronic device 100 may extract a plurality of keywords by processing various analyzes, such as morpheme analysis, syntax analysis, entity name recognition, and sentiment analysis, with respect to the basic sentence. In this case, the electronic device 100 may use at least one artificial intelligence model for extracting a keyword corresponding to a morpheme from within the text.

For example, the electronic device 100 may extract keywords such as 'the hell', 'work', 'how', and 'do' from the basic sentence, “How the hell are you doing?”

Then, the electronic device 100 may identify a similar keyword having a similarity to at least one of the plurality of keywords equal to or greater than a threshold ( S320 ).

In this case, the electronic device 100 converts a plurality of keywords into a vector form through at least one natural language understanding technology, and at least one similarity according to the converted vector is greater than or equal to a threshold (or a deviation is less than the threshold). Similar keywords can be identified.

To this end, the electronic device 100 may use a data structure in which a large number of morphemes, words, sentences, etc. are defined in the form of vectors having one or more dimensions.

For example, with respect to 'work', which is a keyword in a basic sentence, 'work', 'office work', 'work processing', etc. may be identified as similar keywords.

The threshold for the degree of similarity may be set differently according to the number of basic sentences registered for the corresponding consultation intention. Specifically, the electronic device 100 may set the threshold to be higher as the number of pre-registered basic sentences increases.

In addition, the threshold for the degree of similarity is the ratio of the corresponding counseling intention according to the past output history of the artificial intelligence model 10 (eg, the ratio at which the artificial intelligence model 10 selected “dissatisfaction” according to the input text). ) may be set differently. Specifically, the electronic device 100 may set the threshold to be lower as the rate at which the corresponding consultation intention is output is higher. As a result, more training data may be generated (according to a process to be described later) for high frequency counseling intentions.

As a specific example, the electronic device 100 may calculate the threshold according to the following equation (Th(i): similarity threshold for a specific consultation intention (i), N(bs _i ): consultation intention (i) The number of basic sentences registered for , R(i): the ratio at which the consultation intention (i) was output from the AI model, α: a preset constant).

In addition, the electronic device 100 may collect a plurality of sentences from the database based on a plurality of keywords and (identified) similar keywords included in the basic sentence ( S330 ).

The database may include a large number of sentences corresponding to various forms and contents, may be a database built in a system (including the electronic device 100 ) in which a customer consultation service is provided, or may be capable of communicating with the electronic device 100 . It may be a database built in at least one external system (eg, SNS, search engine, other various online DBs).

Specifically, the electronic device 100 may generate a search word based on a plurality of keywords and at least one similar keyword included in the basic sentence.

Here, the electronic device 100 generates a search word by combining keywords of a certain percentage (eg, 30%, 50%, etc.) among all keywords including a plurality of keywords and similar keyword(s) included in the basic sentence. can

In addition, the electronic device 100 may collect a plurality of sentences by performing a search for the generated search word through the database.

Then, the electronic device 100 may select at least one sentence having a similarity to the basic sentence or more from among the plurality of collected sentences ( S340 ).

Specifically, the electronic device 100 may convert each of the plurality of collected sentences into a vector form, and also convert the basic sentence into a vector form, thereby calculating a similarity between the basic sentence and each of the collected sentences.

In addition, the electronic device 100 may select at least one (collected) sentences having a similarity with a basic sentence equal to or greater than a threshold (or a deviation less than a threshold) as a result of comparison in a vector form among a plurality of collected sentences.

The electronic device 100 is based on a Recurrent Neural Network (RNN), Long Shot Term Memory (LSTM), etc. suitable for understanding the meaning of a sentence composed of sequentially successive elements (morpheme, word, phrase, punctuation mark, etc.) A model or an engine may be used, but is not limited thereto.

Here, when the similarity with the basic sentence is equal to or greater than the threshold, it may mean that the basic sentence can be replaced with the corresponding sentence.

For example, the plural sentences collected were “Please do your job properly”, “How the hell are you doing your job”, “You will see this as office work”, etc., and the similarity to the basic sentences of each sentence is 75.2% and 95.3, respectively. %, 57.3% can be assumed.

If the threshold for the similarity is 60%, the electronic device 100 may select "How the hell are you doing your work" and "I'll see you like this", etc., having a similarity greater than or equal to the threshold.

The electronic device 100 may store each of the selected sentences as training data matching the consultation intention (eg, complaint) in which the basic sentence is registered.

That is, in addition to the basic sentences already registered for a specific counseling intention (eg, dissatisfaction), selected sentences may be additionally registered (tagged) for the same counseling intention. As a result, the number of training data (sentences) for which the artificial intelligence model 10 is trained in relation to the corresponding consultation intention (eg, complaints) is automatically increased by the electronic device 100 .

And, the artificial intelligence model 10 may be trained based on training data including basic sentences and selected sentences.

Meanwhile, training of the artificial intelligence model 10 may be performed in the electronic device 100 or at least one external device.

When the artificial intelligence model 10 is included in the memory 110 of the electronic device 100 , the training of the artificial intelligence model 10 using the (increased) training data constructed as described above is performed by the electronic device 100 . It may be performed through the processor 120 of

Alternatively, when the artificial intelligence model 10 is stored in at least one external device connected to the electronic device 100 , the electronic device 100 transmits training data including the basic sentence and the selected sentence to the external device to the external device. can contribute to the training carried out in

In this way, since the process of FIG. 3 is performed for each basic sentence registered for each consultation intention, a large amount of training data for the artificial intelligence model 10 for classifying various consultation intentions can be quickly built.

Meanwhile, according to an embodiment, the electronic device 100 may generate additional training data (sentences) by reconstructing the plurality of sentences S330 collected in the process of FIG. 3 .

In relation to this, FIG. 4 is a flowchart illustrating an operation in which an electronic device additionally generates and selects a sentence according to an embodiment of the present disclosure.

Referring to FIG. 4 , the electronic device 100 determines the similarity between the plurality of keywords included in the previously collected sentences (eg, please do your job properly, how the hell are you doing your job, office work) as a threshold value. A group including the above keywords may be created (S410).

In this case, the electronic device 100 may create a group for each function by comparing similarity between keywords having the same function (noun, adverb, verb, etc.).

For example, the electronic device 100 determines the similarity of 'please', 'correctly', 'here', 'how', and 'this way', which are adverbs, among keywords (morphemes) included in the collected sentences. can be calculated. If the similarity between the keywords corresponding to the adverb is all less than a threshold (eg, 60%), the electronic device 100 may not create a group for the keywords corresponding to the adverb.

Also, the electronic device 100 may calculate the similarity of 'have' and 'than', which correspond to verbs, among keywords included in the collected sentences. If the similarity is equal to or greater than the threshold of 60.4%, the electronic device 100 may create a group including 'do' and 'than'.

Also, the electronic device 100 may calculate similarities between 'work processing', 'task', and 'office work' corresponding to nouns among keywords included in the collected sentences. If all of the calculated similarities are equal to or greater than the threshold, the electronic device 100 may create a group including corresponding keywords corresponding to nouns.

In addition, the electronic device 100 may generate a plurality of reconstructed sentences by replacing a keyword included in one (generated) group with another keyword in the same group in at least one sentence among the plurality of collected sentences. There is (S420).

For example, the electronic device 100 may replace 'than' for 'do' or 'do' for 'do' included in at least one sentence among the collected sentences. For example, 'Please do your work properly' may be changed to 'Please do your work properly'.

Also, the electronic device 100 replaces 'work' or 'office work' for 'work processing' included in at least one sentence among the collected sentences, or replaces 'task' with 'work processing' or 'office work' Or, 'office work' can be replaced with 'work handling' or 'work'.

As a result, the electronic device 100 displays “Please do your work properly”, “Please do your work properly”, “Please do your job properly”, “Please do your job properly”, “How the hell do you view your work”, “What the hell do you do? You can obtain various reconstructed sentences such as “How the hell do you do business”, “How the hell do you view office work”, “I will see work this way”, “I will do business this way”, “I will view work this way”, and so on.

In addition, the electronic device 100 may select at least one sentence having a similarity to the basic sentence of a plurality of reconstructed sentences equal to or greater than a threshold ( S430 ). In this case, the electronic device 100 may store the selected sentence as training data matching the consultation intention (eg, complaint) in which the basic sentence is registered.

As a result, in addition to the basic sentence and the sentence selected in the process of FIG. 3 , the sentence selected through the process of FIG. 4 may also be used for training the artificial intelligence model 10 .

In particular, when a plurality of reconstructed sentences are generated as shown in FIG. 4 , the secured amount of meaningful training data may be rapidly increased.

Meanwhile, although in FIG. 4 described above, the electronic device 100 generates additional sentences by using the plurality of sentences collected in S330 of FIG. 3 , the electronic device 100 performs a process S340 among the plurality of sentences collected in FIG. 3 . Processes S410 to S430 may be performed using only sentences selected through .

Meanwhile, according to an embodiment, the electronic device 100 may store the sentences including the basic sentence and the selected sentences ( FIGS. 3 and/or 4 ) by dividing the sentences into training data and verification data.

As an embodiment, the electronic device 100 may arbitrarily divide and store the sentences including the basic sentence and the selected sentences into training data and verification data.

As a specific example, the electronic device 100 may define (random) 80% of the sentences as training data and store the remaining 20% as verification data, but is not limited thereto.

In another embodiment, the electronic device 100 lists the selected sentences in order according to the degree of similarity to the basic sentence, and according to a constant frequency (eg, one sentence per two sentences, eight sentences per ten sentences, etc.), training data You can choose the sentence that corresponds to . And, sentences other than the training data become verification data.

As a specific example, the electronic device 100 may sequentially store one sentence per two sentences as training data and store the remaining sentences as verification data in a state in which the selected sentences are sequentially arranged according to the degree of similarity (eg, sentence 1). (training data) - sentence 2 (verification data) - sentence 3 (training data) - sentence 4 (validation data) - sentence 5 (training data), ...)

In this case, in each of the training data and the verification data, the similarity distribution with the basic sentence may not be biased.

When the training data and the verification data are classified, the electronic device 100 (or an external device) may train the artificial intelligence model 10 based on a sentence corresponding to the training data. Here, as a sentence corresponding to the training data is input, the weight between nodes of the artificial intelligence model may be updated so as to output a specific consultation intention (eg, dissatisfaction).

Next, the electronic device 100 (or an external device) may verify the artificial intelligence model 10 by inputting a sentence corresponding to the verification data into the artificial intelligence model 10 .

Specifically, the electronic device 100 may input a sentence corresponding to the verification data into the artificial intelligence model 10 to identify whether the artificial intelligence model 10 outputs the corresponding consultation intention (eg, complaint). there is.

In this case, the electronic device 100 may perform verification using all sentences corresponding to the verification data, but may also perform verification using only some of them.

As an embodiment, the electronic device 100 lists sentences corresponding to the verification data in order according to the degree of similarity to the basic sentence, and selects one sentence for every predetermined number of sentences (eg, one sentence per ten sentences). sentence) can be used for verification. In this case, the time required for verification may be reduced by using only some of the sentences corresponding to the verification data.

If, as a result of inputting a sentence corresponding to the verification data, the artificial intelligence model 10 outputs an accurate consultation intention (: dissatisfaction), the electronic device 100 determines that the verification of the artificial intelligence model 10 is successful. can

On the other hand, when the verification fails, the electronic device 100 may additionally train the artificial intelligence model 10 using the verification data as training data.

Here, even if only some sentences among all sentences corresponding to the verification data are used for verification, additional training according to the verification failure may be performed through all sentences corresponding to the verification data. Alternatively, additional training may be performed only through sentences having a similarity with a specific sentence that has failed verification among sentences corresponding to the verification data equal to or less than a preset threshold.

Also, when the verification fails, the electronic device 100 may generate additional training data based on the verification data.

Specifically, when verification of the artificial intelligence model 10 through at least one verification sentence included in verification data fails, the electronic device 100 may extract a plurality of keywords from the verification sentence. Here, the electronic device 100 may identify a similar keyword having a similarity of at least one of a plurality of extracted keywords equal to or greater than a threshold. Next, the electronic device 100 may collect a plurality of sentences from the database based on the plurality of keywords and similar keywords. In addition, the electronic device 100 may select at least one sentence having a similarity to the basic sentence greater than or equal to a threshold among a plurality of sentences, and store/register the basic sentence as training data matching the registered counseling intention.

In this case, the electronic device 100 may generate additional training data by using only a specific sentence in which verification has failed among all sentences corresponding to the verification data. Alternatively, the electronic device 100 may generate additional training data using only sentences having a similarity with a specific sentence for which verification has failed, which is equal to or greater than a preset threshold.

Meanwhile, FIG. 5 is a block diagram illustrating a functional configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 5 , the electronic device 100 may include an input unit 510 , a collection unit 520 , an analysis unit 530 , a storage unit 540 , and the like. Each of the components may include at least one functionally divided module, and each module may be implemented through software and/or hardware.

The input unit 510 is a configuration for receiving and managing data required for constructing training data of the electronic device 100 .

The input unit 510 may include a consultation intention management module 511 , a basic sentence registration module 512 , and the like.

The consultation intention management module 511 corresponds to a module for managing registered consultation intentions, such as change, addition, exclusion, and the like.

The consultation intentions registered on the consultation intention management module 511 may be an output of the artificial intelligence model 10, and the artificial intelligence model 10 is trained to select at least one of the registered consultation intentions. can be

The consultation intention management module 511 may receive and store at least one consultation intention from the user, and may change or add at least one consultation intention according to the user input.

The basic sentence registration module 512 is a module for registering at least one basic sentence for each consultation intention.

For example, the basic sentence registration module 512 may acquire one or more basic sentences for each consultation intention through a user input.

Alternatively, the basic sentence registration module 512 may acquire a related basic sentence by performing a search and analysis in the database for at least one registered consultation intention.

The collection unit 520 is configured to collect various sentences that may be training data of the artificial intelligence model 10 or keywords that may constitute the corresponding sentences.

The collection unit 520 may include a morpheme analysis module 521 , a keyword extraction module 522 , a search module 523 , a sentence collection module 524 , a sentence reconstruction module 525 , and the like.

The analysis unit 530 is configured to perform various analyzes (eg, similarity analysis) on at least one keyword and sentence.

The analysis unit 530 may include a related keyword extraction module 531 , a sentence similarity analysis module 532 , a sentence-counseling intention tagging module 533 , and the like.

The morpheme analysis module 521 may extract a plurality of morphemes from the basic sentence by performing morpheme analysis on the basic sentence.

In addition, the keyword extraction module 522 may obtain at least one keyword from the plurality of extracted morphemes.

In this case, the entire extracted morpheme may be a keyword, or only a morpheme including the meaning of a minimum unit may be extracted as a keyword. To this end, the keyword extraction module 522 may obtain, as a keyword, only morphemes matching vectors having values or coordinates within a specific range by vector-transforming each morpheme. However, the present invention is not limited thereto.

The related keyword extraction module 531 is a module for extracting similar keywords related to or similar to at least one keyword.

The related keyword extraction module 531 may identify at least one similar keyword having a similarity with the keyword extracted through the keyword extraction module 522 equal to or greater than a threshold value. In this case, the related keyword extraction module 531 may convert the keyword into a vector to calculate a similarity or a distance between vectors.

The search module 523 is a module for additionally collecting at least one sentence.

The search module 523 may generate a search word using the keyword extracted through the keyword extraction module 522 and the similar keyword extracted from the related keyword extraction module 531, and obtain various sentences through the generated search word. .

The sentence collection module 524 is a module for collecting sentences that can be training data.

The sentence collection module 524 is a keyword extracted through the keyword extraction module 522 and/or similar keywords extracted through the related keyword extraction module 531 among various sentences previously obtained through the search of the search module 523 . It is possible to collect sentences including a certain number or more.

In this case, the sentence similarity analysis module 532 may compare the similarity between each of the collected sentences and the basic sentence.

Here, the sentence-counseling intention tagging module 533 may tag a sentence having a similarity with a basic sentence or more among the collected sentences with respect to a specific counseling intention (in which the basic sentence is registered).

On the other hand, the sentence reconstruction module 525 is a module for further securing training data by reconstructing the collected sentences.

The sentence reconstruction module 525 may calculate a similarity between each of the keywords in the collected sentences, and group keywords having a similarity greater than or equal to a threshold value into the same group.

Here, the sentence reconstruction module 525 may obtain a plurality of reconstructed sentences by replacing keywords of the same group in the collected sentences.

In this case, the sentence similarity analysis module 532 may calculate a similarity between each of the plurality of reconstructed sentences and the basic sentence.

Here, the sentence-counseling intention tagging module 533 may tag a sentence having a similarity of more than a threshold value with the basic sentence among the reconstructed sentences with respect to a specific counseling intention (in which the basic sentence is registered).

The storage unit 540 is configured to finally classify and store training data for each counseling intention.

The storage unit 540 may include a sentence verification module 541 and a training/verification data classification module 542 .

The sentence inspection module 541 is a module for performing additional inspection on the sentences tagged with respect to the consultation intention.

The sentence checking module 541 may filter at least one sentence according to sentence completion, or may filter at least one sentence according to a user input that has checked each sentence.

The training/verification data classification module 542 is a module for classifying sentences tagged with respect to a specific counseling intention into training data and verification data.

Specifically, the training/verification data classification module 542 may use some of the tagged sentences (classified as training data) as training data to be used for training the artificial intelligence model 10 .

Thereafter, the trained artificial intelligence model 10 may be verified based on a sentence corresponding to the verification data.

Meanwhile, the various embodiments described above may be implemented by combining two or more embodiments as long as they do not conflict with or contradict each other.

Meanwhile, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof.

According to the hardware implementation, the embodiments described in the present disclosure are ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays) ), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing other functions may be implemented using at least one.

In some cases, the embodiments described herein may be implemented by the processor itself. According to the software implementation, embodiments such as the procedures and functions described in this specification may be implemented as separate software modules. Each of the above-described software modules may perform one or more functions and operations described herein.

On the other hand, the computer instructions or computer program for performing the processing operation in the electronic device 100 according to various embodiments of the present disclosure described above is a non-transitory computer-readable medium. can be stored in When the computer instructions or computer program stored in the non-transitory computer-readable medium are executed by the processor of the specific device, the specific device performs the processing operation in the electronic device 100 according to the various embodiments described above.

The non-transitory computer-readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specific examples of the non-transitory computer-readable medium may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is commonly used in the technical field pertaining to the present disclosure without departing from the gist of the present disclosure as claimed in the claims. Various modifications may be made by those having the knowledge of, of course, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

100: electronic device 110: memory
120: processor

Claims (9)

In the method of constructing training data of an electronic device for an artificial intelligence model for classifying counseling intentions,
extracting a plurality of keywords from at least one basic sentence registered for one consultation intention among a plurality of pre-stored consultation intentions;
identifying a first similar keyword having a similarity to at least one of the plurality of extracted keywords equal to or greater than a threshold;
collecting a plurality of sentences from a database based on the plurality of keywords and the first similar keyword; and
Including; selecting at least one first sentence having a similarity to the basic sentence or more from among the plurality of collected sentences;
The artificial intelligence model is
is trained based on the basic sentence and the selected first sentence,
The method of constructing training data of the electronic device,
storing the sentences including the basic sentence and the selected first sentence as training data and verification data;
training the artificial intelligence model to identify the counseling intention for which the basic sentence is registered, based on the sentence corresponding to the training data;
verifying the artificial intelligence model by inputting a sentence corresponding to the verification data into the artificial intelligence model;
extracting a plurality of keywords from the verification sentence when verification of the artificial intelligence model through at least one verification sentence included in the verification data fails;
identifying a second similar keyword having a similarity to at least one of the plurality of extracted keywords equal to or greater than a threshold;
collecting a plurality of sentences from a database based on the plurality of keywords and the second similar keywords; and
selecting at least one third sentence having a similarity to the basic sentence or more from among the plurality of collected sentences;
The artificial intelligence model is
A method of constructing training data of an electronic device, which is trained based on the verification sentence and the selected third sentence. According to claim 1,
The step of collecting the plurality of sentences,
generating a search word by combining keywords of a certain ratio or more among the plurality of keywords and keywords including the first similar keyword;
A method of constructing training data for an electronic device, wherein the plurality of sentences are obtained by performing a search for the generated search word. According to claim 1,
The method of constructing training data of the electronic device,
Storing the selected first sentence as training data matching the consultation intention in which the basic sentence is registered; comprising, a method of constructing training data for an electronic device. According to claim 1,
The method of constructing training data of the electronic device,
generating a group including keywords having a similarity level greater than or equal to a threshold value among a plurality of keywords included in the collected plurality of sentences;
generating a plurality of reconstructed sentences by replacing a keyword included in the group with another keyword in the group in at least one sentence among the plurality of collected sentences; and
Selecting at least one second sentence having a similarity to the basic sentence from among the plurality of reconstructed sentences is greater than or equal to a threshold value;
The artificial intelligence model is
The training data construction method of an electronic device, which is trained based on the basic sentence, the selected first sentence, and the selected second sentence. delete According to claim 1,
The verification step is
Input the sentence corresponding to the verification data into the artificial intelligence model, and identify whether the artificial intelligence model outputs the consultation intention in which the basic sentence is registered,
When the artificial intelligence model outputs the consultation intention in which the basic sentence is registered, it is determined that the verification of the artificial intelligence model is successful. delete An electronic device comprising a memory and a processor, the electronic device comprising:
The processor is
An electronic device for performing the training data construction method of claim 1 by executing the instructions stored in the memory. In a computer program stored in a computer-readable recording medium,
A computer program stored in a computer-readable recording medium that is executed by a processor of an electronic device to cause the electronic device to perform the training data construction method of claim 1 .

Images