KR102424848B1 - Realtime consultation quality evaluation method of electronic apparatus for utilizing consultation pattern model of best consultant based on artificial intelligence, and system - Google Patents

Abstract

A counseling quality evaluation method of an electronic device is disclosed. The counseling quality evaluation method comprises the steps of: providing a voice call between a customer terminal and a counselor terminal; obtaining audio data on the basis of a spoken voice of the counselor input through the counselor terminal while the voice call is provided; converting the audio data to obtain a text; analyzing the audio data to obtain feature information of the audio data, and generating voice quality information by comparing the obtained feature information with pre-stored feature information; analyzing the text to obtain rule information of the text, and generating text quality information by comparing the obtained rule information with pre-stored rule information; and providing voice quality information and text quality information through the counselor terminal while the voice call is provided.

Description

REALTIME CONSULTATION QUALITY EVALUATION METHOD OF ELECTRONIC APPARATUS FOR UTILIZING CONSULTATION PATTERN MODEL OF BEST CONSULTANT BASED ON ARTIFICIAL INTELLIGENCE, AND SYSTEM }

The present disclosure relates to a method for evaluating real-time counseling quality of an electronic device, and more particularly, to an electronic device and system for supporting voluntary counseling quality management of counselors by automatically performing counseling quality evaluation when counselors perform real-time counseling. .

The call center is the top priority channel for responding to customers from a company's point of view, and the quality of response by counselors is a decisive factor in corporate marketing and enhancing brand competitiveness, such as product sales, contract signing, customer complaint resolution, and customer request reception.

Due to the prolonged COVID-19 outbreak, phone consultations between customers and call center counselors are increasing, and in particular, call center work is a representative industry vulnerable to Corona in that a large number of counselors in a limited space perform direct voice communication.

Therefore, in the current state, a system that can support the management of the counseling quality of counselors while being able to perform smooth customer service in a non-face-to-face environment is needed.

In the case of counselor training, at present, most call centers allow counseling managers to randomly select and listen to only a small number of calls from the counselor’s call history for a certain period of time, and according to the set counseling quality management rules, fast-talk, bright tone, and prescribed utterance. It is conducted in the form of hand-evaluating whether or not the script is omitted after listening, and since it takes more time to evaluate the recording and listening than the consultation process time, it is practically impossible to conduct a complete inspection.

In addition, according to the results of the counseling quality evaluation, counseling managers conduct individual or group training for counselors in a face-to-face environment, which is also acting as a risk factor in the Corona crisis.

In order to improve the human and time cost environment of such call center quality management and to improve the problem of random call listening analysis rather than full analysis, in the call center market, speech recognition and text analysis technology are applied to automatically convert speech into text and convert The quality of counseling is evaluated by analyzing the text. However, in this case, it is not possible to evaluate factors that cannot be determined by text alone, ie, speed of speech, tone of voice, and tone.

In order to solve the problems of the prior art described above, the present disclosure trains an artificial intelligence model (engine) according to the counseling case of an excellent counselor to build a standard quality evaluation tool, and utilizes it to train an artificial intelligence model (engine) during real-time counseling of individual counselors. An electronic device and system are provided that provide an assessment of consultation quality in real time.

Specifically, the present disclosure provides an electronic device that separately analyzes characteristic information of a counselor's spoken voice and rule information of a spoken text to evaluate the counseling quality, and provides real-time feedback so that the counselor in counseling can immediately check the evaluation details and systems.

In addition, the present disclosure further includes analysis of audio data rather than using only a single model of existing text data when constructing artificial intelligence learning data through selection of excellent counseling cases to build a standard model for counseling quality evaluation and building a model. It provides a counseling quality evaluation method that builds a sophisticated quality evaluation model and applies the established standard model to counseling in real time so that the quality evaluation result is immediately provided to the counselor.

Objects of the present disclosure are not limited to the above-mentioned purposes, and other objects and advantages of the present disclosure that are not mentioned may be understood by the following description, and will be more clearly understood by examples of the present disclosure. Moreover, it will be readily apparent that the objects and advantages of the present disclosure may be realized by the means and combinations thereof indicated in the claims.

A method for evaluating consultation quality of an electronic device according to an embodiment of the present disclosure includes providing a voice call between a customer terminal and a counselor terminal, based on the counselor's speech voice input through the counselor terminal while the voice call is being provided obtaining audio data, converting the audio data to obtain text, analyzing the audio data to obtain characteristic information of the audio data, and comparing the obtained characteristic information with pre-stored characteristic information. generating voice quality information; analyzing the text to obtain rule information of the text; and comparing the obtained rule information with pre-stored rule information to generate text quality information; while the voice call is provided , providing the voice quality information and the text quality information through the counselor terminal.

The method for evaluating the counseling quality of the electronic device includes generating a first artificial intelligence model for determining a match rate by extracting feature information of audio data; It may include training the first artificial intelligence model based on the matching audio data. In addition, the acquiring of the voice quality information may include comparing the acquired feature information with the pre-stored feature information to obtain a matching rate through the artificial intelligence model. In this case, the pre-stored characteristic information may be characteristic information of audio data that matches the speech voice of at least one excellent counselor.

In addition, the method for evaluating the counseling quality of the electronic device includes the steps of extracting rule information from text and generating a second artificial intelligence model for determining a match rate, text from which the spoken voice of the excellent counselor is converted, and the spoken voice of the inferior counselor It may include training the second artificial intelligence model based on the converted text. In this case, the obtaining of the text quality information may include comparing the obtained rule information with the pre-stored rule information to obtain a matching rate through the artificial intelligence model. In addition, the pre-stored rule information may be rule information of a text in which the speech voice of at least one excellent counselor is converted.

The feature information may include at least one of pitch, resonance, intensity, and speed of a voice. In this case, the voice quality information may include information on a matching rate for each item of the feature information and the previously registered feature information.

The rule information may include one or more items related to at least one of whether a preset script is uttered, a preset number of utterances of the script, and an utterance order of a plurality of scripts. In this case, the text quality information may include information on a matching rate for each item of the rule information and the previously registered rule information.

On the other hand, the method for evaluating the counseling quality of the electronic device includes: when a first spoken voice of a counselor is received through the counselor terminal, displaying a first text converted from the first spoken voice through the counselor terminal; Generating first voice quality information based on a first spoken voice, generating first text quality information based on the first text, through the counselor terminal, the first voice quality information and the first text displaying quality information; when the counselor's second spoken voice is received through the counselor's terminal in a state in which the first text is displayed, the second text converted from the second spoken's voice through the counselor's terminal is displayed as the second text Displaying together with 1 text, generating second speech quality information based on the first spoken voice and the second spoken voice, and generating second text quality information based on the first text and the second text generating, and displaying the second voice quality information and the second text quality information through the counselor terminal.

In this case, the method for evaluating the counseling quality of the electronic device includes comparing the first voice quality information and the second voice quality information to determine whether the counseling quality of the counselor is improving in real time; When it is determined that the consultation quality is being improved in real time, acquiring the uttered voice of the counselor input thereafter in a first cycle and updating the voice quality information for each cycle, it is determined that the counseling quality of the counselor is not improving in real time If it is determined, the method may include acquiring the voice of the counselor's utterance input thereafter with a second period shorter than the first period, and updating the voice quality information for each period.

The method for evaluating the counseling quality of an electronic device according to the present disclosure may provide not only voice counseling with a customer through a counselor's terminal, but also real-time evaluation information on counseling currently being performed.

That is, the counseling quality evaluation method of the electronic device according to the present disclosure enables each counselor to immediately receive feedback on his/her problem through comparison with best practices in real time during his/her counseling and improve it, It can provide an environment in which counselors can achieve self-directed quality improvement in real time without evaluation or training, and this can support counseling quality management and holistic analysis within a non-face-to-face environment. Through this, it is possible to minimize the human and time costs required for separate random call analysis or counselor training, thereby reducing administrative labor costs and enabling non-face-to-face self-directed counseling quality management.

1 is a view for explaining an operation of an electronic device performing communication with a customer terminal and a counselor terminal according to an embodiment of the present disclosure;
2 is a flowchart illustrating a method for evaluating consultation quality of an electronic device according to an embodiment of the present disclosure;
3 is a diagram for explaining an operation in which an electronic device provides evaluation information (quality information) in real time through a counselor terminal according to an embodiment of the present disclosure;
4 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure; and
5 is a block diagram illustrating a functional configuration and operation of an electronic device according to an embodiment of the present disclosure.

Prior to describing the present disclosure in detail, a description will be given of the description of the present specification and drawings.

First, terms used in the present specification and claims have been selected in consideration of functions in various embodiments of the present disclosure. However, these terms may vary depending on the intention or legal or technical interpretation of a person skilled in the art, and the emergence of new technology. Also, some terms are arbitrarily selected by the applicant. These terms may be interpreted in the meaning defined in the present specification, and if there is no specific term definition, it may be interpreted based on the general content of the present specification and common technical knowledge in the art.

Also, the same reference numerals or reference numerals in each drawing attached to this specification indicate parts or components that perform substantially the same functions. For convenience of description and understanding, the same reference numerals or reference numerals are used in different embodiments. That is, even though all components having the same reference number are illustrated in a plurality of drawings, the plurality of drawings do not mean one embodiment.

In addition, in this specification and claims, terms including an ordinal number, such as "first" and "second", may be used to distinguish between elements. This ordinal number is used to distinguish the same or similar elements from each other, and the meaning of the term should not be construed as limited due to the use of the ordinal number. For example, the components combined with such an ordinal number should not be limited in the order of use or arrangement by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and are intended to indicate that one or more other It is to be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In an embodiment of the present disclosure, terms such as “module”, “unit”, “part”, etc. are terms for designating a component that performs at least one function or operation, and such component is hardware or software. It may be implemented or implemented as a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, except when each needs to be implemented as individual specific hardware, and thus at least one processor. can be implemented as

In addition, in an embodiment of the present disclosure, when a part is connected to another part, this includes not only direct connection but also indirect connection through another medium. In addition, the meaning that a certain part includes a certain component means that other components may be further included without excluding other components unless otherwise stated.

1 is a diagram for explaining an operation of an electronic device that communicates with a customer terminal and a counselor terminal according to an embodiment of the present disclosure.

Referring to FIG. 1 , the electronic device 100 may perform communication with a customer terminal 200 and a counselor terminal 300 , respectively.

As an embodiment, the electronic device 100 may be implemented as a server. The server may be configured as a system including one or more computers. The electronic device 100 may correspond to a server for operating a consultation system.

In addition, the electronic device 100 may be implemented as various electronic devices such as a desktop PC, a smart phone, and a tablet PC.

The customer terminal 200 corresponds to a terminal device of a customer receiving a consultation. The customer terminal 200 may correspond to various terminal devices such as a smart phone, a tablet PC, a wearable device, a VR/AR device, a notebook PC, and a desktop PC.

The customer terminal 200 may include an audio input unit (ex. a microphone) and/or an audio output unit (ex. a speaker, an earphone/headphone terminal, a communication unit for communicating with a wireless earphone/headphone) for performing a voice call. can Also, the customer terminal 200 may be connected to a separate audio input device and/or an audio output device.

The counselor terminal 300 corresponds to a counselor's terminal device that provides counseling. The counselor terminal 300 may correspond to various terminal devices, such as a smart phone, a tablet PC, a wearable device, a VR/AR device, a notebook PC, and a desktop PC.

The counselor terminal 300 may include an audio input unit (eg, a microphone) and/or an audio output unit (eg, a speaker, an earphone/headphone terminal, and a communication unit for communicating with a wireless earphone/headphone) for performing a voice call. can Also, the counselor terminal 300 may be connected to a separate audio input device and/or an audio output device.

The electronic device 100 may provide a voice call for phone consultation between the customer terminal 200 and the counselor terminal 300 .

In this case, the electronic device 100 may convert the customer's uttered voice input through the customer terminal 200 into audio data and transmit it to the counselor terminal 300 . Similarly, the electronic device 100 may convert the counselor's utterance voice input through the counselor terminal 300 into audio data and transmit it to the customer terminal 300 .

In addition, the electronic device 100 may generate quality information about the consultation by analyzing the counselor's consultation content while the voice call is being provided, which will be described in more detail below with reference to the drawings.

2 is a flowchart illustrating a method for evaluating consultation quality of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the electronic device 100 may provide a voice call between the customer terminal 200 and the counselor terminal 300 ( S210 ), and is input through the counselor terminal 300 while the voice call is provided. Audio data may be acquired based on the counselor's spoken voice (S220).

Here, the electronic device 100 may generate voice quality information by comparing the characteristic information of the audio data with the previously stored characteristic information ( S230 ).

The feature information may include a value for at least one of pitch, resonance (Formant Frequency), intensity, and speed (Speed) of a voice.

The pre-stored feature information may be feature information extracted from the speech voice of at least one counselor corresponding to the past counseling case. For example, the pre-stored feature information may be feature information (eg, height, resonance, intensity, speed, etc.) extracted from the utterance voice of a counselor in an excellent counseling case, but is not limited thereto.

The voice quality information is information indicating the quality of the counselor's spoken voice. The electronic device 100 may compare feature information of audio data with pre-stored feature information for each item to obtain a match rate, and generate voice quality information including information on the match rate for each item.

The electronic device 100 may extract and compare feature information of audio data by using at least one artificial intelligence model.

In relation to this, the electronic device 100 may generate a first artificial intelligence model for determining a match rate by extracting feature information of the audio data.

Here, the electronic device 100 may extract audio data matching the uttered voice of the excellent counselor and the uttered voice of the inferior counselor based on past counseling cases, and may extract audio data matching the uttered voice of the inferior counselor, and utilize the extracted audio data to extract the first You can train artificial intelligence models.

As an example, the first artificial intelligence model may be implemented as a neural network model. In this case, the first AI model may be trained in such a way that it is updated according to weights between nodes included in different layers. For example, the first AI model may include at least one layer for receiving audio data and extracting feature information, and at least one layer for calculating a matching rate by comparing feature information between different audio data. It can contain layers.

In this way, in a state in which the first artificial intelligence model is trained, the electronic device 100 may input the audio data obtained in step S220 to the first artificial intelligence model.

In this case, the first artificial intelligence model can extract characteristic information of the audio data, and compare the extracted characteristic information with pre-stored characteristic information (eg, characteristic information of an excellent counselor's spoken voice) to output a matching rate for each item. have.

Meanwhile, the electronic device 100 may obtain text by converting audio data corresponding to the counselor's uttered voice ( S240 ). In this case, the electronic device 100 may use at least one engine for performing speech to text (STT) conversion.

Here, the electronic device 100 may analyze the text to obtain rule information of the text, and compare the obtained rule information with pre-stored rule information to generate text quality information ( S250 ).

The rule information may include one or more items related to at least one of whether a preset script is uttered, the number of utterances of the preset script, and an utterance order of a plurality of scripts.

Here, the script may be text corresponding to the entire consultation content, or text corresponding to at least one word or sentence constituting the consultation content.

For example, the rule information may include, but is not limited to, values for various items such as presence/number of greetings, presence/number of understanding words, presence/number of non-repudiation words, and presence/number of waiting comments. .

The pre-stored rule information may be rule information extracted from a text in which the speech voice of at least one counselor corresponding to the past counseling case is converted. For example, the pre-stored rule information may be rule information of a text in which the speech voice of a counselor of an excellent counseling case is converted, but is not limited thereto.

The text quality information is information indicating the quality of text (consultation content) in which the counselor's spoken voice is converted. The electronic device 100 compares rule information of text converted from audio data with pre-stored rule information (eg, rule information of text of excellent counselors) for each item to obtain a match rate, and includes information on the match rate for each item. You can generate text quality information.

The electronic device 100 may extract and compare rule information of the text by using at least one artificial intelligence model.

In relation to this, the electronic device 100 may generate a second artificial intelligence model for determining a match rate by extracting feature information of the audio data.

Here, the electronic device 100 may extract the text in which the speech voice of the excellent counselor is converted and the text in which the speech voice of the inferior counselor is converted based on the past counseling case, and the second artificial intelligence model is used by using the extracted text. can be trained

For example, the second artificial intelligence model includes at least one module for converting text into a vector form, at least one module for understanding the meaning of the converted vector in word or sentence units, whether a word or sentence exists / It may include at least one module for acquiring rule information of text according to the number, but is not limited thereto.

In this way, in a state in which the second artificial intelligence model is trained, the electronic device 100 may input the text obtained in step S240 into the second artificial intelligence model.

In this case, the second artificial intelligence model can extract rule information from the input text, and compare the extracted rule information with pre-stored rule information (eg, rule information of an excellent counselor's utterance) and output the matching rate for each item. can do.

When the voice quality information and the text quality information are generated as described above, the electronic device 100 may provide the voice quality information and the text quality information through the counselor terminal 300, respectively (S260).

Specifically, while a voice call between the customer terminal 200 and the counselor terminal 300 is provided, the electronic device 100 may display voice quality information and text quality information through the counselor terminal 300 . For example, the matching rate for each item constituting the voice quality information and the matching rate for each item constituting the text quality information may be displayed on the screen of the counselor terminal 300 , respectively.

3 is a diagram for explaining an operation in which an electronic device provides evaluation information (quality information) in real time through a counselor terminal according to an embodiment of the present disclosure.

3 illustrates a screen of the counselor terminal 300 while a voice call between the customer terminal 200 and the counselor terminal 300 is in progress. Referring to FIG. 3 , the electronic device 100 includes texts 301 , 302 , 303 , and 304 constituting consultation contents, a user interface (UI) 310 for displaying voice quality information in real time, and text quality information in real time. A UI 320 for representing , etc. may be displayed.

Specifically, the customer's uttered voice input through the customer terminal 200 and the counselor's uttered voice input through the counselor terminal 300 may be converted into text and displayed (301, 302, 303 in FIG. 3). 304).

For example, when the counselor's first spoken voice (eg, “Oh yes, hello”) is received through the counselor terminal 300 , the electronic device 100 transmits the first text 302 in which the first spoken voice is converted. It may be displayed through the counselor terminal 300 .

Here, the electronic device 100 may generate the first voice quality information based on the first spoken voice.

Specifically, the electronic device 100 inputs the first spoken voice to the first artificial intelligence model to extract feature information, and combines the extracted feature information with pre-stored feature information (eg, feature information of an excellent counselor's speech voice) and By comparison, the matching rate for each item can be calculated. As a result, the first voice quality information including the matching rate for each item, such as height, resonance, intensity, and speed, may be displayed in the UI 310 of the counselor terminal 300 .

Also, the electronic device 100 may generate the first text quality information based on the text 302 in which the first spoken voice is converted.

Specifically, the electronic device 100 inputs the text 302 into the second artificial intelligence model to extract rule information, and uses the extracted rule information as pre-stored rule information (eg, of the text in which the excellent counselor's speech is converted). rule information) to calculate the matching rate for each item. As a result, the first text quality information including the matching rate for each item, such as the number of greetings, the number of understanding words, the number of swear words, and the number of waiting comments, may be displayed in the UI 320 of the counselor terminal 300. have.

Thereafter, when the counselor's second spoken voice (eg, “nene”) is received through the counselor terminal 300 , the electronic device 100 transmits the second text 304 in which the second spoken voice is converted to the counselor terminal 300 . ) can be indicated by

In this case, the electronic device 100 may generate second voice quality information based on the first uttered voice and the second uttered voice.

For example, as a result of inputting audio data including both the first uttered voice and the second uttered voice to the first artificial intelligence model, second voice quality information may be generated.

As another example, the second voice quality information may be generated as a result of combining the first voice quality information generated for the first uttered voice and the voice quality information generated for the second uttered voice.

That is, the second voice quality information in which all spoken voices of the counselor provided so far are reflected may be generated and provided through the UI 310 .

Similarly, the electronic device 100 may generate the second text quality information based on the second text 304 from which the second spoken voice is converted and the above-described first text 302 .

For example, as a result of inputting text including both the first text 302 and the second text 302 into the second artificial intelligence model, second text quality information may be generated.

As another example, the second text quality information may be generated as a result of combining the first text quality information generated with respect to the first text 302 and the text quality information generated with respect to the second text 304 .

In this case, second text quality information in which all consultation contents (texts) of the counselor provided so far are reflected may be generated and provided through the UI 320 .

As such, according to the embodiment of FIG. 3 , the quality information (voice quality information, text quality information) displayed on the UIs 310 and 320 is updated in real time while the consultation is in progress (a new spoken voice is continuously input). , the counselor can check the quality information provided in real time, and the feedback process can proceed naturally.

Meanwhile, the electronic device 100 may monitor the quality information updated in real time to determine whether the counseling quality of the counselor is being improved in real time.

As an example, referring to the above-described embodiment of FIG. 3 , the electronic device 100 compares the first voice quality information and the second voice quality information generated at different times to determine whether the counselor's consultation quality is improving in real time. can determine whether

For example, when the match rate for each item gradually increases when the first voice quality information and the second voice quality information are compared, it may be determined that the consultation quality is improving in real time.

If it is determined that the counselor's counseling quality is improving in real time, the electronic device 100 acquires the counselor's uttered voice input later (newly) in the first cycle, and provides voice quality information (eg, UI 310 ) for each cycle. ) can be updated.

Specifically, the electronic device 100 may update the voice quality information by analyzing the spoken voice input during a first time period among a plurality of time periods divided according to the first period, and may provide the updated voice quality information. . Similarly, the electronic device 100 may update and provide voice quality information again by analyzing the spoken voice input during a second time section following the first time section among the plurality of time sections.

On the other hand, when it is determined that the counselor's counseling quality is not improving in real time, the electronic device 100 acquires the counselor's uttered voice, which is input later (newly) more frequently in a second cycle shorter than the first cycle, Quality information (eg, voice quality information displayed on the UI 310) may be updated.

In this case, the slower the improvement of the consultation quality, the faster the quality information is updated, so that it is possible to provide stronger feedback to the counselor.

However, if it is determined that the consultation quality is not improved even though the voice quality information is provided in the second shorter period, the electronic device 100 acquires the spoken voice according to the third period longer than the first period, You can update quality information. In other words, slow feedback may be provided to counselors who do not have the effect of strong feedback.

Meanwhile, according to the above-described embodiment, when the consultation quality is not improved even though the period is variously changed, the electronic device 100 may change the size/weight of the UI 310 including the voice quality information in real time. In this case, the size of the image/text constituting the voice quality information may also be changed.

For example, the electronic device 100 may control the counselor terminal 300 to enlarge the size of the UI 310 displayed on the screen of the counselor terminal 300 (: the specific gravity of the UI 310 in the screen). .

If it is determined that the consultation quality is improved as the size of the UI 310 is enlarged, the enlarged UI 310 may be maintained.

On the other hand, when it is determined that the consultation quality is not improved even though the size of the UI 310 is enlarged, the electronic device 100 displays the size of the UI 310 displayed on the screen of the counselor terminal 300 (: UI in the screen) 310) may be controlled to reduce the size of the counselor terminal 300 more than the existing size. Here, when the consultation quality is improved, the reduced UI 310 may be maintained. On the other hand, when the consultation quality is not improved even though the UI 310 is reduced, the electronic device 100 may control the counselor terminal 300 not to display the UI 310 for a predetermined period of time.

Meanwhile, similar to the above-described embodiment in which it is determined whether the consultation quality is improved through the voice quality information, it may be determined whether the real-time consultation quality is improved through the text quality information.

As an example, referring to the above-described embodiment of FIG. 3 , the electronic device 100 compares the first text quality information and the second text quality information generated at different time periods to determine whether the counselor's consultation quality is improving in real time. can determine whether

For example, when the matching rate for each item gradually increases when the first text quality information and the second text quality information are compared, it may be determined that the consultation quality is improving in real time.

If it is determined that the counselor's counseling quality is improving in real time, the electronic device 100 acquires the text by converting the counselor's uttered voice inputted later (newly) in the first cycle, and converts the obtained text. By using it, text quality information (eg, text quality information displayed on the UI 320 ) may be updated for each cycle.

On the other hand, when it is determined that the counselor's counseling quality is not improving in real time, the electronic device 100 acquires and converts the counselor's uttered voice input later (newly) with a second cycle shorter than the first cycle, thereby acquiring text And, by using the acquired text, text quality information (eg, text quality information displayed on the UI 320 ) may be updated for each cycle.

However, if it is determined that the consultation quality is not improved even though the text quality information is provided in the second shorter period, the electronic device 100 acquires the spoken voice according to the third period longer than the first period and provides text for each period. You can update quality information.

Alternatively, according to an embodiment, the electronic device 100 determines whether the voice quality is being improved according to the first voice quality information and the second voice quality information, and uses the text according to the first text quality information and the second text quality information. It can be judged whether the quality is improving or not.

Here, when it is determined that both the voice quality and the text quality are being improved, the electronic device 100 acquires the (new) input counselor's utterance voice in the first cycle and updates the voice quality information and the text quality information every cycle. can do. In addition, the updated voice quality information and text quality information may be displayed through the UIs 310 and 320 .

On the other hand, when it is determined that at least one of the voice quality and the text quality is not improving, the electronic device 100 acquires the (newly) inputted voice of the counselor in the second period (<first period) and provides a voice Quality information and text quality information may be updated every cycle. In addition, the updated voice quality information and text quality information may be displayed through the UIs 310 and 320 .

Meanwhile, as an embodiment, the electronic device 100 may set a different period for updating the voice quality information and the text quality information for each counseling case performed through the counselor terminal 300 .

For example, while the first consultation is in progress through the counselor terminal 300 , the electronic device 100 collects the spoken voice at a specific cycle (: cycle A) and updates the voice quality information and text quality information for each cycle. (on the screen of the counselor terminal 300) may be provided. Here, as the first consultation ends, the electronic device 100 may finally identify first quality information (voice quality information and text quality information) of the consultation.

In addition, while the second consultation is in progress through the counselor terminal 300 , the electronic device 100 collects the spoken voice at a specific cycle (: cycle B) and updates the voice quality information and the text quality information for each cycle. can Here, as the second consultation ends, the electronic device 100 may finally identify second quality information (voice quality information and text quality information) of the consultation.

In addition, while the third consultation is in progress through the counselor terminal 300 , the electronic device 100 collects the spoken voice at a specific cycle (: cycle C) and updates the voice quality information and text quality information for each cycle. can Here, as the third consultation ends, the electronic device 100 may finally identify third quality information (voice quality information and text quality information) of the consultation.

In this case, by comparing the first to third quality information, the electronic device 100 may identify a consultation having the best quality information among the first to third consultations. Then, the electronic device 100 may select a period in which the quality information is updated and provided for the consultation in which the quality information is the best.

For example, when the second quality information among the first to third quality information is the best, the electronic device 100 identifies the counselor of the counselor terminal 300 as the best fit for the update of the quality information according to the B cycle. can

In addition, with respect to a subsequent consultation case of the counselor terminal 300 , while a consultation (voice call) is provided, the electronic device 100 updates the voice quality information and the text quality information according to the corresponding cycle (eg, the B cycle). It can be displayed on the screen of the counselor terminal 300 in real time.

In this case, a feedback cycle (a cycle in which the quality information is updated and displayed on the UIs 310 and 320) is selected and utilized for each counselor, so that customized feedback can be performed.

Meanwhile, FIG. 4 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 4 , the electronic device 100 may include at least one memory 110 , a communication unit 120 , and a processor 130 .

The memory 110 is a configuration for storing an operating system (OS) for controlling overall operations of the components of the electronic device 100 and at least one instruction or data related to the components of the electronic device 100 . .

The memory 110 may include non-volatile memory such as ROM and flash memory, and may include volatile memory such as DRAM. In addition, the memory 110 may include a hard disk, a solid state drive (SSD), or the like.

The memory 110 may include an artificial intelligence model (eg, a first artificial intelligence model, a second artificial intelligence model) related to the various embodiments described above.

The artificial intelligence model according to the present disclosure may be composed of an ontology-based data structure in which various concepts, conditions, relationships, or agreed knowledge is expressed in a computer-processable form.

The artificial intelligence model may be learned through the electronic device 100 or a separate server/system through various learning algorithms.

Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithm in the present disclosure is specified when It is not limited to the above-described example except for.

The artificial intelligence model may correspond to a network model (neural network model) based on a neural network. The network model may include a plurality of network nodes having weights. A plurality of network nodes may form a connection relationship based on weights between nodes of different layers.

Neural network models include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), and GAN (GAN). Generative Adversarial Network) and deep Q-networks (Deep Q-Networks), etc. may be configured, but is not limited thereto.

The communication unit 120 may include circuits, modules, chips, etc. for performing communication with at least one external device in various wired/wireless communication methods.

The communication unit 120 may be connected to external devices (eg, the customer terminal 200 and the counselor terminal 300) through various networks.

The network may be a personal area network (PAN), a local area network (LAN), a wide area network (WAN), etc. depending on the area or size, and depending on the openness of the network, an intranet, It may be an extranet or the Internet.

The communication unit 120 is a long-term evolution (LTE), LTE Advance (LTE-A), 5th generation (5G) mobile communication, code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS) , WiBro (Wireless Broadband), GSM (Global System for Mobile Communications), DMA (Time Division Multiple Access), WiFi (Wi-Fi), WiFi Direct, Bluetooth, NFC (near field communication), Zigbee, etc. It can be connected to external devices through

Also, the communication unit 120 may be connected to external devices through a wired communication method such as Ethernet, an optical network, a Universal Serial Bus (USB), or a ThunderBolt.

The processor 130 is a configuration for overall controlling the electronic device 100 . Specifically, the processor 130 may perform operations according to various embodiments of the present disclosure by executing at least one instruction stored in the memory 110 while being connected to the memory 110 .

The processor 130 may include a general-purpose processor such as a CPU, an AP, a digital signal processor (DSP), or the like, a graphics-only processor such as a GPU, a vision processing unit (VPU), or the like, or an artificial intelligence-only processor such as an NPU. The AI-only processor may be designed with a hardware structure specialized for training or use of a specific AI model.

5 is a block diagram illustrating a functional configuration and operation of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 5 , the processor 130 may control various functional configurations including the input unit 510 , the learning unit 520 , the analysis unit 530 , and the display unit 540 . The components 510 , 520 , 530 , and 540 of FIG. 5 may correspond to a module or an engine composed of software and/or hardware.

The input unit 510 is configured to receive past counseling cases and build training data (learning data) for training the artificial intelligence model.

Specifically, the input unit 510 may inquire about an existing consultation call corresponding to a past consultation case (S511). Specifically, the input unit 510 may acquire audio data and texts for voices uttered by various counselors constituting past counseling data.

Here, the past consultation data may be stored in advance in the memory 120 of the electronic device 100 or may be stored in at least one external device (database) capable of communicating with the electronic device 100 .

The input unit 510 may select and collect counseling data related to counseling calls of excellent counseling and/or inferior counseling among past counseling data (S512).

In this case, the counseling data selected according to the user input may be identified as excellent counseling or poor counseling, but is not limited thereto.

Meanwhile, the input unit 510 may register the consultation script and rule information (S513). Here, the consultation script and rule information is standard consultation script and rule information, and may be registered according to a user input, but is not limited thereto.

The learning unit 520 is configured to train an artificial intelligence model used for quality evaluation.

The learning unit 520 may collect recorded data (audio data corresponding to spoken voice) from the selected consultation data ( S521 ), and extract characteristic information from the audio data ( S522 ).

In this case, a first artificial intelligence model trained to extract feature information from audio data may be utilized.

In addition, the learning unit 520 may train the first AI model by using audio data according to excellent counseling and/or inferior counseling ( S523 ). In addition, the learning unit 520 may store characteristic information of audio data matched with excellent counseling.

Meanwhile, the learning unit 520 may convert the audio data in the selected consultation data into text (S524) and extract rule information (S525).

In this case, the second artificial intelligence model trained to extract rule information from the text may be utilized.

Then, the learning unit 520 may train the second artificial intelligence model by using the text according to the excellent counseling and/or the inferior counseling (S526). In addition, the learning unit 520 may store rule information of texts matching excellent counseling.

The analysis unit 530 is configured to analyze the quality of a consultation while a consultation (voice call) is in progress in real time.

The analysis unit 530 may collect audio data corresponding to the counselor's uttered voice included in the real-time call voice (S531), and extract characteristic information from the collected audio data (S532).

Then, the analysis unit 530 may obtain a matching rate by comparing the extracted characteristic information with pre-stored characteristic information (eg, characteristic information of audio data of excellent counseling/poor counseling) ( S533 ).

For example, it may be identified that the higher the matching rate with respect to the characteristic information of audio data of excellent counseling is, the better the quality of counseling currently being provided in real time.

As another example, the lower the matching rate with respect to the characteristic information of the audio data of the inferior counseling, the better the quality of the counseling currently being provided in real time may be identified.

On the other hand, the analysis unit 530 converts the audio data collected in real time into text (S534), and compares rule information of the converted text with pre-stored rule information (eg, rule information of text of excellent counseling/poor counseling). to obtain a matching rate (S535).

For example, it may be identified that the higher the matching rate of the text of the excellent consultation with the rule information, the better the quality of the consultation currently being provided in real time.

As another example, it may be identified that the lower the matching rate with respect to the rule information of the text of the inferior consultation is, the better the quality of the consultation currently being provided in real time.

The display unit 540 is configured to provide information on the quality of counseling provided in real time (S541).

The display unit 540 may control the counselor terminal 300 to display a text in which the spoken voice exchanged between the customer terminal 200 and the counselor terminal 300 is converted (S541).

In addition, the display unit 540 may control the counselor terminal 300 to display information (: voice quality information, text quality information) about the matching rate obtained through the analysis unit 530 previously (S542, S543). .

Meanwhile, the various embodiments described above may be implemented by combining two or more embodiments as long as they do not conflict with or contradict each other.

Meanwhile, various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof.

According to the hardware implementation, the embodiments described in the present disclosure are ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays) ), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing other functions may be implemented using at least one.

In some cases, the embodiments described herein may be implemented by the processor itself. According to the software implementation, embodiments such as the procedures and functions described in this specification may be implemented as separate software modules. Each of the above-described software modules may perform one or more functions and operations described herein.

On the other hand, the computer instructions or computer program for performing the processing operation in the electronic device 100 according to various embodiments of the present disclosure described above is a non-transitory computer-readable medium. can be stored in When the computer instructions or computer program stored in such a non-transitory computer-readable medium are executed by the processor of the specific device, the specific device performs the processing operation in the electronic device 100 according to the various embodiments described above.

The non-transitory computer-readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, a cache, a memory, and the like, and can be read by a device. Specific examples of the non-transitory computer-readable medium may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is commonly used in the technical field pertaining to the present disclosure without departing from the gist of the present disclosure as claimed in the claims. Of course, various modifications may be made by those having the knowledge of

100: electronic device 200: customer terminal
300: agent terminal

Claims (9)

A method for evaluating consultation quality of an electronic device, the method comprising:
providing a voice call between the customer terminal and the counselor terminal;
acquiring audio data based on the counselor's spoken voice input through the counselor's terminal while the voice call is being provided;
converting the audio data to obtain text;
acquiring characteristic information of the audio data by analyzing the audio data, and generating voice quality information by comparing the acquired characteristic information with pre-stored characteristic information;
obtaining rule information of the text by analyzing the text, and generating text quality information by comparing the obtained rule information with pre-stored rule information; and
providing the voice quality information and the text quality information through the counselor terminal while the voice call is provided;
The method for evaluating the consultation quality of the electronic device,
when the counselor's first spoken voice is received through the counselor's terminal, displaying the first text converted from the first spoken voice through the counselor's terminal;
generating first voice quality information based on the first spoken voice;
generating first text quality information based on the first text;
displaying the first voice quality information and the first text quality information together with the first text through the counselor terminal;
displaying the second text converted from the second spoken voice together with the first text through the counselor's terminal when the counselor's second spoken voice is received through the counselor's terminal while the first text is displayed ;
generating second voice quality information based on the first uttered voice and the second uttered voice;
generating second text quality information based on the first text and the second text; and
Displaying the second voice quality information and the second text quality information together with the second text through the counselor terminal;
The method for evaluating the consultation quality of the electronic device,
comparing the first voice quality information and the second voice quality information to determine whether the counseling quality of the counselor is being improved in real time;
when it is determined that the counselor's counseling quality is not improving in real time, controlling the counselor's terminal so that the size of a user interface (UI) including voice quality information is larger than the existing size;
when it is determined that the counseling quality of the counselor is not improving in real time, controlling the counselor terminal so that the size of the UI is reduced compared to the existing size; and
When it is determined that the counseling quality of the counselor is not improving in real time, controlling the counselor terminal not to display the UI for a predetermined period of time. According to claim 1,
The method for evaluating the consultation quality of the electronic device,
generating a first artificial intelligence model for determining a match rate by extracting feature information of audio data; and
Training the first artificial intelligence model based on audio data matching the speech voice of the excellent counselor and the audio data matching the speech voice of the poor counselor;
The step of obtaining the voice quality information includes:
Through the artificial intelligence model, the obtained characteristic information is compared with the pre-stored characteristic information to obtain a matching rate,
The pre-stored feature information is,
A method for evaluating counseling quality of an electronic device, which is characteristic information of audio data that matches the speech voice of at least one excellent counselor. According to claim 1,
The method for evaluating the consultation quality of the electronic device,
generating a second artificial intelligence model for determining a match rate by extracting rule information from the text; and
Training the second artificial intelligence model based on the converted text of the excellent counselor's speech and the converted text of the inferior counselor's speech;
The step of obtaining the text quality information includes:
Through the artificial intelligence model, the obtained rule information is compared with the pre-stored rule information to obtain a matching rate,
The pre-stored rule information is
A method for evaluating counseling quality of an electronic device, wherein the speech voice of at least one excellent counselor is rule information of the converted text. According to claim 1,
The characteristic information is
Containing at least one item of pitch, resonance, intensity, and speed of the voice;
The voice quality information is
A method for evaluating counseling quality of an electronic device, comprising information on a matching rate for each item of the feature information and the pre-stored feature information. According to claim 1,
The rule information is
including one or more items related to at least one of whether a preset script is uttered, the number of utterances of a preset script, and an utterance order of a plurality of scripts,
The text quality information is
The method for evaluating consultation quality of an electronic device, comprising information on a matching rate for each item of the rule information and the pre-stored rule information. delete According to claim 1,
Comparing the first voice quality information and the second voice quality information to determine whether the counseling quality of the counselor is improving in real time,
when it is determined that the counseling quality of the counselor is improving in real time, acquiring the uttered voice of the counselor input thereafter in a first cycle and updating the voice quality information for each cycle; and
When it is determined that the counseling quality of the counselor is not improving in real time, acquiring the uttered voice of the counselor, which is subsequently input, in a second cycle shorter than the first cycle, and updating the voice quality information for each cycle; , a method for evaluating the quality of consultation in electronic devices. In an electronic device,
a memory in which at least one instruction is stored; and
The electronic device comprising a; processor for executing the instruction to perform the consultation quality evaluation method of claim 1 . A computer program stored in a computer readable medium,
A computer program stored in a computer-readable medium, which is executed by a processor of an electronic device to cause the electronic device to perform the consultation quality evaluation method of claim 1 .

Images