KR102583434B1 - Method and system for evaluating quality of voice counseling - Google Patents

Abstract

The present invention relates to a method for evaluating the quality of voice consultation. The quality evaluation method of voice consultation includes, when voice consultation is performed, receiving recorded data associated with the voice consultation, and based on the received recorded data, generating text data corresponding to the voices of the counselor and the customer over time. Step, generating a plurality of text sentences by dividing the generated text data into sentence units, generating a Mel spectrogram associated with the recorded data, and verifying the accuracy of the plurality of text sentences generated using the generated Mel spectrogram. It includes the step of performing a quality evaluation of the voice consultation using a plurality of verified text sentences according to the stage and time section and time of the voice consultation.

Description

Quality evaluation method and system for voice counseling {METHOD AND SYSTEM FOR EVALUATING QUALITY OF VOICE COUNSELING}

The present invention relates to a quality evaluation method and system for voice consultation, and specifically, to a quality evaluation method and system for voice consultation that converts voice into text and performs quality evaluation.

With the beginning of the untact era, the growth rate of the online industry is gradually accelerating. As the online industry grows, the importance of customer management has increased, and the contact center that exists at the point of contact is also recognized as important. The work of these contact centers is largely divided into counselor counseling work and counselor management work. As one of the counselor management tasks, it is very important to periodically evaluate the counselor's counseling quality for customer management.

Meanwhile, the consultation quality evaluation may be performed by an educational instructor who performs the quality evaluation. However, because the number of training instructors is significantly small compared to counselors, it is impossible for training instructors to inspect the call contents of all counselors and perform quality evaluation. Therefore, technology for automating the quality evaluation of voice consultation with high accuracy is required.

The present invention provides a method for evaluating the quality of voice consultation, a computer program stored in a recording medium, and a system (device) to solve the above problems.

The present invention may be implemented in various ways, including as a method, system (device), or computer program stored in a readable storage medium.

According to one embodiment of the present invention, a method for evaluating the quality of a voice consultation performed by at least one processor includes, when a voice consultation is performed, receiving recording data associated with the voice consultation, based on the received recording data. , generating text data corresponding to the voices of the counselor and customer over time, dividing the generated text data into sentences to generate a plurality of text sentences, extracting silent sections included in the recording data, Verifying the accuracy of a plurality of text sentences generated based on the extracted silent section and performing a quality evaluation of the voice consultation using the plurality of verified text sentences according to the time section and time of the voice consultation. Includes.

According to an embodiment of the present invention, the step of extracting a silent section included in the recorded data and verifying the accuracy of a plurality of text sentences generated based on the extracted silent section includes generating a mel spectrogram associated with the recorded data. It includes the steps of generating and extracting a silent section using the generated Mel spectrogram.

According to one embodiment of the present invention, the step of performing quality evaluation on voice consultation includes extracting a first set of text sentences included within a specific time range from among a plurality of text sentences, the extracted first set of text It includes dividing a sentence into a plurality of morphemes and determining whether a greeting keyword related to a greeting is included in the plurality of divided morphemes.

According to one embodiment of the invention, performing a quality assessment for a voice consultation includes splitting the plurality of text sentences into a second set of text sentences associated with a counselor and a third set of text sentences associated with a customer; and calculating the number of times at least some of the text sentences in the second set overlap with at least some of the text sentences in the third set.

According to an embodiment of the present invention, the step of performing quality evaluation on voice consultation includes extracting a section in which muting occurred during a specific time based on the time section of the voice consultation and a plurality of text sentences according to time, a plurality of steps, It includes determining a text sentence before the section where muting occurred among the text sentences of and determining whether the determined text sentence before the section where muting occurred includes a waiting request keyword.

According to one embodiment of the invention, performing a quality assessment for a voice consultation includes splitting the plurality of text sentences into a second set of text sentences associated with a counselor and a third set of text sentences associated with a customer; A step of extracting a personal information text sentence containing the customer's personal information from a third set of text sentences, a step of extracting a text sentence immediately following the extracted personal information text sentence from the second set of text sentences, and the extracted personal information text It includes a step of determining whether the text sentence immediately following the sentence contains a keyword corresponding to the customer's personal information.

According to one embodiment of the present invention, the step of performing quality evaluation on voice consultation includes extracting a fourth set of text sentences corresponding to the last specific number of a plurality of text sentences and using an entity name recognition algorithm. and determining whether information associated with the counselor is included in the fourth set of text sentences.

According to an embodiment of the present invention, the method further includes generating a quality evaluation report by visualizing result data for quality evaluation of voice consultation for each quality evaluation item.

A computer program stored in a computer-readable recording medium is provided to execute the above-described voice consultation quality evaluation method according to an embodiment of the present invention on a computer.

The voice consultation quality evaluation system according to an embodiment of the present invention receives recording data associated with the voice consultation when voice consultation is performed, and responds to the voices of the counselor and the customer over time based on the received recording data. A text conversion unit that generates text data, a sentence generator that divides the generated text data into sentence units to generate a plurality of text sentences, extracts the silent section included in the recorded data, and extracts the silent section based on the extracted silent section. It includes a sentence verification unit that verifies the accuracy of the plurality of generated text sentences, and a quality evaluation performance unit that performs a quality evaluation of the voice consultation using the plurality of verified text sentences according to the time section and time of the voice consultation.

In various embodiments of the present invention, the quality of voice counseling can be effectively evaluated without the user having to directly listen to all the voice counseling contents of the counselors.

In various embodiments of the present invention, training for counselors can be efficiently performed using an automatically generated quality evaluation report without generating separate documents for improving the counselor's counseling ability and additional training.

In various embodiments of the present invention, the processor can simply determine whether a greeting containing a predetermined composition has been uttered within a certain time after the voice consultation begins.

In various embodiments of the present invention, the processor can effectively evaluate the counselor's listening ability by recognizing the section where the voices of the counselor and the customer overlap through text sentences.

In various embodiments of the present invention, the processor can perform consultation quality evaluation by effectively extracting only the silent sections in which there is no waiting request among the silent sections that occur during voice counseling.

In various embodiments of the present invention, when the customer's personal information is uttered, the processor can simply recognize whether the counselor has repeated the content and re-confirmed the accuracy of the personal information.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned are clear to a person skilled in the art (referred to as a “person skilled in the art”) in the technical field to which the present invention pertains from the description of the claims. It will be understandable.

Embodiments of the present invention will be described with reference to the accompanying drawings described below, in which like reference numerals represent like elements, but are not limited thereto.
1 is a diagram illustrating an example in which quality evaluation of voice consultation is performed according to an embodiment of the present invention.
Figure 2 is a functional block diagram showing the internal configuration of a voice consultation quality evaluation system according to an embodiment of the present invention.
Figure 3 is a diagram showing an example of generating a Mel spectrogram according to an embodiment of the present invention.
Figure 4 is a flowchart showing an example of a method for evaluating the quality of voice consultation according to an embodiment of the present invention.
Figure 5 is a flowchart showing an example of an introduction evaluation method according to an embodiment of the present invention.
Figure 6 is a flowchart showing an example of a listening ability evaluation method according to an embodiment of the present invention.
Figure 7 is a flowchart showing an example of a silence evaluation method according to an embodiment of the present invention.
Figure 8 is a flowchart showing an example of an information verification evaluation method according to an embodiment of the present invention.
Figure 9 is a flowchart showing an example of a closing greeting evaluation method according to an embodiment of the present invention.

Hereinafter, specific details for implementing the present invention will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present invention.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms, but the present embodiments only serve to ensure that the present invention is complete and that the present invention does not convey the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the function in the present invention, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plural expressions. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

In the present invention, terms such as "comprise", "comprising", etc. may indicate the presence of features, steps, operations, elements and/or components, but may indicate that such terms include one or more other functions, It does not preclude the addition of steps, operations, elements, components and/or combinations thereof.

In the present invention, when a specific component is referred to as being “coupled,” “combined,” “connected,” or “reacting” with any other component, the specific component is directly bonded, combined, and/or connected to the other component. Alternatively, it may be connected or react, but is not limited thereto. For example, one or more intermediate components may exist between a particular component and another component. Additionally, in the present invention, “and/or” may include each of one or more listed items or a combination of at least a portion of one or more items.

In the present invention, terms such as “first” and “second” are used to distinguish specific components from other components, and the components described above are not limited by these terms. For example, the “first” component may be an element of the same or similar form as the “second” component.

In the present invention, 'recorded data' is voice data that records or transcribes a voice consultation between a counselor and a customer, and may include information about the volume (amplitude) and pitch of the counselor's and customer's voices over time.

In the present invention, 'spectrogram' may refer to an image that displays the sound area as the main feature center, and may include the distribution of intensity and frequency of the voice. Additionally, Mel Spectrogram may refer to a spectrogram that recognizes voice in the frequency domain.

1 is a diagram illustrating an example in which quality evaluation of voice consultation is performed according to an embodiment of the present invention. According to one embodiment, the counselor 110 and the customer 120 may perform voice counseling. In this case, the voice consultation recording server 130 can record the voices of the counselor 110 and the customer 120 performing the voice consultation in real time. For example, the voice consultation recording server 130 records the voice of the counselor 110 and the voice of the customer 120 separately, or records the content of the voice consultation, and then records the voice of the counselor 110 and the voice of the customer ( 120) can also be separated by voice.

Recorded data acquired by the voice consultation recording server 130 may be stored and managed in the recording DB 140. For example, the recorded data may be separated by counselor who performed the consultation and stored in the recording DB 140, or separated by customer and stored in the recording DB 140, but is not limited to this. In another example, recorded data may be matched with the names of counselors and customers and stored in the recording DB 140.

The voice consultation quality evaluation system 150 may receive or extract the recording data 142 stored in the recording DB 140. In this case, the voice counseling quality evaluation system 150 may use the received recording data 142 to evaluate the quality of the counselor 110's voice counseling. For example, the quality evaluation of voice counseling may include, but is not limited to, introduction evaluation, listening ability evaluation, silence evaluation, information confirmation evaluation, and closing greeting evaluation, and may include additional evaluation using plus and minus points. More may be included. For example, in quality evaluation, additional points may be given if it is determined that the counselor handled the consultation kindly or if a cushion word was used more than n times (where n is a natural number) during the consultation. In other examples, points may be deducted if the counselor is determined to have handled the counseling unkindly or if the process of recording the history of counseling results falls short of certain standards.

For quality evaluation, the voice counseling quality evaluation system 150 may generate text data corresponding to the voices of the counselor 110 and the customer 120 over time based on the received recording data 142. In this case, any STT (Speech To Text) algorithm may be used to convert speech to text, or any machine learning model may be used. The text data generated in this way can be divided into text data related to the speaking of the counselor 110 and text data related to the speaking of the customer 120. Here, since the text data is generated to correspond to voices over time, the order of speech of the counselor 110 and the customer 120 included in the voice consultation can be maintained the same in the generated text data. In other words, the text data may include time information associated with the voice consultation.

According to one embodiment, the voice consultation quality evaluation system 150 may divide the generated text data into sentences to generate a plurality of text sentences. Here, the voice consultation quality evaluation system 150 may divide text data into sentence units using an arbitrary algorithm, machine learning model, etc. For example, the voice consultation quality evaluation system 150 may determine a portion of text data including a subject and a verb as one text sentence, but is not limited to this.

The voice consultation quality evaluation system 150 can verify the accuracy of the plurality of generated text sentences. In other words, it can be verified whether the text data has been accurately segmented for each sentence. For example, when voice counseling is performed, not only standard sentences but also unstructured sentences may be included, and at least some of these unstructured sentences may not be accurately divided into text sentences. Therefore, by verifying the accuracy of the plurality of initially segmented text sentences, it is possible to accurately identify and re-segment all sentences included in the text data.

According to one embodiment, the voice consultation quality evaluation system 150 may extract a silent section included in the recording data 142 and verify the accuracy of a plurality of text sentences generated based on the extracted silent section. . Here, the voice consultation quality evaluation system 150 generates a mel spectrogram associated with the recording data 142, extracts a silent section using the generated mel spectrogram, and then extracts a silent section based on the extracted silent section. The accuracy of multiple generated text sentences can be verified. For example, if the divided text sentence is composed of “Oh, that is (space) 25,000 won”, the voice consultation quality evaluation system 150 re-divides the text sentence into two new text sentences based on the space. You can.

Thereafter, the voice consultation quality evaluation system 150 may perform a quality evaluation of the voice consultation using a plurality of verified text sentences according to the time section and time of the voice consultation. That is, the voice consultation quality evaluation system 150 performs introduction evaluation, listening ability evaluation, silence evaluation, information confirmation evaluation, and closing greeting evaluation, and combines the results in a defined format to generate a quality report 152. can do. The quality report 152 generated in this way may be transmitted or delivered to user terminals such as the user and/or the counselor 110 performing the quality evaluation.

Figure 2 is a functional block diagram showing the internal configuration of the voice consultation quality evaluation system 150 according to an embodiment of the present invention. As shown, the voice consultation quality evaluation system 150 may include a text conversion unit 210, a sentence generation unit 220, a sentence verification unit 230, and a quality evaluation performance unit 240. As described above, the voice consultation quality evaluation system 150 communicates with a recording database, etc., and can exchange data and/or information necessary for quality evaluation.

The text converter 210 may receive recorded data related to voice counseling and, based on the received recorded data 142, generate text data corresponding to the voices of the counselor and the customer over time. For example, the text conversion unit 210 may convert recorded data into text data using an arbitrary STT algorithm, machine learning model, etc. In this way, when STT conversion is performed, text data may be composed of a set of voice recognition results and time information of the results.

In order to evaluate the utterance of the first greeting in the introduction part of the consultation, the response process, and the utterance of the final greeting in the closing process, it may be required to process the content of the voice consultation on a sentence-by-sentence basis. Accordingly, the sentence generator 220 can change text data into a script in sentence units. For example, the sentence generator 220 may generate a plurality of text sentences by combining words constituting sentences among words included in text data. In this case, the sentence generator 220 may generate a plurality of text sentences to include subjects and verbs of adjacent time ranges, but is not limited to this.

The sentence verification unit 230 may use the recording data 142 to verify a plurality of text sentences generated by the sentence generation unit 220. According to one embodiment, during the counseling process, the counselor and the customer may exchange conversations using incomplete sentences, and accordingly, unstructured sentences may be included in the recording data 142. At least some of these irregular sentences may not be accurately classified by the sentence generator 220. That is, the sentence verification unit 230 can extract the silent section included in the recording data 142 and verify the accuracy of a plurality of text sentences generated based on the extracted silent section. In this case, the sentence verification unit 230 generates a mel spectrogram associated with the recording data 142, extracts a silent section using the generated mel spectrogram, and then converts the text sentences into different types based on the extracted silent section. It can be divided into multiple text sentences.

The quality evaluation performing unit 240 may perform a quality evaluation of the voice consultation using a plurality of text sentences that have been generated and verified. That is, the quality evaluation performing unit 240 may perform a quality evaluation of the voice consultation using a plurality of verified text sentences according to the time section and time of the voice consultation. Quality evaluation of voice counseling can be divided into several stages according to a predetermined method, and arbitrary algorithms and machine learning models for quality evaluation can be used at each stage. In this way, after the quality evaluation is completed, the voice consultation quality evaluation system 150 visualizes the result data (e.g., data in JSON format) for the quality evaluation of the voice consultation for each item of the quality evaluation to produce a quality evaluation report 152. It is possible to generate and provide the generated quality evaluation report 152 to users.

In FIG. 2, each functional configuration included in the voice consultation quality evaluation system 150 is separately described and described in detail, but this is only to aid understanding of the invention, and two or more functions may be performed in one computing device. With this configuration, the user can effectively evaluate the quality of voice counseling without having to directly listen to all the voice counseling contents of the counselors. In addition, training for counselors can be efficiently performed using the automatically generated quality evaluation report 152 without creating separate documents for improving the counselor's counseling ability and additional training.

Figure 3 is a diagram illustrating an example of generating a Mel spectrogram 340 according to an embodiment of the present invention. As described above, the voice consultation quality evaluation system (150 in FIG. 1) may generate a mel spectrogram 340 using the recorded data 310. In this way, the generated mel spectrogram 340 can be used to verify the accuracy of the text sentence. The Mel spectrogram 340 may be visualized and generated in the form of a graph, but is not limited to this.

According to one embodiment, the voice consultation quality evaluation system may generate a spectrum 320 using recorded data 310. For example, the voice consultation quality evaluation system applies Fourier transform to the recording data 310 to extract frequency information associated with the recording data 310, and uses the extracted frequency information to obtain a spectrum 320. can be created. Specifically, the recording data 310 may include information about the amplitude of the voice over time, and the spectrum 320 may be configured to include information about the amplitude according to the frequency.

Additionally, the voice consultation quality evaluation system can generate a spectrogram 330 using the spectrum 320. For example, the voice consultation quality evaluation system may apply Fourier transform for each time of the spectrum 320 to extract information about amplitude according to frequency for each time. Then, the voice consultation quality evaluation system can convert the amplitude into decibels and apply a log scale to the frequency to generate a spectrogram 330.

Afterwards, the voice consultation quality evaluation system can generate a mel spectrogram 340 using the spectrogram 330. For example, when matching the mel scale to the frequency on the spectrogram 330, the mel spectrogram 340 may be generated. Here, the Mel scale is a scale created in consideration of the characteristics of people who react more sensitively to low frequency bands than to high frequencies, and is a scale that represents the relationship between physical frequencies and frequencies perceived by actual people. It can be referred to. As described above, the voice consultation quality evaluation system extracts silent sections on the recording data 310 using the mel spectrogram 340 generated in this way, and determines the accuracy of a plurality of text sentences based on the extracted silent sections. It can be verified. For example, the voice consultation quality evaluation system can divide one text sentence containing two or more sentences into two different text sentences, the sentence before and after the silent section, based on the silent section.

Figure 4 is a flowchart showing an example of a method 400 for evaluating the quality of voice consultation according to an embodiment of the present invention. According to one embodiment, the method 400 for evaluating the quality of voice consultation may be performed by a processor (eg, at least one processor of a voice equivalent quality evaluation system). As shown, the method 400 for evaluating the quality of voice counseling may be initiated by receiving recorded data associated with the voice counseling when voice counseling is performed (S410).

Based on the received recording data, the processor may generate text data corresponding to the voices of the counselor and customer over time (S420). Additionally, the processor may divide the generated text data into sentence units to generate a plurality of text sentences (S430). That is, the processor can extract sentences related to the customer and the counselor who performed the voice consultation from the recorded data.

The processor may extract the silent section included in the recording data and verify the accuracy of a plurality of text sentences generated based on the extracted silent section (S440). For example, the processor may generate a mel spectrogram associated with the recorded data and extract a silent section using the generated mel spectrogram. Additionally, the processor may perform a quality evaluation of the voice consultation using a plurality of verified text sentences according to the time section and time of the voice consultation (S450). In this case, the processor can generate a quality evaluation report by visualizing the result data for the quality evaluation of voice consultation for each quality evaluation item.

Figure 5 is a flowchart showing an example of an introduction evaluation method 500 according to an embodiment of the present invention. According to one embodiment, the introductory evaluation method 500 may be performed by a processor (e.g., at least one processor of a speech equivalent quality evaluation system). As shown, the introduction evaluation method 500 may be initiated by the processor extracting a first set of text sentences included within a specific time range from among a plurality of text sentences (S510). Here, the specific time range may include a range within a predetermined time from the start of the conversation. For example, the specific time range may be a time range that is twice the average time the greeting is uttered. That is, if it is determined that uttering a greeting takes an average of 2 seconds, the specific time range may include a range within 4 seconds from the start of the conversation.

The processor may perform an introductory evaluation based on whether a greeting containing information of greeting, affiliation, and/or name was uttered within a certain time range. For introduction evaluation, the processor may divide the extracted first set of text sentences into a plurality of morphemes (S520). Additionally, the processor may determine whether a greeting keyword is included in the plurality of segmented morphemes (S530). Here, the greeting keyword is a keyword related to the greeting and may be a core common keyword commonly used in various greetings, but is not limited thereto and may further include keywords about the affiliation, name, etc. of a predetermined counselor. For example, the processor may use morphological analysis to determine whether each element is present in the utterance. Since various variations of a greeting may be permitted, the processor may determine that a greeting exists if a greeting keyword is included in the morpheme analysis result.

Additionally or alternatively, the processor may use a Named Entity Recognition (NER) algorithm to determine whether the counselor's name is included. That is, the processor can detect a noun in the form of a person's name through entity name recognition and determine whether the counselor's name is included in the text sentence. In this case, the processor can perform entity name recognition using a machine learning model such as a transformer-based BERT model or ELECTRA model. Additionally or alternatively, the processor may pre-store the counselor's name and determine whether the counselor's name is included based on whether a noun matching the stored counselor's name exists. With this configuration, the processor can simply determine whether a greeting containing a predetermined configuration has been uttered within a specific time after the voice consultation begins.

Figure 6 is a flowchart showing an example of a listening ability evaluation method 600 according to an embodiment of the present invention. According to one embodiment, the listening ability evaluation method 600 may be performed by a processor (e.g., at least one processor of a speech equivalent quality evaluation system). Here, listening ability is a quantitative evaluation factor of how carefully the counselor listened to the customer's speech, and can be judged by detecting the number of times the counselor intervened during the customer's speech. As shown, the listening ability evaluation method 600 may be initiated by the processor dividing a plurality of text sentences into a second set of text sentences associated with the counselor and a third set of text sentences associated with the customer (S610).

Thereafter, the processor may evaluate the counselor's listening ability by calculating the number of times that at least part of the text sentence in the second set overlaps with at least part of the text sentence in the third set (S620). In other words, the processor can calculate the number of times the counselor's utterance overlaps while the customer's utterance is in progress first. In this case, the processor can calculate only the number of times overlap occurs for a certain period of time (e.g., 0.5 seconds or more) and the counselor's utterances occur continuously after the overlap as the number of times the counselor's and customer's utterances overlap. Additionally, even if the customer's utterance occurs after the overlap, if the overlap exceeds another specific time (e.g., 1 second), the counselor's and the customer's utterances may be determined to have overlapped. The higher the number of overlaps, the lower the counselor's listening ability may be evaluated. With this configuration, the processor can effectively evaluate the counselor's listening ability by recognizing the section where the counselor's and the customer's voices overlap through text sentences.

Figure 7 is a flowchart showing an example of a silence evaluation method 700 according to an embodiment of the present invention. According to one embodiment, the silence evaluation method 700 may be performed by a processor (e.g., at least one processor of a speech equivalent quality evaluation system). For example, in cases where external information must be referred to in order to meet the customer's needs during the voice consultation process, muting that occurs may be permitted, but muting that occurs in other cases is grounds for lowering the quality of consultation. can be decided. Accordingly, the processor may perform the silence evaluation method 700 by detecting a mute section in which there is no waiting request. As shown, the silence evaluation method 700 may be initiated by extracting a section in which silence occurs for a specific time based on the time section of the voice consultation and a plurality of text sentences according to time (S710). Here, the section in which silence occurs may refer to a section in which silence lasts for a specific time (e.g., 5 seconds), and may occur, for example, between specific text sentences.

The processor may determine the text sentence preceding the section in which the extracted muting occurred among the plurality of text sentences (S720). Additionally, the processor may determine whether a waiting request keyword is included in the text sentence before the determined muted section (S730). If it is determined that the waiting request keyword is included, it may be an exception to the section in which muting occurred. However, if it is determined that the waiting request keyword is not included, it may be determined to be a section in which muting occurred. According to one embodiment, whether a waiting request keyword is included may be performed similarly to the introduction evaluation method (600 in FIG. 6). For example, the processor evaluates the text sentence before the section where muting occurred in morpheme units. Silence evaluation can be performed by segmenting and determining whether the segmented morphemes contain waiting request keywords. With this configuration, the processor can effectively extract only the silent sections without waiting requests among the silent sections that occur during voice counseling and perform consultation quality evaluation.

Figure 8 is a flowchart showing an example of an information verification evaluation method 800 according to an embodiment of the present invention. According to one embodiment, the information verification evaluation method 800 may be performed by a processor (e.g., at least one processor of a speech equivalent quality evaluation system). Here, information verification is a process to check whether the customer has permission to access the service in order to provide the service requested by the customer, and may refer to checking the customer's name, contact information, account number, etc. To confirm information, the counselor may request information from the customer, repeat the information received from the customer, and then say thank you. As shown, the information verification evaluation method 800 may be initiated by dividing a plurality of text sentences into a second set of text sentences associated with the counselor and a third set of text sentences associated with the customer (S810).

The processor may extract a personal information text sentence containing the customer's personal information from the third set of text sentences (S820). For example, a personal information text sentence may refer to a sentence containing the customer's name, contact information, account number, etc. The processor may extract personal information text sentences using the above-described entity name recognition and/or morphological analysis, but is not limited to this, and may extract personal information using an arbitrary algorithm for extracting text containing specific information. You can also extract text sentences.

The processor may extract a sentence immediately following the extracted personal information text sentence from among the text sentences in the second set (S830). Additionally, the processor may determine whether the sentence immediately following the extracted personal information text sentence includes a keyword corresponding to the customer's personal information (S840). According to one embodiment, the processor extracts a keyword corresponding to the customer's personal information from the personal information text sentence and determines whether the keyword corresponding to the extracted keyword is included in the counselor's utterance immediately after the extracted personal information text sentence. can be judged. Additionally, the processor may determine whether a text sentence including a thank you note exists after the counselor confirms the customer's personal information. With this configuration, when the customer's personal information is uttered, the processor can simply recognize whether the counselor has re-confirmed the accuracy of the personal information by repeating the relevant content.

Figure 9 is a flowchart showing an example of a closing greeting evaluation method 900 according to an embodiment of the present invention. According to one embodiment, the greeting evaluation method 900 may be performed by a processor (eg, at least one processor of a voice equivalent quality evaluation system). Here, the ending greeting may consist of additional inquiries, affiliation, name, and greeting at the end of the voice consultation. That is, the processor may perform a closing greeting evaluation based on whether or not the counselor uttered a closing greeting including additional inquiry, affiliation, name, and/or greeting in the closing portion. As shown, the ending greeting evaluation method 900 may be initiated by the processor extracting a fourth set of text sentences corresponding to the last specific number (e.g., 5) of the plurality of text sentences (S910).

The processor may determine whether the counselor's name is included in the fourth set of text sentences using a Named Entity Recognition (NER) algorithm. That is, the processor can detect a noun in the form of a person's name through entity name recognition and determine whether the counselor's name is included in the text sentence. In this case, the processor can perform entity name recognition using a machine learning model such as a transformer-based BERT model or ELECTRA model.

Additionally, the processor may segment the fourth set of text sentences into a plurality of morphemes. Additionally, the processor may determine whether a final keyword is included in the plurality of divided morphemes. Here, the closing keyword may be a keyword related to a closing greeting, confirmation of additional inquiry, etc., but is not limited to this and may further include keywords related to the affiliation, name, etc. of a predetermined counselor. For example, the processor may use morphological analysis to determine whether each element is present in the utterance. Since various variations of a greeting may be permitted, the processor may determine that a greeting exists if a concluding keyword is included in the morpheme analysis result.

The above-described method and/or various embodiments may be implemented with digital electronic circuitry, computer hardware, firmware, software, and/or combinations thereof. Various embodiments of the present invention are executed by a data processing device, for example, one or more programmable processors and/or one or more computing devices, or as a computer program stored in a computer-readable recording medium and/or a computer-readable recording medium. It can be implemented. The above-described computer program may be written in any form of programming language, including compiled language or interpreted language, and may be distributed in any form such as a stand-alone program, module, or subroutine. A computer program may be distributed via a single computing device, multiple computing devices connected through the same network, and/or multiple computing devices distributed so as to be connected through multiple different networks.

The above-described method and/or various embodiments include one or more processors configured to execute one or more computer programs that process, store, and/or manage any functions, functions, etc., by operating on input data or generating output data. It can be performed by . For example, the method and/or various embodiments of the present invention may be performed by special purpose logic circuits such as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), and the method and/or various embodiments of the present invention may An apparatus and/or system for performing embodiments may be implemented as a special purpose logic circuit, such as an FPGA or ASIC.

The one or more processors executing the computer program may include a general-purpose or special-purpose microprocessor and/or one or more processors in any type of digital computing device. The processor may receive instructions and/or data from each of read-only memory and random access memory, or may receive instructions and/or data from read-only memory and random access memory. In the present invention, components of a computing device performing methods and/or embodiments may include one or more processors for executing instructions and one or more memory devices for storing instructions and/or data.

According to one embodiment, a computing device may exchange data with one or more mass storage devices for storing data. For example, a computing device may receive data from and/or transmit data to a magnetic disc or optical disc. Computer-readable storage media suitable for storing instructions and/or data associated with a computer program include semiconductor memory devices such as EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable PROM), and flash memory devices. It may include, but is not limited to, any form of non-volatile memory. For example, computer-readable storage media may include magnetic disks such as internal hard disks or removable disks, magneto-optical disks, CD-ROM, and DVD-ROM disks.

To provide interaction with a user, the computing device may include a display device (e.g., cathode ray tube (CRT), liquid crystal display (LCD), etc.) for presenting or displaying information to the user and a display device (e.g., cathode ray tube (CRT), liquid crystal display (LCD), etc.) for displaying or presenting information to the user. It may include, but is not limited to, a pointing device (e.g., keyboard, mouse, trackball, etc.) capable of providing input and/or commands. That is, the computing device may further include any other types of devices for providing interaction with the user. For example, a computing device may provide the user with any form of sensory feedback for interaction with the user, including visual feedback, auditory feedback, and/or tactile feedback. In response, the user can provide input to the computing device through various gestures such as sight, voice, and movement.

In the present invention, various embodiments may be implemented in a computing system that includes a back-end component (e.g., a data server), a middleware component (e.g., an application server), and/or a front-end component. In this case, the components may be interconnected by any form or medium of digital data communication, such as a communications network. For example, a communication network may include a Local Area Network (LAN), a Wide Area Network (WAN), etc.

Computing devices based on example embodiments described herein may be implemented using hardware and/or software configured to interact with a user, including a user device, user interface (UI) device, user terminal, or client device. You can. For example, a computing device may include a portable computing device, such as a laptop computer. Additionally or alternatively, computing devices include personal digital assistants (PDAs), tablet PCs, game consoles, wearable devices, internet of things (IoT) devices, virtual reality (VR) devices, AR (augmented reality) devices, etc. may be included, but are not limited thereto. Computing devices may further include other types of devices configured to interact with a user. Additionally, the computing device may include a portable communication device (eg, a mobile phone, smart phone, wireless cellular phone, etc.) suitable for wireless communication over a network, such as a mobile communication network. The computing device may wirelessly connect to a network server using wireless communication technologies and/or protocols, such as radio frequency (RF), microwave frequency (MWF), and/or infrared ray frequency (IRF). It can be configured to communicate with.

The various embodiments herein, including specific structural and functional details, are illustrative. Accordingly, embodiments of the present invention are not limited to those described above and may be implemented in various other forms. Additionally, the terms used in the present invention are intended to describe some embodiments and are not to be construed as limiting the embodiments. For example, the singular forms of words and the like may be construed to include the plural, unless the context clearly indicates otherwise.

In the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which these concepts belong. . Additionally, commonly used terms, such as those defined in dictionaries, should be interpreted as having meanings consistent with their meaning in the context of the relevant technology.

Although the present invention has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present invention as understood by those skilled in the art. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

110: Counselor 120: Customer
130: Voice consultation recording server 140: Recording DB
142: Recorded data 150: Voice consultation quality evaluation system
152: Evaluation report

Claims (10)

A method for evaluating the quality of a voice consultation performed by at least one processor, comprising:
When a voice consultation is performed, receiving recorded data associated with the voice consultation;
Based on the received recording data, generating text data corresponding to the voices of the counselor and the customer over time;
generating a plurality of text sentences by dividing the generated text data into sentence units;
extracting a silent section included in the recorded data and verifying the accuracy of the plurality of generated text sentences based on the extracted silent section; and
performing a quality evaluation of the voice consultation using a time section of the voice consultation and a plurality of verified text sentences according to the time;
In the quality evaluation method of voice consultation, including,
The quality evaluation of the above voice consultation is,
The entire voice counseling process includes an introduction evaluation, listening ability evaluation, silence evaluation, information confirmation evaluation, and closing greeting evaluation performed in chronological order,
In the case of the listening ability evaluation, the step of performing a quality evaluation of the voice consultation is,
splitting the plurality of text sentences into a second set of text sentences associated with the counselor and a third set of text sentences associated with the customer; and
calculating a number of times that at least a portion of the text sentences in the second set overlap with at least a portion of the text sentences in the third set;
Including,
The number of overlaps is,
Overlap occurs over a certain time, and includes the number of times the counselor's utterance occurs continuously after the overlap, and the number of times the overlap exceeds another specific time even if the customer's utterance occurs after the overlap,
In the case of the silent evaluation, the step of performing a quality evaluation of the voice consultation is,
extracting a section in which mute occurs for another specific time based on the time section of the voice consultation and a plurality of text sentences according to time;
determining a text sentence before the section in which the muting occurred among the plurality of text sentences; and
determining whether a waiting request keyword is included in the text sentence before the determined muted section;
Quality evaluation method of voice consultation including.
According to paragraph 1,
The step of extracting a silent section included in the recording data and verifying the accuracy of the plurality of text sentences generated based on the extracted silent section includes,
generating a mel spectrogram associated with the recording data and extracting the silent section using the generated mel spectrogram;
Including, a method for evaluating the quality of voice consultation.
According to paragraph 1,
The step of performing a quality evaluation on the voice consultation is,
extracting a first set of text sentences included within a specific time range from among the plurality of text sentences;
dividing the extracted first set of text sentences into a plurality of morphemes; and
determining whether a greeting keyword associated with a greeting is included in the plurality of divided morphemes;
Including, a method for evaluating the quality of voice consultation.
delete delete According to paragraph 1,
The step of performing a quality evaluation on the voice consultation is,
splitting the plurality of text sentences into a second set of text sentences associated with the counselor and a third set of text sentences associated with the customer; and
extracting a personal information text sentence containing the customer's personal information from the third set of text sentences;
extracting a text sentence immediately following the extracted personal information text sentence from among the second set of text sentences; and
determining whether a text sentence immediately following the extracted personal information text sentence includes a keyword corresponding to the customer's personal information;
Including, a method for evaluating the quality of voice consultation.
According to paragraph 1,
The step of performing a quality evaluation on the voice consultation is,
extracting a fourth set of text sentences corresponding to a last specific number of the plurality of text sentences; and
determining whether information associated with the counselor is included in the fourth set of text sentences using a Named Entity Recognition (NER) algorithm;
Including, a method for evaluating the quality of voice consultation.
According to paragraph 1,
generating a quality evaluation report by visualizing result data for quality evaluation of the voice consultation for each quality evaluation item;
A method for evaluating the quality of voice consultation, further comprising:
A computer program stored in a computer-readable recording medium for executing the quality evaluation method of voice consultation according to any one of claims 1 to 3 and 6 to 8 on a computer.
As a voice consultation quality evaluation system,
When a voice consultation is performed, a text conversion unit that receives recording data associated with the voice consultation and generates text data corresponding to the voices of the counselor and the customer over time based on the received recording data;
a sentence generator that divides the generated text data into sentence units to generate a plurality of text sentences;
a sentence verification unit that extracts a silent section included in the recorded data and verifies the accuracy of the plurality of generated text sentences based on the extracted silent section; and
a quality evaluation performing unit that performs a quality evaluation of the voice consultation using a time section of the voice consultation and a plurality of verified text sentences according to the time;
In the voice consultation quality evaluation system including,
The quality evaluation of the above voice consultation is,
The entire voice counseling process includes an introduction evaluation, listening ability evaluation, silence evaluation, information confirmation evaluation, and closing greeting evaluation performed in chronological order,
In the case of the listening ability evaluation, the quality evaluation performance department,
Splitting the plurality of text sentences into a second set of text sentences associated with the counselor and a third set of text sentences associated with the customer, wherein at least a portion of the second set of text sentences is one of the third set of text sentences. Calculate the number of times it overlaps with at least a part,
The number of overlaps is,
Overlap occurs over a certain time, and includes the number of times the counselor's utterance occurs continuously after the overlap, and the number of times the overlap exceeds another specific time even if the customer's utterance occurs after the overlap,
In the case of the silent evaluation, the quality evaluation performance unit,
Based on the time section of the voice consultation and a plurality of text sentences according to time, a section in which muting occurred during another specific time is extracted, and a text sentence before the section in which the muting occurred among the plurality of text sentences is determined. and determining whether a waiting request keyword is included in the text sentence before the section in which the determined muting occurred,
Voice consultation quality evaluation system.