JP6799574B2 - Method and device for determining satisfaction with voice dialogue - Google Patents

Description

Embodiments of the present invention relate to the field of voice dialogue, and particularly to methods and devices for determining satisfaction with voice dialogue.

With the development of artificial intelligence technology, the development and use of smart voice dialogue products is drawing attention. Note that the smart voice dialogue is a dialogue mode by voice input, the user can input his / her own request by voice, and the product can respond to the contents of the correspondence according to the intention of the request.

In the field of voice dialogue, it is important to evaluate the satisfaction of voice dialogue of the product in order to make a better manufacture and upgrade for the smart voice dialogue product. This is because the evaluation of the satisfaction level of the voice dialogue can reflect whether or not the voice dialogue function of the product is recognized by the user. In the prior art, the evaluation of satisfaction mainly uses the intention of one request by the user and the content of the response by the terminal as evaluation processing data, and calculates the relevance of the intention and the content by some text processing techniques. The user's satisfaction level according to the response is labeled.

However, it is not possible to obtain a real and complete evaluation of the voice dialogue from the user in order to obtain the user's satisfaction only by the relationship between the content returned from the terminal and the user's intention. It is difficult to evaluate the satisfaction level of the dialogue.

Embodiments of the present invention provide a method and apparatus for determining the satisfaction level of a voice dialogue that can provide a realistic and complete evaluation for the voice dialogue.

As a first aspect, the embodiment of the present invention is a method for determining the satisfaction level of a voice dialogue, and is a step of acquiring a voice dialogue feature including objective data of the voice dialogue and subjective data of the voice dialogue. , The objective data of the voice dialogue and the subjective data of the voice dialogue are data for the same subject. The step and the objective data are evaluated to obtain an objective evaluation, and the subjective data is evaluated. A step of performing processing and acquiring a subjective evaluation, and a step of acquiring the satisfaction of the voice dialogue output from the satisfaction evaluation model by using the objective evaluation and the subjective evaluation as inputs of the satisfaction evaluation model. Provided is a method for determining the satisfaction level of a voice dialogue including.

In a possible design, the objective data of the voice dialogue includes the user intent and the response content, the response delay, and the current playback time of the response content, and the subjective data of the voice dialogue interrupts the reproduction of the response content. Contains text information that corresponds to voice input by the user after it has been done.

In a possible design, the step of performing evaluation processing on the objective data and acquiring the objective evaluation is a step of acquiring the first objective evaluation based on the degree of intentional match between the user's intention and the response content. , A step of acquiring a second objective evaluation based on the response delay and the standard delay, and a step of acquiring a third objective evaluation based on the current reproduction time of the response content and the standard reproduction time of the response content. The step of acquiring the satisfaction level of the voice dialogue output from the satisfaction evaluation model by inputting the objective evaluation and the subjective evaluation into the satisfaction evaluation model includes the first objective evaluation and the second objective evaluation. It includes a step of acquiring the satisfaction level of the voice dialogue output from the satisfaction evaluation model by inputting the evaluation, the third objective evaluation, and the subjective evaluation into the satisfaction evaluation model.

In a possible design, the step of performing evaluation processing on the subjective data to obtain the subjective evaluation is to perform a word meaning analysis on the text information and use emotional attributes or subject attributes corresponding to the text information. It includes a step of acquiring a certain content attribute and a step of acquiring a subjective evaluation based on the content attribute corresponding to the text information.

In a possible design, if the content attribute is a subject attribute, the steps to acquire the subjective evaluation based on the content attribute corresponding to the text information are the step to acquire the subject type corresponding to the text information and the step. If the subject type corresponding to the text information matches the subject type corresponding to the user's intention, the step of determining that the subjective evaluation is lower than a predetermined evaluation value is included.

In a possible design, if the content attribute is an emotional attribute , the steps of acquiring the subjective evaluation based on the content attribute corresponding to the text information include a step of extracting the emotional keyword in the text information and the emotional keyword. The step of acquiring one of the mood types consisting of positive mood, negative mood, and neutral mood based on the correspondence between mood type and mood type, and the mood type and predetermined mood type. Includes a step to obtain a subjective evaluation based on the correspondence with the evaluation of.

In a possible design, the step of acquiring the voice dialogue feature is to select the first log data whose time interval between the second log data in the immediately preceding time zone and the third log data in the immediately following time zone is larger than a predetermined threshold. Based on the step to acquire, the step to acquire the subject corresponding to each of the two adjacent voice inputs by the user from the first log data, and the subject corresponding to each of the two adjacent voice inputs. Includes a step of acquiring the spoken dialogue feature.

In a possible design, the method further takes the objective evaluation and the subjective evaluation as inputs to the satisfaction evaluation model and further before the step of acquiring the satisfaction of the voice dialogue output from the satisfaction evaluation model. The first sample evaluation obtained by performing evaluation processing on objective sample data, the second sample evaluation obtained by performing evaluation processing on subjective sample data, and the satisfaction input by the user. A step of obtaining a training sample set that includes a degree, wherein the objective sample data and the subjective sample data are data for the same subject, based on the step and the training sample set. It includes a step of acquiring the satisfaction evaluation model by iteration training.

As a second aspect, the embodiment of the present invention is a device for determining the satisfaction level of voice dialogue, and is an acquisition module for acquiring voice dialogue features including objective data of voice dialogue and subjective data of voice dialogue. The objective data of the voice dialogue and the subjective data of the voice dialogue are data for the same subject. The acquisition module and the objective data are evaluated to acquire the objective evaluation, and the subjective data is subjected to evaluation processing. The processing module that performs the evaluation process to acquire the subjective evaluation, and the objective evaluation and the subjective evaluation are input to the satisfaction evaluation model, and the satisfaction level of the voice dialogue output from the satisfaction evaluation model is acquired. Provided is a device for determining the satisfaction level of a voice dialogue including a module.

In a possible design, the objective data of the voice dialogue includes the user intention and the response content, the response delay, and the current playback time of the response content, and the subjective data of the voice dialogue includes the reproduction of the response content. It includes text information corresponding to voice input by the user after the interruption, or text information input by the user after the reproduction of the response content is completed.

In a possible design, the processing module specifically obtains a first objective evaluation based on the degree of matching between the user's intention and the response content, and based on the response delay and standard delay. The second objective evaluation is acquired, and the third objective evaluation is acquired based on the current reproduction time of the response content and the standard reproduction time of the response content, and the confirmation module is specifically the first. It is configured to acquire the satisfaction level of the voice dialogue output from the satisfaction evaluation model by using the one objective evaluation, the second objective evaluation, the third objective evaluation, and the subjective evaluation as inputs of the satisfaction evaluation model. There is.

In a possible design, the processing module specifically performs a word meaning analysis on the text information to acquire a content attribute which is an emotional attribute or a subject attribute corresponding to the text information, and uses the text information as the text information. It is configured to acquire subjective evaluations based on the corresponding content attributes.

In a possible design, when the content attribute is a subject attribute, the processing module specifically acquires the subject type corresponding to the text information, and the subject type corresponding to the text information is intended by the user. If it matches the corresponding subject type, the subjective evaluation is configured to be determined to be lower than a predetermined evaluation value.

In a possible design, when the content attribute is an emotional attribute , the processing module specifically extracts an emotional keyword in the text information, and based on the correspondence between the emotional keyword and the mood type, Acquire one of the mood types in the group consisting of positive mood, negative mood, and neutral mood, and acquire subjective evaluation based on the correspondence between the mood type and a predetermined evaluation. It is configured to do.

In a possible design, the acquisition module specifically captures first log data in which the time interval between the second log data in the immediately preceding time zone and the third log data in the immediately following time zone is greater than a predetermined threshold. Acquired, from the first log data, the subject corresponding to each of the two adjacent voice inputs by the user is acquired, and the voice dialogue feature is based on the subject corresponding to each of the two adjacent voice inputs. Is configured to get.

In a possible design, the device further comprises a training module, the training module satisfying the voice dialogue output from the satisfaction evaluation model with the objective evaluation and the subjective evaluation as inputs to the satisfaction evaluation model. The first sample evaluation obtained by performing evaluation processing on objective sample data and the second sample evaluation obtained by performing evaluation processing on subjective sample data before acquiring the degree. , A training sample set including the satisfaction level input from the user is obtained, and the objective sample data and the subjective sample data are data for the same subject, and based on the training sample set, It is configured to obtain the satisfaction evaluation model by iteration training.

As a third aspect, an embodiment of the present invention is a device for determining the satisfaction level of a voice dialogue, which includes at least one processor and a memory, and stores commands that can be executed by a computer in the memory. The at least one processor is described in various possible designs of the first aspect or the first aspect of the at least one processor by executing a command that can be executed by the computer stored in the memory. Provided is a device for determining the satisfaction level of the voice dialogue, which executes the method for determining the satisfaction level of the voice dialogue.

As a fourth aspect, the embodiment of the present invention is a computer-readable storage medium, in which commands that can be executed by a computer are stored in the computer-readable storage medium, and commands that can be executed by the computer. When executed by a processor, it provides a computer-readable storage medium that implements the method of determining the satisfaction of a voice dialogue as described in the various possible designs of the first aspect or the first aspect.

The method and apparatus for determining the satisfaction level of a voice dialogue provided by an embodiment of the present invention acquires a voice dialogue feature including objective data of the voice dialogue and subjective data of the voice dialogue, which are data for the same subject. By acquiring the subjective data and the objective data of the same subject, the data for evaluation of satisfaction can be acquired in a real and complete manner, and the realism and completeness of the evaluation data can be ensured. Satisfaction is more comprehensive and closer to the user's realistic evaluation. Satisfaction evaluation model by performing evaluation processing on objective data to acquire objective evaluation, performing evaluation processing on subjective data to acquire subjective evaluation, and using objective evaluation and subjective evaluation as input of satisfaction evaluation model By acquiring the satisfaction level of the voice dialogue output from, the satisfaction level can be obtained quickly and accurately by acquiring the satisfaction level by the means of the satisfaction evaluation model. Therefore, when the method has a large amount of voice dialogue. Will be applicable to.

In order to more clearly explain the technical proposal in the embodiment or the prior art of the present invention, the drawings necessary for the explanation of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present invention, and other drawings can be further obtained based on these drawings on the premise that no creative labor is applied to those skilled in the art.
It is a schematic diagram of the architecture of the satisfaction determination system of the voice dialogue provided by the embodiment of the present invention. It is a schematic diagram of acquisition of the satisfaction evaluation model provided by the Example of this invention. It is the first flowchart of the method of determining the satisfaction degree of the voice dialogue provided by the Example of this invention. It is a schematic diagram of the log data provided by the Example of this invention. It is a second flowchart of the method of determining the satisfaction degree of the voice dialogue provided by the Example of this invention. It is a flowchart of acquisition of the subjective evaluation provided by the Example of this invention. It is a schematic diagram of the configuration of the device for determining the satisfaction level of the voice dialogue provided by the embodiment of the present invention. It is a block diagram of the hardware of the device for determining the satisfaction of the voice dialogue provided by the embodiment of the present invention.

In order to further clarify the purpose, technical proposal and advantages of the examples of the present invention, the technical proposal in the examples of the present invention will be described in a clear and complete manner together with the drawings in the examples of the present invention. It goes without saying that the examples described are not all examples, but some examples of the present invention. All other embodiments obtained by one of ordinary skill in the art based on the embodiments of the present invention on the premise that no creative labor is applied also belong to the scope of protection of the present invention.

FIG. 1 is a schematic diagram of the architecture of the voice dialogue satisfaction determination system provided by the embodiments of the present invention. As shown in FIG. 1, the system provided by this embodiment includes a terminal 101 and a server 102. The terminal 101 may be a child story machine, a mobile phone, a tablet, an in-vehicle terminal, or the like. In this embodiment, the means for realizing the terminal 101 is not limited, as long as the terminal 101 can execute a voice dialogue with the user.

Speech Interaction is a technology such as speech recognition, speech synthesis, and natural language analysis that gives the terminal the function of "listening, speaking, and understanding" in multiple types of actual application scenes. A smart human-machine interaction experience that applies to multiple application scenes, including scenes such as smart questions and answers, smart playback, and intelligent search.

The user inputs a search term to the terminal 101 by voice. The terminal 101 can acquire the user's intention by the search term, acquire the response content corresponding to the intention locally or from the server 102 according to the intention, and feed back the response content to the user. For example, booking food, booking tickets, searching for music, movies, or certain products.

Since the terminal 101 can provide various voice dialogue services to the user, it is important for the development and upgrade of the voice dialogue in the terminal to obtain a real and complete evaluation of the voice dialogue from the user.

The embodiments of the present invention provide a method for determining the satisfaction level of a voice dialogue. The method can provide a realistic and complete evaluation of spoken dialogue. The method of determining the satisfaction level of the voice dialogue can be executed by the terminal 101 shown in FIG. The terminal 101 can determine the satisfaction level based on the log data, then feed back the satisfaction level to the server 102, and cause the server 102 to execute further processing based on the satisfaction level. The method of determining the satisfaction level of the voice dialogue of this embodiment may be executed by the server 102 shown in FIG. The terminal 101 transmits the log data to the server 102, and causes the server 102 to determine the satisfaction level and execute further processing.

In this embodiment, there is no particular restriction on the executing subject of the method for determining the satisfaction level of the voice dialogue, and the method may be executed by the terminal 101 or the server 102. In this embodiment, the terminal 101 and the server 102 may both adopt the same method to determine the satisfaction level of the voice dialogue after acquiring the log data.

In this embodiment, in order to quickly determine the satisfaction level, the satisfaction level is determined by the satisfaction evaluation model. In this embodiment, first, the process of acquiring the satisfaction evaluation model will be described together with FIG.

FIG. 2 is a schematic diagram of acquisition of the satisfaction evaluation model provided by the embodiment of the present invention. In this embodiment, satisfaction is obtained by evaluating objective data and subjective data. As a result, when acquiring the satisfaction evaluation model, the objective sample data and the subjective sample data are simultaneously considered to acquire a real and complete evaluation of the voice dialogue from the user. The satisfaction evaluation model may be acquired by the terminal or the server, and after the satisfaction evaluation model is acquired by the server or the server, the satisfaction evaluation model is installed in the installation package. You may send it to the terminal.

As shown in FIG. 2, a training sample set is first obtained. The training sample set includes a first sample evaluation, a second sample evaluation, and a satisfaction level input by the user. As can be understood by those skilled in the art, a satisfaction evaluation model is performed by performing iteration training on appropriate amounts of data in the training sample set, that is, first sample evaluation, second sample evaluation, and satisfaction input from the user. Can be obtained.

The first sample evaluation is obtained by performing evaluation processing on objective sample data, and the second sample evaluation is obtained by performing evaluation processing on subjective sample data. In this embodiment, the objective sample data and the subjective sample data are data for the same subject.

The objective sample data is data that does not include the emotional nuances of the user, for example, data relating to a terminal. The subjective sample data is data including emotional nuances of the user. If it is data for the same subject, it is possible to acquire all of them and distinguish between subjective and objective.

In a specific example, "I want to hear Zhang San's new song" is acquired as the user's intention according to the user's voice input, and the response content returned from the terminal according to the intention is "XX song". It is an audio file of. When the user gives subjective feedback about the "XX song", the user stops playing the audio file at once, and then inputs "This song is not what I want" by voice. , Enter your own satisfaction with the terminal. In this process, the playback time and response delay of the "XX song" are acquired. As a result, the user had two dialogues with the terminal on the subject of the "song".

In this process, the objective sample data may include the user's intention and response content, response delay, and the current playback time of the response content. As for the objective evaluation, the objective evaluation can be obtained from these objective sample data by a predetermined algorithm. The predetermined algorithm may be a function or the like including the objective sample data, and is not limited to the predetermined algorithm in this embodiment.

The subjective sample data includes text information corresponding to the voice input by the user after the reproduction of the response content is interrupted. It is possible to obtain a subjective evaluation by performing a word meaning analysis on the text information.

As can be understood by those skilled in the art, the objective evaluation and the subjective evaluation may be specific evaluation values, and the satisfaction level input by the user may also be a specific value. A satisfaction evaluation model can be obtained by performing iteration training on these values.

It is also possible to selectively perform evaluation processing on different objective sample data to obtain the corresponding evaluation values. For example, the evaluation value x1 can be acquired according to the user's intention and the response content, the evaluation value y1 can be acquired according to the response delay, and the evaluation value z1 can be acquired according to the current reproduction time of the response content. The evaluation value corresponding to the subjective evaluation is p1, and the satisfaction level input from the user is n1.

In this example, a possible satisfaction evaluation model was given. The satisfaction evaluation model may be Ax + By + Cz + Dp = n. A, B, C, and D can be obtained by substituting each evaluation value into the satisfaction evaluation model and performing iteration training. As a result, a satisfaction evaluation model can be obtained. After the evaluation value is acquired, if it is directly substituted into the model, the final satisfaction level n can be acquired. In this embodiment, a kind of model is illustrated to explain how to construct a satisfaction evaluation model, but other types of models can provide satisfaction for subjective evaluation and objective evaluation. , Both fall within the scope of protection of this embodiment. In this embodiment, there are no restrictions on the specific means of realizing the satisfaction evaluation model.

Hereinafter, a detailed example will be adopted to explain how to obtain the satisfaction level by the satisfaction evaluation model in the embodiment of the present invention.

FIG. 3 is a first flowchart of the method for determining the satisfaction level of the voice dialogue provided by the embodiment of the present invention. The execution subject of the method may be the terminal or server shown in FIG. As shown in FIG. 3, the method includes the following steps.

In S301, the voice dialogue feature is acquired. The voice dialogue feature includes objective data of voice dialogue and subjective data of voice dialogue. The objective data and the subjective data are data for the same subject.

The voice dialogue feature of this embodiment can be acquired from the log data of the terminal. In the specific realization procedure, the terminal acquires the voice input from the user, converts the voice into text information, acquires the user intention based on the text information, and acquires the response content according to the user intention. And feed back the response contents to the user. The user can give subjective feedback according to the content of the response.

Regarding the dialogue data between the user and the terminal, after acquiring the corresponding subject for each data, it is possible to acquire the subjective data and the objective data for the same subject. Specifically, it is possible to generate subjective data and objective data for the same subject by performing word meaning analysis, content analysis, time series analysis at the time of data generation, etc. on the dialogue data. In this embodiment, the realization means for acquiring data of the same subject is not limited.

Optionally, in a specific example, the voice input time, the text information and the response content indicator, the attribute, and the response time all form log data. As can be understood by those skilled in the art, the time is recorded for each log in the log data, and the first log data is acquired based on the time information. The first log data is a series of log data in which the user and the terminal continuously interact with each other.

In a specific implementation procedure, the time interval between each log recording is acquired, and when the two time intervals are larger than a predetermined threshold value, the log data between the two time intervals, that is, the first log data is acquired. To do. The first log data is all data between the two time intervals.

As will be appreciated by those skilled in the art, the time interval between the first log data and the second log data in the time zone before the adjacency and the third log data in the adjacent next time zone is larger than a predetermined threshold value. The second log data and the third log data may be understood as one log data, or may be understood as all data between two time intervals similar to the first log data. FIG. 4 is a schematic diagram of log data provided by an embodiment of the present invention. As shown in FIG. 4, on the time axis, the central portion is the recording time of each data in the first log data. As can be understood by those skilled in the art, the first log data, the second log data, and the third log data correspond to data partitioned according to a session.

After the first log data is acquired, the voice dialogue feature is acquired in the first log data based on the text information corresponding to the two adjacent voice inputs by the user.

As can be understood by those skilled in the art, when performing a voice dialogue, the voice dialogue procedure between the user and the terminal is voice input-response content-word meaning input-response content .... As a result, the voice input and the response content are repeatedly generated.

After the voice input is converted into text information, the word meaning analysis can be performed on the text information. The text information may be the user's intention or the user's feedback. In this embodiment, when each of the two adjacent text information is the user intention and the user feedback, the voice dialogue feature is extracted by extracting the information related to the user intention, the user feedback, and the response content of the terminal feedback. To get. That is, the feature information is extracted in the procedure of voice input-response content-word meaning input. The procedure is user intent-response content-user feedback. If the user's feedback does not have any content related to another subject, it is considered that the user's feedback still corresponds to the subject corresponding to the user's intention, that is, the subject does not change. The subjective data is user feedback, and the objective data includes information related to the user's intention and response content.

As will be appreciated by those skilled in the art, if some continuous voice inputs involve two different subjects at the same time, the voice inputs can be divided into two parts of content. For example, if the front part is related to one subject and the rear part is related to another subject, the front part can be divided into one subject to extract voice dialogue features, and the rear part can be divided into other subjects to extract voice features. it can.

In S302, the objective data is evaluated and the objective evaluation is acquired, and the subjective data is evaluated and the subjective evaluation is acquired.

As will be appreciated by those skilled in the art, objective data and subjective data are some data. These data have different formats, different formats, or different types. In order to unify the input of the satisfaction evaluation model by unifying the data, the objective data is evaluated and the objective evaluation is acquired, and the subjective data is evaluated and the subjective evaluation is acquired. Can be done.

Both the objective evaluation and the subjective evaluation are evaluation values. The evaluation value may be calculated based on the total value of satisfaction, or may be determined for different data types. Alternatively, the evaluation value may take a value of -1 to 1.

As can be understood by those skilled in the art, the evaluation processing method for objective data and the evaluation processing method for subjective data are the same as the evaluation processing method for establishing the satisfaction evaluation model, thereby evaluating the evaluation value and satisfaction. Ensured the rationality of the model.

In S303, the objective evaluation and the subjective evaluation are input to the satisfaction evaluation model, and the satisfaction level of the voice dialogue output from the satisfaction evaluation model is acquired.

After acquiring the objective evaluation and the subjective evaluation, the objective evaluation and the subjective evaluation are input to the satisfaction evaluation model, and the calculation and analysis are performed by the satisfaction evaluation model. The output of the satisfaction evaluation model is the satisfaction of the voice dialogue.

The method of determining the satisfaction level of the voice dialogue provided by the embodiment of the present invention is to acquire a voice dialogue feature including objective data of the voice dialogue and subjective data of the voice dialogue, which are data for the same subject. By acquiring the subjective data and the objective data of the same subject, the data for evaluation of satisfaction can be acquired in a real and complete manner, and the realism and completeness of the evaluation data can be ensured. Is more comprehensive and closer to the user's realistic evaluation. Evaluation processing is performed on the objective data to obtain the objective evaluation, evaluation processing is performed on the subjective data to obtain the subjective evaluation, and the objective evaluation and the subjective evaluation are input from the satisfaction evaluation model from the satisfaction evaluation model. By acquiring the satisfaction level of the output voice dialogue, the satisfaction level can be obtained quickly and accurately by acquiring the satisfaction level by the means of the satisfaction evaluation model. Therefore, when the method has a large amount of voice dialogue. Will be applicable to.

The realization method for acquiring objective evaluation and subjective evaluation by adopting a specific realization method will be described in detail below. In this embodiment, the objective data of the voice dialogue includes the user intention and the response content, the response delay, and the current playback time of the response content. The subjective data of the voice dialogue includes text information corresponding to the voice input by the user after the reproduction of the response content is interrupted.

It means that the higher the degree of intentional match between the user's intention and the response content, the shorter the response delay, and the longer the current playback time of the response content, the greater the user's satisfaction and the larger the objective evaluation value.

Acquire the mood information of the text information. The better the user feels, the larger the subjective evaluation value.

Hereinafter, a detailed description will be given together with FIGS. 5 and 6. The following implementation method for acquiring the objective evaluation and the subjective evaluation may be applied to the model establishment example shown in FIG. 2 above, or may be applied to the satisfaction determination example shown in FIG. Is also good.

FIG. 5 is a second flowchart of the method for determining the satisfaction level of the voice dialogue provided by the embodiment of the present invention. As shown in FIG. 5, the method includes the following.

In S501, the voice dialogue feature is acquired. The voice dialogue feature includes objective data of voice dialogue and subjective data of voice dialogue. The objective data of the voice dialogue includes the user intention and the response content, the response delay, and the current playback time of the response content.

Specifically, the method of acquiring the voice dialogue feature in S501 is similar to the method of acquiring the voice dialogue feature in S101. The objective data of this embodiment includes the user intention and the response content, the response delay, and the current playback time of the response content.

The objective data includes various feedbacks on the user's intention and the user's intention from the terminal. It should be noted that the intention is an operation on the area data, which is generally named by a guest phrase, such as a weather inquiry and a music search. Feedback to the user's intention from the terminal includes the response content, the response delay, and the current playback time of the response content.

In S502, the first objective evaluation is acquired based on the intention match degree between the user intention and the response content.

Semantic parsing is performed on the user's intention to acquire the contents such as the area attribute in the user's intention, and the contents such as the area attribute of the response content are extracted. The degree of intentional match is determined based on the similarity of the attributes of the area. The value of the intentional match degree may be 0% -100%. The intentional match degree can be converted into the first objective evaluation, and the value of the first objective evaluation is 0 to 1 accordingly. For example, when the user's intention is Zhang San's A song, the response content is Zhang San's B song, and the area attributes are the singer name and song name, the similarity is 50% and the corresponding first objective evaluation is 0.5. Is.

In S503, a second objective evaluation is acquired based on the response delay and the standard delay.

The standard delay may be a preset acceptable delay for the user. For example, the standard delay is 200 ms, and the second objective evaluation corresponding to the standard delay is 0. When the response delay is larger than the standard delay, the second objective evaluation is less than 0, and when the response delay is smaller than the standard delay, the second objective evaluation is larger than 0.

In a possible implementation, a standardization process may be performed based on the standard delay so that the second objective evaluation is between -1 and 1. For example, when the response delay is larger than the standard delay, if the difference between the response delay and the standard delay is larger than the standard delay, the value of the second objective evaluation is 1, and if the difference is smaller than the standard delay, the response is concerned. Take a negative ratio between the difference and the standard delay to get a second objective evaluation. When the response delay is smaller than the standard delay, the difference between the standard delay and the response delay is acquired, and the ratio of the difference to the standard delay is used as the second objective evaluation.

In S504, the third objective evaluation is acquired based on the current reproduction time of the response content and the standard reproduction time of the response content.

In this embodiment, the longer the current playback time of the response content, the higher the user's satisfaction. The ratio of the current playback time to the standard playback time can be used as the third objective evaluation.

In S505, the subjective evaluation is obtained by performing an evaluation process on the subjective data.

In S506, the satisfaction level of the voice dialogue output from the satisfaction evaluation model is acquired by using the first objective evaluation, the second objective evaluation, the third objective evaluation, and the subjective evaluation as inputs of the satisfaction evaluation model. ..

The first objective evaluation, the second objective evaluation, the third objective evaluation, and the subjective evaluation are input to the satisfaction evaluation model, and the satisfaction level of the voice dialogue is directly output from the satisfaction evaluation model by the processing of the satisfaction evaluation model.

As will be appreciated by those skilled in the art, when establishing a satisfaction evaluation model, model inputs are input from the first objective evaluation, the second objective evaluation, the third objective evaluation, the subjective evaluation, and the user corresponding to the sample data. Satisfaction. Then, iterate training is performed to acquire a satisfaction evaluation model.

In the embodiment of the present invention, various objective elements such as content, delay, and playback time are obtained in the objective evaluation by acquiring objective evaluations for the user intention, the response content, the response delay, and the current playback time of the response content, respectively. It is possible to obtain the user's satisfaction level by integrating the above, and the satisfaction level becomes accurate and complete.

FIG. 6 is a flowchart for acquiring the subjective evaluation provided by the embodiment of the present invention. The objective evaluation provided by this embodiment is applicable to any one of the above embodiments. The subjective data of the voice dialogue corresponding to the objective evaluation includes text information corresponding to the voice input by the user after the reproduction of the response content is interrupted, or text information input by the user after the reproduction of the response content is completed. That is, feedback on the response content from the user is included. The feedback may be direct feedback, mood feedback, or the like. As shown in FIG. 6, the method includes the following.

In S601, the word meaning analysis is performed on the text information, and the content attribute corresponding to the text information is acquired. The content attribute is an emotional attribute or a subject attribute.

After the text information corresponding to the voice feedback input to the response content from the user is acquired, the word meaning analysis is performed on the text information to acquire the content attribute corresponding to the text information. In this embodiment, the content attributes are divided into emotional attributes and subject attributes. The emotional attribute refers to the content that the user conveys emotions, and the subject attribute refers to further operations from the user to the current subject.

In a specific realization process, a keyword in text information can be extracted by word meaning analysis, and the keyword can be matched with a word in an emotion bank or a subject bank to determine a content attribute corresponding to the text information.

In S602, the subject type corresponding to the text information is acquired.

In S603, if the subject type corresponding to the text information matches the subject type corresponding to the user's intention, it is determined that the subjective evaluation is lower than the predetermined evaluation value.

Note that S602-S603 is set for the content attribute of the text information being the subject attribute. If the subject type corresponding to the text information matches the subject type corresponding to the user's intention, it means that the user is not satisfied with the response content fed back about the current subject, so the input content is repeated again. Therefore, the subjective evaluation is lower than the predetermined evaluation value. A specific evaluation value can be determined based on the number of times the input by the user is repeated and the length of the input character.

In S604, the emotional keyword in the text information is extracted.

In S605, one of the mood types is acquired based on the correspondence between the emotion keyword and the mood type. The mood types include positive mood, negative mood and neutral mood.

In S606, a subjective evaluation is acquired based on the correspondence between the mood type and the predetermined evaluation.

In addition, S604-S606 is set for the content attribute being a mood attribute. Emotion keywords in text information can be extracted. In this embodiment, a mood database can be set up in advance. The mood database includes a positive mood subdatabase, a negative mood subdatabase, and a neutral mood subdatabase.

Match the emotional keyword with a word in the mood database. When the emotional keyword is matched to the positive mood subdatabase, the mood type is positive mood and the corresponding subjective evaluation value is 1. When the emotional keyword is matched to the negative mood subdatabase, the mood type is negative mood and the corresponding subjective evaluation value is -1. When the emotional keyword is matched to the neutral mood subdatabase, the mood type is neutral mood and the corresponding subjective evaluation value is 0.

In the embodiment of the present invention, the subjective satisfaction with the voice dialogue from the user can be obtained by analyzing the subjective feedback of the user. An embodiment of the present invention can be evaluated for various operations of the user by analyzing the mood of the user and repeated instructions for the same subject from the user when determining the subjective evaluation. It is possible to give real feedback on user satisfaction without cheating on the expression of.

In each of the above examples, the examples of the present invention have shown the process of determining satisfaction with a dialogue on one subject. In the concrete realization process, the satisfaction level of dialogue on a plurality of subjects can be continuously obtained. For the method of determining the satisfaction level of the dialogue on each subject, the above-described embodiment can be referred to. Some specific examples will be provided below to explain how to obtain satisfaction with dialogues on multiple subjects.

One specific example is user intention 1-response content 1-user intention 2-response content 2-subjective feedback + user intention 3-response content 3 .... As can be seen from the above contents, since the user does not give feedback to the user regarding the user intention 1 and the response content 1, it is sufficient to obtain an objective evaluation. The above-described embodiment can be referred to as a method for realizing the objective evaluation. The satisfaction evaluation model may be involved only in the objective evaluation, and may be involved in both the objective evaluation and the subjective evaluation, but the input of the subjective evaluation may be 0. User intention 2-Response content 2-Subjective feedback can be realized by adopting the above-mentioned method for determining satisfaction. User Intent 3-Satisfaction level can be determined for response content 3 based on the specific realization status.

After acquiring the plurality of satisfaction levels, the terminal or the server performs various analyzes on the data of the plurality of satisfaction levels to acquire the product performance of the terminal.

As will be understood by those skilled in the art, the evaluation value valuation method corresponding to each of the above subjective evaluations or objective evaluations is an exemplary valuation method, but other valuation methods may be adopted. For example, the pricing method differs depending on the difference between the 5-point system, the 10-point system, and the 100-point system, and the present embodiment is not limited to a specific pricing method.

FIG. 7 is a schematic configuration diagram of a device for determining the satisfaction level of the voice dialogue provided by the embodiment of the present invention. As shown in FIG. 7, the apparatus 70 includes an acquisition module 701, a processing module 702, a confirmation module 703, and optionally further a training module 704.

The acquisition module 701 acquires voice dialogue features. The voice dialogue feature includes objective data of voice dialogue and subjective data of voice dialogue. The objective data and the subjective data are data for the same subject.

The processing module 702 performs evaluation processing on the objective data to acquire the objective evaluation, and performs evaluation processing on the subjective data to acquire the subjective evaluation.

The confirmation module 703 takes the objective evaluation and the subjective evaluation as inputs of the satisfaction evaluation model, and acquires the satisfaction level of the voice dialogue output from the satisfaction evaluation model.

Optionally, the objective data of the voice dialogue includes the user's intention and response content, response delay, and the current playback time of the response content. The subjective data of the voice dialogue includes text information corresponding to the voice input by the user after the reproduction of the response content is interrupted, or text information input by the user after the reproduction of the response content is completed.

Optionally, the processing module 702 specifically acquires the first objective evaluation based on the intention match degree between the user intention and the response content, and performs the second objective evaluation based on the response delay and the standard delay. The third objective evaluation is acquired based on the current reproduction time of the response content and the standard reproduction time of the response content.

Specifically, the confirmation module 703 uses the first objective evaluation, the second objective evaluation, the third objective evaluation, and the subjective evaluation as inputs of the satisfaction evaluation model, and the voice dialogue output from the satisfaction evaluation model. Get satisfaction.

Optionally, the processing module 702 specifically performs a word meaning analysis on the text information to acquire a content attribute which is an emotional attribute or a subject attribute corresponding to the text information, and corresponds to the text information. Obtain a subjective evaluation based on the content attributes.

Optionally, if the content attribute is a subject attribute, the processing module 702 specifically acquires the subject type corresponding to the text information, and the subject type corresponding to the text information is the subject corresponding to the user's intention. If it matches the type, it is determined that the subjective evaluation is lower than the predetermined evaluation value.

Alternatively, if the content attribute is an emotional attribute , the processing module 702 specifically extracts an emotional keyword in the text information and has a positive mood based on the correspondence between the emotional keyword and the mood type. , A mood type consisting of a negative mood and a neutral mood is acquired, and a subjective evaluation is acquired based on the correspondence between the mood type and a predetermined evaluation.

Optionally, the acquisition module 701 specifically defines a time interval between the second log data in the immediately preceding (previous adjacent) time zone and the third log data in the immediately preceding (later adjacent) time zone. Acquires the first log data larger than the threshold value of, acquires the subject corresponding to each of the two adjacent voice inputs by the user from the first log data, and corresponds to each of the two adjacent voice inputs. Acquire the spoken dialogue feature based on the subject.

Selectively, the training module 704 sets the first sample evaluation, the first sample evaluation, before acquiring the satisfaction level of the voice dialogue output from the satisfaction evaluation model by inputting the objective evaluation and the subjective evaluation into the satisfaction evaluation model. (Ii) A training sample set including the sample evaluation and the satisfaction level input from the user is acquired, and the satisfaction evaluation model is acquired by iteration training based on the training sample set. The first sample evaluation is obtained by performing an evaluation process on objective sample data, and the second sample evaluation is obtained by performing an evaluation process on subjective sample data. The objective sample data and the subjective sample data are data for the same subject.

The voice dialogue satisfaction determination device provided by this embodiment is capable of executing the embodiment of the above method. Since the implementation method and the technical effect are similar, they will not be described in detail in this embodiment.

FIG. 8 is a hardware configuration diagram of the device for determining the satisfaction level of the voice dialogue provided by the embodiment of the present invention. As shown in FIG. 8, the device 80 for determining the satisfaction level of the voice dialogue includes at least one processor 801 and a memory 802. A computer execution command is stored in the memory 802. The at least one processor 801 executes a computer execution command stored in the memory 802 to cause the at least one processor 801 to execute the method of determining the satisfaction level of the voice dialogue.

For the specific implementation process of the processor 801 can refer to the embodiment of the above method. Since the realization principle and the technical effect are similar, they will not be described in detail in this embodiment.

The determination device 80 is further provided with a communication component 803. The processor 801 and the memory 802 and the communication component 803 are connected by the bus 804.

In the embodiment of FIGS. 7 and 8 above, the device for determining the satisfaction level of the voice dialogue may be the terminal or server shown in FIG.

The embodiments of the present invention further provided computer readable storage media. The computer execution command is stored in the computer-readable storage medium. When the computer execution command is executed by the processor, the method of determining the satisfaction level of the voice dialogue is realized.

It should be understood that the devices and methods disclosed in the above embodiments may be implemented by other means. For example, the examples of the devices described so far are merely exemplary. For example, the division of the module is merely a division by a logic function, and other division means may be adopted when it is actually realized. For example, a plurality of modules may be combined or integrated with other systems, and some features may be ignored or not executed. Also, the coupling, direct coupling or communication connection between the displayed or discussed may be an indirect coupling or communication connection by some interface, device or module, electrical, mechanical or It may be by other means.

The module described above as a separate component may or may not be physically separated, and the component labeled as a module may or may not be a physical unit. That is, it may be located in one place or may be distributed to a plurality of network units. The purpose of the technical proposal of this embodiment can be realized by selecting some or all of the modules according to actual needs.

Further, each functional module in each embodiment of the present invention may be integrated in one processing unit, may physically exist independently as each module, and two or more modules may be one unit. It may be accumulated in. The modularized unit may be realized in the form of hardware, or may be realized by adopting a software functional unit together with the hardware.

The integrated module realized by the software function module may be stored in a computer-readable storage medium. The software function module is stored in a storage medium, and a computer device (which may be a personal computer, a server, a network device, or the like) or a processor is a step of a part of the method in each embodiment of the present application. Includes multiple instructions to execute.

It should be understood that the processor may be a central processing unit (abbreviated as CPU), other general-purpose processors, digital signal processors (abbreviated as Digital Signal Processor, DSP), dedicated integrated. It may be a circuit (Application Special Integrated Circuit, abbreviated as ASIC) or the like. The general-purpose processor may be a microprocessor, and the processor may be any standard processor or the like. The steps of the method disclosed together with the invention may be executed and completed directly by the hardware processor, or may be executed and completed by the hardware and software modules in the processor in combination.

The memory may include a high speed RAM memory and may further include a non-volatile storage NVM, such as at least one magnetic disk memory, and may be a memory card, mobile hard disk, read-only memory, magnetic disk or optical disk. ..

The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For convenience of expression, the bus in the drawings of the present application is not limited to one and is not limited to one type.

The storage medium may be implemented in any type of volatile or non-volatile storage device or a combination thereof, eg static random access memory (SRAM), electroerasable programming read-only memory (EEPROM), and the like. It may be an erasable programming read-only memory (EPROM), a programming read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk. The storage medium may be any usable medium accessible to a general purpose or dedicated computer.

When an exemplary storage medium is coupled to the processor, the processor is capable of reading information from the storage medium and writing information to the storage medium. Of course, the storage medium may be part of the processor configuration. The processor and the storage medium may be installed in a dedicated integrated circuit (Application Specific Integrated Circuits, abbreviated as ASIC). Of course, the processor and the storage medium may be installed in the terminal or the server as separate parts.

As will be appreciated by those skilled in the art, all or part of the steps to implement the embodiments of each of the above methods can be programmatically completed by instructing the relevant hardware. The program may be stored on a computer-readable storage medium. When the program is executed, it executes steps including examples of each of the above methods. The storage medium includes various media capable of storing program codes such as ROM, RAM, magnetic disk, optical disk, and the like.

Finally, it should be explained that each of the above embodiments is not restrictive and merely describes the technical proposal of the present invention. As will be appreciated by those skilled in the art, the present invention has been described in detail with reference to each of the above embodiments, but still amends the technical proposals described in each of the above embodiments, or some or all of the techniques. Features can be replaced equivalently. These amendments or substitutions do not deviate from the scope of the technical proposal of each embodiment of the present invention.

Claims (17)

A method of determining the satisfaction level of a voice dialogue performed by a computer.
A step of acquiring a voice dialogue feature including an objective data of a voice dialogue and a subjective data of a voice dialogue, wherein the objective data of the voice dialogue and the subjective data of the voice dialogue are data for the same subject. ,
A step of performing evaluation processing on the objective data to obtain an objective evaluation and performing evaluation processing on the subjective data to obtain a subjective evaluation.
The objective evaluation and the subjective evaluation are input to the satisfaction evaluation model, and a step of acquiring the satisfaction of the voice dialogue output from the satisfaction evaluation model is included.
The objective data of the voice dialogue includes the user intention and the response content, the response delay, and the current playback time of the response content.
A method characterized in that the subjective data of the voice dialogue includes text information corresponding to voice input by the user after the reproduction of the response content is interrupted. The step of performing evaluation processing on the objective data and acquiring the objective evaluation is
The step of acquiring the first objective evaluation based on the intention match degree between the user intention and the response content, and
The step of obtaining a second objective evaluation based on the response delay and the standard delay, and
Including a step of acquiring a third objective evaluation based on the current playback time of the response content and the standard playback time of the response content.
The step of acquiring the satisfaction level of the voice dialogue output from the satisfaction evaluation model by inputting the objective evaluation and the subjective evaluation into the satisfaction evaluation model is
The first objective evaluation, the second objective evaluation, the third objective evaluation, and the subjective evaluation are input to the satisfaction evaluation model, and the step of acquiring the satisfaction level of the voice dialogue output from the satisfaction evaluation model is included. The method according to claim 1, wherein the method is characterized by the above. The step of performing evaluation processing on the subjective data and acquiring the subjective evaluation is
A step of performing a word meaning analysis on the text information and acquiring a content attribute which is an emotional attribute or a subject attribute corresponding to the text information.
The method according to claim 1, wherein a step of acquiring a subjective evaluation based on a content attribute corresponding to the text information is included. If the content attribute is a subject attribute, the step of acquiring the subjective evaluation based on the content attribute corresponding to the text information is
Steps to get the subject type corresponding to the text information,
3. The third aspect of the present invention includes a step of determining that the subjective evaluation is lower than a predetermined evaluation value if the subject type corresponding to the text information matches the subject type corresponding to the user's intention. The method described. If the content attribute is an emotional attribute, the step of acquiring the subjective evaluation based on the content attribute corresponding to the text information is
The step of extracting emotional keywords in the text information and
Based on the correspondence between the emotional keyword and the mood type, the step of acquiring one of the mood types of the group consisting of positive mood, negative mood, and neutral mood, and
The method according to claim 3, wherein a step of acquiring a subjective evaluation based on a correspondence relationship between the mood type and a predetermined evaluation is included. The steps to acquire voice dialogue features are
The step of acquiring the first log data in which the time interval between the second log data in the immediately preceding time zone and the third log data in the immediately following time zone is larger than a predetermined threshold value, and
From the first log data, a step of acquiring a subject corresponding to each of two adjacent voice inputs by the user, and
The method of claim 1, comprising the step of acquiring the voice dialogue feature based on the subject corresponding to each of the two adjacent voice inputs. In the method, the objective evaluation and the subjective evaluation are input to the satisfaction evaluation model, and before the step of acquiring the satisfaction of the voice dialogue output from the satisfaction evaluation model, more objective sample data is obtained. Training that includes the first sample evaluation obtained by performing evaluation processing on the subject, the second sample evaluation obtained by performing evaluation processing on subjective sample data, and the satisfaction level input by the user. In the step of acquiring the sample set, the objective sample data and the subjective sample data are data for the same subject.
The method according to any one of claims 1 to 6, wherein the step of acquiring the satisfaction evaluation model by iteration training based on the training sample set is included. It is a device for determining the satisfaction level of voice dialogue,
A acquiring module for acquiring voice interaction feature that contains and subjective data objective data and voice dialogue spoken dialogue, subjective data objective data and voice interaction of the sound voice interaction is the data for the same subject, Get module and
A processing module that performs evaluation processing on the objective data to acquire an objective evaluation, and performs evaluation processing on the subjective data to acquire a subjective evaluation.
The objective evaluation and the subjective evaluation are input to the satisfaction evaluation model, and a confirmation module for acquiring the satisfaction of the voice dialogue output from the satisfaction evaluation model is provided.
The objective data of the voice dialogue includes the user intention, the response content, the response delay, and the current playback time of the response content.
The subjective data of the voice dialogue includes text information corresponding to the voice input by the user after the reproduction of the response content is interrupted, or text information input by the user after the reproduction of the response content is completed. , A device characterized by that. Specifically, the processing module
Based on the degree of matching between the user's intention and the response content, the first objective evaluation is obtained.
Obtain a second objective evaluation based on the response delay and standard delay.
It is configured to obtain a third objective evaluation based on the current playback time of the response content and the standard playback time of the response content.
Specifically, the confirmation module
The first objective evaluation, the second objective evaluation, the third objective evaluation, and the subjective evaluation are used as inputs of the satisfaction evaluation model, and the satisfaction level of the voice dialogue output from the satisfaction evaluation model is acquired. The device according to claim 8, wherein the device is provided. Specifically, the processing module
Semantic parsing is performed on the text information to acquire the content attribute which is an emotional attribute or a subject attribute corresponding to the text information.
The apparatus according to claim 8, wherein the subjective evaluation is acquired based on the content attribute corresponding to the text information. When the content attribute is a subject attribute, the processing module specifically
Obtain the subject type corresponding to the text information and
The claim is characterized in that if the subject type corresponding to the text information matches the subject type corresponding to the user's intention, the subjective evaluation is determined to be lower than a predetermined evaluation value. 10. The apparatus according to 10. When the content attribute is an emotional attribute, the processing module specifically
Extract emotional keywords in the text information
Based on the correspondence between the emotional keywords and mood types, one of the mood types in the group consisting of positive mood, negative mood, and neutral mood is acquired.
The device according to claim 10, wherein a subjective evaluation is acquired based on a correspondence relationship between the mood type and a predetermined evaluation. Specifically, the acquisition module
Acquire the first log data in which the time interval between the second log data in the immediately preceding time zone and the third log data in the immediately following time zone is larger than a predetermined threshold value.
From the first log data, the subject corresponding to each of the two adjacent voice inputs by the user is acquired.
The device according to claim 8, wherein the voice dialogue feature is configured to be acquired based on a subject corresponding to each of the two adjacent voice inputs. The device further includes a training module, which uses the objective evaluation and the subjective evaluation as inputs of the satisfaction evaluation model before acquiring the satisfaction level of the voice dialogue output from the satisfaction evaluation model. ,
The first sample evaluation obtained by performing evaluation processing on objective sample data, the second sample evaluation obtained by performing evaluation processing on subjective sample data, and the satisfaction input from the user. A training sample set including the degree is obtained, and the objective sample data and the subjective sample data are data for the same subject.
The apparatus according to any one of claims 8 to 13, characterized in that the satisfaction evaluation model is obtained by iteration training based on the training sample set. It is a device for determining the satisfaction level of voice dialogue,
With at least one processor and memory
Commands that can be executed by a computer are stored in the memory.
The satisfaction level of the voice dialogue according to any one of claims 1 to 7 to the at least one processor by executing a command that can be executed by the computer stored in the memory. A device characterized in that the determination method of is executed. A computer-readable storage medium
The computer-readable storage medium stores commands that can be executed by the computer.
A storage medium according to any one of claims 1 to 7, wherein when a command that can be executed by the computer is executed by the processor, the method for determining the satisfaction level of the voice dialogue according to any one of claims 1 to 7 is realized. It ’s a computer program
A computer program that, when executed by a processor, realizes the method according to any one of claims 1-7.

Images