JP7176325B2 - Speech processing program, speech processing method and speech processing device - Google Patents

Description

The present invention relates to an audio processing program and the like.

In recent years, in many companies, there is a need to acquire various information from communication between store clerks and customers and use it for improving customer satisfaction and marketing strategies. For example, since the voice segments in which customers hesitate to make decisions contain a lot of information related to decision-making materials, we can detect voice segments in which customers hesitate to make decisions, and use the voice information of the detected voice segments. By analyzing, it is possible to estimate customer needs.

For example, there is a conventional technology that detects as thinking time the time from when the agent asks the user a question to when the user answers, until the end of silence and meaningless utterance intervals. Speech intervals that do not make sense are called “filler intervals”.

JP 2017-111760 A JP-A-2000-194386 WO2017/085992

However, the conventional technology described above has a problem that it is impossible to accurately estimate the section in which the user is hesitant.

FIG. 15 is a diagram for explaining the problem of the conventional technology. In the conventional technology, the filler section is detected as thinking time (speech section in which the user hesitates to make a decision). In some cases.

In FIG. 15, the store clerk asks, "Are there any models that you are interested in?" answering. In this example, a section T1 in which "Uh, what? What is it?" is detected as a filler section. This section T1 can be said to be a section in which the customer performs the storage operation.

The store clerk said, "That's XX. It's popular. However, XX doesn't have YY...", and the customer said, "That's right." Is it necessary?" In this example, the section T2 in which the utterance "That's right..." is detected as the filler section. This section T2 can be said to be a section in which the customer hesitates.

In the conventional technology, the sections T1 and T1 are detected as thinking time, but the section T1 is a section in which the customer performs a memory operation, not a section in which the customer hesitates to make a decision.

In one aspect, an object of the present invention is to provide a speech processing program, a speech processing method, and a speech processing device capable of accurately estimating a section in which a user hesitates.

The first option is to have the computer perform the following processing. A computer detects a plurality of speech segments from the voice information. The computer specifies, as a filler segment, an utterance segment in which filler is detected from a plurality of utterance segments. The computer determines whether or not the voice information in the filler section is voice information to be spoken when the user hesitates to make a decision, based on the feature amount of the voice information in the filler section.

It is possible to accurately estimate the section in which the user is lost.

FIG. 1 is a diagram for explaining processing of the speech processing device according to the first embodiment. FIG. 2 is a diagram showing the configuration of the system according to the first embodiment. FIG. 3 is a functional block diagram showing the configuration of the speech processing device according to the first embodiment. FIG. 4 is a functional block diagram showing the configuration of the state determination unit according to the first embodiment. FIG. 5 is a diagram for explaining the processing of the brightness calculator. FIG. 6 is a flow chart showing the processing procedure of the speech processing device according to the first embodiment. FIG. 7 is a diagram showing an example of experimental results. FIG. 8 is a diagram for explaining the transition of the frequency of audio information. FIG. 9 is a diagram showing the configuration of a system according to the second embodiment. FIG. 10 is a functional block diagram showing the configuration of the recording device according to the second embodiment. FIG. 11 is a functional block diagram showing the configuration of the speech processing device according to the second embodiment. FIG. 12 is a functional block diagram showing the configuration of the state determination unit according to the second embodiment. FIG. 13 is a flow chart showing the processing procedure of the speech processing device according to the second embodiment. FIG. 14 is a diagram showing an example of the hardware configuration of a computer that implements functions similar to those of the audio processing device. FIG. 15 is a diagram for explaining the problem of the conventional technology.

Hereinafter, embodiments of the audio processing program, the audio processing method, and the audio processing apparatus disclosed in the present application will be described in detail based on the drawings. In addition, this invention is not limited by this Example.

FIG. 1 is a diagram for explaining processing of the speech processing device according to the first embodiment. In the first embodiment, as an example, voice information of the first speaker and the second speaker is acquired to detect an utterance section, and an utterance section with the second speaker is detected as an utterance section by the second speaker, A case will be described in which it is determined whether or not it is an interval where the In the following description, among the utterance sections, a section in which the user hesitates to make a decision will be referred to as a "questionable section" as appropriate.

When the speech information is acquired, the speech processing device detects an utterance period of the first speaker and an utterance period of the second speaker. Identify the utterance segment that will be the segment. A speech segment is a segment sandwiched between silent segments. A filler section is an utterance section in which meaningless utterances are made. For example, a section in which utterances such as "Well, what is that?"

In FIG. 1, the utterance periods of the first speaker are T11, T12, and T13, and the utterance periods of the second speaker are T21, T22, T23, and T24. The speech processing device analyzes the speech information of the speech sections T21 and T22 following the speech section T11 and the speech information of the speech sections T23 and T24 following the speech section T12 to identify filler sections. For example, the speech processing device will be described assuming that the utterance sections T21 and T23 are identified as filler sections. The speech processing device determines whether or not the filler section is a questionable section based on the feature amount of the speech information of the filler section.

A graph G1 in FIG. 1 shows a temporal change in the feature amount (brightness of voice) of voice information. The horizontal axis of graph G1 is the time axis. The vertical axis of the graph G1 is the axis corresponding to the brightness of the speaker's voice. Indicates dark. For example, when the brightness in the filler section is "bright", the speech processing device determines that the filler section is "not a doubtful section". On the other hand, if the brightness in the filler section is "dark", the speech processing device determines that the filler section is "a doubtful section".

To explain using FIG. 1, the speech processing apparatus determines that the filler section T21 is not a doubtful section because the brightness of the voice is "bright" in the filler section T21. Since the brightness of the voice in the filler section T23 is "dark", the speech processing device determines that the filler section T23 is a "deterministic section". In this way, the speech processing apparatus according to the present embodiment determines whether or not it is a questionable section based on the feature amount of the speech information of the filler section, so that the section in which the user hesitates can be accurately estimated. can do.

Note that the speech processing device uses not only the feature amount of speech information in the filler section, but also the feature amount of speech information in the response section following the filler section, the average brightness of the entire dialogue, and the response time to determine the filler section. , it may be determined whether or not it is an indecisive section. As a result, it is possible to determine the doubtful decision section with higher accuracy. It should be noted that the response period is the utterance period other than the filler period among the utterance periods of the second speaker. In the example shown in FIG. 1, utterance sections T22 and T24 are response sections. The response time is the time from the utterance period of the first speaker to the response period of the second speaker. For example, the response time is from the end time of the utterance period T11 to the start time of the utterance period (response period) T22.

A case will be described in which the speech processing device determines a questionable section based on a filler section and a response section following the filler section. When the brightness of the speaker's voice in the filler section is "dark" and the brightness of the speaker's voice in the response section is "bright", the speech processing device determines that the filler section is the "uncertain section". judge. If this condition is not satisfied, the speech processing device determines that the filler section is "not a questionable section."

For example, since the brightness of the voice in the filler section T23 is "dark" and the brightness of the voice in the response section T24 is "bright", the speech processing device satisfies the above conditions and treats the filler section T23 as the "difficult to judge" section. It is determined that On the other hand, since the voice brightness of the filler section T21 is "bright" and the voice brightness of the response section T24 is "bright", the speech processing device does not satisfy the above conditions, so the filler section T23 is regarded as the "uncertain section is not determined.

Next, a case will be described in which the speech processing device determines whether or not it is an indecisive section based on the filler section, the response section following the filler section, the average brightness of the entire dialogue, and the response time. do. Only when the average brightness of the entire dialogue is equal to or greater than the threshold TH_D' and the response time is equal to or greater than the threshold TH_R, the speech processing device divides the filler section and the response section following the filler section. make a decision based on On the other hand, when the average brightness of the entire dialogue is less than the threshold TH_D', the speech processing device can be said that the speaker is not interested in the speaker. Regardless of the characteristics of the audio information, it is determined that the filler section is "not a dubious section". In addition, when the response time is less than the threshold TH_R, the speech processing device determines that the filler section is not a doubtful section regardless of the characteristics of the filler section, the response section following the filler section, and the audio information. judge.

By using the average brightness of the entire dialogue and the response time, the speech processing device can determine whether the filler section is “determined” without analyzing the feature amount of the speech information in the filler section and the feature amount of the speech information in the response section. It can be determined that it is not a lost section.

Next, the configuration of the system according to the first embodiment will be described. FIG. 2 is a diagram showing the configuration of the system according to the first embodiment. As shown in FIG. 2, this system includes a voice processing device 1 and microphones 10a and 10b. The audio processing device 1 is connected to microphones 10a and 10b.

The microphone 10a is a microphone that collects the voice of the user U101. The microphone 10 a outputs the collected sound information to the sound processing device 1 . The microphone 10b is a microphone that collects the voice of the user U102. The microphone 10 a outputs the collected sound information to the sound processing device 1 . For example, user U101 corresponds to the first speaker and user U102 corresponds to the second speaker. In the following description, the voice information of the user U101 will be referred to as "first voice information", and the voice information of the user U102 will be referred to as "second voice information".

The voice processing device 1 acquires voice information from the microphones 10a and 10b, determines a doubtful judgment interval from the utterance interval of the user U102 based on the acquired voice information, and stores the determined voice information of the doubtful judgment interval. It is a device that stores in the device.

FIG. 3 is a functional block diagram showing the configuration of the speech processing device according to the first embodiment. As shown in FIG. 3, this audio processing device 1 is connected to the microphones 10a and 10b described with reference to FIG. The speech processing device 1 includes AD converters 20a and 20b, a preprocessing unit 30, a storage unit 40, and a state determination unit 100. Each processing unit of the AD conversion units 20a and 20b, the preprocessing unit 30, and the state determination unit 100 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Also, each processing unit may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The AD conversion unit 20a is an AD conversion circuit (Analog-to-digital converter) that converts the first audio information input from the microphone 10a from an analog signal to a digital signal. The AD conversion unit 20 a outputs the first audio information converted into a digital signal to the preprocessing unit 30 . In the following description, the first audio information converted into a digital signal is simply referred to as first audio information.

The AD conversion unit 20b is an AD conversion circuit that converts the second audio information input from the microphone 10b from an analog signal to a digital signal. The AD conversion section 20b outputs the second audio information converted into a digital signal to the preprocessing section 30 . In the following description, the second audio information converted into a digital signal is simply referred to as second audio information.

The preprocessing unit 30 performs various preprocessing on the first audio information and the second audio information, and outputs the preprocessed first audio information and the second audio information to the state determination unit 100. Department. For example, in the system shown in FIG. 2, voices of both users U101 and U102 may be collected by microphones 10a and 10b. Therefore, the preprocessing unit 30 performs preprocessing to remove the voice of the user U102 from the first voice information so that only the voice of the user U101 is included in the first voice information. The preprocessing unit 30 performs preprocessing to remove the voice of the user U101 from the second voice information so that only the voice of the user U102 is included in the second voice information.

The state determination unit 100 acquires the first audio information and the second audio information, detects the utterance period, and identifies the filler period from each utterance period. The state determination unit 100 determines whether or not the filler section is a questionable section based on the feature amount of the audio information included in the specified filler section. The state determination unit 100 stores the voice information of the uncertain judgment interval in the storage unit 40 when the filler interval is the doubtful judgment interval. In the present embodiment, a case will be described in which a filler section is specified from the utterance section of the second audio information, and whether or not it is an indecisive section is determined, but the present invention is not limited to this. The state determination unit 100 may identify a filler segment from the utterance segment of the first audio information and determine whether or not it is a doubtful determination segment.

The storage unit 40 is a storage device that stores audio information of the doubtful judgment section. The storage unit 40 corresponds to, for example, a storage device such as a semiconductor memory device such as a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory.

FIG. 4 is a functional block diagram showing the configuration of the state determination unit according to the first embodiment. As shown in FIG. 4, the state determination unit 100 includes speech period detection units 110a and 110b, an identification unit 120, a response time calculation unit 130, a brightness calculation unit 140, a long-term average calculation unit 150, and a threshold calculation unit. It has a unit 160 , a threshold DB 170 and a determination unit 180 .

The utterance period detection unit 110a receives input of the first voice information of the first speaker (user U101), and calculates a period sandwiched between silent periods of low power of the first voice information as a utterance period. Speech period detection section 110 a outputs information on the speech period of the first audio information to response time calculation section 130 .

For example, the speech period detection unit 110a detects the power of the first audio information x1(n) and sets a variable "v1(t)" indicating the presence or absence of speech. n indicates the sample number of voice information. t indicates a frame number. One frame is assumed to be 30 ms. If the power of the first audio information x1(n) in one frame is less than the threshold, the speech period detection unit 110a sets the variable "v1(t)=0" as "no speech". When the power of the first audio information x1(n) in one frame is equal to or greater than the threshold, the speech period detection unit 110a sets the variable "v1(t)=1" as "there is speech". Note that the speech period detection unit 110a may determine the presence or absence of speech based on the technique described in WO2009/145192. In addition, speech period detection section 110a may determine the presence or absence of speech using SNR (signal-noise ratio) instead of the power of voice information. The same applies to the speech period detection unit 110b, which will be described later.

Based on the variable V1(t), the utterance period detection unit 110a detects the start time T1s(k1) of the utterance period of the first speaker and the end time T1e(k1) of the utterance period of the first speaker. Output to the response time calculation unit 130 . "k1" is a speech segment number that identifies a speech segment of the first audio information.

For example, speech period detection section 110a outputs T1s(k1) and T1e(k1) based on the following equations (1) and (2). As shown in Equation (1), speech segment detection section 110a sets the timing at which the value of v1(t) changes from "0" to "1" as the start time of the speech segment. As shown in Equation (2), speech segment detection section 110a determines the timing at which the value of v1(t) changes from "1" to "0" as the end time of the speech segment.

The utterance period detection unit 110b receives input of the second voice information of the second speaker (user U102), and calculates a period sandwiched between silent periods with low power of the second voice information as a utterance period. Speech period detection section 110 b outputs speech period information to identification section 120 and brightness calculation section 140 .

For example, the speech period detection unit 110b detects the power of the second audio information x2(n) and sets a variable "v2(t)" indicating the presence or absence of speech. If the power of the second audio information x2(n) in one frame is less than the threshold, the speech period detection unit 110b sets the variable "v2(t)=0" as "no speech". When the power of the second audio information x2(n) in one frame is equal to or greater than the threshold, the speech period detection unit 110b sets the variable "v2(t)=1" as "there is speech".

Based on the variable V2(t), the speech period detection unit 110b identifies the start time T2s(k2) of the second speaker's speech period and the end time T2e(k2) of the second speaker's speech period. Output to the unit 120 and the brightness calculation unit 140 . "k2" is a speech segment number that identifies a speech segment of the second audio information.

For example, speech period detection section 110b outputs T2s(k2) and T2e(k2) based on the following equations (3) and (4). As shown in Equation (3), speech period detection section 110b sets the timing at which the value of v2(t) changes from "0" to "1" as the start time of the speech period. As shown in Equation (4), speech segment detection section 110b sets the timing at which the value of v2(t) changes from "1" to "0" as the end time of the speech segment.

The identification unit 120 is a processing unit that receives the second voice information and the information on the second speaker's utterance period and determines whether the utterance period is a filler period. Identifying section 120 outputs the determination result to response time calculating section 130 and determining section 180 .

The specifying unit 120 inputs the speech information included in the second speech information x2(n) from the start time T2s(k2) to the end time T2e(k2) of the utterance period to the speech recognition engine, and outputs the speech recognition result. By referring to a filler DB (data base), it is determined whether or not the speech section is a filler section. For example, the filler DB stores various kinds of nonsense information about speech, ``That's right, um, what is it?...''. The identifying unit 120 identifies an utterance segment as a filler segment when the speech recognition result hits the speech information in the filler DB. Further, the specifying unit 120 may determine whether or not the speech segment is a filler segment using the technique described in Japanese Patent Application Laid-Open No. 2015-082087.

The identifying unit 120 outputs the filler determination result F(k2) to the response time calculating unit 130 and the determining unit 180 . When identifying section 120 determines that the utterance section with utterance section number “k2” of the second audio information is a filler section, identifying section 120 outputs “F(k2)=1” to determining section 180 . When identifying unit 120 determines that the utterance segment with utterance segment number “k2” of the second audio information is not a filler segment, specifying unit 120 outputs “F(k2)=0” to determination unit 180 .

Speech recognition generally means converting speech into characters. When the specifying unit 120 detects fillers, it is possible to perform determination after converting to characters, but it is possible to detect fillers without converting to characters. For example, the identifying unit 120 may determine whether or not the utterance segment is a filler segment based on prosodic features (such as accent, intonation, rhythm, etc.) and acoustic features (such as feature amounts that form the basis of speech recognition) included in the utterance segment. may be detected.

The response time calculation unit 130 is a processing unit that calculates the response time from the end of the utterance interval by the first speaker to the start of the utterance interval (not the filler interval) by the second speaker. . Response time calculation section 130 outputs response time information to threshold calculation section 160 and determination section 180 .

For example, response time calculator 130 calculates response time R(k2) based on equation (5). In Equation (5), T1e(k1) is the end time of the first speaker's utterance. T2s(k2') is an utterance segment of the second speaker's utterance segment number k2, which is included from the first speaker's utterance end time T1e(k1) to the next utterance start time T1s(k1+1), Indicates the start time of the first speech segment that is not a filler segment. Note that a plurality of utterance intervals of the second speaker may be included between the end time T1e(k1) of the first speaker's utterance and the start time T1s(k1+1) of the next utterance.

The brightness calculation unit 140 is a processing unit that receives the second voice information and the information of the speech period of the second speaker, and calculates the estimated value D of the brightness of the speech period of the second speaker. The brightness calculation unit 140 outputs the information of the brightness estimated value D to the long-term average calculation unit 150 , the threshold calculation unit 160 and the determination unit 180 . The brightness estimate value D corresponds to the variance of each fundamental frequency included in the speech period.

An example of the processing of the brightness calculation unit 140 will be described below. First, the brightness calculator 140 calculates the fundamental frequency P(t) from the second voice information included in the second speaker's utterance period T2s(k2) to T2e(k2). Brightness calculator 140 calculates the autocorrelation function of the second audio information, and calculates the fundamental frequency for each frame based on the position where the value of the autocorrelation function peaks. The brightness calculator 140 may calculate the fundamental frequency using the technique described in Japanese Patent Laid-Open No. 8-44395.

The brightness calculator 140 converts the fundamental frequency P(t) [Hz] into a fundamental frequency P'(t) [semitone] based on Equation (6). The fundamental frequency P'(t) is indicated by a scale in the logarithmic domain that matches the human auditory pitch of the voice.

The brightness calculation unit 140 calculates a long-term average P_ave_long(t) of the fundamental frequency from the time-series data P'(t) of the fundamental frequency by moving average of a predetermined frame. For example, brightness calculator 140 calculates P_ave_long(t) based on equation (7). "L" included in equation (7) indicates the movement width at the time of average calculation.

The brightness calculation unit 140 calculates the average amount of difference (variance) in the speech segment with the speech segment number “k2” as the estimated brightness value D(k2). For example, the brightness calculation unit 140 calculates the brightness estimated value D(k2) based on Equation (8).

FIG. 5 is a diagram for explaining the processing of the brightness calculator. In FIG. 5, the horizontal axis of the graph G2 is the axis corresponding to the number of frames, and the vertical axis is the axis corresponding to the fundamental frequency P'(t) [semitone]. The horizontal axis of the graph G3 is the axis corresponding to the speech period number k2 of the second speaker, and the vertical axis is the axis corresponding to the estimated brightness value D(k2). For example, by dividing P'(t) included in area A1 of graph G2 by the long-term average P_ave_long(t) of P'(t) included in frame numbers tL to L, the estimated value D _A1 is Calculated.

The brightness calculation unit 140 calculates the estimated value D of the brightness of each utterance period of the second speaker by repeatedly executing the above process for each utterance period of the second speaker.

Returning to the description of FIG. The long-term average calculation unit 150 calculates a long-term average brightness D′(k2) from the time-series data D(k2) of the estimated brightness value obtained from the brightness calculation unit 140, using a moving average of a predetermined frame. . For example, the long-term average calculating unit 150 calculates the long-term average D'(k2) of brightness based on Equation (9). In Expression (9), L2 indicates the time after a predetermined time from the end time of the speech segment with the speech segment number k2. L1 indicates the time a predetermined time before the start time of the speech segment with the speech segment number k2. The long-term average calculation unit 150 utilizes it as an index indicating the brightness of the conversation situation before and after. Long-term average calculation section 150 outputs long-term average D′(k2) of brightness to determination section 180 .

The threshold calculation unit 160 is a processing unit that calculates various thresholds and outputs information on the calculated thresholds to the determination unit 180 . For example, threshold calculator 160 calculates threshold TH_D, threshold TH_R, and threshold TH_D'. The threshold TH_D is a threshold to be compared with the estimated brightness value D of the speech period. The threshold TH_R is the threshold that is compared with the response time. The threshold TH_D' is the threshold that is compared with the brightness long-term average D'.

It is assumed that initial values of the threshold TH_D, the threshold TH_R, and the threshold TH_D' are recorded in the threshold DB 170 . For example, let the initial value of the threshold TH_D be "1.5 [semitone]". Assume that the initial value of the threshold TH_R is "200 [Frame]". Assume that the initial value of the threshold TH_D' is "1.0 [semitone]".

A process of calculating the threshold HT_D by the threshold calculator 160 will be described. The threshold calculator 160 acquires the estimated brightness value D of each speech period from the brightness calculator 140, and calculates the average AVE_D and the variance VAR_D of the multiple estimated values D thus acquired. The threshold calculator 160 updates the threshold HT_D based on Equation (10). Threshold calculation section 160 outputs information on updated threshold HT_D to determination section 180 . α in equation (10) is a coefficient, for example, “α=0.5”.

HT_D=AVE_D-α×VAR_D (10)

A process of calculating the threshold HT_R by the threshold calculator 160 will be described. The threshold calculator 160 acquires each response time R from the response time calculator 130 and calculates the average AVE_R and variance VAR_R of the multiple response times R thus acquired. The threshold calculator 160 updates the threshold HT_R based on Equation (11). Threshold calculation section 160 outputs information on updated threshold HT_R to determination section 180 . β in equation (11) is a coefficient, for example, "β=0.5".

HT_R=AVE_R-β×VAR_R (11)

Threshold calculation section 160 outputs threshold TH_D' to determination section 180 as it is without performing update processing.

The threshold calculation unit 160 periodically performs processing for updating the threshold TH_D and the threshold TH_R, and outputs the updated threshold TH_D and the threshold TH_R to the determination unit 180 at the update timing. In addition, the threshold calculation unit 160 stores the threshold TH_D and the threshold TH_R in the threshold DB 170 in association with the identification information of the second speaker. In the case of "not in doubtful judgment interval", a threshold value corresponding to the second speaker may be retrieved from the stored threshold values, and the retrieved threshold value may be output to the determination unit 180. As a result, processing can be performed using a threshold optimized for the voice information of the second speaker.

The determination unit 180 acquires the presence/absence F of the filler section, the response time R, the estimated value D, and the long-term average D′ for each utterance section, and performs the following processing on the utterance section determined to be the filler section, thereby obtaining the filler It is determined whether or not the section is “a doubtful judgment section”. The determination unit 180 stores the second audio information of the uncertain judgment interval in the storage unit 40 when judging that it is the doubtful judgment interval.

For example, the determination unit 180 performs long-term average determination processing, response time determination processing, filler brightness determination processing, and response brightness determination processing. Note that the determination unit 180 skips the long-term average determination process, the response time determination process, the filler brightness determination process, and the response brightness determination process for an utterance segment (F(k2)=0) that is not a filler segment, and skips the “determination It is not a lost section."

The long-term average determination processing will be explained. The determination unit 180 compares the long-term average brightness D′(k2) of the speech segment with the speech segment number “k2” of the filler segment (F(k2)=1) with the threshold value TH_D′ to determine the brightness. Determines whether the long-term average is “bright” or “dark”. If the long-term average D'(k2) is equal to or greater than the threshold TH_D', the determining unit 180 determines that the long-term average brightness of the speech segment with the speech segment number k2 is "bright." If the long-term average D'(k2) is less than the threshold TH_D', the determining unit 180 determines that the long-term average brightness of the speech segment with the speech segment number k2 is "dark."

Response time determination processing will be described. The determination unit 180 compares the response time R(k2) corresponding to the utterance segment with the utterance segment number “k2” of the filler segment (F(k2)=1) with the threshold TH_R, and determines the response time R Determine whether (k2) is "long" or "short". If the response time R(k2) is equal to or greater than the threshold TH_R, the determination unit 180 determines that the response time corresponding to the speech segment with the speech segment number k2 is "long". If the response time R(k2) is less than the threshold TH_R, the determination unit 180 determines that the response time corresponding to the speech segment with the speech segment number k2 is "short".

When the determination result of the long-term average determination process is "bright" and the response time determination result is "long", the determination unit 180 performs subsequent filler brightness determination processing and response brightness determination processing. On the other hand, when the determination result of the long-term average determination process is “dark” or the response time determination result is “short,” determination unit 180 determines that the utterance segment with utterance segment number “k2” It is not an interval", and the filler brightness determination process and the response brightness determination process are skipped.

The filler brightness determination processing will be described. The determination unit 180 compares the estimated brightness value D(k2) of the speech segment with the speech segment number “k2” of the filler segment (F(k2)=1) with the threshold value TH_D. Determine whether it is "light" or "dark". If the estimated value D(k2) is equal to or greater than the threshold TH_D, the determination unit 180 determines that the brightness of the speech period with the speech period number k2 is "bright". If the estimated value D(k2) is less than the threshold TH_D, the determination unit 180 determines that the brightness of the speech period with the speech period number k2 is "dark".

The response brightness determination process will be described. The determination unit 180 compares the estimated brightness value D(k2′) of the response segment following the speech segment with the speech segment number “k2” of the filler segment (F(k2)=1) with the threshold value TH_D. and determines whether the brightness is “bright” or “dark”. The determination unit 180 determines that the brightness of the response section is "bright" when the estimated value D(k2') is equal to or greater than the threshold TH_D. The determination unit 180 determines that the brightness of the response section is "dark" when the estimated value D(k2') is less than the threshold TH_D.

If the determination result of the filler brightness determination process is "dark" and the determination result of the response brightness determination process is "bright", determination unit 180 determines that the utterance segment of utterance segment number "k2" is " It is a judgment uncertain section."

Note that the determination unit 180 executes the long-term average determination process, the response time determination process, the filler brightness determination process, and the response brightness determination process, and collectively uses each determination result to determine the utterance segment number “k2”. It may be determined whether or not the utterance segment is “a doubtful determination segment”. The determination unit 180 determines that the result of the long-term average determination process is "bright", the result of the response time determination process is "long", the result of the filler brightness determination process is "dark", and the result of the response brightness determination process is "bright". , it may be determined that the utterance segment with the utterance segment number “k2” is “a doubtful determination segment”.

Next, an example of the processing procedure of the state determination unit 100 of the speech processing device 1 of the first embodiment will be described. FIG. 6 is a flow chart showing the processing procedure of the speech processing device according to the first embodiment. As shown in FIG. 6, the state determination unit 100 of the speech processing device 1 acquires first speech information and second speech information (step S101).

The speech period detection unit 110a of the state determination unit 100 detects the speech period of the first speaker, and the speech period detection unit 110b detects the speech period of the second speaker (step S102). The specifying unit 120 of the state determination unit 100 detects the filler section (step S103). If there is no filler section (step S104, No), the state determination unit 100 proceeds to step S111.

On the other hand, if there is a filler section (step S104, Yes), the state determination unit 100 proceeds to step S105. The response time calculation unit 130 of the state determination unit 100 calculates the response time R (step S105). The brightness calculation unit 140 of the state determination unit 100 calculates an estimated value D of brightness (step S106).

The long-term average calculator 150 of the state determination unit 100 calculates the long-term average brightness D' (step S107). The determination unit 180 of the state determination unit 100 executes each threshold determination (step S108). In step S108, the determination unit 180 executes a long-term average determination process, a response time determination process, a filler brightness determination process, and a response brightness determination process.

The determination unit 180 determines whether or not the utterance period is the "difficult decision period" (step S109). If the utterance period is the "unsure judgment period" (step S109, Yes), the determination unit 180 stores the voice information of the uncertain judgment period in the storage unit 40 (step S110). On the other hand, if the utterance period is not the "unsure judgment period" (step S109, No), the determination unit 180 proceeds to step S111.

If there is a next conversation (step S111, Yes), the state determination unit 100 proceeds to step S103. If there is no next conversation (step S111, No), the state determination unit 100 terminates the process.

Next, effects of the speech processing device 1 according to the first embodiment will be described. The speech processing device 1 identifies a filler section from a speaker's utterance section, and determines the filler section as a questionable section when the estimated brightness value of the identified filler section is less than a threshold. This makes it possible to accurately estimate the section in which the user is hesitant.

In addition to the estimated value of the brightness of the filler section, the speech processing device 1 determines whether the filler section is an uncertain section based on a combination of the estimated value of the brightness of the response section, the response time, and the determination result of the long-term average. By determining whether or not, the determination accuracy can be improved.

It is said that the brightness of a speaker's voice is related to the pitch of the voice, and the higher the voice, the brighter the speaker feels. The inventor conducted an experiment regarding the relationship between the median frequency of speech information in the filler section and the state of the filler section for different speakers. FIG. 7 is a diagram showing an example of experimental results. As shown in FIG. 7, as a whole (with some exceptions), the frequency of the filler section in the state of "determination doubtful section" is higher than that of the filler section state of "memory operation (not judgment doubtful section)". When the brightness of the speaker's voice is low and the brightness of the speaker's voice is "dark", it can be said that the filler section is a "difficult judgment section".

FIG. 8 is a diagram for explaining the transition of the frequency of audio information. The horizontal axis of graphs G4 and G5 in FIG. 8 is the time axis, and the vertical axis is the axis corresponding to frequency (pitch of voice). For example, in the graph G4, the state of the filler section T31 is "memory operation (not indecisive section)". The frequency of this filler section T31 is around 200 to 250 Hz, which is a bright voice. On the other hand, in the graph G5, the state of the filler section 32 is the "deterministic section". The frequency of this filler section T31 is around 150 to 200 Hz, which is a dark voice. The frequency of the utterance section T33 following the filler section 32 is a bright voice around 200-300 Hz.

FIG. 9 is a diagram showing the configuration of a system according to the second embodiment. As shown in FIG. 9, this system has an audio processing device 2, microphones 10a and 10b, and a recording device 300. In FIG. Microphones 10 a and 10 b are connected to recording equipment 300 . The recording device 300 is connected via the network 5 to the audio processing device 2 on the cloud. Although illustration is omitted, the voice processing device 2 may be configured by a plurality of servers.

The microphone 10a is a microphone that collects the voice of the user U101. The microphone 10 a outputs collected audio information to the recording device 300 . The microphone 10b is a microphone that collects the voice of the user U102. The microphone 10 a outputs collected audio information to the recording device 300 . For example, user U101 corresponds to the first speaker and user U102 corresponds to the second speaker. In the following description, the voice information of the user U101 will be referred to as "first voice information", and the voice information of the user U102 will be referred to as "second voice information".

The recording device 300 is a device that records the first audio information and the second audio information. The recording device 300 converts the first audio information and the second audio information into an audio file and transmits the audio file to the audio processing device 2 .

FIG. 10 is a functional block diagram showing the configuration of the recording device according to the second embodiment. As shown in FIG. 10, this recording device 300 is connected to microphones 10a and 10b. The recording device 300 also has AD converters 310 a and 310 b , an audio file generator 320 and a transmitter 330 . Each of the AD converters 310a and 310b, the audio file generator 320, and the transmitter 330 is realized by, for example, a CPU, MPU, or the like. Also, each processing unit may be implemented by an integrated circuit such as an ASIC or FPGA, for example.

The AD conversion unit 310a is an AD conversion circuit that converts the first audio information input from the microphone 10a from an analog signal to a digital signal. The AD converting section 310 a outputs the first audio information converted into a digital signal to the audio filing section 320 . In the following description, the first audio information converted into a digital signal is simply referred to as first audio information.

The AD conversion unit 310b is an AD conversion circuit that converts the second audio information input from the microphone 10b from an analog signal to a digital signal. The AD converting section 310b outputs the second audio information converted into a digital signal to the audio filing section 320. FIG. In the following description, the second audio information converted into a digital signal is simply referred to as second audio information.

The audio filing unit 320 acquires the first audio information from the AD converting unit 310a and creates an audio file from the acquired first audio information. Audio filing unit 320 outputs the audio filed first audio information to transmission unit 330 . In addition, the audio filing unit 320 acquires the second audio information from the AD conversion unit 310b and forms the acquired second audio information into an audio file. The voice filing unit 320 outputs the voice filed second voice information to the transmission unit 330 .

The transmission unit 330 transmits the first audio information converted into an audio file to the audio processing device 2 via the network 5 . Also, the transmission unit 330 transmits the second audio information converted into an audio file to the audio processing device 2 via the network 5 .

The audio processing device 2 receives audio files of the first audio information and the second audio information from the recording device 300 . The speech processing device 2 determines a doubtful judgment section from the utterance section of the user U102 based on each piece of sound information included in the sound file. The speech processing device 2 stores the speech information of the doubtful decision section in the storage device.

FIG. 11 is a functional block diagram showing the configuration of the speech processing device according to the second embodiment. As shown in FIG. 11 , this speech processing device 2 has a receiving section 21 , a preprocessing section 22 , a storage section 23 and a state determination section 200 . Each processing unit of the receiving unit 21, the preprocessing unit 22, the storage unit 23, and the state determination unit 200 is realized by, for example, a CPU, an MPU, or the like. Also, each processing unit may be implemented by an integrated circuit such as an ASIC or FPGA, for example.

The receiving unit 21 is a processing unit that receives the first audio information and the second audio information converted into audio files from the recording device 300 via the network 5 . In the following description, the first audio information and the second audio information converted into audio files are simply referred to as the first audio information and the second audio information. The receiving section 21 outputs the first audio information and the second audio information to the preprocessing section 22 .

The preprocessing unit 22 performs various preprocessing on the first audio information and the second audio information, and outputs the preprocessed first audio information and the second audio information to the state determination unit 200. Department. For example, in the system shown in FIG. 9, voices of both users U101 and U102 may be collected by microphones 10a and 10b. Therefore, the preprocessing unit 22 performs preprocessing to remove the voice of the user U102 from the first voice information so that only the voice of the user U101 is included in the first voice information. The preprocessing unit 22 performs preprocessing to remove the voice of the user U101 from the second voice information so that only the voice of the user U102 is included in the second voice information.

The state determination unit 200 acquires the first audio information and the second audio information, detects the utterance period, and identifies the filler period from each utterance period. The state determination unit 200 determines whether or not the filler section is a questionable section based on the feature amount of the audio information included in the identified filler section. The state determination unit 200 stores the voice information of the uncertain judgment interval in the storage unit 23 when the filler interval is the doubtful judgment interval. In the present embodiment, a case will be described in which a filler section is specified from the utterance section of the second audio information, and whether or not it is an indecisive section is determined, but the present invention is not limited to this. The state determination unit 200 may identify a filler segment from the utterance segment of the first audio information, and determine whether or not it is a questionable segment.

For example, the state determination unit 200 converts the second audio information into an input spectrum by performing a short-time discrete Fourier transform on the second audio information. The state determination section 200 determines the brightness of the speech period using the feature amount related to the input spectrum.

The storage unit 23 is a storage device that stores voice information of the doubtful decision section. The storage unit 23 corresponds to a storage device such as a semiconductor memory device such as RAM, ROM, flash memory, or the like.

FIG. 12 is a functional block diagram showing the configuration of the state determination unit according to the second embodiment. As shown in FIG. 12 , the state determination section 200 has speech period detection sections 210 a and 210 b , an identification section 220 , a response time calculation section 230 , a brightness calculation section 240 and a determination section 250 .

The utterance period detection unit 210a receives input of the first voice information of the first speaker (user U101), and calculates a period sandwiched between silent periods of low power of the first voice information as the utterance period. Speech period detection section 210 a outputs information on the speech period of the first audio information to response time calculation section 130 . The process of calculating the speech period by the speech period detection unit 210a is the same as the process of the speech period detection unit 110a.

The utterance period detection unit 210b receives input of the second voice information of the second speaker (user U102), and calculates a period sandwiched between silent periods of low power of the second voice information as a utterance period. The speech period detection unit 210 b outputs information on the speech period of the second audio information to the identification unit 220 and the brightness calculation unit 240 . The process of calculating the utterance period by the utterance period detection section 210b is the same as the process of the utterance period detection section 110b.

The specifying unit 220 is a processing unit that receives the second voice information and the information on the speech period of the second speaker and determines whether the speech period is a filler period. Identifying section 120 outputs the determination result to response time calculating section 230 and determining section 250 . The processing by the identifying unit 220 for determining whether or not it is a filler section is the same as the processing by the identifying unit 120 described in the first embodiment.

The response time calculation unit 230 is a processing unit that calculates the response time from the end of the utterance period by the first speaker to the start of the utterance period (not the filler period) by the second speaker. . Response time calculation section 230 outputs response time information to determination section 250 . The process of calculating the response time by the response time calculator 230 is the same as the process of the response time calculator 130 described in the first embodiment.

The brightness calculation unit 240 is a processing unit that receives the second voice information and the information of the speech period of the second speaker, and calculates the feature amount of the brightness of the speech period of the second speaker. Brightness calculation section 240 outputs information on the brightness feature amount of each speech period to determination section 250 .

An example of the processing of the brightness calculation unit 240 will be described below. First, the brightness calculation unit 240 performs a short-term discrete Fourier transform on the second speech information x2(n) included in the utterance period T2s(k2) to T2e(k2) of the second speaker. , to generate the input spectrum X2(l) of the second speech information.

The brightness calculation unit 240 is described in the document (F. Eyben et al., “The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing,” in IEEE Transactions on Affective Computing, vol. 7, no. 2, pp. .190-202, April-June 1 2016.), etc., calculate statistics such as the average, variance, and median of multiple feature values regarding the input spectrum, and obtain a feature vector V (ks, s). Here, "s" of the feature vector V indicates the number of elements of the feature quantity.

The feature amount related to the input spectrum includes a feature amount related to the shape of the spectrum, a feature amount related to volume, and a feature amount related to speech speed. Features related to the shape of the spectrum include "Alpha Ratio, Hammarberg Indes, Spectral Slope 0-500Hz, Spectral Slope 500-1500Hz, Formant 1,2, and relative energy, Harmonic difference H1-H2, Harmonic difference H1-A3", etc. included. The volume-related feature quantity includes "Loudness, Rate of loudness peaks" and the like. Features related to speech speed include "Number of continuous voiced regions per second (pseudo syllable rate)".

The brightness calculation unit 240 may generate a feature vector V(ks, s) from a plurality of feature amounts related to the input spectrum, or may generate a feature vector V(ks, s ) may be generated. The brightness calculation unit 240 may also use speech recognition to calculate the number of characters per second as the speech speed. Brightness calculation section 240 calculates a feature vector for each utterance period of the second speaker and outputs it to determination section 250 .

In addition, the brightness calculation unit 240 performs the following pre-processing. When receiving teacher data from the determination unit 250, the brightness calculation unit 240 calculates the feature vector V by performing the above processing on the teacher data. The brightness calculation section 240 outputs the feature vector V calculated for the teacher data to the determination section 250 . A feature vector V corresponding to the training data is used when determining the threshold.

The determination unit 250 acquires the presence/absence F of the filler segment, the response time R, and the feature vector V for each utterance segment, and performs the following processing to determine whether or not the filler segment is a doubtful segment. The determination unit 250 stores the second audio information of the uncertain judgment interval in the storage unit 23 when judging that it is the uncertain judgment interval.

First, pre-processing of the determination unit 250 will be described. By executing such preprocessing, a threshold TH_V(s) for classifying the feature vector V(k2, s) into "bright" or "dark" is prepared. The judging unit 250 collects in advance teacher data of speech information in the filler section and speech information in the response section in which the brightness judgment result is determined to be “bright”. The determination unit 250 outputs the teacher data to the brightness calculation unit 240 and acquires the feature vector V(k2, s) corresponding to the teacher data. The determination unit 250 inputs the set of the feature vector V(k2, s) and the correct label to the support vector machine, and performs two-class classification of “bright” or “dark”. The determination unit 250 sets the boundary surface of the two-class classification of “bright” and “dark” as the brightness threshold TH_V(s). It is assumed that the value of the response time threshold TH_R is set in advance.

A process of determining whether or not the determination section 250 is in an indecisive section will be described. The determination unit 250 compares the feature vector V(k2, s) of the filler section with the threshold TH_V(s), and if the feature vector V(ks, s) is equal to or greater than the threshold TH_V(s), the filler section It is determined that the interval is “bright”. The determination unit 250 compares the feature vector V(k2, s) of the filler section with the threshold TH_V(s), and if the feature vector V(ks, s) is less than the threshold TH_V(s), the filler section The section is determined to be "dark". Note that the determination unit 250 may input the feature vector V(k2, s) of the filler section to a learned vector machine to determine "bright" or "dark".

The determination unit 250 compares the feature vector V(k2, s) of the response interval with the threshold TH_V(s), and if the feature vector V(ks, s) is equal to or greater than the threshold TH_V(s), the response It is determined that the interval is “bright”. The determination unit 250 compares the feature vector V(k2, s) of the response interval with the threshold TH_V(s), and if the feature vector V(ks, s) is less than the threshold TH_V(s), the response The section is determined to be "dark". Note that the determination unit 250 may input the feature vector V(k2, s) of the response interval to a learned vector machine to determine “bright” or “dark”.

In addition, similarly to the determination unit 180 described in the first embodiment, the determination unit 250 determines the response time corresponding to the utterance segment with the utterance segment number “k2” (F(k2)=1) determined as the filler segment. R(k2) is compared with the threshold TH_R to determine whether the response time R(k2) is "long" or "short".

When the determination result of the filler section is "dark", the determination result of the response section is "bright", and the response time is "long", the determination unit 250 determines that the filler section is a "deterministic section." ” is determined. On the other hand, when the determination result of the filler section is "bright", the determination result of the response section is "dark", or the response time is "long", the determination unit 250 determines that the filler section is the "difficult to judge" section. not."

Next, an example of the processing procedure of the state determination unit 200 of the speech processing device 2 of the second embodiment will be described. FIG. 13 is a flow chart showing the processing procedure of the speech processing device according to the second embodiment. As shown in FIG. 13, the state determination unit 200 of the speech processing device 2 acquires first speech information and second speech information (step S201).

The speech period detection unit 210a of the state determination unit 200 detects the speech period of the first speaker, and the speech period detection unit 210b detects the speech period of the second speaker (step S202). The specifying unit 220 of the state determination unit 200 detects the filler section (step S203). If there is no filler section (step S204, No), the state determination unit 200 proceeds to step S209.

On the other hand, if there is a filler section (step S204, Yes), the state determination unit 200 proceeds to step S205. The response time calculation unit 230 of the state determination unit 200 calculates the response time R (step 205). The brightness calculation unit 240 of the state determination unit 200 calculates a brightness feature vector (step S206).

The determination unit 250 determines whether or not the utterance period is the "difficult decision period" (step S207). If the utterance period is the "unsure judgment period" (step S207, Yes), the determination unit 250 stores the voice information of the uncertain judgment period in the storage unit 23 (step S208). On the other hand, if the utterance period is not the "unsure judgment period" (step S207, No), the determination unit 250 proceeds to step S209.

If there is a next conversation (step S209, Yes), the state determination unit 200 proceeds to step S203. If there is no next conversation (step S209, No), state determination unit 200 ends the process.

Next, effects of the speech processing device 2 according to the second embodiment will be described. The speech processing device 2 identifies a filler section from the speaker's utterance section, and determines the filler section as a judgment uncertain section based on the spectral feature amount of the identified filler section. This makes it possible to accurately estimate the section in which the user is hesitant.

By the way, the processing of the state determination unit 200 described above is an example, and other processing may be performed. For example, the brightness calculation unit 240 may generate a feature amount of brightness in advance by supervised machine learning. The brightness calculation unit 240 collects in advance teacher data of voice information in the filler section and voice information in the response section for which the brightness determination result is determined to be “bright”.

The brightness calculation unit 240 inputs the above teacher data to the DNN model described in the literature (SoundNet: Learning Sound Representations from Unlabeled Video Yusuf Aytar, Carl Vondrick, Antonio Torralba NIPS 2016) to learn the classifier. .

The brightness calculation unit 240 learns such a classifier in advance, inputs the speech information of the utterance period to the classifier, and converts the information output from the layer immediately before the output layer (final layer) of the classifier to It is calculated as a feature amount vector W(k2, u). Here, "u" of the feature vector W indicates the number of elements of the feature vector. Brightness calculation section 240 outputs feature amount vector W to determination section 250 .

Note that the determination unit 250 performs preprocessing to prepare a threshold TH_W(u) for classifying the feature vector W(k2, u) into "bright" or "dark". The judging unit 250 collects in advance teacher data of speech information in the filler section and speech information in the response section for which the brightness judgment result is determined to be “bright”. The determination unit 250 outputs the teacher data to the brightness calculation unit 240 and acquires the feature amount vector W(k2, u) corresponding to the teacher data. The determination unit 250 inputs the set of the feature amount vector W(k2, u) and the correct label to the support vector machine, and performs two-class classification of “bright” or “dark”. The determination unit 250 sets the boundary surface of the two-class classification of “bright” and “dark” as the brightness threshold TH_W(s). It is assumed that the value of the response time threshold TH_R is set in advance.

The determination unit 250 compares the feature vector W (k2, u) of the filler section with the threshold TH_W (s), and if the feature vector W (ks, u) is equal to or greater than the threshold TH_W (u), , the filler section is determined to be “bright”. The determination unit 250 compares the feature amount vector V(k2, u) of the filler section with the threshold TH_W(u), and if the feature vector W(ks, u) is less than the threshold TH_W(u), The filler section is determined to be "dark". Note that the determination unit 250 may input the feature vector V(k2, s) of the filler section to a learned vector machine to determine “bright” or “dark”.

When the determination result of the filler section is "dark", the determination result of the response section is "bright", and the response time is "long", the determination unit 250 determines that the filler section is a "deterministic section." ” is determined. On the other hand, when the determination result of the filler section is "bright", the determination result of the response section is "dark", or the response time is "long", the determination unit 250 determines that the filler section is the "difficult to judge" section. not."

Next, an example of the hardware configuration of a computer that implements the same functions as those of the speech processing apparatuses 1 and 2 shown in the above embodiments will be described. FIG. 14 is a diagram showing an example of the hardware configuration of a computer that implements functions similar to those of the audio processing device.

As shown in FIG. 14, a computer 300 has a CPU 301 that executes various arithmetic processes, an input device 302 that receives data input from a user, and a display 303 . The computer 300 also has a reading device 304 that reads programs and the like from a storage medium, and an interface device 305 that exchanges data with recording devices and the like via a wired or wireless network. The computer 300 also has a RAM 306 that temporarily stores various information, and a hard disk device 307 . Each device 301 - 307 is then connected to a bus 308 .

The hard disk device 307 has a speech period detection program 307a, a specific program 307b, a response time calculation program 307c, a brightness calculation program 307d, and a long-term average calculation program 307e. The hard disk device 307 also has a threshold calculation program 207f and a determination program 307g. The CPU 501 reads each program 307 a to 307 g and develops them in the RAM 306 .

The speech segment detection program 307a functions as a speech segment detection process 306a. Specific program 307b functions as specific process 306b. The response time calculation program 307c functions as a response time calculation process 306c. The brightness calculation program 307d functions as a brightness calculation process 306d. The long-term average calculation program 307e functions as a long-term average calculation process 306e. The threshold calculation program 307f functions as a threshold calculation process 306f. The determination program 307g functions as a determination process 306f.

The processing of the speech segment detection process 306a corresponds to the processing of the speech segment detection units 110a, 110b, 210a, and 210b. The identification process 306 b corresponds to the processing of the identification units 120 and 220 . The processing of the response time calculation process 306 c corresponds to the processing of the response time calculators 130 and 230 . A brightness calculation process 306 d corresponds to the processing of the brightness calculation units 140 and 240 . The long-term average calculation process 306 e corresponds to the processing of the long-term average calculation unit 150 . A threshold calculation process 306 f corresponds to the processing of the threshold calculation unit 160 . The determination process 306g corresponds to the processing of the determination unit 360g.

Note that the programs 307a to 307g do not necessarily have to be stored in the hard disk device 307 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk (FD), CD-ROM, DVD disk, magneto-optical disk, IC card, etc. inserted into the computer 300 . Then, the computer 500 may read and execute the programs 307a to 307g.

The following additional remarks are disclosed regarding the embodiments including the above examples.

(Appendix 1) Detecting a plurality of utterance intervals from voice information,
identifying an utterance segment in which filler is detected from the plurality of utterance segments as a filler segment;
causing a computer to execute a process of determining whether or not the voice information of the filler section is voice information to be uttered when the user hesitates to make a decision based on the feature amount of the voice information of the filler section; A speech processing program characterized by:

(Supplementary Note 2) The determination process is based on the feature amount of the voice information of the filler section and the feature amount of the voice information of the utterance section following the filler section. The speech processing program according to appendix 1, wherein the speech processing program determines whether or not the speech information is spoken when the judgment is uncertain.

(Supplementary note 3) The determination process further uses the response time from the first user's utterance period to the second user's utterance period, and the voice information of the filler period is used by the second user to make a determination. 3. The speech processing program according to appendix 1 or 2, wherein it is determined whether or not the speech information is uttered when hesitating.

(Supplementary note 4) The determination process calculates an average value of each feature amount calculated from each utterance period of the second user, and further uses the average value of each feature amount to determine the second user 3. The voice processing program according to appendix 3, wherein it is determined whether or not the voice information is to be uttered when the determination is uncertain.

(Appendix 5) The audio processing program according to any one of appendices 1 to 4, wherein the determining process uses a variance calculated based on the fundamental frequency of the audio information as the feature amount. .

(Appendix 6) The audio processing program according to any one of appendices 1 to 4, wherein the determination process converts the audio information into a spectrum, and uses a feature of the spectrum as the feature amount.

(Supplementary Note 7) In the determination process, a classifier trained using teacher data that associates the feature amount with information indicating whether or not the audio information corresponding to the feature amount is bright audio is given the feature Supplementary note 1 characterized by determining whether or not the voice information in the filler section is voice information to be uttered when the user hesitates to make a decision based on the result obtained by inputting the amount. the audio processing program described in .

(Supplementary Note 8) The determination process acquires a first threshold, a second threshold, and a third threshold corresponding to the second user, and compares the feature amount of the speech information of the filler section with the first threshold. As a result, the comparison result between the feature amount of the speech information in the utterance section following the filler section and the first threshold, the comparison result between the response time and the second threshold, the average value of each feature amount and the third 4. The method according to appendix 4, wherein it is determined whether or not the voice information in the filler section is voice information to be spoken when the user hesitates to make a decision, based on the comparison result with the threshold. sound processing program.

(Appendix 9) A computer-executed audio processing method,
Detect multiple utterance segments from voice information,
identifying an utterance segment in which filler is detected from the plurality of utterance segments as a filler segment;
Determining whether or not the voice information of the filler section is voice information to be uttered when the user hesitates to make a decision based on the feature amount of the voice information of the filler section. An audio processing method characterized by:

(Supplementary Note 10) The determination process is based on the feature amount of the voice information of the filler section and the feature amount of the voice information of the utterance section following the filler section. The speech processing method according to appendix 9, wherein it is determined whether or not the speech is speech information to be spoken when the judgment is uncertain.

(Supplementary note 11) The determination process further uses the response time from the first user's utterance period to the second user's utterance period, and the voice information of the filler period is used by the second user for determination. 11. The voice processing method according to appendix 9 or 10, wherein it is determined whether or not the voice information is uttered when in doubt.

(Supplementary note 12) In the determination process, an average value of each feature amount calculated from each utterance period of the second user is calculated, and further using the average value of each feature amount, the second user 12. The speech processing method according to appendix 11, wherein it is determined whether or not the speech information is to be spoken when the judgment is uncertain.

(Appendix 13) The speech processing method according to any one of appendices 9 to 12, wherein the determination process uses a variance calculated based on the fundamental frequency of the speech information as the feature amount. .

(Appendix 14) The speech processing method according to any one of appendices 9 to 12, wherein the determination process converts the speech information into a spectrum and uses a feature of the spectrum as the feature amount.

(Supplementary note 15) In the determination process, a classifier learned by teacher data that associates the feature amount with information indicating whether or not the speech information corresponding to the feature amount is bright voice, and the feature Supplementary note 9 characterized by determining whether or not the voice information in the filler section is voice information to be uttered when the user hesitates to make a decision based on the result obtained by inputting the amount. 2. The audio processing method described in .

(Supplementary Note 16) The determination process acquires a first threshold, a second threshold, and a third threshold corresponding to the second user, and compares the feature amount of the speech information of the filler section with the first threshold. As a result, the comparison result between the feature amount of the speech information in the utterance section following the filler section and the first threshold, the comparison result between the response time and the second threshold, the average value of each feature amount and the third 13. The method according to appendix 12, wherein it is determined whether or not the voice information in the filler section is voice information to be spoken when the user hesitates to make a decision based on the comparison result with the threshold. Audio processing method.

(Appendix 17) an utterance period detection unit that detects a plurality of utterance periods from voice information;
a specifying unit that specifies, as a filler segment, an utterance segment in which a filler is detected from the plurality of utterance segments;
a determination unit that determines whether or not the voice information in the filler section is voice information to be uttered when the user hesitates to make a decision, based on the feature amount of the voice information in the filler section; A speech processing device characterized by:

(Supplementary note 18) The determination unit determines, based on the feature amount of the voice information of the filler section and the feature amount of the voice information of the utterance section following the filler section, that the voice information of the filler section is 18. The speech processing device according to appendix 17, wherein it is determined whether or not the information is voice information to be uttered when in doubt.

(Supplementary note 19) The determination unit further uses the response time from the first user's utterance period to the second user's utterance period to determine whether the speech information in the filler period is 19. The speech processing device according to appendix 17 or 18, wherein it is determined whether or not the speech information is to be uttered when the voice is spoken.

(Supplementary Note 20) The determination unit calculates an average value of each feature amount calculated from each utterance period of the second user, and further uses the average value of each feature amount to determine whether the second user 19. The speech processing device according to appendix 19, wherein it is determined whether or not the information is voice information to be spoken when the user is uncertain about the determination.

(Appendix 21) The audio processing apparatus according to any one of appendices 17 to 20, wherein the determination unit uses a variance calculated based on the fundamental frequency of the audio information as the feature amount.

(Appendix 22) The speech processing apparatus according to any one of appendices 17 to 20, wherein the determination unit converts the speech information into a spectrum and uses a feature of the spectrum as the feature amount.

(Supplementary Note 23) The determination unit assigns the feature amount to a classifier trained using teacher data that associates the feature amount with information indicating whether or not the audio information corresponding to the feature amount is bright audio. Supplementary note 17, wherein it is determined whether or not the voice information in the filler section is voice information to be spoken when the user is unsure of the decision based on the result obtained by inputting A sound processing device as described.

(Additional remark 24) The determination unit acquires a first threshold, a second threshold, and a third threshold corresponding to the second user, and compares the feature amount of the speech information of the filler section with the first threshold. , a comparison result between the feature amount of the speech information in the utterance section following the filler section and the first threshold, a comparison result between the response time and the second threshold, an average value of each feature amount and the third threshold Based on the result of comparison with the voice according to Supplementary Note 20, it is determined whether or not the voice information in the filler section is voice information to be spoken when the user is unsure of the decision. processing equipment.

Reference Signs List 1, 2 Speech processing device 5 Network 10a, 10b Microphones 20a, 20b, 310a, 310b AD conversion unit 21 Receiving unit 22, 30 Preprocessing unit 23, 40 Storage unit 100, 200 State determination unit 110a, 110b, 210a, 210b Speech Section detection unit 120, 220 Identification unit 130, 230 Response time calculation unit 140, 240 Brightness calculation unit 150 Long-term average calculation unit 160 Threshold calculation unit 170 Threshold DB
180, 250 Determination unit 300 Recording device 320 Audio file generation unit 330 Transmission unit

Claims (9)

Detect multiple utterance segments from voice information,
identifying an utterance segment in which filler is detected from the plurality of utterance segments as a filler segment;
identifying an utterance segment included in the plurality of utterance segments, the utterance segment following the filler segment as an utterance segment following the filler segment;
Based on the combination of the feature amount of the voice information of the filler section and the comparison result of the first threshold and the comparison result of the feature amount of the voice information of the utterance section following the filler section and the second threshold, the filler section A voice processing program characterized by causing a computer to execute a process of determining whether or not the voice information of is voice information to be uttered when the user hesitates to make a decision. The determination process further uses the response time from the first user's utterance period to the second user's utterance period, and the voice information of the filler period is used when the second user is unsure of the judgment 2. The voice processing program according to claim 1, wherein it is determined whether or not the voice information is voice information to be uttered. In the determination processing, an average value of each feature amount calculated from each utterance period of the second user is calculated, and the average value of each feature amount is further used to determine whether the second user hesitates to make a decision. 3. The voice processing program according to claim 2 , wherein it is determined whether or not the voice information is spoken when the voice is spoken. 4. The audio processing program according to claim 1 , wherein said determining process uses a variance calculated based on a fundamental frequency of said audio information as said feature quantity. 4. The audio processing program according to claim 1 , wherein said determining process converts said audio information into a spectrum and uses a feature of said spectrum as said feature quantity. In the determination process, the feature amount is input to a classifier learned by teacher data that associates the feature amount with information indicating whether or not the speech information corresponding to the feature amount is a bright voice. 2. The method according to claim 1, wherein, based on the result obtained by the above-mentioned processing, it is determined whether or not the voice information in the filler section is voice information to be uttered when the user hesitates to make a decision. sound processing program. The determination process acquires the first threshold, the second threshold, and the third threshold corresponding to the second user, and compares the feature amount of the voice information of the filler section with the first threshold, A comparison result between the feature amount of the speech information in the utterance section following the filler section and the first threshold, a comparison result between the response time and the second threshold, an average value of each feature amount and the third threshold 4. The voice according to claim 3 , wherein it is determined whether or not the voice information in the filler section is voice information to be uttered when the user hesitates to make a decision based on the comparison result of processing program. A computer-implemented audio processing method comprising:
Detect multiple utterance segments from voice information,
identifying an utterance segment in which filler is detected from the plurality of utterance segments as a filler segment;
identifying an utterance segment included in the plurality of utterance segments, the utterance segment following the filler segment as an utterance segment following the filler segment;
Based on the combination of the feature amount of the voice information of the filler section and the comparison result of the first threshold and the comparison result of the feature amount of the voice information of the utterance section following the filler section and the second threshold, the filler section A voice processing method characterized by determining whether or not the voice information of is voice information to be uttered when the user hesitates to make a decision. an utterance segment detection unit that detects a plurality of utterance segments from voice information;
An utterance segment in which a filler is detected from the plurality of utterance segments is specified as a filler segment, and an utterance segment included in the plurality of utterance segments and next to the filler segment is an utterance segment following the filler segment. a specific part identified as
Based on the combination of the feature amount of the voice information of the filler section and the comparison result of the first threshold and the comparison result of the feature amount of the voice information of the utterance section following the filler section and the second threshold, the filler section and a judgment unit for judging whether or not the speech information of is speech information to be uttered when the user hesitates to make a decision.

Images