JP2015068897A - Evaluation method and device for utterance and computer program for evaluating utterance - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an utterance evaluation system having a useful extraction method for extracting clearness as a feature of evaluating utterance.SOLUTION: An utterance evaluation device includes: an input part to which a voice signal of the free utterance of an utterer is input; a feature extraction part for extracting a feature to be used for evaluation from the input voice signal; a feature evaluation part for comparing the feature extracted by the feature extraction part with a preliminarily stored reference feature; and an output part for outputting a comparison result. The feature to be used for evaluation includes at least the clearness of utterance, and the clearness of utterance is represented with a ratio of a resonance sound to an obstruction sound in the input voice signal. The feature extraction part acquires the ratio of the resonance sound to the obstruction sound by using means for dividing the input voice signal into a plurality of segments and means for classifying the obtained segments into the obstruction sound and the resonance sound.

Description

The present invention relates to an evaluation method and apparatus for utterances (how to speak, speak, speech, etc.) and a computer program for evaluating utterances.

Every human being has a certain habit or characteristic in how to speak. For example, some people speak fast, some people have loud voices, and some have bad tongues. In addition, there are many cases where the individual does not recognize the wrinkles that the individual has.

As a method of correcting the way of speaking, it is common to have a third party listen to speech and receive feedback later. It is also possible to record speech and evaluate the recorded speech later.

In addition to not being very pleasant to be evaluated by others about your speech, it takes some time to evaluate and gives the speaker the opportunity to correct or correct it during the speech Is difficult.

Therefore, it is conceivable to evaluate utterances by machines rather than humans. There are a number of known user utterance features that can affect the human perceivable quality. For example, the degree of stuttering, poorness, accuracy of word pronunciation, word generation speed, pitch variation, and loudness all make a difference in the perception of speech quality. For example, the loudness and pitch of a voice are features that can be easily extracted by both humans and machines, and there are existing methods. However, the clarity of the utterance is not easily noticed by the utterer himself, and there is no useful existing method for recognizing this.

Various researches have been conducted on speech processing since ancient times. Representative speech research includes speech synthesis, speech recognition, speaker recognition, etc. These objectives are not to evaluate normal natural speech and feed it back to the speaker. For example, the purpose of automatic speech recognition is conversion from utterance to text, and even though a speech recognition module is installed in a smartphone, it currently requires speech input with a certain degree of clear pronunciation, and natural free speech Is not intended for processing.

Speech evaluation is performed using speech processing in the field of language learning (Non-Patent Documents 1 and 2). Specifically, the speaker's speech on the text facilitated in advance and the text recorded in advance. It is intended to evaluate the voice of a native speaker, and not to evaluate free speech.

In voice processing, various feature amounts are extracted according to the purpose, but in order to use them as feedback to the speaker, the feature amounts must be controllable by humans. In a typical example, a human being can easily control a feature amount such as voice volume. On the other hand, in the field of phonetics, attempts have been made to extract and evaluate features by speech processing (Non-patent Document 3), but the features extracted there are limited to human control. Such features are difficult to use as feedback to the speaker.

In addition, the inventors of the present application have considered how to extract useful speech and provide useful feedback, particularly using a mobile device such as a smartphone. In order to perform utterance evaluation in real time, particularly on a mobile device such as a smartphone, it is important that the amount of feature calculation is small.
Franco, H., Abrash, V., Precoda, K., et al. The SRI EduSpeakTMsystem: Recognition and pronunciation scoring for language learning.Proc. InSTILL '00 (2000). Lai, Y., Tsai, H., and Yu, P. A multimedia English learning systemusing HMMs to improve phonemic awareness for English learning.Educational Technology and Society (2009). Rusilo, L., and de Camargo, Z. The validity of some acousticmeasures to predict voice quality settings: trends between acoustic and perceptual correlates of voice quality.In ExLing '11 (2011).

An object of this invention is to provide the speech evaluation system provided with the clarity as the characteristic which evaluates speech.
Another object of the present invention is to provide an utterance evaluation system that provides an evaluation result to a speaker in real time on a mobile device such as a smartphone.

The technical means adopted by the present invention are:
An input unit for inputting a voice signal of a speaker's free utterance;
A feature extraction unit that extracts features used for evaluation from the input audio signal;
A feature evaluation unit that compares the feature extracted by the feature extraction unit with a reference feature stored in advance;
An output unit for outputting a comparison result;
With
Features used for the evaluation include at least clarity of utterance,
The intelligibility of the utterance is represented by an inhibition sound to resonance ratio in the input audio signal,
The feature extraction unit acquires the inhibition sound-to-resonance sound ratio using means for dividing the input audio signal into a plurality of segments and means for classifying the obtained segment into an inhibition sound and a resonance sound. ,
An utterance evaluation device.

  In one aspect, the feature used for the evaluation further includes a voice volume, and the feature extraction unit obtains the voice volume from the energy of the input voice signal.

In one aspect, the features used for the evaluation further include a pause rate,
The pause rate is the percentage of time that speech is not spoken in the length of the audio signal,
The feature extraction unit acquires the ratio by a voice section detection unit.
In one aspect, the voice segment detection means detects a voice segment using the energy of the input voice signal.

In one aspect, the features used for the evaluation further include pitch.
The feature extraction unit acquires a pitch by a pitch extraction unit.
In one aspect, the pitch extraction means is means using an autocorrelation method.
In an embodiment described later, ESACF (enhanced summary autocorrelation function) is used as the autocorrelation method.

In one aspect, the features used for the evaluation further include the speed of beat,
The speed of the beat is represented by the number of utterance events within a predetermined time,
The feature extraction unit obtains the speed of speaking by counting the number of utterance events within a predetermined time.
In one aspect, the utterance event is a syllable.

In one aspect, the function of the voice representing loudness, the vertical axis is energy, and the horizontal axis is time is called a loudness function.
The feature extraction unit sets a local energy minimum point of the loudness function as a syllable boundary.
In one aspect, the feature extraction unit includes:
A first means for obtaining a loudness function from the input audio signal;
A second means for calculating a convex hull function from the loudness function;
Searching for a point where the difference between the calculated convex hull function and the corresponding loudness function is maximum, and if the difference is greater than or equal to a threshold value, a third means for determining the point as an energy local minimum point;
With
The first means, the second means, and the third means are recursively repeated.

In one aspect, in the means for dividing the input audio signal into a plurality of segments, the segments are syllables,
The feature extraction unit acquires a plurality of segments using syllable boundaries when acquiring the speed of the beat.

In one aspect, the means for classifying the segment into an inhibition sound and a resonance sound uses supervised machine learning means.
In one aspect, the supervised machine learning means uses a time length of a segment, periodicity, and a mel frequency cepstrum coefficient (MFCC) as a feature vector. For example, periodicity can be obtained in pitch extraction. The calculation of the mel frequency cepstrum coefficient is well known to those skilled in the art.
In one aspect, the supervised machine learning means is a support vector machine (SVM).
Although SVM may be heavy learning, SVM is advantageous because classification is fast and is performed with light computation.
The supervised machine learning means that can be applied to the present invention is not limited to the SVM, but a simple Bayes classifier, K-neighbor method, Bayesian network, HMM (Hidden Markov Model), GMM (Mixed Gaussian model), clustering , Discriminant analysis (linear discriminant function or Mahalanobis distance), logistic regression analysis, decision tree, and other existing means.

In one aspect, the inhibitory sound to resonance sound ratio is a ratio of the number of inhibition sound segments to the number of resonance sound segments.
It can be considered that the higher the energy of the inhibitory sound, the higher the clarity, and the higher the number of inhibitory sound segments in the utterance for a given time (as seen by the resonance ratio), the higher the energy, It can be determined that the clarity is high.

In one aspect, the segment is a phone or a phoneme, and the inhibition sound to resonance sound ratio is an inhibition sound energy to resonance sound energy ratio.
It is assumed that a single sound or a phoneme break in a speech signal is extracted, each single sound or phoneme is classified as either an inhibition sound or a resonance sound, the energy of the inhibition sound is measured, and the higher the energy of the inhibition sound, the clearer the sound.
The level of the energy of the inhibitory sound can be determined by the level of the inhibitory sound energy in the utterance for a predetermined time. And a substantially equivalent index). If the ratio is high (inhibition sound energy is high), it is determined that the clarity of the utterance is high.
As the energy of the inhibition sound, for example, RMS can be used, but another method of measuring the volume may be used.

  In one aspect, the reference feature stored in the feature evaluation unit is a feature that characterizes an utterance targeted by each user.

In one aspect, the output unit is a display unit that visually displays a comparison result.
In one aspect, the feature extraction unit, the feature evaluation unit, and the output unit are executed in real time on the input audio signal.

  In one aspect, the speech evaluation apparatus is configured from a mobile device such as a smartphone.

In one aspect, the feature is an average value of the feature at a set time.
In one aspect, the standard deviation of the feature at a set time is further calculated.

Other technical means adopted by the present invention are:
Inputting a speech signal of a speaker's free speech;
A feature extraction step of extracting features used for evaluation from the input speech signal;
A feature evaluation step for comparing the feature extracted in the feature extraction step with a pre-stored reference feature;
Outputting a comparison result; and
With
Features used for the evaluation include at least clarity of utterance,
The intelligibility of the utterance is represented by an inhibition sound to resonance ratio in the input audio signal,
The feature extraction step includes a step of dividing the input audio signal into a plurality of segments, a step of classifying the obtained segment into an inhibition sound and a resonance sound, and a step of acquiring an inhibition sound to resonance sound ratio. Contains,
This is an utterance evaluation method.

  The description about many aspects described in the said speech evaluation apparatus can be used as an aspect which limits a speech evaluation method.

  Another technical means employed by the present invention is a computer program for causing a computer to execute an utterance evaluation method.

According to the present invention, it is possible to provide an utterance evaluation method including one of the features of “clarity of utterance”, which is difficult to notice by the utterer himself and there is no useful recognition method.
According to the present invention, it is possible to find and correct a problem in one's own utterance by feeding back features that the speaker can control.
According to the present invention, by storing the reference features for the target or ideal utterance, the user can make corrections that bring his utterance closer to the target or ideal speech.
The features used in the present invention are relatively light in computation and enable real-time feedback in mobile devices with limited resources such as smartphones.

1 is an overall schematic diagram of an utterance evaluation system according to the present invention. It is a front view of the smart phone equipped with the speech evaluation system, the left figure shows an overview screen, and the right figure shows a detail screen. It is a state transition diagram of the user interface of an utterance evaluation system. It is a figure which shows the outline | summary of the architecture of an utterance evaluation system. It is a figure which shows the CPU usage rate of the said system in the smart phone by which the speech evaluation system is mounted. It is a figure which shows the detail of the feature extraction part of the speech evaluation system which concerns on this embodiment. The autocorrelation value obtained by the ESACF algorithm when applied to two 500 ms samples of speech is shown. The audio was sampled at 16000 Hz and ESACF was applied at W = 1024. The upper figure shows high periodicity, low frequency, and the lower figure shows low periodicity, noise, and high frequency. The upper figure shows the original waveform and sound data from the TIMIT speech corpus, and the lower figure shows the loudness function obtained therefrom. The utterance is speaker FAEM0, sample SA2: “[...] Carry an oily rag [...].”. The upper diagram is the same as FIG. 8, and the lower diagram shows the convex hull and segment boundary of the loudness function in the loudness function of FIG. The result of the speed extraction algorithm of this embodiment about the speech corpus sample is shown in comparison with the actual speed. It is a figure similar to FIG. 9, and shows a syllable segment boundary in a loudness function. The syllable segments in FIG. 11 are classified into inhibition sounds (O) and resonance sounds (C). In this example, clarity score = number of inhibition sounds / number of all segments = 4 / (4 + 3) = 0.57. Clearspeechjph Shows the distribution of clearness values for clear speech (upper figure) and plain speech (lower figure) in S teaching materials. (a) shows the average value of each feature of the four speakers over 1 minute. (b) shows the standard deviation of the pitch and speaking speed for each speaker.

[A] Outline of Speech Evaluation System As shown in FIG. 1, the speech evaluation system according to the present embodiment includes an input unit (microphone) to which a speech signal of a speaker's free speech is input and an input speech signal. A feature extraction unit that extracts features used for evaluation, a feature evaluation unit that compares a feature extracted by the feature extraction unit with a reference feature stored in advance, and an output unit (screen) that outputs a comparison result The system can be configured from a computer (including a storage unit such as an input unit, a display, a RAM, and a ROM, and a processing unit mainly including a CPU).

The feature extraction unit is composed of a processing unit of a computer, and extracts a feature of an utterance by processing a sound signal obtained by a microphone. In the present embodiment, there are five features acquired by the feature extraction unit: voice volume, pause rate, pitch, speaking speed, and clarity. These features will be described in detail later.

The reference feature storage unit is composed of a storage unit of a computer, and reference features corresponding to the features extracted by the feature extraction unit, specifically, voice volume, pause rate, pitch, speaking speed, and clarity 5 One reference feature is stored. The reference features are extracted by extracting the features of the reference utterance (input from the microphone or the media library in which the utterance is recorded) in the learning mode, and processing them in the feature learning unit. Stored in the storage unit. The actual value of the reference feature can vary from user to user.

The feature evaluation unit is composed of a processing unit of a computer, compares the feature of the input utterance extracted by the feature extraction unit with the reference feature stored in the reference feature storage unit, and displays the comparison result on the screen.

The left figure in Fig. 2 is a screen that displays an overview of the evaluation results, called Energy (voice volume), Pause (pause rate), Pitch (pitch), Speed (speaking speed), and Clarity (clarity). For five features, reference features (learning data) are arranged at the vertices of a regular pentagon. The feature of the user's utterance is displayed on a relative scale with respect to the feature to be referred to. The average value of each feature of the user utterance at a predetermined time (time set by the user) is a vertex, and the white thick line is an edge connecting the vertices. A deviation between each vertex of the pentagon of the user's utterance feature and the regular pentagon represents a deviation from the reference feature.

On the screen, an area with an average value of ± 1 standard deviation is shown in a band shape along a pentagonal side connecting user characteristics (see the left figure in FIG. 2). As the standard deviation of the feature increases, the width of the band increases, and the band does not have the same width. The standard deviation indicates the change of each feature over a predetermined time. By showing temporal changes in features, secondary features can be visually displayed. For example, large changes in loudness and pitch over a period of time mean dynamic speech (on the contrary, small changes are monotone speech), and small changes in speed mean a stable rhythm. To do.

Below the overview screen is a slider for setting the time frame for calculating the mean and standard deviation of the features. The slider is composed of a horizontal track and a circular button that can be moved on the track. For example, by dragging the slider with the thumb, the time frame can be set to a desired time (for example, any time between 1 to 30 seconds). Time).

A learning button (Train) and a detail display button (Details) are arranged on the upper right of the overview screen. When the detail display button is touched, the screen is switched to the detail display (see the right figure in FIG. 2). Touch the learning button to switch to learning mode. Learning of the speech evaluation system can be performed by loading live recording of speech from the microphone of the device or audio recording of the media library stored in the storage unit of the device (see FIG. 3). System learning can be done offline.

The user interacts with the utterance rating system by speaking and receives visual feedback, and the user can compare his / her way of speaking with that of others (which may be a sample of his / her way of speaking). By learning the utterance evaluation system with a model of speaking that you want to imitate, you can recognize the difference between the target speaking method and your own speaking method so that it approaches the target speaking method You can try to correct or correct. The application of the speech evaluation system is installed in a smartphone, and when the user's voice is input, the five features of voice volume, pause rate, pitch, speaking speed, and clarity are extracted, and the format is easy for the user to understand. Provides graphical feedback to users in real time. Speech evaluation can be performed during speech or presentation, or during a call (for example, when the screen can be viewed using a headset, hands-free unit, or speakerphone). it can.

In the present embodiment, the utterance evaluation system is mounted on a smartphone. A smartphone includes hardware elements of a general-purpose computer, and an utterance evaluation system can be configured from the smartphone. However, as shown in Table 1, the capacity of the CPU and the storage unit is limited compared to a personal computer, and the calculation of digital signal processing and the display processing on the screen can be executed with a smaller calculation amount. Required.

FIG. 4 shows an outline of the architecture of the speech evaluation system according to this embodiment. In the present embodiment, an architecture with a low calculation amount is supported by a multi-thread model. The threads are formed as an audio capture thread, an audio processing thread, and a rendering thread, and exchange data with each other by a Job Queue managed by Grand Central Dispatch. Audio capture is performed by iOS's Audio Unit API, which generates a 25 ms non-overlapping frame stream of 16 kHz audio, resulting in a system that processes frames at 40 Hz. Speech processing algorithms (vector, matrix, Fast Fourier Transform (FFT) calculations) are performed by a vDSP that provides a single instruction, multiple data (SIMD) CPU instruction API. Graphic rendering uses both Core Graphics (“Quartz”) and OpenGL. Quartz is used when its advanced rendering capabilities are needed, and Quartz image data is cached in the OpenGL texture memory on the device's GPU. OpenGL is used for all other renderings, and cached Quartz image data can be accessed directly if needed.

FIG. 5 shows the CPU usage when the speech evaluation system is executed (10 seconds) on a smartphone (iPhone 4 ’s single-core A4 CPU, with iOS 5.1.). The speech evaluation system accounts for 31.9% of CPU usage (A). Among them, voice processing accounts for 70.8% (B). A breakdown of the proportion of each feature in speech processing is shown (C). It can be seen that the CPU time increases as the features become more advanced.

[B] Feature Extraction There are many known features of user utterances that can affect the quality that humans can recognize. For example, the degree of stuttering, poorness, accuracy of word pronunciation, word generation speed, pitch variation, and loudness all make a difference in the perception of speech quality. The inventors of the present application have considered how a speech device can be extracted using a mobile device such as a smartphone to provide useful feedback. The present embodiment proposes a technique that can be executed online while minimizing calculation and memory requirements. In this embodiment, five features (voice volume, pause rate, pitch, speaking speed, and clarity) are selected as features to be extracted.

Table 2 shows the hierarchy of speech processing and utterance features and the acoustic / speech features. The processing is advanced in the order of digital signal processing (DSP), automatic speech recognition (ASR), and natural language processing (NLP), and the characteristics obtained by each processing are also different. In the present embodiment, five features (voice volume, pause rate, pitch, speaking speed, and clarity) are extracted by the DSP. Note that the features that can be used in the present invention are not limited to these five features, and may include other features (such as “stuttering” and “scoring such as ending”).

Among the five features used in this embodiment, some can be obtained using existing techniques as shown in Table 3. Lower level features support higher features. For example, the clarity of an utterance depends on the speed and pitch of the utterance, and the speed of the utterance depends on the volume of the voice. In the following description, a simpler feature (voice volume) will be described in turn to a more complicated feature (clarity).

As shown in FIG. 6, the voice signal input from the microphone is subjected to FFT processing, RMS amplitude is acquired, and voice loudness characteristics are acquired. Using this RMS amplitude, the voice interval is detected by the VAD algorithm, and the pause rate is acquired. Pitch extraction is performed on the audio signal by applying an autocorrelation method (ESACF). In the calculation of the fundamental frequency in pitch extraction, unvoiced intervals are removed using a VAD algorithm for detecting speech intervals. Mermelstein's for the loudness function obtained for the audio signal (vertical function is energy and horizontal function is time)
A segment is obtained by applying a syllabic-unit segmentation algorithm (hereinafter referred to as “Mermelstein algorithm”), and the speaking speed is obtained from the number of segments per unit time. Using this segment, the inhibitory sound / resonant sound is separated by the inhibitory sound / resonant sound classifying means to obtain the inhibitory sound to resonant sound ratio, and used as the speech clarity score. The scores of the five features (voice volume, pause rate, pitch, speaking speed, clarity) are average values for a predetermined time. In addition, a standard deviation at a predetermined time is calculated for these features. For example, the standard deviation about the pitch can be used as an index (secondary feature) of speech inflection.

[B-1] Voice loudness
In the present embodiment, in order to express a loudness level that matches human perception, well-known sound intensity or intensity is used. The sound intensity is calculated for each acoustic frame (for example, every 25 ms). The intensity of sound is a logarithmic quantity of acoustic energy (here, using a decibel scale) and approximates human perception of sound volume. As the acoustic energy, for example, RMS amplitude can be used.

That is, in the present embodiment, the loudness of the voice is expressed by the following expression.
Here, x is a vector representing an acoustic frame composed of N sample speech waveform (amplitude) values.
You can refer to the following papers for sound intensity. Jurafsky, D., Martin,
J., Kehler, A., et al. Speech and Language Processing: An Introduction to
Natural Language Processing, Computational Linguistics, and Speech Recognition,
second ed. Prentice Hall New Jersey, 2008.

The strength of the sound is low in calculation amount and versatile. According to experiments, it was confirmed that the technique using sound intensity works well if the user's distance from the microphone does not change significantly.

[B-2] Pause Ratio
The pause rate is defined as the amount of non-speaking time for the total length of the signal. A voice activity detection (VAD) algorithm is used to identify when the acoustic signal contains a human voice. Discriminating speech from noise is a basic task in the speech processing domain, and various methods have been proposed as shown in the following papers. Ramirez, J., G´orriz, J., and Segura, J. Voice activity detection.
fundamentals and speech recognition system robustness.
Recognition and Understanding (2007).

In the present embodiment, an energy-based method is used to guarantee feature estimation with a small amount of calculation. The latest t-second energy value (RMS amplitude) is stored in a circular buffer, and the minimum energy value is obtained as noise. It is assumed that energy exceeding a predetermined threshold with respect to the minimum energy value is utterance content. In the present embodiment, as a non-limiting numerical value, t = 5 seconds and the threshold is 6 dB. This technique is sensitive to noise, but is effective and requires less computation in a relatively quiet environment. The above method uses the RMS amplitude acquired to acquire the loudness of the voice, and does not separately calculate VAD, so that the calculation amount is small. Various VAD calculation methods are known, and the VAD that can be used in the present invention is not limited to the above method.

[B-3] Pitch
The fundamental frequency (f ₀ ) of the speech signal is estimated using an autocorrelation algorithm, and this estimation is used as a measured value of the pitch. In the present embodiment, the fundamental frequency f ₀ is estimated by applying an enhanced summary autocorrelation function (ESACF) to each frame. ESACF is a time domain pitch estimation method (because it is a full autocorrelation method), and it has the best balance between accuracy and light computation among several algorithms we have tried. ESACF is publicly known, and for specific contents, for example, the following documents can be referred to. Tolonen, T., and Karjalainen, M. A computationally efficient
multipitch analysis model. IEEE Trans. Speech Audio Process (2000). Mazzoni, D., and Dannenberg,
R. Melody matching directly from audio.In Proc.ISMIR '01, Indiana University
(2001).

In the present embodiment, the above method is selected without using a general method using a frequency spectrum. The simplest approach is to present the spectrogram in the frequency domain of the speech signal to the user, but it is not easy to interpret the spectrogram. Although it is possible to estimate the fundamental frequency f ₀ from the spectrum by analyzing the peak position, it is not realistic. Human speech is complex and includes many different frequencies and harmonics.

Autocorrelation algorithms, including ESACF, are performed by calculating the correlation of the signal to itself at a number of time lags. The overlapping window of W sample length is moved with respect to the signal S, and the algorithm calculates the correlation for the deviation (lag) 1 <τ <W / 2. Each lag τ represents a frequency of sample rate / τHz. The value in this lag represents the strength of that frequency in the signal. Therefore, the fundamental frequency f ₀ (that is, the pitch) can be estimated by the following equation.

For frames that do not contain audio signals, ESACF tends to generate large values indicating high f ₀ at small time lags. FIG. 7 illustrates this problem with a large peak at 3200 Hz in the unvoiced sample (bottom). In this embodiment, these peaks are filtered using the above VAD algorithm, the unvoiced sound is removed from the f ₀ estimation calculation, and the sudden increase that exceeds twice the frequency (one octave) over one frame is ignored. And filter silent but high energy utterances. Various types of pitch estimation methods are known, and the pitch estimation method that can be used in the present invention is not limited to ESACF. Known pitch extraction methods for obtaining the fundamental frequency and pitch period of voiced sound include time-domain techniques (such as autocorrelation) and frequency-domain techniques (methods of detecting fundamental frequency from the harmonic structure on the spectrum). These methods can also be used.

[B-4] Speed
To measure speed, the number of speech events is counted over time. These events are separated by energy peaks in the speech signal and represent consonants (subsounds), syllable nuclei (central vowels), or complex syllables.

In this embodiment, the speed is extracted in real time based on a modification of the Mermelstein algorithm. The Mermelstein algorithm itself is known, and for example, the following documents can be referred to.
Mermelstein, P. Automatic segmentation of speech into syllabic
units. Journal of the Acoustical Society of America (JASA) (1975).
Villing, R., Ward, T., and Timoney, J. Performance limits for
envelope based automatic syllable segmentation. In Proc. ISSC '06, IET (2006).

The present inventors adopted the Mermelstein algorithm after comparing the output results of ASR (automatic speech recognition) and speed calculation using other algorithms. First, in the text output of an ASR (automatic speech recognition) system, it is conceivable to count words, syllables, or single notes. Litman, DJ, Hirschberg, KB, and Swerts, M. Predicting
automatic speech recognition performance using prosodic cues.In Proc.NAACL
'00, ACM (2000). However, in addition to the language / dialogue dependency that can be inherent in ASR, the current ASR is a technology level that can accurately recognize utterances consisting of real-time and natural large vocabulary for smartphones. Is not yet.

On the other hand, in the DSP method, we considered directly searching for the minimum in the stream of intensity values generated by the loudness feature calculation and making the sound part a speech event. Resolution, which can affect the performance of the clarity features described below. Note that the segmentation method for speed calculation in the present invention is not limited to the one using the Mermelstein algorithm, and does not exclude the output result of ASR (automatic speech recognition) or the one using another algorithm. .

The Mermelstein algorithm is applied directly to the waveform and is in line with light computational requirements, and the downsampling of this application provides a sample for processing in convex hull generation with little reduction in resolution. The number can be reduced.

The Mermelstein algorithm calculates the convex hull of the loudness function of the sound signal and detects high energy intervals that correspond well to syllables. An example of this procedure is shown in FIGS. The upper graph shows the original waveform (amplitude), and the lower graph shows the loudness function. The convex hull is indicated by diagonal lines. The maximum difference between the loudness function and its convex hull is used to find the minimum in the loudness function. The maximum in the difference that exceeds the threshold is determined as the syllabic-unit segment boundary, where the signal is split. This algorithm is applied iteratively until no further significant minima is found.

The Mermelstein algorithm in this embodiment differs from the original algorithm in that it is executed online. In the first proposed algorithm, the total length of the signal is determined and stored in memory (runs offline). This is necessary to find the maximum of the loudness function to generate a convex hull. The signal begins with a minimum (silence) of the loudness function and ends, during which there is a maximum of the loudness function, where the maximum changes from a monotonically increasing function to a monotonically decreasing function. In this embodiment, in order to execute the Mermelstein algorithm online, convex hull generation and iterative minimum search are applied to the variable window of the sound signal that starts at the most recent segment boundary and ends at the most recent frame end. The most recent segment boundary is the minimum of the loudness function, so this windowed approach yields the same boundary as the original approach. In order to improve the performance of this online deformation technique, the waveform of the sound signal, the loudness function, the convex hull, and the distance between the loudness function and the convex hull are stored in the cache from the nearest segment boundary. The cache is updated when new sound frames are added, and the cache is removed when new segments are found. For example, when the first segment boundary is found at αs, the data before αs is deleted, and the window (variable window) is extended from αs until the next segment boundary is found. When a segment boundary is found at βs in the window, data up to βs is deleted to shorten the window. Repeat this process.

To evaluate this algorithm, we used a text-to-speech engine (Apple's Speech Synthesis Manager with "Daniel" (male, British English)) instead of generating speech with accurate speed and minimal error by humans. . To perform measurements with this algorithm, small corpus of speech samples was generated by speaking at different rates of 120-400 wpm (words / minute) at linear intervals. In the experiment, we used the first 10 paragraphs from chapter 1 of Alice in the magical country of Lewis Carroll, downloaded from Project Gutenberg (http://www.gutenberg.org/ebooks/11). An online variant of Mermelstein algorithm was applied to this data. The results are shown in FIG. The number of segments per second recognized by the algorithm shows a speed (words / minute) and a linear increase. This linear result indicates that the algorithm has successfully identified the speech rate.

[B-5] Enunciation Clarity
In this utterance evaluation system, the clarity of utterance is considered as the ability to pronounce clearly. This is intimately related to pronunciation and other pronunciation problems that can reduce intelligibility. Although the clarity of occurrence has been studied for many years, there are few useful prior literature on algorithms for estimating clarity. The concept of “clear speech” in phonetics should be adopted under certain circumstances, such as when talking to a non-native speaker, talking to a child, or changing to a computer interface. It is a clear and unnatural way of speaking.

Pisoni literature (Pisoni, D. Clear Speech.
In The Handbook of Speech Perception. Blackwell Publishing, 2005, ch. 9, 218-227), global features that are considered as elements of clear speech include sound intensity, speaking speed, pause, fundamental frequency, and spectral tilt. Time envelope modulation is mentioned. In this specification, the first four are mentioned.

Phonetic features have also been investigated and a consonant-to-vowel ratio (CVR) has been proposed. This is the ratio of the consonant energy to the vowel energy, and a high consonant energy has been proposed as an indication of clarity of speech. However, different results have been obtained using CVR (Hazan, V., and Markham, D. Acoustic-phonetic correlates of talker
intelligibility for adults and children.Journal of the Acoustical Society of
America (JASA) (2004)). This is considered due to the following basic defects. Although consonants and vowels can be reasonably distinguished on text as characters, there are various types of consonant sounds, and many consonants have acoustic features similar to vowel sounds. There are also reports that not all consonants are equally important in estimating clarity, but plosives and strong frictional sounds are relatively important, and nasal sounds are less important (Kennedy, E., Levitt, H., Neuman, AC, and Weiss, M.
Consonant-vowel intensity ratios for maximizing consonant recognition by
hearing-impaired listeners. Journal of the Acoustical Society of America (JASA)
(1998)).

Therefore, the present inventors propose the idea of comparing the inhibition sound and the resonance sound ratio in order to estimate the clarity. Therefore, in the present invention, an obstructive-to-sonorant ratio (OSR) is used as a measure of articulation clarity instead of a consonant-to-vowel ratio. In order to acquire the OSR, first, it is necessary to specify a single tone boundary in the audio signal. This would be possible using an ASR-like system that uses a Gaussian mixture model (GMM) trained with frame features such as mel frequency cepstrum coefficients (MFCCs).

However, in the present embodiment, the boundary generated by the above-described velocity estimation algorithm is reused as a means that reduces the amount of calculation to substitute for finding all single-tone boundaries. Although not all phone boundaries are found in this algorithm, all obtained boundaries identify the energy minimum in the signal loudness function. At the boundary located between the resonance and the inhibition sound, both sounds should have sufficient energy to form a significant minimum between these energy peaks. A low energy inhibitor will not have enough energy to produce this minimum. Here, by classifying the obtained segments as either resonance sound or inhibition sound, high energy inhibition sound can be counted. A large number of obstruction sounds means a high OSR, resulting in a clear way of speaking. In the present embodiment, the number of inhibition sound segments with respect to the total number of segments is used as a scale. FIG. 11 shows the syllable segment boundary in the loudness function, and seven segments are obtained. As shown in FIG. 12, the seven segments were separated into the inhibition sound 4 and the resonance sound 3. At this time, the clarity score = the number of inhibition sounds / the total number of segments = 4 / (4 + 3) = 0.57.

The distinction between resonant and inhibitory sounds is a binary classification problem. In order to solve this problem, the present embodiment trains a support vector machine (SVM) to classify these two categories of sounds. Support vector machines are well known to those skilled in the art, for example, see Cortes, C., and Vapnik, V. Support-vector networks. Machine Learning (1995). The support vector machine can quickly perform classification (although it takes a long time to learn), and has proved to be an effective means of maintaining light computational complexity. The model is learned off-line and the application itself performs only binary classification, resulting in a low computational complexity.

In order to generate learning data and test data, each sentence in a corpus (phone-tagged corpus) tagged with sound was divided using the speed estimation algorithm. Then, using sound annotation to find the sound in the segment, it was determined whether each segment was a resonance sound. A segment containing any resonance is identified as a resonance.

The feature vector that describes each segment consists of 14 elements: time length, periodicity, and 12 mel frequency cepstrum coefficients (MFCCs). 13 mel frequency cepstrum coefficients (MFCCs) were generated, but the first coefficient is excluded because it includes a dependency on absolute energy levels (which are relatively unchanged in TIMIT corpus but change in the actual acoustic environment) did. For periodicity, the ESACF output for pitch extraction can be used. The calculation of the mel frequency cepstrum coefficients (MFCCs) is well known to those skilled in the art and will not be described.

TIMIT was used for the speech corpus. Garofolo, J., Lamel, L., Fisher, W., et al. TIMIT acoustic-phonetic continuous
speech corpus. Linguistic Data Consortium (1993). The TIMIT corpus contains 6300 sentences spoken in 8 different US English dialects. The corpus was divided into learning data (4,620 sentences, 464 speakers) and test data (1,680 sentences, 168 speakers). Some sentences are the same, but different speakers.

Using the velocity estimation algorithm, all sentences were divided and the segments were labeled based on whether or not they contained resonance sounds. Vowels, semi-vowels, and glide were treated as resonances. A training set consisting of voiced sound segments 49,409 (70.2%) and unvoiced sound segments 21,003 (29.8%) and a test set consisting of voiced sound segments 18,109 (69.8%) and unvoiced sound segments 7,848 (30.2%) were obtained.

SVM ^light was used for SVM implementation. SVM kernel has RBF (radial basis
function). To find the optimal C and γ values for the RBF kernel, 2,000 random samples were taken from the training data and LIBSVM's [Chang, C., and Lin, C.
LIBSVM: a library for support vector machines.ACM Trans.Intelligent Systems
and Technology (2011).] grid parameter search tool [Hsu, C., Chang, C., Lin,
C., et al. A practical guide to support vector classification, 2003.]. By using the features of segment time length, periodicity, and twelve mel frequency cepstrum coefficients, segments could be classified with 93.27% accuracy and 94.85% recall. When only the time length and periodicity were used, the accuracy was 75.51% and the recall was 58.47%.

The corpus material S (3 paragraphs, separated into 34 phrases) with the title “clearspeechjph” was used as a corpus to verify the evaluation of actual speech by this algorithm. The definition of clear speech is not the same as that of the present invention, but was adopted as a preliminary verification of the accuracy of the OSR measure according to the present invention. The speaker speaks each sample in both a clear and understandable style. The results are shown in FIG. In the upper diagram, clear speech samples are distributed in the vicinity of 1, and in the lower diagram, the speech samples are distributed in the vicinity of 0. A simple speech sample is not necessarily a bad utterance, and thus is distributed over a certain extent.

[C] Evaluation of actual utterances Using the utterance evaluation system according to the present embodiment, four utterances of Barack Obama, Woody Allen, Meryl Streep, and Lisa Simpson were evaluated. The utterance time is about 1 minute. The utterances were played from the speakers to the iPhone's internal microphone to get closer to actual use. The results are shown in FIG. In actual use of the speech evaluation system, the user interface displays the features of the user's utterance and the utterance features of the model speaker selectively. In FIG. 14A, the evaluation results of four speakers are displayed. Shown together and compared. The regular pentagon indicates the average value of the characteristics of the four speakers. In the actual screen, as shown in the left diagram of FIG. 2, the standard deviation, that is, dynamism is shown in a band shape. In FIG. 14A, the standard deviation is not shown, and is shown separately in FIG. 14B.

The fact that all energy levels are almost equal is due to the recording being played back at the same volume level in the same environment. The similar pose rate is probably due to the fact that all speech was in the form of monologue speech. The pitch indicates the pitch of each speaker and the variation of the pitch. Obama speaks in a relatively low voice, whereas Allen speaks in a high voice for men. Streep speaks low for women, but Lisa speaks on a comical high pitch.

In speaking speed, Streep is a regular, powerful monologue, whereas Allen is a choppy, fast-paced neurotic style.

The clarity results show that Allen pronounces the word very carefully and clearly despite the fast pace. Obama's speech has the lowest clarity score, and Obama's low-energy consonants give a more rounded sound than Allen's sharp pronunciation. The dynamism values (pitch and speed) in FIG. 14 (b) show that Obama is monotonic and stable in rhythm / pace, whereas Allen's pitch is fluctuating and the pace is also fluctuating, Give a more dynamic feeling.

The present invention can be installed in a mobile device such as a smartphone as a speech correction application.

Claims (23)

An input unit for inputting a voice signal of a speaker's free utterance;
A feature extraction unit that extracts features used for evaluation from the input audio signal;
A feature evaluation unit that compares the feature extracted by the feature extraction unit with a reference feature stored in advance;
An output unit for outputting a comparison result;
With
Features used for the evaluation include at least clarity of utterance,
The intelligibility of the utterance is represented by an inhibition sound to resonance ratio in the input audio signal,
The feature extraction unit acquires the inhibition sound-to-resonance sound ratio using means for dividing the input audio signal into a plurality of segments and means for classifying the obtained segment into an inhibition sound and a resonance sound. ,
Utterance evaluation device. Features used for the evaluation further include loudness,
The feature extraction unit obtains a voice volume from energy of an input voice signal;
The utterance evaluation apparatus according to claim 1. The features used for the evaluation further include a pause rate,
The pause rate is the percentage of time that speech is not spoken in the length of the audio signal,
The feature extraction unit acquires the ratio by a voice section detection unit.
The utterance evaluation apparatus according to claim 1. The features used for the evaluation further include pitch,
The feature extraction unit obtains a pitch by a pitch extraction unit;
The utterance evaluation apparatus according to claim 1. The characteristics used for the evaluation further include the speed of beat,
The speed of the beat is represented by the number of utterance events within a predetermined time,
The feature extraction unit obtains the speed of speaking by counting the number of utterance events within a predetermined time.
The utterance evaluation apparatus according to claim 1.   The utterance evaluation apparatus according to claim 5, wherein the utterance event is a syllable. In the means for dividing the input audio signal into a plurality of segments, the segments are syllables,
In the feature extraction unit, a plurality of segments are acquired using a syllable boundary when acquiring the speed of the beat.
The utterance evaluation apparatus according to claim 6. The means for classifying the segment into an inhibition sound and a resonance sound uses supervised machine learning means.
The utterance evaluation apparatus according to claim 1. The supervised machine learning means uses a segment time length, periodicity, and Mel frequency cepstrum coefficient as a feature vector.
The speech evaluation apparatus according to claim 8.   The speech evaluation apparatus according to claim 1, wherein the inhibition sound to resonance sound ratio is a ratio of inhibition sound segment number to resonance sound segment number.   The speech evaluation apparatus according to claim 1, wherein the segment is a single sound or a phoneme, and the inhibition sound to resonance sound ratio is an inhibition sound energy to resonance sound energy ratio. Inputting a speech signal of a speaker's free speech;
A feature extraction step of extracting features used for evaluation from the input speech signal;
A feature evaluation step for comparing the feature extracted in the feature extraction step with a pre-stored reference feature;
Outputting a comparison result; and
With
Features used for the evaluation include at least clarity of utterance,
The intelligibility of the utterance is represented by an inhibition sound to resonance ratio in the input audio signal,
The feature extraction step includes a step of dividing the input audio signal into a plurality of segments, a step of classifying the obtained segment into an inhibition sound and a resonance sound, and a step of acquiring an inhibition sound to resonance sound ratio. Contains,
Utterance evaluation method. Features used for the evaluation further include loudness,
The feature extraction step includes obtaining a voice volume from energy of an input voice signal.
The speech evaluation method according to claim 12. The features used for the evaluation further include a pause rate,
The pause rate is the percentage of time that speech is not spoken in the length of the audio signal,
The feature extraction step includes obtaining the ratio by a voice section detection unit;
The speech evaluation method according to any one of claims 12 and 13. The features used for the evaluation further include pitch,
The feature extracting step includes obtaining a pitch by a pitch extracting means,
The speech evaluation method according to any one of claims 12 to 14. The characteristics used for the evaluation further include the speed of beat,
The speed of the beat is represented by the number of utterance events within a predetermined time,
The feature extraction step includes obtaining the speed of the beat by counting the number of utterance events within a predetermined time,
The speech evaluation method according to any one of claims 12 to 15.   The speech evaluation method according to claim 16, wherein the speech event is a syllable. In the step of dividing the input audio signal into a plurality of segments, the segments are syllables;
In the feature extraction step, a plurality of segments are acquired using a syllable boundary when acquiring the speed of the beat.
The speech evaluation method according to claim 17. The step of classifying the segment into an inhibitory sound and a resonant sound uses supervised machine learning means.
The speech evaluation method according to any one of claims 12 to 18. The supervised machine learning means uses a segment time length, periodicity, and Mel frequency cepstrum coefficient as a feature vector.
The utterance evaluation method according to claim 19.   The speech evaluation method according to any one of claims 12 to 20, wherein the inhibition sound-to-resonance sound ratio is a ratio of the number of inhibition sound segments to the number of resonance sound segments.   The speech evaluation method according to any one of claims 12 to 20, wherein the segment is a single sound or a phoneme, and the inhibition sound to resonance sound ratio is an inhibition sound energy to resonance sound energy ratio. A computer program for causing a computer to execute the method according to any one of claims 12 to 22.