CN115359782A - An evaluation method for reading aloud ancient poems based on the fusion of quality and prosodic features - Google Patents

Abstract

The invention provides an ancient poetry reading evaluation method based on quality and rhythm feature fusion, which comprises the steps of establishing an objective voice quality evaluation model based on an MOS (metal oxide semiconductor), extracting mel frequency spectrum features, extracting signal high-dimensional features by a mask _ res residual convolution network, and aggregating MOS scores of single ancient poetry reading by a UnMask output module; establishing a characteristic fusion-based rhythm evaluation model, extracting fundamental frequency, energy, zero crossing rate and other signal basic characteristics, calculating parameters of light and heavy tones, intonation and rhythm characteristic through a multi-characteristic analysis model, establishing a rhythm scoring function, and mapping actual rhythm score; and establishing a comprehensive measurement system based on polynomial fitting, and establishing a reference-free evaluation model based on the fusion of quality and prosodic features based on the target of an optimal solution and a minimized model.

Description

An evaluation method for reading aloud ancient poems based on the fusion of quality and prosodic features

technical field

The invention belongs to the technical field of speech signal processing, and in particular relates to an evaluation method for reading aloud ancient poems based on the fusion of quality and prosody features.

Background technique

Ancient poetry is the collective name of ancient poetry and words, and ancient poetry is divided into various types of poetry, such as Yuefu poems, metrical poems, etc., but these ancient poems also have some common characteristics, such as the arrangement of syllables. It has certain rules, pays attention to level and rhyme, so it will have a sense of cadence when reading, and it is suitable as a reading material. Through reading aloud, the content and emotion to be expressed in the poem can be better conveyed. When evaluating ancient poetry, voice quality and pronunciation rhythm are two important evaluation dimensions. Simply put, they are the two levels of "sound" and "rhyme". The former refers to the ability to distinguish clear pronunciation, which is the most basic level for evaluating the quality of speech reading aloud; the latter refers to the rhythm, severity, tone, intonation, etc. displayed during recitation.

Reading aloud ancient poems is an important part of learning content. However, the evaluation technology that is widely used at present is only limited to the correctness of specific phonemes, and it seems stretched to have multi-dimensional quality evaluation for reading aloud ancient poems. Or just scoring by comparing spoken speech with a reference speech has many limitations in terms of flexibility and reach. In summary, it is necessary to propose a quantified, objective, and non-reference-based evaluation method for reading aloud ancient poems through the analysis of speech signals for reading aloud, combined with acoustic feature parameters. There are certain rules, pay attention to the level and oblique rhyme, through the rhythm of reading, readers can express the content and emotion of the poem vividly.

Contents of the invention

In view of this, the present invention aims to propose a method for evaluating the reading aloud of ancient poems based on the fusion of quality and prosodic features, using acoustic features and perceptual features as important indicators to evaluate the Chinese classical poetry of prosody, signal-to-noise ratio, and clarity, by extracting The pitch frequency of the poem, we quantify the reference evaluation function to obtain a prosody score based on the degree of deviation. The predicted scores have a good correlation with human ratings, effectively reflecting the reading level of readers of classical Chinese poetry and the quality of the audio itself.

In order to achieve the above object, technical solution of the present invention is achieved in that way:

As shown in Figure 1, the present invention provides a kind of ancient poetry reading evaluation method based on quality and prosodic feature fusion, comprises the following steps:

(1) Establish an objective speech quality evaluation model based on MOS. Extract the mel spectral features, use the mask_res residual convolution network to extract high-dimensional features of the signal, and aggregate the MOS score of a single ancient poetry recitation in the UnMask output module.

(2) Establish a prosody evaluation model based on feature fusion, extract basic signal features such as fundamental frequency, energy, and zero-crossing rate, convert them into light and heavy sounds, intonation, and rhythm prosody features according to the multi-feature analysis method, and map them into actual prosody scores through the prosody scoring function .

(3) Establish a comprehensive measurement system based on polynomial fitting. For the two scoring models obtained by task division in steps 1 and 2, based on the goal of optimal solution and minimization model, construct a no-reference ancient model based on the fusion of quality and rhythm features. The poetry reading evaluation mapping function g() is:

S=g(w ₁ S _R ,w ₂ S _MOS )

S _R is prosodic feature fusion prosodic score, S _MOS is quality model score, w ₁ and w ₂ are the weights of the evaluation model, which are determined by the polynomial regression equation.

Further, step (1) includes:

(11) feature extraction, calculate Mel subframe from input signal, and divide overlapping section, and complement the length of different speech segments;

(12) Carry out quality analysis according to the prosodic features obtained in step (11), and perform feature dimensionality reduction with Mel subframes as input, and predict the speech sequence, specifically: use the residual convolutional layer network to extract high-dimensional features, in Downward convolution in BasicBlock achieves 3 feature dimension reductions. Then output through the fully connected layer, set the output feature dimension to 20, and flatten the output through the view.

(13) Perform UnMask output according to the high-dimensional features obtained in step (12). The speech time is based on the restored feature length and feature aggregation to estimate a single MOS value. Specifically: first, according to the original length recorded before, get the UnMask mask and compare it with The unmask value at the corresponding position of the eigenvector is multiplied to complete the zero-removal operation to obtain the actual speech segment length. Then, through the maximum pooling layer, for each effective feature vector, the maximum value of all feature numbers is taken to obtain the MOS score output of a single speech.

Further, step (2) includes:

(21) Prosodic feature extraction, divide the input into frames, use a rectangular window, take N as a sampling rate of 0.05 times, calculate the short-term average amplitude function and pitch curve of ancient poetry, and extract each peak value in the function curve, get The relative standard deviation of the peak value. Calculate the fundamental frequency, and estimate the cepstrum for each frame. Use mean filtering to smooth the fundamental frequency curve, and fine-tune the threshold parameter to mark the main peak.

(22) Multi-feature analysis. According to the prosodic feature obtained in step (21), the feature parameter is calculated. Calculate the standard deviation of each peak of the short-term average amplitude to reflect the stress change of the sound; calculate the relative standard deviation parameter of each adjacent peak time interval to reflect the rhythm characteristics of the voice; calculate the relative standard deviation parameter of each peak, Reflects how readers process intonation; calculates the relative standard deviation of the syllable length of each word in a poem to reflect the pause or lengthening of syllables; calculates the silence time to reflect whether the pause in reading is reasonable.

(23) Prosodic scoring model. According to the feature parameters obtained in step (22), use the scoring formula to map the actual prosody evaluation score:

θ _i ∈{σ,RSD _p ,RSD _t ,RSD,t _s }

是对应特征参数θ_i∈{σ,RSD_p,RSD_t,RSD,t_s}的量化值，λ是映射分数的放大系数。among them

is the quantized value of the corresponding feature parameter θ _i ∈ {σ,RSD _p ,RSD _t ,RSD,t _s }, and λ is the amplification factor of the mapping score.

Convert the characteristic parameters of the reading samples into percentile scores, and formulate reference values based on the experimental values of the best reading samples. Score the different features of the sample, and take the weighted average as the final score.

Compared with the prior art, a kind of ancient poetry reading assessment method based on the fusion of quality and prosodic features of the present invention has the following advantages:

The present invention combines the traditional prosody evaluation method with the neural network-based voice quality evaluation method, and proposes a no-reference evaluation of Chinese classical poetry, which uses acoustic features and perceptual features as important indicators to evaluate prosody, signal-to-noise ratio, and clarity, respectively. method, by extracting the pitch frequencies of the poems, we quantify the reference evaluation function to obtain a prosody score based on the degree of deviation. The predicted scores have a good correlation with human ratings, effectively reflecting the reading level of readers of classical Chinese poetry and the quality of the audio itself. On the one hand, the underlying spectral-temporal structure of target speech and noise is captured. On the other hand, this technology analyzes the acoustic feature parameters of speech and gives a weighted prosody score, which has certain reference value and application prospect. By objectifying the subjective evaluation criteria, we can reasonably quantify the overall quality of Chinese classical poetry audio from the perspective of audibility and aesthetics, and further reveal the objective psychological laws of the audience's evaluation of reading quality, which can provide some new ideas for theoretical research. A comprehensive scoring model that combines deep learning methods with prosody analysis theory, jointly fits human subjective perception from these two aspects, and obtains a feasible evaluation system.

Description of drawings

The drawings constituting a part of the present invention are used to provide a further understanding of the present invention, and the schematic embodiments and descriptions of the present invention are used to explain the present invention, and do not constitute an improper limitation of the present invention. In the attached picture:

Fig. 1 is the structural representation of ancient poetry reading evaluation model of the present invention;

Figure 2 is the result graph of the best evaluation model.

Detailed ways

It should be noted that, in the case of no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

In describing the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", " The orientations or positional relationships indicated by "vertical", "horizontal", "top", "bottom", "inner" and "outer" are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and Simplified descriptions, rather than indicating or implying that the device or element referred to must have a particular orientation, be constructed and operate in a particular orientation, and thus should not be construed as limiting the invention. In addition, the terms "first", "second", etc. are used for descriptive purposes only, and should not be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, a feature defined as "first", "second", etc. may expressly or implicitly include one or more of that feature. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

In the description of the present invention, it should be noted that unless otherwise specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection. Connected, or integrally connected; it can be mechanically connected or electrically connected; it can be directly connected or indirectly connected through an intermediary, and it can be the internal communication of two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention based on specific situations.

The present invention will be described in detail below with reference to the accompanying drawings and examples.

The present invention proposes a method for evaluating the reading of ancient poems based on the fusion of quality and prosody features, including the following steps:

1. Establish an objective speech quality evaluation model based on MOS.

Use perceptual features to evaluate perceivable speech quality such as signal-to-noise ratio. It is divided into 3 parts: feature extraction, quality analysis, and output from UnMask. The feature extraction module calculates the Mel subframe from the input signal, divides it into overlapping segments, and completes the length of different speech segments; the quality analysis module uses the Mel subframe as input to perform feature dimensionality reduction and predict the speech sequence. In the UnMask output module, the voice time is estimated based on the restored feature length and feature aggregation to estimate a single MOS value.

1. Feature extraction module.

The signal-to-noise ratio characteristics of speech are expressed by mel spectrograms. Under different additive noises, there are obvious differences in the spectrograms, and these features are learned through neural networks. Set the number of mel bands to 24 units, divide subframes with a length of 7 units, and do a frame shift of 2 units. The overlapping rate of adjacent subframes is 71.4%. Using the short-term stationary characteristics of speech, the feature changes are smooth. The time lengths of different voices are generally different, and zeros are added during batch processing to ensure that the feature vector lengths of each voice are the same.

The mel spectrogram can well express the signal-to-noise ratio characteristics of speech. There are obvious differences in the spectrogram under different additive noises, and the neural network can well represent these features. The extraction steps are as follows:

(1) At first the voice signal is pre-emphasized, framed and windowed;

(2) Perform Discrete Fourier Transform (DFT) on each frame of speech signal, and then square the DFT coefficient of each frame to obtain short-term spectrum energy, and finally arrange these frames in time order to obtain energy spectrogram;

(3) Perform Mel filtering by weighting and summing the Mel filter bank and the energy spectrogram. The formula of the Mel filter bank is shown in formula (1), and the weighted sum formula is shown in formula (2), where f(m) is the center frequency of the Mel filter bank, m represents the order of the Mel filter, k represents the number of the midpoint of the FFT, and |X(k)| ² represents the spectrum energy;

(4) Filter the spectral features after DFT to obtain m filter bank energies, and finally perform log operation to obtain Mel spectrograms, where |X(k)| ² represents spectral energy.

2. Quality analysis module.

According to the spectrogram features extracted by the feature extraction module, the residual convolutional layer network is used to extract high-dimensional features. The specific method is: downward convolution in BasicBlock to achieve three times of feature dimensionality reduction. Then output through the fully connected layer, set the output feature dimension to 20, and flatten the output through the view.

3. UnMask output module.

According to the high-dimensional features obtained by the quality analysis module, firstly perform the de-zero operation on the UnMask layer to obtain the actual length of each speech segment. The unmask value is multiplied, and a minimum value will be obtained at the zero padding. Afterwards, through the maximum pooling layer, for each effective feature vector, the maximum value of all feature numbers is taken as the MOS score output of a single speech.

2. Establish a prosody evaluation model based on feature fusion.

The present invention designs an evaluation method for the quality of recitation of ancient poems from the aspects of lightness and stress, intonation and rhythm by using acoustic features. They are important feature dimensions that determine prosodic perception. Pitch control reflects the change of pitch in the reading audio; rhythm characteristics reflect the change of rhythm and the length of pauses; the change of lightness and severity reflects the reader's grasp of light reading and heavy reading.

1. Prosodic feature extraction.

Calculate the short-term average amplitude function and smooth pitch-frequency curve of ancient poetry, and extract each peak in the function curve, and get the relative value of the peak as the standard deviation, as a value that reflects the changing characteristics of the two. Then, the relative standard deviation of the peak-to-peak interval and the ratio of the sum of silent durations to the total duration were calculated, reflecting the rhythmic characteristics of the samples. Enter framing. Set the length of each frame to be 0.02 times the sampling rate. Calculate the fundamental frequency, and estimate the cepstrum for each frame. Use mean filtering to smooth the fundamental frequency curve and fine-tune the threshold parameter to mark the main peak. Use a rectangular window and take N as 0.05 times the sampling rate to calculate the short-term average amplitude curve.

2. Multi-feature analysis.

According to the obtained prosodic features, feature parameters are calculated. Calculate the standard deviation of each peak of the short-term average amplitude to get σ to reflect the stress change of the sound; calculate the relative standard deviation parameter of each adjacent peak time interval to get RSD _t to reflect the rhythm characteristics of the speech; calculate the The Relative Standard Deviation parameter yields RSD _p , which reflects how readers process intonation; calculates the relative standard deviation of the syllable lengths of each word in a poem to yield RSD, which reflects pauses or prolongations of syllables; the total silence duration divided by the total The audio duration is obtained by t _s silence time to reflect whether the pause in reading is reasonable.

3. Prosodic scoring model.

According to the obtained feature parameters, the actual prosodic evaluation scores are mapped. The characteristic parameters of the reading samples are converted into percentile scores, and the module parameters are formulated according to the experimental values of the best reading samples. Score the different features of the sample, and take the weighted average as the final score. The scoring formula is to select the function with the highest reference value and decrease to both sides, and use formula 3 to convert a single parameter into a score:

θ _i ∈{σ,RSD _p ,RSD _t ,RSD,t _s }

是对应特征参数θ_i∈{σ,RSD_p,RSD_t,RSD,t_s}的量化值，λ是映射分数的放大系数。in

is the quantized value of the corresponding feature parameter θ _i ∈ {σ,RSD _p ,RSD _t ,RSD,t _s }, and λ is the amplification factor of the mapping score.

3. Establish a comprehensive measurement system based on polynomial fitting

For the two scoring models obtained in steps 1 and 2, consider the S overall scoring model, and based on the goal of the smallest model, in the case of the optimal solution, the maximum model reliability and effectiveness are obtained:

S=g(w ₁ S _R ,w ₂ S _MOS )

S _R is prosodic feature fusion prosodic score, S _MOS is quality model score, w ₁ and w ₂ are the weights of the evaluation model, which are determined by the polynomial regression equation.

The invention combines the traditional prosody evaluation method with the neural network-based speech quality evaluation method. We complete the exploration of 6 network structures for evaluating signal-to-noise properties, 4 prosodic scoring functions, and an optimal polynomial regression. We use root mean square error (RMSE) and Pearson correlation coefficient R to measure the performance of the module, with lower RMSE and higher R indicating better performance. As shown in Table 1 and Table 2, Table 1 is the performance index of the signal-to-noise network, and Table 2 is the performance index of the comprehensive evaluation model. RMSE and R are defined as follows:

为所有X_i的平均值，

为所有x_i的平均值，分别反映主观MOS与客观MOS的偏差和相关性。Where _xi is the average target MOS of a single input, and _Xi is the corresponding subjective MOS.

is the average value of all _Xi ,

is the average value of all _xi , respectively reflecting the deviation and correlation between subjective MOS and objective MOS.

In Table 2, α and β represent the polynomial order of the quality model and the prosody model respectively, and the maximum value of α and β is used as the order of the fitting polynomial. As the order increases, the prediction performance is improved at the cost of increasing the complexity of the system, but it also brings the risk of premature overfitting. As the order of the regression polynomial increases, the performance of the model gets better and better. The parameters and polynomial order are designed by determining the optimal order of the two input parts, which balances system complexity and accurate prediction while avoiding overfitting. Based on the combination of resnet-18, max_UnMask, linear scoring function and second-order polynomial, this best-performing model achieves overall good results of R = 0.90 and RMSE = 0.39 in terms of indicators, which are similar to existing single-quality evaluation methods. And it also incorporates the prosody evaluation system. Figure 2 is a visualized two-dimensional distribution diagram of the model, which reflects the function model of the overall score of poetry quality with the MOS value and prosody score. In Figure 2, the x-axis represents the predicted objective prosody score MOS_pred; the y-axis represents the predicted prosody score Prosody_pred; the z-axis represents the overall score Score of ancient poetry reading under feature fusion. Each axis is mapped to a range of values from 0-5. On the one hand, it captures the fundamental spectral-temporal structure of target speech and noise. On the other hand, the acoustic feature parameters of speech are analyzed, and the weighted prosodic score is given, which has certain reference value and application prospect. Through objective subjective evaluation criteria, the overall quality of Chinese classical poetry audio is reasonably quantified from the perspective of audibility and aesthetics, and further reveals the objective psychological laws of the audience's evaluation of reading quality. This is unique and innovative, and can be used for theoretical research on Chinese ancient poetry reading. Provide some new ideas.

Table I

Table II

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims (4)

1. An ancient poetry reading evaluation method based on quality and rhythm feature fusion is characterized in that: the method comprises the following steps:

(1) Establishing an objective voice quality evaluation model based on an MOS, extracting mel frequency spectrum characteristics, extracting signal high-dimensional characteristics by using a mask _ res residual convolution network, and aggregating MOS scores of single classical poetry recitation at a UnMask output module;

(2) Establishing a prosody evaluation model based on feature fusion, extracting basic features of signals, converting the basic features into light and heavy tones, intonation and rhythm prosody features according to a multi-feature analysis method, and mapping the features into actual prosody scores through a prosody scoring function;

(3) And aiming at the two evaluation models, establishing a reference-free ancient poetry reading evaluation model based on the fusion of quality and rhythm characteristics based on polynomial fitting, and grading by using the model.

2. The ancient poetry reading evaluation method based on quality and prosodic feature fusion as claimed in claim 1, characterized in that: the step (1) specifically comprises the following steps:

(11) The characteristic extraction is carried out on the basis of the characteristic,

calculating Mel sub-frames from the input signal, dividing into overlapping segments, filling the lengths of different speech segments, and learning via neural network to obtain speech spectrogram characteristics;

(12) The mass analysis is carried out on the mixture of the components,

performing quality analysis according to the spectrogram characteristics obtained in the step (11), performing characteristic dimension reduction by taking Mel subframes as input, and predicting the voice sequence, wherein the method specifically comprises the following steps: extracting high-dimensional characteristics by using a residual convolutional layer network, carrying out downward convolution in a BasicBlock to realize 3-time characteristic dimension reduction, then outputting through a full connection layer, setting the output characteristic dimension to be 20, and realizing output flattening through view;

(13) From UnMask output

And (4) carrying out UnMask output according to the high-dimensional characteristics obtained in the step (12), carrying out characteristic aggregation on the voice time according to the restoration characteristic length, and estimating a single MOS value, wherein the specific steps are as follows: firstly, according to the recorded original length, obtaining a UnMask mask and multiplying the UnMask mask with an UnMask value at a position corresponding to a feature vector, completing zero-removing operation to obtain the actual voice segment length, and then, taking the maximum value of all feature numbers for each effective feature vector through a maximum pooling layer to obtain the MOS score output of a single voice.

3. The ancient poetry reading evaluation method based on quality and prosodic feature fusion as claimed in claim 1, characterized in that: the step (2) specifically comprises the following steps:

(21) The prosodic features are extracted, and the prosodic features are extracted,

framing the input, using a rectangular window, taking a sampling rate with N being 0.05 times, calculating a short-time average amplitude function and a pitch curve of the ancient poetry, extracting each peak value in the function curve, obtaining a relative standard deviation of the peak values, calculating a fundamental frequency, estimating a cepstrum of each frame, smoothing the fundamental frequency curve by using mean filtering, and finely adjusting a threshold parameter to mark a main peak;

(22) The multi-feature analysis is carried out on the data,

calculating characteristic parameters according to the prosodic features obtained in the step (21), and calculating the standard deviation sigma of each peak value of the short-term average amplitude to reflect the accent change of the voice; calculating a relative standard deviation parameter RSD for each adjacent peak time interval _t To reflect the characteristics of the voice rhythm; calculating a relative standard deviation parameter RSD of each peak _p Reflecting the processing mode of the reader for the intonation; calculating relative standard deviation RSD of syllable length of each word in the poetry so as to reflect pause or extension of syllables; calculating the mute time t _s To reflect inWhether the pause of reading is reasonable or not;

(23) A prosodic scoring model that is used to generate a prosodic score,

according to the characteristic parameters obtained in the step (22), mapping the actual rhythm evaluation score by using a scoring formula:

θ _i ∈{σ，RSD _p ，RSD _t ，RSD，t _s }

wherein

Is a corresponding characteristic parameter theta _i ∈{σ，RSD _p ，RSD _t ，RSD，t _s The quantized value of { lambda is the magnification factor of the mapping fraction;

and converting the characteristic parameters of the reading sample into percentage scores, formulating reference values according to the experimental values of the best reading sample, scoring different characteristics of the sample, and taking the weighted average of the characteristics as a final score.

4. The ancient poetry reading evaluation method based on the fusion of quality and prosodic features as claimed in claim 1, characterized in that: in the step (3), regarding the two scoring models obtained in the steps 1 and 2, considering the S overall scoring model, and based on the target of the minimum model, under the condition of an optimal solution, constructing a no-reference ancient poetry reading evaluation mapping function g () based on the fusion of quality and prosodic features as follows:

S＝g(w ₁ S _R ，w ₂ S _MOS )

S _R is prosodic feature fusion prosody scoring, S _MOS Is a quality model score, w ₁ 、w ₂ The weight of the evaluation model is determined by a polynomial regression equation.

Images