CN114170997A - Pronunciation skill detection method, device, storage medium and electronic device - Google Patents

Abstract

A pronunciation skill detection method, device, storage medium and electronic device. The method includes acquiring the text to be detected, converting the text to be detected into a corresponding phoneme sequence; acquiring the audio to be detected obtained by the speaker speaking the text to be detected, and extracting the acoustic features of the audio to be detected; The trained pronunciation skill detection model performs pronunciation skill detection processing to obtain a first detection result and a second detection result; wherein, the first detection result is used to represent whether the pronunciation skill needs to be used to speak the text to be detected, and the second detection result is used to represent Whether the speaker uses pronunciation skills to speak the text to be detected. The present application can improve the accuracy of pronunciation skill detection.

Description

Pronunciation skill detection method, device, storage medium and electronic device

technical field

The present application relates to the technical field of speech recognition, and in particular to a pronunciation skill detection method, device, storage medium and electronic device.

Background technique

At present, for any language, whether it is Chinese or English, spoken language is the top priority for mastering these languages. For example, for English learners, oral pronunciation is often the focus and weak point in their English learning process. Whether the pronunciation skills such as continuous reading, loss of blasting and voicing are used accurately reflects the oral ability of English learners . In the related art, artificial hearing detection is usually used to detect the pronunciation ability of the speaker. However, factors such as artificial subjective judgment and auditory fatigue will affect the accuracy of the pronunciation skills detection result.

SUMMARY OF THE INVENTION

The present application provides a pronunciation skill detection method, device, storage medium and electronic device, which can improve the accuracy of pronunciation skill detection.

The pronunciation skills detection methods provided in this application include:

Obtain the text to be detected, and convert the text to be detected into a corresponding phoneme sequence;

Obtain the audio to be detected obtained by the speaker speaking the text to be detected, and extract the acoustic features of the audio to be detected;

Input the phoneme sequence and the acoustic feature into the trained pronunciation skill detection model for pronunciation skill detection processing, and obtain the first detection result and the second detection result;

The first detection result is used to represent whether the text to be detected needs to be spoken by using pronunciation skills, and the second detection result is used to represent whether the speaker uses pronunciation skills to speak the text to be detected.

The pronunciation skill detection device provided by this application includes:

The first acquisition module is used to acquire the text to be detected, and convert the text to be detected into a corresponding phoneme sequence;

The second acquisition module is used to acquire the audio to be detected obtained by the speaker speaking the text to be detected, and to extract the acoustic features of the audio to be detected;

The detection module is used to input the phoneme sequence and acoustic features into the trained pronunciation skill detection model for pronunciation skill detection processing, and obtain the first detection result and the second detection result;

The first detection result is used to represent whether the text to be detected needs to be spoken by using pronunciation skills, and the second detection result is used to represent whether the speaker uses pronunciation skills to speak the text to be detected.

The storage medium provided by the present application has a computer program stored thereon, and when the computer program is loaded by the processor, the steps in the pronunciation skill detection method provided by the present application are executed.

The electronic device provided by the present application includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the steps in the pronunciation skill detection method provided by the present application by loading the computer program.

In this application, the text to be detected and the audio to be detected obtained by the speaker speaking the text to be detected are obtained, and the audio to be detected and the text to be detected are used to detect the speaker's pronunciation, wherein the text to be detected is converted into corresponding The phoneme sequence and the acoustic features of the audio to be detected are extracted, and then the phoneme sequence and acoustic features are input into the trained pronunciation skill detection model for pronunciation skill detection processing, and the first detection result and the second detection result are obtained. The first detection result is used to represent whether the text to be detected needs to be spoken by using pronunciation skills, and the second detection result is used to represent whether the speaker uses pronunciation skills to speak the text to be detected. Compared with the related art, the present application adopts the artificial intelligence-based pronunciation skill detection method to replace the traditional artificial hearing detection, which can avoid artificial subjective judgment and auditory fatigue, thereby improving the accuracy of pronunciation skill detection.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic diagram of a scene of a pronunciation skill detection system provided by an embodiment of the present application.

FIG. 2 is a schematic flowchart of a pronunciation skill detection method provided by an embodiment of the present application.

FIG. 3 is an example diagram of extracting acoustic features in an embodiment of the present application.

FIG. 4 is a structural block diagram of a pronunciation skill detection model provided by an embodiment of the present application.

Figure 5 is a structural block diagram of the phoneme feature extraction network in the pronunciation skill detection model.

FIG. 6 is a structural block diagram of the phoneme feature extraction sub-module inside the phoneme feature module in the phoneme feature extraction network.

FIG. 7 is another structural block diagram of the phoneme feature extraction network in the pronunciation skill detection model.

Figure 8 is a structural block diagram of the feature encoding module inside the acoustic feature enhancement network in the pronunciation skill detection model.

Figure 9 is a block diagram of the refined structure of the acoustic feature enhancement module inside the acoustic feature enhancement network in the pronunciation skill detection model.

Figure 10 is a block diagram of the feature fusion network in the pronunciation skill detection model.

Fig. 11 is a structural block diagram of the first pronunciation skill detection network in the pronunciation skill detection model.

FIG. 12 is a structural block diagram of a branch detection network inside the second pronunciation skill detection network in the pronunciation skill detection model.

FIG. 13 is a structural block diagram of a pronunciation skill detection apparatus provided by an embodiment of the present application.

FIG. 14 is a structural block diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

It should be noted that the principles of the present application are exemplified by being implemented in a suitable computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be construed as limiting other specific embodiments of the present application not detailed herein. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

The relational terms such as first and second involved in the following embodiments of the present application are only used to distinguish one object or operation from another object or operation, and are not used to limit the existence of actual relationship between these objects or operations. sequence relationship. In the description of the embodiments of the present application, "plurality" means two or more, unless otherwise expressly and specifically defined.

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which, deep learning (DL) is a new research direction in machine learning, which is introduced into machine learning to make it closer to the original goal , namely artificial intelligence. At present, deep learning is mainly used in computer vision, natural language processing and other fields.

Deep learning is to learn the inherent laws and representation levels of sample data, and the information obtained during these learning processes is of great help to the interpretation of data such as text, images and sounds. Using deep learning technology and corresponding training data sets, network models that implement different functions can be trained. For example, a deep learning network for gender classification can be trained based on a training data set, and a deep learning network for gender classification can be trained based on another training data set. Image-optimized deep learning networks, etc.

In order to improve the efficiency of pronunciation skill detection, the present application introduces deep learning into pronunciation skill detection, and accordingly provides a pronunciation skill detection method, a pronunciation skill detection device, a storage medium and an electronic device. Wherein, the pronunciation skill detection method can be performed by an electronic device.

Please refer to FIG. 1 , the present application also provides a pronunciation skill detection system. As shown in FIG. 1 , the pronunciation skill detection system includes an electronic device 100. For example, the electronic device can obtain the text to be detected for pronunciation skill detection, and The text to be detected is converted into a corresponding phoneme sequence, and when the electronic device is also equipped with a microphone, audio collection can be performed during the speaker speaking the text to be detected, so as to obtain the audio to be detected obtained by the speaker speaking the text to be detected, And extract the acoustic features of the audio to be detected, and then further input the obtained phoneme sequence and acoustic features into the trained pronunciation skill detection model for pronunciation skill detection processing, and obtain the first detection result and the second detection result. The detection result is used to represent whether the text to be detected needs to be spoken using pronunciation skills, and the second detection result is used to represent whether the speaker uses pronunciation skills to speak the text to be detected.

The electronic device 100 may be any device equipped with a processor and capable of processing, such as a mobile electronic device with a processor such as a smart phone, a tablet computer, a PDA, a notebook computer, or a desktop computer, a TV, a server, and the like with a processor. stationary electronic equipment.

In addition, as shown in FIG. 1 , the pronunciation skill detection system may also include a storage device 200 for storing data, including but not limited to the original data, intermediate data and result data obtained during the pronunciation skill detection process, for example, an electronic device 100 can store the acquired text to be detected, the audio to be detected, the phoneme sequence converted from the text to be detected, the acoustic features extracted from the audio to be detected, and the first detection result and the second detection result output by the pronunciation skill detection model. into the storage device 200.

It should be noted that the schematic diagram of the scene of the pronunciation skill detection system shown in FIG. 1 is only an example. The pronunciation skill detection system and the scene described in the embodiments of the present application are for the purpose of illustrating the technical solutions of the embodiments of the present application more clearly, not It is known to those of ordinary skill in the art that, with the evolution of the pronunciation skill detection system and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application have the same effect on similar technical problems. Be applicable.

Please refer to FIG. 2 , which is a schematic flowchart of a pronunciation skill detection method provided by an embodiment of the present application. As shown in Figure 2, the process of the pronunciation skill detection method provided by the embodiment of the present application may be as follows:

In S310, the text to be detected is acquired, and the text to be detected is converted into a corresponding phoneme sequence.

Wherein, the text to be detected refers to the text used for pronunciation skill detection, and the pronunciation skill detection includes detecting whether to use pronunciation skills to speak the text to be detected, and detecting whether the speaker uses pronunciation skills to speak the text to be detected.

It should be noted that different languages such as Chinese and English have their own pronunciation skills. Taking English as an example, there are pronunciation skills such as continuous reading, loss of blasting and turbidization.

Among them, continuous reading refers to the natural spelling of the first and last phonemes of two adjacent words together without a pause in between.

Loss of blasting means that when two blasting sounds (such as p, d, t, k, g) are adjacent, the former blasting sound is only mouth-shaped according to its pronunciation position to form an obstacle, but it does not blast out, and waits for a little pause. Then pronounce the following consonants. The previous plosive is called a lost plosive, as in goo(d)bye.

Voiced is when the sound before a voiceless consonant is /s/, the voiceless consonant has its corresponding voiced consonant, and there is a vowel after the voiceless consonant, then the voiceless consonant is read as its corresponding voiced consonant consonant. Taking speak as an example, the unvoiced consonant /p/ is preceded by the sound /s/, the corresponding voiced consonant of /p/ is /b/, and /p/ is followed by the vowel /i:/. At this time, the original /spi is used. :k/ is read as /sbi:k/.

As above, the pronunciation skill detection method provided by the present application can be used for pronunciation skill detection of pronunciation skills in any language. Correspondingly, according to the actual pronunciation skill detection requirements, the text to be detected can be text in any language. After acquiring the text to be detected, the electronic device further converts the acquired text to be detected into a corresponding phoneme sequence. For example, the electronic device can convert the acquired text to be detected into a corresponding phoneme sequence according to the pronunciation dictionary.

In an optional embodiment, the text to be detected is converted into a corresponding phoneme sequence, including:

Remove the silent text units in the text to be detected to obtain a new text to be detected;

Convert each text unit in the new text to be detected into a corresponding phoneme unit to obtain a phoneme sequence.

It can be understood that, for any text, when the text is spoken, not all text units in the text need to be pronounced, for example, punctuation marks in the text do not need to be pronounced.

Therefore, in order to eliminate the interference of silent text units and improve the accuracy of pronunciation skill detection, when the electronic device converts the text to be detected into the corresponding phoneme sequence, it first removes the silent text units (such as punctuation marks) in the text to be detected. , emoticons, etc.) to obtain a new text to be detected, and then convert each text unit in the new text to be detected into a corresponding phoneme sequence according to the pronunciation dictionary.

For example, when it is necessary to detect English pronunciation skills, the electronic device obtains the text to be detected in English as "Please turn on the light.", and the text unit-punctuation mark "." in the text to be detected is silent text unit, after removing the silent text unit ".", the new text to be detected is "Please turn on the light", and further according to the pronunciation dictionary, each text unit in the new text to be detected is converted into a corresponding phoneme unit, get phoneme sequence

In addition, in order to more clearly characterize the phoneme sequence, the electronic device may also add a start mark and an end mark before and after the phoneme sequence, the start mark indicates the start of the phoneme sequence, and the end mark indicates the end of the phoneme sequence. The specific configuration of the start flag and the end flag is not specifically limited here, and can be configured by those skilled in the art according to actual needs.

添加起始标志和结束标志后变更为

For example, you can configure the start flag as "<bos>" and configure the end flag as "<eos>", and the electronic device can be configured in the above phoneme sequence

After adding the start flag and end flag, change it to

In S320, the to-be-detected audio obtained by the speaker speaking the to-be-detected text is acquired, and the acoustic features of the to-be-detected audio are extracted.

In this embodiment, in addition to converting the text to be detected into a corresponding phoneme sequence, the electronic device also acquires the audio to be detected obtained by the speaker speaking the text to be detected. There is no specific limitation on the data format of the audio to be detected here, which can be configured by those skilled in the art according to actual detection needs.

Among them, the speaker can be a real person or a virtual person.

For example, when the speaker is a real person, the electronic device can use the configured audio capture device (either a built-in audio capture device or an external audio capture device) to detect the text to be detected spoken by the real person. The audio is collected by voice, and the collected audio is used as the audio to be detected; in addition, the electronic device can also obtain the audio to be detected from other electronic devices that have been collected by other electronic devices, and a real person speaks the text to be detected. Correspondingly, using the audio to be detected obtained at this time, the electronic device can use the pronunciation skill detection method provided by the present application to detect the pronunciation ability of the real person.

For another example, when the speaker is a virtual person, such as speech synthesis software based on artificial intelligence, the electronic device can directly input the text to be detected into the speech synthesis software, the speech synthesis software performs speech synthesis, and outputs the synthesized audio, and This audio is used as the audio to be detected. Correspondingly, using the audio to be detected obtained at this time, the electronic device can be used to detect the speech synthesis capability of the speech synthesis software in the pronunciation skill detection method provided in this application.

As above, after acquiring the audio to be detected obtained by the speaker speaking the text to be detected, the electronic device further extracts the acoustic features of the audio to be detected. Among them, the acoustic feature refers to the physical quantity that represents the acoustic characteristics of speech, and is also the general term for the acoustic performance of various elements of the sound, such as the energy concentration area, formant frequency, formant intensity and bandwidth that represent the timbre, as well as the duration, base, etc. that represent the prosody characteristics of speech. frequency, average speech power, etc.

In an optional embodiment, in order to further improve the accuracy of pronunciation skill detection, the acoustic features of the audio to be detected are extracted, including:

Extract the Filterbank feature, fundamental frequency feature and energy feature of the audio to be detected;

Acoustic features are obtained by fusing Filterbank features, fundamental frequency features and energy features.

In this embodiment, Filterbank features, fundamental frequency features and energy features are used as acoustic features related to pronunciation skills. Correspondingly, when extracting acoustic features for pronunciation skills detection, the electronic device extracts the Filterbank features, base features of the audio to be detected. frequency characteristics and energy characteristics. The dimension of the extracted Filterbank feature is not specifically limited here, and can be configured by those skilled in the art according to actual needs. For example, in this embodiment, the electronic device can extract the 40-dimensional Filterbank feature of the audio to be detected.

As above, after extracting the Filterbank feature, fundamental frequency feature and energy feature of the audio to be detected, the electronic device further fuses the Filterbank feature, fundamental frequency feature and energy feature according to the configured fusion strategy to obtain a fusion feature, and uses the fusion feature as a Acoustic features for pronunciation skill detection. The configuration of the fusion policy is not specifically limited here, and can be configured by those skilled in the art according to actual needs.

For example, referring to FIG. 3 , the fusion strategy configured in this embodiment is to splicing Filterbank features, fundamental frequency features, and energy features according to the time dimension, so as to obtain acoustic features for pronunciation skill detection.

In S330, the phoneme sequence and the acoustic feature are input into the trained pronunciation skill detection model for pronunciation skill detection processing, and a first detection result and a second detection result are obtained.

It should be noted that the present application pre-trains corresponding pronunciation skill detection models for different languages. For example, for Chinese, a pronunciation skill detection model for detecting pronunciation skills in Chinese is pre-trained. Pronunciation skill detection model for pronunciation skill detection in English. The structure and training method of the pronunciation skill detection model are not specifically limited here, and can be selected by those skilled in the art according to actual needs.

The pronunciation skill detection model is configured to take the acoustic features of the audio to be detected derived from the speaker speaking the text to be detected, and the phoneme sequence derived from the text to be detected as input, and the corresponding output is used to indicate whether the pronunciation skill is required to be used. The detection result of the text to be detected is obtained, and the detection result used to characterize whether the speaker uses pronunciation skills to speak the text to be detected.

Correspondingly, in this embodiment, after obtaining the above phoneme sequence and acoustic feature, if the phoneme sequence and acoustic feature input that the electronic device is about to obtain matches the language of the text to be detected, the pronunciation skill has been trained in the pronunciation skill detection model. The detection process is performed to obtain the first detection result and the second detection result output by the pronunciation skill detection model. The first detection result is used to represent whether the text to be detected needs to be spoken by using pronunciation skills, and the second detection result is used to represent whether the speaker uses pronunciation skills to speak the text to be detected.

Among them, taking the above-mentioned text to be detected "Please turn on the light" as an example, according to expert knowledge, the phonemes "n" and "ɑ" are linked phonemes, and turn and on need to be linked together. The phoneme sequence and acoustic features, the first detection result output by the pronunciation skill detection model will indicate the need to use the pronunciation skill "continuous reading" to output the text to be detected, depending on whether the speaker uses the pronunciation skill "continuous reading" to speak the text to be detected, and the pronunciation The skill detection model will output a second detection result for .

In addition, the phoneme sequence can be input as the original phoneme sequence, or after digital encoding, the phoneme sequence can be converted into a phoneme sequence in digital form, that is, the corresponding phoneme is represented by numbers in the converted phoneme sequence. Correspondingly, if the phoneme sequence is input in the form of numbers, when training the pronunciation skill detection model, it needs to be trained with the phoneme sequence samples in the form of numbers.

In an optional embodiment, the pronunciation skill detection model includes a phoneme feature extraction network, an acoustic feature enhancement network, a feature fusion network, a first pronunciation skill detection network, and a second pronunciation skill detection network. The trained pronunciation skill detection model performs pronunciation skill detection processing to obtain the first detection result and the second detection result, including:

Input the phoneme sequence into the phoneme feature extraction network for feature extraction processing to obtain a phoneme feature matrix;

Input the phoneme feature matrix into the first pronunciation skill detection network to perform pronunciation skill detection processing, and obtain the first detection result;

If the first detection result representation needs to use pronunciation skills to speak the text to be detected, the acoustic features are input into the acoustic feature enhancement network for feature enhancement processing, and an enhanced acoustic feature matrix is obtained;

Input the enhanced acoustic feature matrix and the phoneme feature matrix into the feature fusion network for feature fusion processing to obtain a fusion feature matrix;

The fusion feature matrix is input into the second pronunciation skill detection network for pronunciation skill detection processing, and the second detection result is obtained.

Please refer to FIG. 4 , the pronunciation skill detection model provided by this embodiment consists of five major parts, namely, a phoneme feature extraction network, an acoustic feature enhancement network, a feature fusion network, a first pronunciation skill detection network, and a second pronunciation skill detection network.

Wherein, the phoneme feature extraction network is configured to perform feature extraction on the input phoneme sequence to obtain a phoneme feature matrix reflecting the mutual relationship between the phonemes in the phoneme sequence.

The acoustic feature enhancement network is configured to perform feature enhancement processing on the input acoustic features, and enhance the features more relevant to pronunciation skills to obtain an enhanced acoustic feature matrix.

The feature fusion network is configured to perform feature fusion on the input phoneme feature matrix and the enhanced acoustic feature matrix, and obtain the fusion feature matrix without the interaction information between the phoneme and the acoustic feature.

The first pronunciation skill detection network is configured to perform pronunciation skill detection processing on the input phoneme feature matrix, and output a first detection result used to represent whether a pronunciation skill needs to be used to speak the text to be detected.

The second pronunciation skill detection network is configured to perform pronunciation skill detection processing on the input fusion feature matrix, and output a second detection result used to represent whether the speaker uses pronunciation skills to speak the text to be detected.

Correspondingly, in this embodiment, when the phoneme sequence and acoustic features are input into the trained pronunciation skill detection model for pronunciation skill detection processing, the electronic device can perform feature extraction processing on the phoneme sequence phoneme feature extraction network to obtain a phoneme feature matrix, Then, the phoneme feature matrix is input into the first pronunciation skill detection network for pronunciation skill detection processing, and the first detection result is obtained.

At the same time, the electronic device inputs the acoustic feature into the acoustic feature enhancement network for feature enhancement processing to obtain an enhanced acoustic feature matrix, and inputs the enhanced acoustic feature feature and the phoneme feature matrix into the feature fusion network for feature fusion processing to obtain a fusion feature matrix, and the fusion The obtained fusion feature matrix is input into the second pronunciation skill detection network for pronunciation skill detection processing, and the second detection result is obtained.

In addition, the electronic device may also determine whether to output the second detection result according to the first detection result. Wherein, after obtaining the first detection result and the second detection result obtained by the pronunciation skill detection model by the pronunciation skill detection model, the electronic device determines whether to use the pronunciation skill to speak the text to be detected according to the first detection result. Use pronunciation skills to speak the text to be detected, then the electronic device outputs the first detection result and the second detection result at the same time, the first detection result at this time indicates that the text to be detected needs to be spoken by pronunciation skills, and the second detection result at this time The detection result represents whether the speaker uses pronunciation skills to speak the text to be detected; if it is determined that the text to be detected does not need to be spoken by pronunciation skills, the electronic device can discard the second detection result and output only the first detection result.

In other embodiments, if the first detection result indicates that the text to be detected needs to be spoken using pronunciation skills, the electronic device inputs the acoustic features into an acoustic feature enhancement network for feature enhancement processing to obtain an enhanced acoustic feature matrix. After that, the electronic device further inputs the enhanced acoustic feature feature and the phoneme feature matrix into the feature fusion network to perform feature fusion processing to obtain a fusion feature matrix. Finally, the electronic device inputs the fusion feature matrix obtained by fusion into the second pronunciation skill detection network for pronunciation skill detection processing, and obtains a second detection result.

In addition, if the first detection result indicates that pronunciation skills do not need to be used to say the text to be detected, there is no need to perform further pronunciation skills detection. At this time, the electronic device no longer uses acoustic features to detect pronunciation skills, and can only output the first Test results.

In an optional embodiment, the phoneme feature extraction network includes a phoneme embedding module and a phoneme feature extraction module, and the phoneme sequence is input into the phoneme feature extraction network for feature extraction processing to obtain a phoneme feature matrix, including:

Input the phoneme sequence into the phoneme embedding module for embedding processing, and obtain the phoneme vector matrix;

The phoneme vector matrix is input into the phoneme feature extraction module for feature extraction processing to obtain a phoneme feature matrix.

Referring to FIG. 5 , in this embodiment, the phoneme feature extraction network consists of two parts, namely a phoneme embedding module and a phoneme feature extraction module, wherein the phoneme embedding module is configured to perform embedding processing on the input phoneme sequence, and the vector to obtain a phoneme vector matrix; the phoneme feature extraction module is configured to perform feature extraction on the input phoneme vector matrix to obtain a phoneme feature matrix reflecting the relationship between the phonemes in the phoneme sequence.

Correspondingly, in this embodiment, when inputting the phoneme sequence into the phoneme feature extraction network for feature extraction processing, the electronic device first inputs the phoneme sequence into the phoneme embedding module for embedding processing to obtain a phoneme vector matrix, and then inputs the phoneme vector matrix into the phoneme. The feature extraction module performs feature extraction processing to obtain a phoneme feature matrix.

The phoneme feature extraction module includes at least one phoneme feature extraction sub-module, and the phoneme vector matrix is input into the phoneme feature extraction module for feature extraction processing to obtain a phoneme feature matrix, including:

When the number of phoneme feature extraction submodules is one, the phoneme vector matrix is input into the phoneme feature extraction submodule for feature extraction processing to obtain a phoneme feature matrix; or,

When the number of phoneme feature extraction submodules is N, the phoneme vector matrix is input into the N phoneme feature extraction submodules to perform feature extraction processing in sequence to obtain a phoneme feature matrix, where N is an integer greater than 1. The value of N is not specifically limited here, and can be configured by those skilled in the art according to actual needs. For example, N can be configured as 2.

In this embodiment, the audio feature extraction module may be composed of one phoneme feature extraction sub-module, or may be composed of N phoneme feature extraction sub-modules connected in sequence. Wherein, when the audio feature extraction module consists of N phoneme feature extraction sub-modules, each audio extraction sub-module performs the same feature extraction process. The following takes the feature extraction process of one audio feature extraction sub-module as an example for description.

Please refer to FIG. 6 , the audio feature extraction sub-module consists of 3 sub-layers, namely the first matrix conversion layer, the first multi-head attention layer and the first matrix fusion layer, wherein,

The first matrix conversion layer is configured to perform matrix conversion processing on the input matrix, and convert the input matrix into a query matrix, a key matrix and a value matrix respectively;

The first multi-head attention layer is configured to perform attention enhancement processing on the input query matrix, key matrix and value matrix to obtain an attention enhancement matrix;

The first matrix fusion layer is configured to perform matrix fusion processing on the input matrix of the first matrix conversion layer and the output matrix of the first multi-head attention layer to obtain a fusion matrix.

Correspondingly, when the number of phoneme feature extraction submodules is one, the electronic device can extract the phoneme feature matrix as follows:

Input the phoneme vector matrix into the first matrix conversion layer for matrix conversion processing to obtain a query matrix, a key matrix and a value matrix, which are respectively denoted as a first query matrix, a first key matrix and a first value matrix;

Input the first query matrix, the first key matrix and the first value matrix into the first multi-head attention layer for attention enhancement processing to obtain an attention enhancement matrix, which is denoted as the first attention enhancement matrix;

The first attention enhancement matrix and the phoneme vector matrix are input into the first matrix fusion layer for matrix fusion processing to obtain a fusion matrix, which is recorded as a phoneme feature matrix.

It can be understood that when there are N phoneme feature extraction submodules, it is only necessary to input the phoneme vector matrix into the first phoneme feature extraction submodule in the N successively connected phoneme feature extraction submodules. The sub-modules sequentially perform feature extraction processing in the above manner, and use the fusion feature output by the N-th phoneme feature sub-module as a phoneme feature matrix.

There is no specific limitation on how the first matrix fusion layer performs matrix fusion in this embodiment, and can be configured by those skilled in the art according to actual needs.

For example, the first matrix fusion layer may include two sub-layers, namely an addition layer and a layer normalization layer. When performing matrix fusion, the addition layer first adds the two input matrices to obtain a sum value matrix, and then Then, the sum value matrix is normalized by the layer normalization layer, and the fusion matrix is obtained.

In an optional embodiment, please refer to FIG. 7 , the phoneme feature extraction network also includes a first position encoding module and a second matrix fusion layer, and the phoneme vector matrix is input into the phoneme feature extraction module for feature extraction processing to obtain a phoneme feature matrix. Before, also included:

Input the phoneme vector matrix into the first position encoding module for position encoding processing to obtain the first position encoding matrix;

Inputting the first position encoding matrix and the phoneme vector matrix into the second matrix fusion layer to perform matrix fusion processing to obtain a phoneme position fusion matrix;

Input the phoneme vector matrix into the phoneme feature extraction module for feature extraction processing to obtain a phoneme feature matrix, including:

The phoneme position fusion matrix is input into the phoneme feature extraction module for feature extraction processing, and the phoneme feature matrix is obtained.

In this embodiment, in order to further improve the accuracy of pronunciation skill detection, in this embodiment, the original phoneme vector matrix is not input into the phoneme feature extraction module for feature extraction, but after position encoding, the position information will be carried. The phoneme vector matrix is input to the phoneme feature extraction module for feature extraction.

Wherein, the electronic device first inputs the phoneme vector matrix into the first position coding module for position coding processing to obtain a position coding matrix, which is denoted as the first position coding matrix, and the first position coding matrix represents the position of each matrix unit in the phoneme vector matrix The information can be relative position information or absolute position information.

After obtaining the first position encoding matrix as above, the electronic device inputs the first position encoding matrix and the phoneme vector matrix into the second matrix fusion layer for matrix fusion processing to obtain a fusion matrix, which is recorded as a phoneme position fusion matrix. After that, the electronic device further inputs the phoneme position fusion matrix carrying the position information into the phoneme feature extraction module to perform feature extraction processing to obtain a phoneme feature matrix. Wherein, for how the phoneme feature extraction module performs the feature extraction process, please refer to the relevant description of the above embodiment for details, and details are not repeated here.

In addition, it should also be noted that this embodiment does not specifically limit the matrix fusion manner of the second matrix fusion layer, which can be configured by those skilled in the art according to actual needs. For example, the second matrix fusion layer is configured to perform addition processing on two input matrices, and output the sum value matrix obtained by the addition as a fusion matrix.

In an optional embodiment, the acoustic feature enhancement network includes a feature encoding module and at least one acoustic feature enhancement module, and the acoustic features are input into the acoustic feature enhancement network for feature enhancement processing to obtain an enhanced acoustic feature matrix, including:

Input the acoustic features into the feature encoding module for feature encoding processing to obtain an acoustic feature matrix;

When the number of acoustic feature enhancement modules is one, input the acoustic feature matrix into the acoustic feature enhancement module for feature enhancement processing to obtain an enhanced acoustic feature matrix; or,

When the number of acoustic feature enhancement modules is M, the acoustic feature matrix is input into the M acoustic feature enhancement modules to perform feature enhancement processing in turn to obtain an enhanced acoustic feature matrix, where M is an integer greater than 1. The value of M is not specifically limited here, and can be configured by those skilled in the art according to actual needs. For example, M can be configured as 4.

It should be noted that the Filterbank features, fundamental frequency features and energy features obtained in the above embodiments are all presented in the form of feature maps. Correspondingly, the acoustic features obtained by the fusion of Filterbank features, fundamental frequency features and energy features are also represented by feature maps. presented in the form of.

In order to effectively perform feature enhancement processing on acoustic features, in this embodiment, the acoustic feature enhancement network consists of one feature encoding module and at least one acoustic feature enhancement module, wherein the feature encoding module is configured to encode acoustic features Process, compress the feature dimension, and obtain the corresponding acoustic feature matrix; the acoustic feature enhancement module is configured to perform feature enhancement processing on the input acoustic feature matrix, enhance the features related to pronunciation skills, and obtain the enhanced acoustic feature matrix.

Correspondingly, when the acoustic features are input into the acoustic feature enhancement network for feature enhancement processing, the electronic device first inputs the acoustic features into the feature encoding module for feature encoding processing to obtain an acoustic feature matrix.

Please refer to Figure 8, the feature encoding module consists of 4 sub-layers, including a first convolution layer, a first pooling layer, a second convolution layer and a second pooling layer, wherein,

The first convolution layer is configured to perform convolution processing on the input feature map to obtain a corresponding convolution result;

The first pooling layer is configured to perform pooling processing on the convolution result output by the first convolution layer to obtain a corresponding pooling result;

The second convolution layer is configured to perform convolution processing on the pooling result output by the first pooling layer to obtain a corresponding convolution result;

The second pooling layer is configured to perform pooling processing on the convolution result output by the second convolution layer to obtain a feature matrix corresponding to the feature map.

Correspondingly, the electronic device can input the acoustic owner into the feature encoding module for feature encoding processing in the following manner:

Input the acoustic features into the first convolution layer for convolution processing, and obtain the convolution result, which is recorded as the first convolution result;

The first convolution result is input into the first pooling layer for pooling processing, and the pooling result is obtained, which is recorded as the first pooling result;

The first pooling result is input into the second convolution layer for convolution processing, and the convolution result is obtained, which is recorded as the second convolution result;

The second convolution result is input into the second pooling layer for pooling processing, and the acoustic feature matrix is obtained.

It should be noted that in this embodiment, the convolution kernel size, stride size, and padding size of the first convolutional layer and the second convolutional layer, the pooling kernel size of the first pooling layer and the second pooling layer, The step size is not specifically limited, and can be configured by those skilled in the art according to actual needs.

For example, in this embodiment, the size of the convolution kernel of the first convolutional layer is [3,3], the stride is [1,1], the padding size is [1,1], and the second convolutional layer is configured The size of the convolution kernel is [3,3], the stride is [1,1], the padding size is [1,1], the pooling type of the first pooling layer is max pooling, and the pooling kernel size is [2,2], the step size is [1,1], the pooling type of the second pooling layer is configured as max pooling, the pooling kernel size is [2,2], and the step size is [1,1].

Further, when the number of acoustic feature enhancement modules is one, the electronic device directly inputs the acoustic feature matrix into the acoustic feature enhancement module for feature enhancement processing to obtain an enhanced acoustic feature matrix.

The following takes the feature enhancement processing process of one acoustic feature enhancement module as an example for description.

Please refer to Figure 9, the acoustic feature enhancement module consists of 6 sub-layers, namely the second matrix conversion layer, the second multi-head attention layer, the third matrix fusion layer, the third convolution layer, the deconvolution layer and the fourth layer. Matrix fusion layer. in,

The second matrix conversion layer is configured to perform matrix conversion processing on the input matrix, and convert the input matrix into a query matrix, a key matrix and a value matrix respectively;

The second multi-head attention layer is configured to perform attention enhancement processing on the input query matrix, key matrix and value matrix to obtain an attention enhancement matrix;

The third matrix fusion layer is configured to perform matrix fusion processing on the input matrix of the second matrix conversion layer and the output matrix of the second multi-head attention layer to obtain a fusion matrix;

The third convolution layer is configured to perform convolution processing on the input fusion matrix to obtain a convolution result;

The deconvolution layer is configured to perform deconvolution processing on the input convolution result to obtain the deconvolution result in the form of a matrix;

The fourth matrix fusion layer is configured to perform matrix fusion processing on the fusion matrix output by the third matrix fusion layer and the deconvolution result output by the deconvolution layer to obtain a fusion matrix.

Correspondingly, when the number of acoustic feature enhancement modules is one, the electronic device can enhance the obtained enhanced acoustic feature matrix as follows:

Input the acoustic feature matrix into the second matrix conversion layer for matrix conversion processing to obtain a query matrix, a key matrix and a value matrix, which are respectively denoted as the second query matrix, the second key matrix and the second value matrix;

Input the second query matrix, the second key matrix and the second value matrix into the second multi-head attention layer for attention enhancement processing to obtain the attention enhancement matrix, which is denoted as the second attention enhancement matrix;

The second attention enhancement matrix and the acoustic feature matrix are input into the third matrix fusion layer for matrix fusion processing, and the fusion matrix is obtained, which is recorded as the acoustic fusion matrix;

Input the acoustic fusion matrix into the third convolution layer for convolution processing, and obtain the third convolution result;

The third convolution result is input into the deconvolution layer for deconvolution processing, and the deconvolution result in matrix form is obtained;

The acoustic fusion matrix and the deconvolution result are input into the fourth matrix fusion layer for matrix fusion processing, and the fusion matrix is obtained, which is denoted as the enhanced acoustic feature matrix.

There is no specific limitation on how the third matrix fusion layer and the fourth matrix fusion layer perform matrix fusion in this embodiment, and can be configured by those skilled in the art according to actual needs.

For example, the third matrix fusion layer and the fourth matrix fusion layer have the same structure, and both include two sub-layers, which are the addition layer and the layer normalization layer. Add to get the sum value matrix, and then perform layer normalization processing on the sum value matrix through the layer normalization layer to obtain the fusion matrix.

It should be noted that when there are M acoustic feature enhancement modules, each acoustic feature enhancement module performs the same feature enhancement processing, and only needs to input the acoustic feature matrix into the first one of the M sequentially connected acoustic feature enhancement modules. In the acoustic feature enhancement module, the M acoustic feature enhancement modules perform feature enhancement processing in sequence in the above manner, and the fusion feature output by the Mth acoustic feature enhancement module is used as the enhanced acoustic feature matrix.

In addition, this embodiment does not impose specific restrictions on the configuration of the convolution kernel size, stride size, and padding size in the third convolution layer and the deconvolution layer, which can be selected by those skilled in the art according to actual needs.

It can be understood that this implementation can more effectively enhance the acoustic features in the form of feature maps by enhancing the convolution processing and deconvolution processing in the process of enhancing the acoustic features, and finally extract the features that are more related to pronunciation skills. feature.

In an optional embodiment, the acoustic feature enhancement network further includes a second position encoding module and a fifth matrix fusion layer, the acoustic feature matrix is input into the acoustic feature enhancement module for feature enhancement processing, and before the enhanced acoustic feature matrix is obtained, it also includes: :

Input the acoustic feature matrix into the second position encoding module for position encoding processing to obtain the second position encoding matrix;

The second position encoding matrix and the acoustic feature matrix are input into the fifth matrix fusion layer for matrix fusion processing to obtain an acoustic position fusion matrix;

Input the acoustic feature matrix into the acoustic feature enhancement module for feature enhancement processing, and obtain the enhanced acoustic feature matrix, including:

The acoustic position fusion matrix is input into the acoustic feature enhancement module for feature enhancement processing, and the enhanced acoustic feature matrix is obtained.

In this embodiment, in order to further improve the accuracy of pronunciation skill detection, in this embodiment, the original acoustic feature matrix is not input into the acoustic feature enhancement module for feature enhancement, but the position information will be carried after position encoding. The acoustic feature matrix is input to the acoustic feature enhancement module for feature enhancement.

The electronic device first inputs the acoustic feature matrix into the second position encoding module for position encoding processing to obtain a position encoding matrix, denoted as the second position encoding matrix, and the second position encoding matrix represents the position of each matrix unit in the acoustic feature matrix The information can be relative position information or absolute position information.

After obtaining the second position encoding matrix as above, the electronic device inputs the second position encoding matrix and the acoustic feature matrix into the fifth matrix fusion layer for matrix fusion processing to obtain a fusion matrix, which is recorded as an acoustic position fusion matrix. After that, the electronic device further inputs the acoustic position fusion matrix carrying the position information into the acoustic feature enhancement module for feature enhancement processing to obtain an enhanced acoustic feature matrix. For details on how the acoustic feature enhancement module performs feature enhancement processing, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

In addition, it should also be noted that this embodiment does not specifically limit the matrix fusion manner of the fifth matrix fusion layer, which can be configured by those skilled in the art according to actual needs. For example, the fifth matrix fusion layer is configured to perform addition processing on the two input matrices, and output the sum value matrix obtained by the addition as a fusion matrix.

In an optional embodiment, please refer to FIG. 10 , the feature fusion network includes a third matrix transformation layer, a fourth matrix transformation layer, a third multi-head attention layer, a sixth matrix fusion layer, a feedforward network layer, and a third matrix transformation layer. Seven matrix fusion layers, where,

The third matrix conversion layer is configured to perform matrix conversion processing on the input matrix to obtain a key matrix and a value matrix;

The fourth matrix conversion layer is configured to perform matrix conversion processing on the input matrix to obtain a query matrix;

The third multi-head attention layer is configured to perform attention enhancement processing on the key matrix and the value matrix output by the third matrix conversion layer and the query matrix output by the fourth matrix conversion layer to obtain an attention enhancement matrix;

The sixth matrix fusion layer is configured to perform matrix fusion processing on the attention enhancement matrix output by the third multi-head attention layer to obtain a fusion matrix;

The feedforward network layer is configured to perform feedforward calculation processing on the fusion matrix output by the sixth matrix fusion layer to obtain a feedforward matrix;

The seventh matrix fusion layer is configured to perform matrix fusion processing on the feedforward matrix output by the feedforward network layer and the fusion matrix output by the sixth matrix fusion layer to obtain a fusion feature matrix.

Correspondingly, the electronic device can input the enhanced acoustic feature matrix and the phoneme feature matrix into the feature fusion network for feature fusion processing in the following manner:

Input the enhanced acoustic feature matrix into the third matrix conversion layer for matrix conversion processing to obtain a key matrix and a value matrix, which are respectively denoted as the third key matrix and the third value matrix;

Input the phoneme feature matrix into the fourth matrix conversion layer for matrix conversion processing to obtain a query matrix, which is denoted as the third query matrix;

Input the third query matrix, the third key matrix and the third value matrix into the third multi-head attention layer for attention enhancement processing, and obtain the attention enhancement matrix, which is recorded as the third attention enhancement matrix;

Input the third attention enhancement matrix and the third query matrix into the sixth matrix fusion layer for matrix fusion processing to obtain a fusion matrix, which is recorded as an acoustic phoneme fusion matrix;

Input the acoustic phoneme fusion matrix into the feedforward network layer for feedforward calculation processing to obtain a feedforward matrix;

The feedforward matrix and the acoustic phoneme fusion matrix are input into the seventh matrix fusion layer for matrix fusion processing to obtain a fusion matrix, which is denoted as fusion feature matrix.

There is no specific limitation in this embodiment on how the sixth matrix fusion layer and the seventh matrix fusion layer perform matrix fusion, and can be configured by those skilled in the art according to actual needs.

For example, the sixth matrix fusion layer and the seventh matrix fusion layer have the same structure, and both include two sublayers, namely the addition layer and the layer normalization layer. Add to get the sum value matrix, and then perform layer normalization processing on the sum value matrix through the layer normalization layer to obtain the fusion matrix.

In an optional embodiment, please refer to FIG. 11 , the first pronunciation skill detection network includes the first fully connected layer and the first classification function layer, and the phoneme feature matrix is input into the first pronunciation skill detection network to carry out pronunciation skill detection processing, Get the first test result, including:

Input the phoneme feature matrix into the first full connection layer for full connection processing to obtain the first full connection result;

The first full connection result is input into the first classification function layer for classification processing to obtain the first detection result.

It should be noted that, since the present embodiment is to perform pronunciation skill detection on multiple types of pronunciation skills, correspondingly, the first classification function layer may adopt any multiple classification function.

Taking the Softmax function as an example, the dimension of the output vector of the Softmax function matches the number of pronunciation skills expected to be detected. For example, taking the English language as an example, the expected detection pronunciation skills include continuous reading, loss of blasting and turbidity, then the output of the Softmax function The vector includes four-dimensional elements, of which 1 element is used to indicate whether the pronunciation skill "continuous reading" is required to speak the text to be detected, and 1 element is used to indicate whether the pronunciation skill "lost blasting" needs to be used to speak the to-be-detected text Text, 1 element is used to represent whether the text to be detected needs to be spoken by the pronunciation skill "voicing", and one element is used to represent the text to be detected without using the pronunciation skill.

Correspondingly, the first full connection result is input into the Softmax function to obtain a 4-dimensional output vector of the Softmax function, the 4-dimensional output vector is used as the first detection result, and according to the first detection result, it can be determined whether to use pronunciation skills To output the text to be detected, and when pronunciation skills are required to speak the text to be detected, what kind of pronunciation skills need to be used to speak the text to be detected.

In an optional embodiment, the second pronunciation skill detection network includes L branch detection networks, each branch detection network corresponds to a different pronunciation skill, L is an integer greater than 1, and the fusion feature matrix is input into the second pronunciation skill detection. The network performs pronunciation skill detection processing to obtain a second detection result, including:

The fusion feature matrix is input into each branch detection network for pronunciation skill detection, and the branch pronunciation skill detection result of each branch detection network is obtained. The branch pronunciation skill detection result of each branch detection network indicates whether the speaker adopts the corresponding branch detection network. The pronunciation skills speak the text to be detected;

The second detection result is obtained according to the branch pronunciation skill detection result of each branch detection network.

Among them, each branch detection network corresponds to a pronunciation skill, and is configured to detect whether the speaker uses its corresponding pronunciation skill to speak the text to be detected. Correspondingly, how many branch detection networks there are will get as many branch pronunciation skills detection results. ,

For example, taking the English language as an example, when the desired pronunciation skills include continuous reading, loss of blasting and voicing, the value of L is 3, that is, the second pronunciation skill detection network will include 3 branch detection networks, of which 1 branch The detection network corresponds to the pronunciation skill "Continuous Reading", the 1 branch detection network corresponds to the pronunciation skill "Lost blasting", and the 1 branch detection network corresponds to the pronunciation skill "turbidization". Correspondingly, the three branch detection networks will each output one branch pronunciation skill detection result, and a total of three branch pronunciation skill detection results are obtained and combined into the second detection result. At this time, the second detection result represents whether the speaker uses pronunciation skills to speak the text to be detected, and when the speaker uses pronunciation skills to speak the text to be detected, what pronunciation skills are used to speak the text to be detected.

It should be noted that the structure of each branch detection network is the same. The following will take a branch detection network as an example for description. Please refer to FIG. 12. The branch detection network includes a second fully connected layer and a second classification function layer, and the feature matrix Input each branch detection network for pronunciation skill detection, and obtain the branch pronunciation skill detection results of each branch detection network, including:

Input the fusion feature matrix into the second fully connected layer for full connection processing to obtain the second fully connected result;

The second full connection result is input into the second classification function layer for classification processing, and the branch pronunciation skill detection result is obtained.

It should be noted that the second classification function layer can adopt any binary classification function.

Taking the sigmoid function as an example, the output value of the sigmoid function is located in [0, 1]. After model training, the output of the sigmoid function can represent the probability that the speaker uses the pronunciation skills corresponding to the branch detection network to speak the text to be detected. For example, when the output value of the sigmoid function reaches a preset threshold (experience values can be obtained by those skilled in the art according to actual needs), it can be determined that the speaker uses the pronunciation skills corresponding to the branch detection network where he is located to speak the text to be detected.

Correspondingly, the second full connection result is input into the sigmoid function to obtain the output value of the sigmoid function, and the output value is used as the branch pronunciation skill detection result. According to the detection result of the branch pronunciation skill, it can be determined whether the speaker speaks the text to be detected by using the pronunciation skill corresponding to the branch detection network where the speaker is located.

In an optional embodiment, before acquiring the text to be detected and converting the text to be detected into a corresponding phoneme sequence, the method further includes:

Obtain multiple types of first sample texts that are known to be spoken by different pronunciation skills, and convert each type of first sample text into a corresponding positive sample phoneme sequence;

Obtaining a sample user uses different pronunciation skills to speak the first sample audio of each type of first sample text, and extracts positive sample acoustic features of the first sample audio of each type of first sample text;

Obtaining a second sample text that is known not to be spoken by pronunciation skills, and converting the second sample text into a corresponding negative sample phoneme sequence;

Obtain the second sample audio of the second sample text spoken by the sample user, and extract the negative sample acoustic features of the second sample audio;

Model training is performed according to each type of positive sample phoneme sequence, each type of positive sample acoustic feature, negative sample phoneme sequence and negative sample acoustic feature, and a pronunciation skill detection model is obtained.

In this embodiment, acoustic feature samples and phoneme sequence samples are not constructed artificially, but a data-driven approach is used to allow the model to learn different pronunciation skills from a large amount of data. The following description takes a specific language as an example.

For this language, the electronic device obtains, respectively, multiple types of first sample texts that are known to need to be spoken with different pronunciation skills. There is no specific limitation on the quantity of the acquired first sample texts for each type of pronunciation skill, which can be configured by those skilled in the art according to actual needs.

For the first sample text obtained for each type of pronunciation skill (hereinafter referred to as the first sample text of each type), the electronic device converts each type of first sample text into corresponding phoneme sequences, which are marked as positive Sample phoneme sequence. Wherein, how to convert the first sample text into a phoneme sequence can be implemented with reference to the method of converting the text to be detected into a phoneme sequence in the above embodiment, which will not be repeated here.

The electronic device also obtains the audio of the sample user speaking each type of first sample text using different pronunciation skills, which is recorded as the first sample audio, and extracts the acoustic features of the first sample audio of each type of first sample text. , denoted as positive sample acoustic features. Among them, the sample user can be a real person with pronunciation skills, or a virtual person with pronunciation skills. Correspondingly, for how to obtain the first sample audio of each type of first sample text, and how to extract the positive The sample acoustic features can be implemented with reference to the manners of acquiring the audio to be detected and extracting the acoustic features of the audio to be detected in the above embodiments, which will not be repeated here.

In addition, the electronic device also acquires the text known to be spoken without using pronunciation skills, denoted as the second sample text, and converts the second sample text into a corresponding phoneme sequence, denoted as a negative sample phoneme sequence. Wherein, how to convert the second sample text into a phoneme sequence can be implemented with reference to the method of converting the text to be detected into a phoneme sequence in the above embodiment, which will not be repeated here.

The electronic device further acquires the audio of the sample user speaking the second sample text, denoted as the second sample audio, and extracts the acoustic feature of the second sample audio, denoted as the negative sample acoustic feature. Wherein, how to obtain the second sample audio of the second sample text and how to extract the negative sample acoustic features can be implemented with reference to the methods of obtaining the audio to be detected and extracting the acoustic features of the audio to be detected in the above embodiment, which is not repeated here. Repeat.

It should be noted that this embodiment does not specifically limit the number of sample users, which can be configured by those skilled in the art according to actual needs. For example, in this embodiment, 500 sample users are used to obtain the above positive sample phoneme sequence, positive sample Acoustic features, negative sample phoneme sequences, and negative sample acoustic features.

After acquiring the positive sample phoneme sequence, positive sample acoustic feature, negative sample phoneme sequence and negative sample acoustic feature as above, the electronic device will determine according to each type of positive sample phoneme sequence, each type of positive sample acoustic feature, negative sample phoneme sequence and The negative sample acoustic features are used for model training until the preset stopping conditions are met, and the pronunciation skill detection model is obtained. The preset stop condition may be configured as the number of iterations of the model reaches the preset number of times during the training process, or the model converges.

Please refer to FIG. 13 , in order to better implement the pronunciation skill detection method provided by the present application, the present application further provides a pronunciation skill detection device 400, as shown in FIG. 13 , the pronunciation skill detection device 400 includes:

The first obtaining module 410 is used to obtain the text to be detected, and convert the text to be detected into a corresponding phoneme sequence;

The second acquisition module 420 is configured to acquire the audio to be detected obtained by the speaker speaking the text to be detected, and to extract the acoustic features of the audio to be detected;

The detection module 430 is used for inputting the phoneme sequence and the acoustic feature into the trained pronunciation skill detection model for pronunciation skill detection processing to obtain the first detection result and the second detection result;

The first detection result is used to represent whether the text to be detected needs to be spoken by pronunciation skills, and the second detection result is used to represent whether the speaker uses pronunciation skills to speak the text to be detected.

In an optional embodiment, the pronunciation skill detection model includes a phoneme feature extraction network, an acoustic feature enhancement network, a feature fusion network, a first pronunciation skill detection network, and a second pronunciation skill detection network, and the detection module 430 is used for:

Input the phoneme sequence into the phoneme feature extraction network for feature extraction processing to obtain a phoneme feature matrix;

Input the phoneme feature matrix into the first pronunciation skill detection network for pronunciation skill detection processing, and obtain the first detection result;

If the first detection result representation needs to use pronunciation skills to speak the text to be detected, the acoustic features are input into the acoustic feature enhancement network for feature enhancement processing, and an enhanced acoustic feature matrix is obtained;

Input the enhanced acoustic feature matrix and the phoneme feature matrix into the feature fusion network for feature fusion processing to obtain a fusion feature matrix;

The fusion feature matrix is input into the second pronunciation skill detection network for pronunciation skill detection processing, and the second detection result is obtained.

In an optional embodiment, the phoneme feature extraction network includes a phoneme embedding module and a phoneme feature extraction module, and the detection module 430 is used for:

Input the phoneme sequence into the phoneme embedding module for embedding processing, and obtain the phoneme vector matrix;

The phoneme vector matrix is input into the phoneme feature extraction module for feature extraction processing to obtain a phoneme feature matrix.

In an optional embodiment, the phoneme feature extraction module includes at least one phoneme feature extraction sub-module, and the detection module 430 is used for:

When the number of phoneme feature extraction submodules is one, the phoneme vector matrix is input into the phoneme feature extraction submodule for feature extraction processing to obtain a phoneme feature matrix; or,

When the number of phoneme feature extraction submodules is N, the phoneme vector matrix is input into the N phoneme feature extraction submodules to perform feature extraction processing in sequence to obtain a phoneme feature matrix, where N is an integer greater than 1.

In an optional embodiment, the phoneme feature extraction sub-module includes a first matrix conversion layer, a first multi-head attention layer and a first matrix fusion layer, and the detection module 430 is used for:

Input the phoneme vector matrix into the first matrix conversion layer for matrix conversion processing to obtain the first query matrix, the first key matrix and the first value matrix;

Inputting the first query matrix, the first key matrix and the first value matrix into the first multi-head attention layer for attention enhancement processing to obtain a first attention enhancement matrix;

The first attention enhancement matrix and the phoneme vector matrix are input into the first matrix fusion layer for matrix fusion processing to obtain the phoneme feature matrix.

In an optional embodiment, the phoneme feature extraction network further includes a first position encoding module and a second matrix fusion layer, and before the phoneme vector matrix is input into the phoneme feature extraction module for feature extraction processing, and before the phoneme feature matrix is obtained, the detection module 430 is also used for:

Input the phoneme vector matrix into the first position encoding module for position encoding processing to obtain the first position encoding matrix;

Inputting the first position encoding matrix and the phoneme vector matrix into the second matrix fusion layer to perform matrix fusion processing to obtain a phoneme position fusion matrix;

When the phoneme vector matrix is input into the phoneme feature extraction module for feature extraction processing to obtain a phoneme feature matrix, the detection module 430 is configured to input the phoneme position fusion matrix into the phoneme feature extraction module for feature extraction processing to obtain a phoneme feature matrix.

In an optional embodiment, the acoustic feature enhancement network includes a feature encoding module and at least one acoustic feature enhancement module, and the detection module 430 is used for:

Input the acoustic features into the feature encoding module for feature encoding processing to obtain an acoustic feature matrix;

When the number of acoustic feature enhancement modules is one, input the acoustic feature matrix into the acoustic feature enhancement module for feature enhancement processing to obtain an enhanced acoustic feature matrix; or,

When the number of acoustic feature enhancement modules is M, the acoustic feature matrix is input into the M acoustic feature enhancement modules to perform feature enhancement processing in turn to obtain an enhanced acoustic feature matrix, where M is an integer greater than 1.

In an optional embodiment, the feature encoding module includes a first convolution layer, a first pooling layer, a second convolution layer, and a second pooling layer, and the detection module 430 is used for:

Input the acoustic features into the first convolution layer for convolution processing to obtain the first convolution result;

Input the first convolution result into the first pooling layer for pooling processing to obtain the first pooling result;

Input the first pooling result into the second convolution layer for convolution processing to obtain the second convolution result;

The second convolution result is input into the second pooling layer for pooling processing, and the acoustic feature matrix is obtained.

In an optional embodiment, the acoustic feature enhancement module includes a second matrix transformation layer, a second multi-head attention layer, a third matrix fusion layer, a third convolution layer, a deconvolution layer, and a fourth matrix fusion layer. , the detection module 430 is used for:

Input the acoustic feature matrix into the second matrix conversion layer for matrix conversion processing to obtain a second query matrix, a second key matrix and a second value matrix;

Input the second query matrix, the second key matrix and the second value matrix into the second multi-head attention layer for attention enhancement processing to obtain the second attention enhancement matrix;

Input the second attention enhancement matrix and the acoustic feature matrix into the third matrix fusion layer for matrix fusion processing to obtain the acoustic fusion matrix;

Input the acoustic fusion matrix into the third convolution layer for convolution processing, and obtain the third convolution result;

Input the third convolution result into the deconvolution layer for deconvolution processing to obtain the deconvolution result;

The acoustic fusion matrix and the deconvolution result are input into the fourth matrix fusion layer for matrix fusion processing, and the enhanced acoustic feature matrix is obtained.

In an optional embodiment, the acoustic feature enhancement network further includes a second position encoding module and a fifth matrix fusion layer, and before the acoustic feature matrix is input into the acoustic feature enhancement module for feature enhancement processing, and before the enhanced acoustic feature matrix is obtained, detection is performed. Module 430 is also used to:

Input the acoustic feature matrix into the second position encoding module for position encoding processing to obtain the second position encoding matrix;

The second position encoding matrix and the acoustic feature matrix are input into the fifth matrix fusion layer for matrix fusion processing to obtain an acoustic position fusion matrix;

When the acoustic feature matrix is input into the acoustic feature enhancement module for feature enhancement processing to obtain the enhanced acoustic feature matrix, the detection module 430 is used for:

The acoustic position fusion matrix is input into the acoustic feature enhancement module for feature enhancement processing, and the enhanced acoustic feature matrix is obtained.

In an optional embodiment, the feature fusion network includes a third matrix conversion layer, a fourth matrix conversion layer, a third multi-head attention layer, a sixth matrix fusion layer, a feedforward network layer, and a seventh matrix fusion layer, The detection module 430 is used to:

Input the enhanced acoustic feature matrix into the third matrix conversion layer for matrix conversion processing to obtain the third key matrix and the third value matrix;

Input the phoneme feature matrix into the fourth matrix conversion layer for matrix conversion processing to obtain a third query matrix;

Input the third query matrix, the third key matrix and the third value matrix into the third multi-head attention layer for attention enhancement processing to obtain the third attention enhancement matrix;

Input the third attention enhancement matrix and the third query matrix into the sixth matrix fusion layer for matrix fusion processing, and obtain the acoustic phoneme fusion matrix;

Input the acoustic phoneme fusion matrix into the feedforward network layer for feedforward calculation processing to obtain a feedforward matrix;

The feedforward matrix and the acoustic phoneme fusion matrix are input into the seventh matrix fusion layer for matrix fusion processing, and the fusion feature matrix is obtained.

In an optional embodiment, the first pronunciation skill detection network includes a first fully connected layer and a first classification function layer, and the detection module 430 is used for:

Input the phoneme feature matrix into the first full connection layer for full connection processing to obtain the first full connection result;

The first full connection result is input into the first classification function layer for classification processing to obtain the first detection result.

In an optional embodiment, the second pronunciation skill detection network includes L branch detection networks, each branch detection network corresponds to a different pronunciation skill, L is an integer greater than 1, and the detection module 430 is used for:

The fusion feature matrix is input into each branch detection network for pronunciation skill detection, and the branch pronunciation skill detection result of each branch detection network is obtained. The branch pronunciation skill detection result of each branch detection network indicates whether the speaker adopts the corresponding branch detection network. The pronunciation skills of say the text to be detected;

The second detection result is obtained according to the branch pronunciation skill detection result of each branch detection network.

In an optional embodiment, the branch detection network includes a second fully connected layer and a second classification function layer, and the detection module 430 is used for:

Input the fusion feature matrix into the second fully connected layer for full connection processing to obtain the second fully connected result;

The second full connection result is input into the second classification function layer for classification processing, and the branch pronunciation skill detection result is obtained.

In an optional embodiment, the pronunciation skill detection device provided by the present application further includes a training module for:

Obtain multiple types of first sample texts that are known to be spoken by different pronunciation skills, and convert each type of first sample text into a corresponding positive sample phoneme sequence;

Obtaining a sample user uses different pronunciation skills to speak the first sample audio of each type of first sample text, and extracts positive sample acoustic features of the first sample audio of each type of first sample text;

Obtaining a second sample text that is known not to be spoken by pronunciation skills, and converting the second sample text into a corresponding negative sample phoneme sequence;

Obtain the second sample audio of the second sample text spoken by the sample user, and extract the negative sample acoustic features of the second sample audio;

Model training is performed according to each type of positive sample phoneme sequence, each type of positive sample acoustic feature, negative sample phoneme sequence and negative sample acoustic feature, and a pronunciation skill detection model is obtained.

In an optional embodiment, the second obtaining module 420 is used for:

Extract the Filterbank feature, fundamental frequency feature and energy feature of the audio to be detected;

Acoustic features are obtained by fusing Filterbank features, fundamental frequency features and energy features.

In an optional embodiment, the first obtaining module 410 is used for:

Remove the silent text units in the text to be detected to obtain a new text to be detected;

Convert each text unit in the new text to be detected into a corresponding phoneme unit to obtain a phoneme sequence.

It should be noted that the pronunciation skill detection apparatus 400 provided in the embodiment of the present application and the pronunciation skill detection method in the above embodiment belong to the same concept, and the specific implementation process thereof is detailed in the above related embodiments, and will not be repeated here.

An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the processor is configured to execute the steps in the pronunciation skill detection method provided by the present embodiment by calling a computer program stored in the memory.

Please refer to FIG. 14 , which is a schematic structural diagram of an electronic device 100 according to an embodiment of the present application.

The electronic device 100 may include components such as a network interface 110, a memory 120, a processor 130, and a screen assembly. Those skilled in the art can understand that the structure of the electronic device 100 shown in FIG. 14 does not constitute a limitation on the electronic device 100, and may include more or less components than the one shown, or combine some components, or different components layout.

The network interface 110 may be used to make network connections between devices.

Memory 120 may be used to store computer programs and data. The computer programs stored in the memory 120 contain executable codes. A computer program can be divided into various functional modules. The processor 130 executes various functional applications and data processing by executing computer programs stored in the memory 120 .

The processor 130 is the control center of the electronic device 100, uses various interfaces and lines to connect various parts of the entire electronic device 100, executes or executes the computer program stored in the memory 120, and invokes the data stored in the memory 120 to execute. Various functions and processing data of the electronic device 100 to perform overall control of the electronic device 100 .

In this embodiment of the present application, the processor 130 in the electronic device 100 loads executable codes corresponding to one or more computer programs into the memory 120 according to the following instructions, and the processor 130 executes the executable codes provided by the present application. The steps in the pronunciation skill detection method of , such as:

Obtain the text to be detected, and convert the text to be detected into a corresponding phoneme sequence;

Obtain the audio to be detected obtained by the speaker speaking the text to be detected, and extract the acoustic features of the audio to be detected;

Input the phoneme sequence and the acoustic feature into the trained pronunciation skill detection model for pronunciation skill detection processing, and obtain the first detection result and the second detection result;

The first detection result is used to represent whether the text to be detected needs to be spoken by using pronunciation skills, and the second detection result is used to represent whether the speaker uses pronunciation skills to speak the text to be detected.

It should be noted that the electronic device 100 provided by the embodiment of the present application and the pronunciation skill detection method in the above embodiment belong to the same concept, and the specific implementation process thereof is detailed in the above related embodiments, which will not be repeated here.

The present application further provides a computer-readable storage medium on which a computer program is stored, and when the stored computer program is executed on the processor of the electronic device provided by the embodiment of the present application, the processor of the electronic device executes the above Steps in any pronunciation skill detection method suitable for electronic equipment. The storage medium may be a magnetic disk, an optical disk, a read only memory (Read Only Memory, ROM), or a random access device (Random Access Memory, RAM), or the like.

A pronunciation skill detection method, device, storage medium and electronic device provided by the present application have been introduced in detail above. The principles and implementations of the present application are described with specific examples. The description of the above embodiments is only used for In order to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope. In summary, the content of this specification It should not be construed as a limitation of this application.

Claims (20)

1. A pronunciation skill detection method, comprising:

acquiring a text to be detected, and converting the text to be detected into a corresponding phoneme sequence;

acquiring a to-be-detected audio obtained by a speaker speaking the to-be-detected text, and extracting acoustic characteristics of the to-be-detected audio;

inputting the phoneme sequence and the acoustic features into a trained pronunciation skill detection model to perform pronunciation skill detection processing to obtain a first detection result and a second detection result;

the first detection result is used for representing whether the text to be detected needs to be spoken by pronunciation skills, and the second detection result is used for representing whether the speaker speaks the text to be detected by pronunciation skills.

2. The pronunciation skill detection method according to claim 1, wherein the pronunciation skill detection model comprises a phoneme feature extraction network, an acoustic feature enhancement network, a feature fusion network, a first pronunciation skill detection network and a second pronunciation skill detection network, and the inputting the phoneme sequence and the acoustic features into the trained pronunciation skill detection model for pronunciation skill detection processing to obtain a first detection result and a second detection result comprises:

inputting the phoneme sequence into the phoneme feature extraction network for feature extraction processing to obtain a phoneme feature matrix;

inputting the phoneme feature matrix into the first pronunciation skill detection network to perform pronunciation skill detection processing to obtain the first detection result;

inputting the acoustic features into the acoustic feature enhancement network for feature enhancement processing to obtain an enhanced acoustic feature matrix;

inputting the enhanced acoustic feature matrix and the phoneme feature matrix into the feature fusion network for feature fusion processing to obtain a fusion feature matrix;

and inputting the fusion feature matrix into the second pronunciation skill detection network to perform pronunciation skill detection processing to obtain the second detection result.

3. The pronunciation skill detection method as claimed in claim 2, wherein the phoneme feature extraction network comprises a phoneme embedding module and a phoneme feature extraction module, and the inputting the phoneme sequence into the phoneme feature extraction network for feature extraction to obtain a phoneme feature matrix comprises:

inputting the phoneme sequence into the phoneme embedding module for embedding to obtain a phoneme vector matrix;

and inputting the phoneme vector matrix into the phoneme feature extraction module for feature extraction processing to obtain the phoneme feature matrix.

4. The pronunciation skill detection method as claimed in claim 3, wherein the phone feature extraction module comprises at least 1 phone feature extraction sub-module, and the inputting the phone vector matrix into the phone feature extraction module for feature extraction to obtain the phone feature matrix comprises:

when the number of the phoneme feature extraction submodules is 1, inputting the phoneme vector matrix into the phoneme feature extraction submodule for feature extraction processing to obtain the phoneme feature matrix; or,

and when the number of the phoneme feature extraction submodules is N, inputting the phoneme vector matrix into the N phoneme feature extraction submodules to sequentially perform feature extraction processing to obtain the phoneme feature matrix, wherein N is an integer greater than 1.

5. The pronunciation skill detection method as claimed in claim 4, wherein the phoneme feature extraction submodule comprises a first matrix conversion layer, a first multi-attention layer and a first matrix fusion layer, and the inputting the phoneme vector matrix into the phoneme feature extraction submodule for feature extraction to obtain the phoneme feature matrix comprises:

inputting the phoneme vector matrix into the first matrix conversion layer for matrix conversion processing to obtain a first query matrix, a first key matrix and a first value matrix;

inputting the first query matrix, the first key matrix and the first value matrix into the first multi-head attention layer for attention enhancement processing to obtain a first attention enhancement matrix;

and inputting the first attention enhancing matrix and the phoneme vector matrix into the first matrix fusion layer for matrix fusion processing to obtain the phoneme feature matrix.

6. The pronunciation skill detection method as claimed in claim 3, wherein the phoneme feature extraction network further comprises a first position coding module and a second matrix fusion layer, and before inputting the phoneme vector matrix into the phoneme feature extraction module for feature extraction to obtain the phoneme feature matrix, the method further comprises:

inputting the phoneme vector matrix into the first position coding module for position coding processing to obtain a first position coding matrix;

inputting the first position coding matrix and the phoneme vector matrix into the second matrix fusion layer for matrix fusion processing to obtain a phoneme position fusion matrix;

the inputting the phoneme vector matrix into the phoneme feature extraction module for feature extraction processing to obtain the phoneme feature matrix includes:

and inputting the phoneme position fusion matrix into the phoneme feature extraction module for feature extraction processing to obtain the phoneme feature matrix.

7. The pronunciation skill detection method according to claim 2, wherein the acoustic feature enhancement network comprises a feature coding module and at least 1 acoustic feature enhancement module, and the inputting the acoustic features into the acoustic feature enhancement network for feature enhancement processing to obtain an enhanced acoustic feature matrix comprises:

inputting the acoustic features into the feature coding module for feature coding processing to obtain an acoustic feature matrix;

when the number of the acoustic feature enhancing modules is 1, inputting the acoustic feature matrix into the acoustic feature enhancing module for feature enhancing processing to obtain the enhanced acoustic feature matrix; or,

and when the number of the acoustic feature enhancing modules is M, inputting the acoustic feature matrix into the M acoustic feature enhancing modules to sequentially perform feature enhancement processing to obtain the enhanced acoustic feature matrix, wherein M is an integer greater than 1.

8. The pronunciation skill detection method as claimed in claim 7, wherein the feature encoding module comprises a first convolution layer, a first pooling layer, a second convolution layer and a second pooling layer, and the inputting the acoustic features into the feature encoding module for feature encoding to obtain an acoustic feature matrix comprises:

inputting the acoustic features into the first convolution layer to carry out convolution processing to obtain a first convolution result;

inputting the first convolution result into the first pooling layer for pooling treatment to obtain a first pooling result;

inputting the first pooling result into the second convolution layer for convolution processing to obtain a second convolution result;

and inputting the second convolution result into the second pooling layer for pooling processing to obtain the acoustic feature matrix.

9. The pronunciation skill detection method according to claim 7, wherein the acoustic feature enhancement module comprises a second matrix conversion layer, a second multi-attention layer, a third matrix fusion layer, a third convolution layer, a reverse convolution layer and a fourth matrix fusion layer, and the inputting the acoustic feature matrix into the acoustic feature enhancement module for feature enhancement processing to obtain the enhanced acoustic feature matrix comprises:

inputting the acoustic feature matrix into the second matrix conversion layer for matrix conversion processing to obtain a second query matrix, a second key matrix and a second value matrix;

inputting the second query matrix, the second key matrix and the second value matrix into the second multi-head attention layer for attention enhancement processing to obtain a second attention enhancement matrix;

inputting the second attention enhancement matrix and the acoustic feature matrix into the third matrix fusion layer for matrix fusion processing to obtain an acoustic fusion matrix;

inputting the acoustic fusion matrix into the third convolution layer for convolution processing to obtain a third convolution result;

inputting the third convolution result into the deconvolution layer for deconvolution processing to obtain a deconvolution result;

and inputting the acoustic fusion matrix and the deconvolution result into the fourth matrix fusion layer for matrix fusion processing to obtain the enhanced acoustic feature matrix.

10. The pronunciation skill detection method according to claim 7, wherein the acoustic feature enhancement network further comprises a second location coding module and a fifth matrix fusion layer, and before inputting the acoustic feature matrix into the acoustic feature enhancement module for feature enhancement processing to obtain the enhanced acoustic feature matrix, the method further comprises:

inputting the acoustic feature matrix into the second position coding module for position coding processing to obtain a second position coding matrix;

inputting the second position coding matrix and the acoustic feature matrix into the fifth matrix fusion layer for matrix fusion processing to obtain an acoustic position fusion matrix;

the inputting the acoustic feature matrix into the acoustic feature enhancement module for feature enhancement processing to obtain the enhanced acoustic feature matrix includes:

and inputting the acoustic position fusion matrix into the acoustic feature enhancement module for feature enhancement processing to obtain the enhanced acoustic feature matrix.

11. The pronunciation skill detection method according to claim 2, wherein the feature fusion network comprises a third matrix conversion layer, a fourth matrix conversion layer, a third multi-attention layer, a sixth matrix fusion layer, a feedforward network layer and a seventh matrix fusion layer, and the inputting the enhanced acoustic feature matrix and the phoneme feature matrix into the feature fusion network for feature fusion processing to obtain a fusion feature matrix comprises:

inputting the enhanced acoustic feature matrix into the third matrix conversion layer for matrix conversion processing to obtain a third key matrix and a third value matrix;

inputting the phoneme feature matrix into the fourth matrix conversion layer for matrix conversion processing to obtain a third query matrix;

inputting the third query matrix, the third key matrix and the third value matrix into the third multi-head attention layer for attention enhancement processing to obtain a third attention enhancement matrix;

inputting the third attention enhancing matrix and the third query matrix into the sixth matrix fusion layer for matrix fusion processing to obtain an acoustic phoneme fusion matrix;

inputting the acoustic phoneme fusion matrix into the feedforward network layer to perform feedforward calculation processing to obtain a feedforward matrix;

and inputting the feedforward matrix and the acoustic phoneme fusion matrix into the seventh matrix fusion layer for matrix fusion processing to obtain the fusion feature matrix.

12. The pronunciation skill detection method according to claim 2, wherein the first pronunciation skill detection network comprises a first fully connected layer and a first classification function layer, and the inputting the phoneme feature matrix into the first pronunciation skill detection network for pronunciation skill detection processing to obtain the first detection result comprises:

inputting the phoneme feature matrix into the first full-connection layer to perform full-connection processing to obtain a first full-connection result;

and inputting the first full-connection result into the first classification function layer for classification processing to obtain the first detection result.

13. The pronunciation skill detection method according to claim 2, wherein the second pronunciation skill detection network comprises L branch detection networks, each branch detection network corresponding to a different pronunciation skill, L being an integer greater than 1, and the inputting the fused feature matrix into the second pronunciation skill detection network for pronunciation skill detection processing to obtain the second detection result comprises:

inputting the fusion feature matrix into each branch detection network to perform pronunciation skill detection to obtain a branch pronunciation skill detection result of each branch detection network, wherein the branch pronunciation skill detection result of each branch detection network represents whether the speaker speaks the text to be detected by adopting pronunciation skill corresponding to each branch detection network;

and obtaining the second detection result according to the branch pronunciation skill detection result of each branch detection network.

14. The pronunciation skill detection method according to claim 13, wherein the branch detection network comprises a second fully connected layer and a second classification function layer, and the inputting the fused feature matrix into each branch detection network for pronunciation skill detection to obtain the branch pronunciation skill detection result of each branch detection network comprises:

inputting the fusion characteristic matrix into the second full-connection layer to perform full-connection processing to obtain a second full-connection result;

and inputting the second full-connection result into the second classification function layer for classification processing to obtain the branch pronunciation skill detection result.

15. The pronunciation skill detection method as claimed in claim 13, wherein before the obtaining the text to be detected and converting the text to be detected into the corresponding phoneme sequence, the method further comprises:

obtaining multiple types of first sample texts which are known to need to be spoken by different pronunciation skills, and converting each type of first sample text into a corresponding positive sample phoneme sequence;

acquiring first sample audio of each type of first sample text spoken by a sample user by adopting different pronunciation skills, and extracting positive sample acoustic features of the first sample audio of each type of first sample text;

obtaining a second sample text which is known not to be spoken by adopting pronunciation skills, and converting the second sample text into a corresponding negative sample phoneme sequence;

acquiring a second sample audio of the second sample text spoken by the sample user, and extracting negative sample acoustic features of the second sample audio;

and performing model training according to each type of the positive sample phoneme sequence, each type of the positive sample acoustic features, the negative sample phoneme sequence and the negative sample acoustic features to obtain the pronunciation skill detection model.

16. The pronunciation skill detection method according to any one of claims 1-15, wherein the extracting the acoustic features of the audio to be detected comprises:

extracting Filterbank characteristics, fundamental frequency characteristics and energy characteristics of the audio to be detected;

and fusing the Filterbank feature, the fundamental frequency feature and the energy feature to obtain the acoustic feature.

17. The pronunciation skill detection method according to any one of claims 1-15, wherein the converting the text to be detected into a corresponding phoneme sequence comprises:

removing unvoiced text units in the text to be detected to obtain a new text to be detected;

and converting each text unit in the new text to be detected into a corresponding phoneme unit to obtain the phoneme sequence.

18. A pronunciation skill detection apparatus, comprising:

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a text to be detected and converting the text to be detected into a corresponding phoneme sequence;

the second acquisition module is used for acquiring the audio to be detected obtained by the speaker speaking the text to be detected and extracting the acoustic characteristics of the audio to be detected;

the detection module is used for inputting the phoneme sequence and the acoustic features into a trained pronunciation skill detection model to carry out pronunciation skill detection processing so as to obtain a first detection result and a second detection result;

the first detection result is used for representing whether the text to be detected needs to be spoken by pronunciation skills, and the second detection result is used for representing whether the speaker speaks the text to be detected by pronunciation skills.

19. A storage medium having stored thereon a computer program for performing the steps of the pronunciation skill detection method as claimed in any one of claims 1-17 when the computer program is loaded by a processor.

20. An electronic device comprising a processor and a memory, said memory storing a computer program, wherein said processor is adapted to perform the steps of the pronunciation skill detection method as claimed in any one of claims 1 to 17 by loading said computer program.

Images