CN113160798B - A method and system for speech recognition of civil aviation air traffic control in China - Google Patents

Abstract

The present invention discloses a method and system for recognizing Chinese civil aviation air traffic control speech. The method comprises: obtaining speech feature data, which is time series feature information extracted based on speech signals; inputting the speech feature data into a trained acoustic model to obtain a recognition result, which represents the Chinese terminology text of air traffic control corresponding to the speech signal; the acoustic model comprises: a TRM module, a BiGRU module, a fully connected layer and a CTC module connected in sequence, the TRM module comprises a multi-head self-attention layer, a first residual connection and a layer normalization layer, a feedforward layer and a second residual connection and a layer normalization layer connected in sequence, the BiGRU module comprises a bidirectional gated recurrent unit network, the CTC module comprises a connection time series classification layer, and the acoustic model is trained by speech samples of air traffic control instruction terms with Chinese character labels. The present invention has the advantage of high recognition accuracy.

Description

A method and system for speech recognition of civil aviation air traffic control in China

Technical Field

The present invention relates to the technical field of speech recognition, and in particular to a speech recognition method and system for civil aviation air traffic control.

Background technique

Air traffic control mainly directs and dispatches aircraft taxiing on the ground and flying on routes. It is an important guarantee for air traffic safety and efficiency, and it relies heavily on air traffic controllers. The air-to-ground conversations between air traffic controllers and crew members are closely related to flight safety. It is necessary to convert the air-to-ground conversations into text records and archive them.

The existing speech recognition technology used in the field of civil aviation air traffic control speech recognition is mainly based on the "CLDNN" neural network based on deep learning, which is composed of multi-layer CNN, multi-layer LSTM, and multi-layer fully connected neural network. However, the recognition accuracy of the existing technical solutions needs to be improved.

Summary of the invention

The object of the present invention is to provide a method and system for recognizing speech of civil aviation air traffic control in China with high recognition accuracy.

To achieve the above object, the present invention provides the following solutions:

A method for speech recognition of civil aviation air traffic control in China, comprising:

Acquire speech feature data, where the speech feature data is time series feature information extracted based on a speech signal;

The speech feature data is input into a trained acoustic model to obtain a recognition result, wherein the recognition result represents the Chinese terminology of air traffic control corresponding to the speech signal; the acoustic model comprises: a TRM module, a BiGRU module, a fully connected layer and a CTC module connected in sequence, the TRM module comprises a multi-head self-attention layer, a first residual connection and a layer normalization layer, a feedforward layer and a second residual connection and a layer normalization layer connected in sequence, the BiGRU module comprises a bidirectional gated recurrent unit network, the CTC module comprises a connection temporal classification layer, and the acoustic model is trained by speech samples of air traffic control instruction terminology with Chinese character labels.

Optionally, before acquiring the voice feature data, the method further includes:

Performing a frame operation on the speech signal to obtain a plurality of speech frames;

The speech feature data is determined according to the speech frame; each piece of speech feature data corresponds to a plurality of continuous speech frames.

Optionally, each of the speech feature data corresponds to a reference speech frame and a set number of speech frames before the reference speech frame and a set number of speech frames after the reference speech frame.

Optionally, when the reference speech frame is the first m frames or the last n frames of the speech signal, the speech feature data belonging to the reference speech frame are padded with zeros in front or behind, respectively, so that the data lengths of each speech feature data are the same, wherein m and n are both positive integers.

Optionally, determining the voice feature data according to the voice frame specifically includes:

Sampling the speech frame to obtain a plurality of sampling points;

Based on the sampling points, the speech feature data are determined, each of the speech feature data corresponds to a sampling point in a plurality of continuous speech frames.

Optionally, the speech feature data is Mel-frequency cepstral coefficients of the speech.

Optionally, before performing the framing operation on the speech signal, the method further includes:

The voice signal is de-mute processed.

Optionally, adjacent speech frames in the speech signal have an overlapping area with a set ratio.

The present invention also provides a Chinese civil aviation air traffic control speech recognition system, comprising:

A speech feature data acquisition module, used to acquire speech feature data, wherein the speech feature data is time series feature information extracted based on speech signals;

A speech recognition module is used to input the speech feature data into a trained acoustic model to obtain a recognition result, wherein the recognition result represents the Chinese terminology of air traffic control corresponding to the speech signal; the acoustic model comprises: a TRM module, a BiGRU module, a fully connected layer and a CTC module connected in sequence, the TRM module comprises a multi-head self-attention layer, a first residual connection and a layer normalization layer, a feedforward layer and a second residual connection and a layer normalization layer connected in sequence, the BiGRU module comprises a bidirectional gated recurrent unit network, the CTC module comprises a connection temporal classification layer, and the acoustic model is trained by speech samples of air traffic control instruction terminology with Chinese character labels.

Optionally, the civil aviation air traffic control speech recognition system further includes:

A de-mute module, used for performing de-mute processing on the voice signal;

A framing module, used for performing a framing operation on the speech signal to obtain a plurality of speech frames, wherein adjacent speech frames have an overlapping area of a set ratio;

The speech feature data determination module is used to determine the speech feature data according to the speech frame; each of the speech feature data corresponds to a plurality of continuous speech frames, and the speech feature data is the Mel-frequency cepstral coefficient of the speech.

According to the specific embodiments provided by the present invention, the following technical effects are disclosed: In the acoustic model structure provided by the embodiments of the present invention, the TRM module can encode the input speech features, realize the mutual connection between the input speech frames through the self-attention mechanism, and obtain a feature representation of the associated context speech information. BiGRU is the product of combining a bidirectional recurrent neural network with a gated recurrent unit network, and has the advantages of both. It can process temporal dependencies like a gated recurrent unit network, and has context information like a bidirectional recurrent neural network. CTC solves the problem of misalignment between the speech input sequence and the label sequence, thereby realizing end-to-end speech recognition. Based on the above reasons, the civil aviation air traffic control speech recognition method provided in the embodiments of the present invention has the advantage of high recognition accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

FIG1 is a schematic flow chart of a method for speech recognition of civil aviation air traffic control provided in an embodiment of the present invention;

FIG2 is a flow chart of another method for speech recognition of civil aviation air traffic control provided by an embodiment of the present invention;

FIG3 is a schematic diagram of the structure of an acoustic model in an embodiment of the present invention;

FIG4 is a schematic diagram of the structure of a TRM module in an embodiment of the present invention;

FIG5 is a schematic diagram of the structure of Multihead Self-Attention in the TRM module in an embodiment of the present invention;

FIG6 is a schematic diagram of the structure of a BiGRU module in an embodiment of the present invention;

FIG7 is a schematic diagram of the structure of a GRU in a BiGRU module according to an embodiment of the present invention;

FIG8 is a schematic diagram of an identification process in an embodiment of the present invention;

FIG9 is a schematic diagram of the structure of a Chinese civil aviation air traffic control speech recognition system provided in an embodiment of the present invention.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The object of the present invention is to provide a method and system for recognizing speech of civil aviation air traffic control in China with high recognition accuracy.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

Example 1

Referring to FIG. 1 , this embodiment provides a method for recognizing speech in civil aviation air traffic control, the method comprising the following steps:

Step 101: Acquire speech feature data, where the speech feature data is time series feature information extracted based on a speech signal;

Step 102: Input the speech feature data into the trained acoustic model to obtain a recognition result, wherein the recognition result indicates the Chinese terminology of air traffic control corresponding to the speech signal; the acoustic model comprises: a TRM module, a BiGRU module, a fully connected layer FC and a CTC module connected in sequence, the TRM module comprises a multi-head self-attention layer, a first residual connection and a layer normalization layer, a feedforward layer and a second residual connection and a layer normalization layer connected in sequence, the BiGRU module comprises a bidirectional gated recurrent unit network, the CTC module comprises a connection time series classification layer, and the acoustic model is trained by speech samples of air traffic control instruction terminology with Chinese character labels. The training of the acoustic model uses the Adam optimizer to fit the training data through the back-propagation algorithm, adjusts the parameters on the validation set, and evaluates the model quality on the test data.

In the acoustic model structure provided in the embodiment of the present invention, the TRM module can encode the input speech features, calculate the similarity between each frame feature and all frame data of the input speech through the self-attention mechanism, fully consider the pronunciation and semantics of the input speech frames, and recalculate a feature representation associated with contextual speech information. BiGRU is a product of combining a bidirectional recurrent neural network with a gated recurrent unit network, and has the advantages of both. It can process temporal dependencies like a gated recurrent unit network, and has contextual information like a bidirectional recurrent neural network, and is suitable as an important module of a speech recognition acoustic model. CTC is used to solve the problem that input sequences and output sequences are difficult to correspond one-to-one, and speech is a typical problem of misalignment between input sequences and label sequences. CTC is aimed at such a problem, so that the deep learning model automatically learns alignment, thereby realizing end-to-end speech recognition. Based on the above reasons, the civil aviation air traffic control speech recognition method provided in the embodiment of the present invention has the advantage of high recognition accuracy. In addition, the acoustic model structure provided by the embodiment of the present invention is composed of only TRM and BiGRU layers, and is not prone to problems such as gradient disappearance and gradient explosion. The model training process is easy to converge, and the requirement for the amount of training data is relatively low, and the cost of data set annotation is low.

In this embodiment, in order to solve the problem of incompatibility between Chinese ATC instructions and Mandarin Chinese pronunciation, an ATC speech data set was built, a deep learning architecture TRM-BiGRU-CTC including a self-attention mechanism was designed, and training and verification were performed on the ATC data set to obtain a Chinese civil aviation air traffic control speech recognition acoustic model. The Chinese civil aviation air traffic control speech recognition acoustic model provided by the present invention has a high accuracy rate for test speech recognition and can recognize ATC speech content recorded at a fast speed in a noisy environment. A large number of professional terms such as numbers, heights, letters, etc. in ATC speech that are different from Mandarin pronunciation can also be automatically converted into corresponding text sequences.

As an implementation method, the Chinese civil aviation air traffic control speech recognition method provided in this embodiment first collects Chinese civil aviation air traffic control speech; then builds an ATC air traffic control speech data set and performs data preprocessing, the data preprocessing includes removing silent segments, extracting features of air traffic control speech and performing feature processing; designs an acoustic model TRM-BiGRU-CTC including a self-attention mechanism, and trains it on the preprocessed ATC speech data set; the air traffic control speech to be recognized is input into the acoustic model obtained after training after feature extraction and feature processing; the output of the acoustic model is decoded by connectionist temporal classification (CTC) to obtain a Chinese text sequence corresponding to the air traffic control speech content.

Furthermore, the format of traffic control voice in civil aviation is stipulated to be WAV format. If it is in other different formats, such as MP3, OGG, etc., it is necessary to convert the format first to ensure that the voice data has a unified WAV format.

The speech data in the self-built ATC speech dataset all come from the actual operating environment of a certain air traffic control area, and the collected ATC speech is manually annotated and proofread according to the "Guide to Air Traffic Radio Call Phrases". The data volume is sufficient to cover the call phrases in most cases in the air traffic control area, ensuring that the speech recognition model trained on this dataset fits the actual environment.

Before step 101 in this embodiment, the following steps may also be included:

Performing a frame operation on the speech signal to obtain a plurality of speech frames, preferably, adjacent speech frames in the speech signal have an overlapping area of a set ratio;

According to the speech frame, the speech feature data is determined; each of the speech feature data corresponds to a plurality of continuous speech frames. Preferably, the speech feature data in this embodiment is the Mel-frequency cepstral coefficient of the speech. Furthermore, each of the speech feature data may correspond to a reference speech frame and a set number of speech frames before the reference speech frame and a set number of speech frames after the reference speech frame. Wherein, when the reference speech frame is the first m frames or the last n frames of the speech signal, the speech feature data belonging to the reference speech frame is padded with zeros in front or behind, respectively, so that the data length of each of the speech feature data is the same. In addition, according to the speech frame, the speech feature data is determined, specifically including: sampling the speech frame to obtain a plurality of sampling points; based on the sampling points, the speech feature data is determined, and each of the speech feature data corresponds to a sampling point in a plurality of continuous speech frames.

Speech feature data refers to the Mel Frequency Cepstral Coefficent (MFCC) feature of speech, which has characteristics that conform to human hearing. Since each frame feature only contains speech information within a very short period of time, most speech frames are not enough to express a Chinese character, so further feature processing is required for this problem. Specifically, a left-right splicing operation is performed on each frame of data of the extracted MFCC feature, that is, for the current frame, the MFCC features of the left m frames and the right n frames are taken and spliced with the current frame as the features of the current speech frame. The purpose of this operation is to make each frame of data input to the acoustic model have more context-related information.

In this embodiment, before performing the frame operation on the voice signal, the method further includes: performing a mute removal process on the voice signal.

Referring to FIG. 2 , this embodiment may include two stages: a training stage and a recognition stage.

First, due to the particularity of air traffic control speech, we need to build our own ATC speech dataset. A large amount of ATC speech is collected in the actual operation environment of a certain air traffic control area, and the format of the speech file is standardized into WAV format, with a bit rate of 128kbps and a sampling rate of 8000Hz. The ATC audio is manually annotated and proofread according to the "Guide to Air Traffic Radio Calling Phrases". The following is an explanation of the targeted annotations corresponding to some special pronunciations in the annotations: (1) The pronunciation of letters is marked as capital letters. For example, the letter A, whose air traffic control pronunciation is Alpha, is marked as A; (2) The marking of numbers is marked as Arabic numerals as much as possible. In addition, regarding the marking of altitude numbers, different air traffic controllers or pilots may have different pronunciations for the same altitude. For example, if 2100 meters is pronounced as two one, it is marked as 21, and if it is pronounced as two thousand one, it is marked as 2 thousand one; (3) For some special waypoints, such as NASPA, it is directly marked as NASPA. When identifying, it is treated as a modeling unit instead of being regarded as five independent modeling units of N, A, S, P and A. The self-built ATC data set in this embodiment has a total of 9,300 voice samples, with a total duration of about 47 hours, of which 7,700 data are used as training sets, 540 data are used as verification sets, and 1,060 data are used as test sets. Compared with the air traffic control voice data set mentioned in the prior art, the ATC voice data set of this embodiment has a smaller data volume, and the data annotation cost is also reduced accordingly. However, the data volume of the data set is still sufficient to cover the vast majority of characters that may appear in the ground-to-air dialogue in the air traffic control field, and since the number of Chinese character categories in this data set is not much different from the number of pinyin categories, Chinese characters can be directly selected as modeling units. The most direct advantage is that no additional language model is required for conversion, and only the acoustic model needs to be trained.

Next, feature extraction is performed on the training speech data after removing the silent segment. Feature extraction first requires a frame operation, that is, N sampling points are synthesized into an observation unit. Since the speech signal has short-term stability (the speech signal can be considered to be approximately unchanged within 10-30ms), the time covered by a frame is set to 25ms in this embodiment. Since the sampling rate of the air traffic control speech data is 8000Hz, there are 200 sampling points in each frame. In order to avoid sudden changes in the feature parameters of the adjacent frames, there is generally an overlapping area between adjacent frames. The overlapping area set in this embodiment is 12.5ms, that is, a frame is taken every 12.5ms. However, the frame number of some ATC speech samples in this embodiment is too long, resulting in higher requirements for the equipment. In this embodiment, a downsampling method of frame sampling is adopted to relieve equipment pressure. The embodiment of the present invention extracts 26-dimensional MFCC features, that is, the feature vector of each frame has 26 features. Since a frame of speech contains only 25ms of content, which is generally not enough to express a syllable, the extracted MFCC features need to be processed by frame splicing, that is, for the current frame, the MFCC features of the left m frames and the right n frames are taken, and then spliced with the current frame as the features of the current frame. In this embodiment, m=7 and n=7 are set. If the current frame is located in the first 7 frames of this segment of speech or the last 7 frames, a zero filling operation is performed, and if the current frame is located in the middle, the MFCC features of the left and right 7 frames of the current frame are spliced to the current frame. In this way, a frame originally only has 26-dimensional MFCC features. After the frame splicing process, each frame has 390 ((7+7+1)*26) dimensional features, including 375ms of speech content information, which solves the problem of less information contained in a single frame data, and can also make each frame have context-related information.

Then, an acoustic model of the Chinese civil aviation air traffic control speech recognition method including a self-attention mechanism is built, and its model structure is shown in Figure 3. In the present invention application, this network structure is referred to as a "TRM-BiGRU-CTC" network, wherein the TRM module refers to the Encoder Block in the Transformer model, and each TRM module consists of a multi-head self-attention (MultiheadSelf-Attention) layer and a feedforward (FeedForward) layer, and each layer is added with residual connections and layer normalization; the BiGRU module refers to a bidirectional gated recurrent unit network; and the CTC module refers to connection timing classification.

An important component of the acoustic model is TRM, and its structure is shown in Figure 4. The TRM module is mainly divided into two parts: the Multihead Self-Attention layer and the Feed Forward layer. The "Add&Norm" in the figure represents the residual connection and layer normalization. The residual connection can effectively improve the gradient disappearance and network degradation problems, while the layer normalization can accelerate the network convergence.

The Feed Forward layer of the TRM module consists of two linear layers. In this embodiment, the number of hidden units in the two linear layers is 1024 and 390 respectively. The output and input dimensions are the same for residual connection. The first linear layer adds a ReLU activation function without Dropout. The second linear layer has no activation function, and the Dropout ratio is 0.3.

The structure of the Multihead Self-Attention layer of the TRM module is shown in Figure 5. For the input X, it first passes through the linear layer Linear to map X into three different representations Q, K, and V respectively. Q represents query, K represents key, and V represents value. In this embodiment, the output dimension of the linear layer is 390. The number of multi-head attention heads in this embodiment is set to 5. The input X is transformed into 5 different self-attention mapping representations through 5 different linear transformations, which are concatenated as new feature codes. Therefore, X is divided into 5 parts according to the feature dimension order after the three mappings Q, K, and V of the Linear layer, as shown in a1, a2, a3, a4, and a5 in Figure 5. The feature dimension of each a is 78 (390/5), and it contains three representations of Q, K, and V. These three representations are then input into the Self-Attention layer. The calculation process of the Self-Attention representation in Figure 5 is Where d _k represents the feature dimension of Q, K, V, which is 78 in this embodiment. Through this calculation, the similarity of each frame of the input multi-time frame data can be calculated with all frames respectively, and the similarity is used as the weight to perform weighted summation on all frames to obtain the association representation of each frame with respect to all input frames, i.e. b1-b5 in FIG5 . The more correlated the frames are, the greater the influence on each other's association representation is, which is manifested as a greater weight representing the similarity. Concatenate b1-b5 to obtain the output Y. The feature dimension of Y is still 390, which is consistent with the input X, so as to perform residual connection.

Another important component of the acoustic model is BiGRU. In a general gated recurrent unit network (GRU), the transmission direction of its hidden layer state is unidirectional from front to back, that is, the state value of the position is only related to the input from position 0 to position i, and has nothing to do with the input from position i+1 to the end, that is, the current state is only related to the "previous" content. However, in the task of speech recognition, the state of the current position often needs to be combined with "context" information to be more effective. The basic idea of the bidirectional gated recurrent unit network (Bi-directional GRU, BiGRU) is to superimpose two unidirectional gated recurrent unit networks up and down. The structure of BiGRU is shown in Figure 6. The first row of circles connected by right-pointing arrows from bottom to top represents the forward GRU, and the second row of circles connected by left-pointing arrows from bottom to top represents the backward GRU. The same training sequence [x1, x2, x3, x4, x5] is input into the forward GRU from front to back to obtain the sequence [b1, b2, b3, b4, b5], and then input into the backward GRU from back to front to obtain the sequence [a1, a2, a3, a4, a5]. The two sequences are concatenated together, for example, b1 and a1 are concatenated into y1, and the output sequence is [y1, y2, y3, y4, y5]. This output sequence can provide complete context information for the state at each moment, and the output dimension of BiGRU is twice that of unidirectional GRU, so BiGRU has stronger expression ability than unidirectional GRU.

The principle structure of the unidirectional GRU in the BiGRU module is shown in Figure 7. Xt represents the input at time t, Ht represents the hidden state at time t, Ht-1 represents the hidden state at time t-1, the circle with an asterisk represents the Hadamard multiplication operation, and the circle with a plus sign represents the addition operation. In principle, GRU realizes long-term temporal dependency through an update gate and a reset gate. The update gate is calculated by the formula Zt=f(Wz[Xt, Ht-1]), where f is the Sigmoid function, and a value between 0-1 is obtained, which determines how much information of the current time step and the past time step should continue to be transmitted. The reset gate is calculated by the formula Rt=f(Wr[Xt, Ht-1]), where f is also the Sigmoid function, and a value between 0-1 is obtained, which determines how much information of the past time step should be forgotten. Although the reset gate and the update gate have the same calculation formula, they have different parameters to achieve different functions. The memory content of the current step is calculated by the formula H`t=tanh(W[Ht, Rt*Ht-1]), and the hidden state at time t is calculated by the formula Ht=H`t*Zt+(1-Zt)*Ht-1, where H`t*Zt represents the information updated in the t time step, and (1-Zt)*Ht-1 represents the information continued to be transmitted in the past time step. The two are combined to obtain the output after t time steps.

In this embodiment, the relevant parameters of the acoustic model are set as follows: the number of heads of the multi-head mechanism of the Multihead Self-Attention layer in TRM is set to 5, the parameter of the Linear layer is 390, then the parameter of each head is 78 (390/5), and the Dropout ratio is 0.3; the number of hidden nodes of the two feedforward sublayers of the Feed Forward layer in TRM are 1024 and 390 respectively, and the Dropout ratio is 0.3; the number of neuron nodes of the forward and backward GRUs in the BiGRU module are both 256, the activation function is the Tanh activation function, and the Dropout ratio is set to 0.3; the hidden nodes of the first fully connected layer FC after BiGRU are set to 256, the activation function is the ReLU activation function, and the Dropout ratio is 0.3; since the final recognition target of the acoustic model is Chinese characters, there are 745 Chinese character classes in the ATC data set, so the hidden nodes of the fully connected layer FC connected to CTC are set to 746 (745+blank) and there is no activation function and no Dropout; the loss function uses the CTC loss function. The Adam optimizer is used to update the network parameters for model training. The initial learning rate is set to 0.0005, and the momentum values in Adam are 0.9 and 0.99 respectively. During the training process, the MFCC features of 12 audios are selected as input in each batch. Since the audio lengths of each batch are different, and the input data of each batch of the neural network are required to be aligned, the MFCC features of the shorter 11 voices in each batch of 12 audios are padded with zeros.

After obtaining the trained acoustic model, the ATC voice data to be recognized can be recognized. As shown in the recognition flow chart of Figure 8, the ATC voice data is first collected through the air traffic control communication equipment, and its format is ensured to be in WAV format. If it is not in WAV format, it needs to be converted to WAV format first. Then remove the silent segment, extract its MFCC features after pre-emphasis, framing, and windowing, and input it into the trained acoustic model after left and right frame splicing operations. The output of the acoustic model is decoded by CTC to obtain the corresponding voice content prediction text. CTC decoding adopts the beam search method. BeamSearch decoding can be considered as a breadth-first search that retains the suboptimal solution. For general breadth-first search, all historical paths are retained in the process, while Beam Search only retains the historical path of TOP-N (called beam width Beam_width). In this embodiment, the beam_width during decoding is set to 5. Assuming that the vocabulary size is 100, when generating the first word, since beam_width is equal to 5, the 5 words with the highest probability are selected from the vocabulary. When generating the second word, the possible sequence of the previous word is any one of the selected 5 words, which are combined with the words in the vocabulary to obtain 5*100 new sequences, and then 10 sequences with the highest confidence are selected from them as the current sequence. When selecting the third word, there will be 5*100 possible sequences, and the 5 sequences with the highest confidence are still selected from them. After that, the above process is repeated until the terminator is encountered, and finally the 5 sequences with the highest scores are selected. The Beam Search method belongs to the idea of greedy algorithm and may not be able to achieve the global optimal solution. However, considering that the number of speech frames is very large and the corresponding number of character classes is also relatively large, if you want to get the global optimal solution, the search space and path will be extremely huge, and the search efficiency will be very low. Therefore, although the Beam Search obtained in this embodiment is a relatively local optimal solution, it is acceptable in terms of engineering effect.

The effect of the present invention is verified as follows

The commonly used evaluation indicator for Chinese speech recognition is the Character Error Rate (CER). The character error rate is calculated as follows: In order to make the recognized sequence consistent with the correct sequence, some characters need to be replaced, deleted or inserted. The total number of these inserted, replaced or deleted characters as a percentage of the total number of characters in the correct sequence is the CER. The calculation formula is as follows:

There are many existing technologies in the field of speech recognition technology, but the existing technologies applied in the field of Chinese air traffic control speech recognition are mainly the "CLDNN" structure, that is, a deep learning architecture composed of multi-layer CNN, multi-layer LSTM, and multi-layer fully connected neural network (from the patent "A method and system for analyzing ground-to-air call data based on speech recognition" with application publication number CN 110335609 A). Since the speech data sets used in the field of Chinese air traffic control speech recognition are generally self-built databases, their speech quality, duration, acquisition equipment, recording environment, etc. are different, so it is impossible to evaluate the advantages and disadvantages of the recognition method by directly comparing the accuracy of different data sets. Therefore, the present invention only conducted a comparative experiment with "CLDNN" and adjusted its structure according to the characteristics of the ATC data set. The specific model structure of the "CLDNN" mentioned in the embodiment of the present invention is as follows: two layers of CNN layers, the convolution kernel size is 3*3, the step size is 1, the number of filters is 32 and 64 respectively, the first layer of CNN is followed by a maximum pooling layer, the pooling window is 2*2, the windows do not overlap, and the second layer of CNN is not followed by a pooling layer; the output of the convolution layer is used as the input of the fully connected network layer to reduce the dimension, and the number of neurons in the fully connected network layer is 512; the fully connected network layer is followed by three layers of LSTM, the number of neurons is 256, followed by a fully connected network layer and a softmax layer, the number of neurons is 256 and the number of character categories respectively. The preprocessing of the speech data is the same as the embodiment of the present invention, and the MFCC features of the speech are extracted and the left 7 frames and the right 7 frames are spliced. The optimizer also uses the Adam optimizer, the initial learning rate is 0.005, and the momentum values in Adam are taken as 0.9 and 0.99 respectively. This "CLDNN" model has high requirements for the amount of data. Due to the limitation of data scale, when the number of LSTM layers in its structure is too large, serious gradient vanishing problem will occur. And due to the limitation of hardware performance, the number of LSTM layers is set to 3 in this experiment.

This "CLDNN" model has been parameterized for the ATC dataset, reducing the parameter scale. However, the number of model parameters is 10,050,163, while the number of model parameters in this embodiment is 3,837,483. In contrast, the parameter scale of this embodiment is smaller and the model maintenance cost is lower.

Table 1 shows the recognition effect of the acoustic model "TRM-BiGRU-CTC" on the test set. The table also lists the word error rates of BiGRU and CLDNN for comparison:

Table 1

It can be seen from the above results that, under the same data preprocessing method, the recognition effect of the present invention on the ATC data set is better than that of "CLDNN" and also better than the benchmark model BiGRU.

The present invention has the following advantages:

(1) ATC speech advantage: The present invention specifically designs an ATC air traffic control speech dataset based on the characteristics of ATC speech.

(2) Model advantages: In the acoustic model structure provided by the present invention, the TRM module can encode the input speech features, calculate the similarity of each frame feature with all the frame data of the input speech through the self-attention mechanism, fully consider the pronunciation and semantics of the input speech frames, and recalculate a feature representation associated with contextual speech information. BiGRU is a product of combining a bidirectional recurrent neural network with a gated recurrent unit network, which has the advantages of both. It can process temporal dependencies like a gated recurrent unit network, and has contextual information like a bidirectional recurrent neural network. CTC is used to solve the problem that it is difficult to correspond one-to-one between input sequences and output sequences, and speech is a typical problem of misalignment between input sequences and label sequences. CTC is aimed at such problems, so that the deep learning model automatically learns alignment, thereby realizing end-to-end speech recognition. In summary, the acoustic model structure is reasonable. At the same time, its main structure is composed of only TRM and BiGRU layers, and it is not easy to have problems such as gradient disappearance and gradient explosion. The model training process is easy to converge, and the data volume requirement is relatively low, and the data set annotation cost is low. Compared with the prior art, the present invention achieves better recognition effect on the ATC air traffic control speech data set with relatively small data volume.

Example 2

Referring to FIG. 9 , this embodiment provides a Chinese civil aviation air traffic control speech recognition system, the system comprising:

The speech feature data acquisition module 901 is used to acquire speech feature data, where the speech feature data is time series feature information extracted based on the speech signal;

The speech recognition module 902 is used to input the speech feature data into the trained acoustic model to obtain a recognition result, wherein the recognition result represents the Chinese terminology of air traffic control corresponding to the speech signal; the acoustic model comprises: a TRM module, a BiGRU module, a fully connected layer FC and a CTC module connected in sequence, the TRM module comprises a multi-head self-attention layer, a first residual connection and a layer normalization layer, a feedforward layer and a second residual connection and a layer normalization layer connected in sequence, the BiGRU module comprises a bidirectional gated recurrent unit network, the CTC module comprises a connection time series classification layer, and the acoustic model is trained by speech samples of air traffic control instruction terminology with Chinese character labels.

As an implementation of this embodiment, the Chinese civil aviation air traffic control speech recognition system further includes:

A de-mute module, used for performing de-mute processing on the voice signal;

A framing module, used for performing a framing operation on the speech signal to obtain a plurality of speech frames, wherein adjacent speech frames have an overlapping area of a set ratio;

The speech feature data determination module is used to determine the speech feature data according to the speech frame; each of the speech feature data corresponds to a plurality of continuous speech frames, and the speech feature data is the Mel-frequency cepstral coefficient of the speech.

In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims (10)

1. A method for speech recognition of civil aviation air traffic control, characterized in that it comprises: Acquire speech feature data, where the speech feature data is time series feature information extracted based on a speech signal; The speech feature data is input into a trained acoustic model to obtain a recognition result, wherein the recognition result represents the Chinese terminology of air traffic control corresponding to the speech signal; the acoustic model comprises: a TRM module, a BiGRU module, a fully connected layer and a CTC module connected in sequence, the TRM module comprises a multi-head self-attention layer, a first residual connection and a layer normalization layer, a feedforward layer and a second residual connection and a layer normalization layer connected in sequence, the feedforward layer consists of a first linear layer and a second linear layer, the first linear layer adds a ReLU activation function without Dropout, and the second linear layer has no activation function; the number of heads of the multi-head attention is set to 5, and the input is subjected to 5 different linear transformations to obtain 5 Different self-attention mapping representations are concatenated and used as new feature codes; the BiGRU module includes a bidirectional gated recurrent unit network, in which the same training sequence [x1, x2, x3, x4, x5] is input into the forward GRU from front to back to obtain a sequence [b1, b2, b3, b4, b5], and then input into the backward GRU from back to front to obtain a sequence [a1, a2, a3, a4, a5], and the two sequences are concatenated together to obtain an output sequence [y1, y2, y3, y4, y5]; the CTC module includes a connection temporal classification layer, and the acoustic model is trained by air traffic control instruction terminology speech samples with Chinese character labels. 2. The method for speech recognition of civil aviation air traffic control according to claim 1, characterized in that before acquiring speech feature data, it also includes: Performing a frame operation on the speech signal to obtain a plurality of speech frames; The speech feature data is determined according to the speech frame; each piece of speech feature data corresponds to a plurality of continuous speech frames. 3. The Chinese civil aviation air traffic control speech recognition method according to claim 2 is characterized in that each of the speech feature data corresponds to a reference speech frame and a set number of speech frames before the reference speech frame and a set number of speech frames after the reference speech frame. 4. The Chinese civil aviation air traffic control speech recognition method according to claim 3 is characterized in that when the reference speech frame is the first m frames or the last n frames of the speech signal, the speech feature data belonging to the reference speech frame are padded with zeros in front or behind respectively so that the data lengths of each of the speech feature data are the same, wherein m and n are both positive integers. 5. The method for recognizing speech of civil aviation air traffic control according to any one of claims 2 to 4, characterized in that the step of determining the speech feature data according to the speech frame specifically comprises: Sampling the speech frame to obtain a plurality of sampling points; Based on the sampling points, the speech feature data are determined, each of the speech feature data corresponds to a sampling point in a plurality of continuous speech frames. 6. The Chinese civil aviation air traffic control speech recognition method according to claim 5 is characterized in that the speech feature data is the Mel-frequency cepstral coefficient of the speech. 7. The method for recognizing speech of civil aviation air traffic control according to claim 2, characterized in that before the speech signal is framed, it further comprises: The voice signal is de-mute processed. 8. The Chinese civil aviation air traffic control speech recognition method according to claim 2 is characterized in that adjacent speech frames in the speech signal have an overlapping area of a set ratio. 9. A Chinese civil aviation air traffic control speech recognition system, characterized by comprising: A speech feature data acquisition module, used to acquire speech feature data, wherein the speech feature data is time series feature information extracted based on speech signals; A speech recognition module is used to input the speech feature data into a trained acoustic model to obtain a recognition result, wherein the recognition result represents the Chinese terminology of air traffic control corresponding to the speech signal; the acoustic model comprises: a TRM module, a BiGRU module, a fully connected layer and a CTC module connected in sequence, the TRM module comprises a multi-head self-attention layer, a first residual connection and a layer normalization layer, a feedforward layer and a second residual connection and a layer normalization layer connected in sequence, the feedforward layer consists of a first linear layer and a second linear layer, the first linear layer adds a ReLU activation function without Dropout, and the second linear layer has no activation function; the number of heads of the multi-head attention is set to 5, and the input is subjected to 5 different linear The transformation obtains 5 different self-attention mapping representations, which are spliced together as new feature codes; the BiGRU module includes a bidirectional gated recurrent unit network, in which the same training sequence [x1, x2, x3, x4, x5] is input into the forward GRU from front to back to obtain the sequence [b1, b2, b3, b4, b5], and then input into the backward GRU from back to front to obtain the sequence [a1, a2, a3, a4, a5], and the two sequences are spliced together to obtain the output sequence [y1, y2, y3, y4, y5]; the CTC module includes a connection time series classification layer, and the acoustic model is trained by air traffic control instruction terminology speech samples with Chinese character labels. 10. The Chinese civil aviation air traffic control speech recognition system according to claim 9, characterized in that the Chinese civil aviation air traffic control speech recognition system further comprises: A de-mute module, used for performing de-mute processing on the voice signal; A framing module, used for performing a framing operation on the speech signal to obtain a plurality of speech frames, wherein adjacent speech frames have an overlapping area of a set ratio; The speech feature data determination module is used to determine the speech feature data according to the speech frame; each of the speech feature data corresponds to a plurality of continuous speech frames, and the speech feature data is the Mel-frequency cepstral coefficient of the speech.