CN110364142B

CN110364142B - Speech phoneme recognition method and device, storage medium and electronic device

Info

Publication number: CN110364142B
Application number: CN201910578724.7A
Authority: CN
Inventors: 苏丹; 陈杰
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-06-28
Filing date: 2019-06-28
Publication date: 2022-03-25
Anticipated expiration: 2039-06-28
Also published as: CN110364142A; CN110473518B; CN110534092A; CN110473518A; CN110428809A; CN110428809B; CN110534092B

Abstract

The invention discloses a method and a device for recognizing speech phonemes, a storage medium and an electronic device. Wherein, the method comprises the following steps: extracting a plurality of first voice features which are in one-to-one correspondence with a plurality of voice frames from the plurality of voice frames which are sequenced according to the time sequence; determining a plurality of key voice features from the plurality of first voice features, wherein the probability that each key voice feature corresponds to one phoneme in the phoneme set is greater than or equal to a target probability threshold; determining a voice feature set corresponding to each key voice feature, wherein each voice feature set comprises the corresponding key voice feature and one or more voice features adjacent to the corresponding key voice feature in the plurality of first voice features; respectively carrying out feature fusion on the voice features in each voice feature set to obtain a plurality of fused voice features, wherein each voice feature set corresponds to one fused voice feature; the phonemes corresponding to each fused speech feature are separately identified in the phoneme set.

Description

Speech phoneme recognition method and device, storage medium and electronic device

Technical Field

The present invention relates to the field of computers, and in particular, to a method and an apparatus for recognizing speech phonemes, a storage medium, and an electronic apparatus.

Background

Currently, in acoustic model modeling, an end-to-end modeling method mainly applied is a frame-level feature coding mode, for example, a CTC (connection Temporal Classification) model. This approach has a condition independent assumption (i.e., the current output is only relevant to the input features and not to the historical output).

However, since there is a correlation between voice data, the above condition-independent assumption makes the acoustic model obtained by modeling have the advantages of simplicity and stability, but the recognition result has a low accuracy and a poor recognition effect. That is, the speech phoneme recognition method in the related art has a problem that the accuracy of the recognition result is low.

Disclosure of Invention

The embodiment of the invention provides a method and a device for recognizing speech phonemes, a storage medium and an electronic device, which are used for at least solving the technical problem that the accuracy of a recognition result is low in a speech phoneme recognition method in the related technology.

According to an aspect of an embodiment of the present invention, there is provided a speech phoneme recognition method, including: extracting a plurality of first voice features which are in one-to-one correspondence with a plurality of voice frames from the plurality of voice frames which are sequenced according to the time sequence; determining a plurality of key voice features from the plurality of first voice features, wherein the probability that each key voice feature corresponds to one phoneme in the phoneme set is greater than or equal to a target probability threshold; determining a voice feature set corresponding to each key voice feature, wherein each voice feature set comprises the corresponding key voice feature and one or more voice features adjacent to the corresponding key voice feature in the plurality of first voice features; respectively carrying out feature fusion on the voice features in each voice feature set to obtain a plurality of fused voice features, wherein each voice feature set corresponds to one fused voice feature; the phonemes corresponding to each fused speech feature are separately identified in the phoneme set.

According to another aspect of the embodiments of the present invention, there is also provided a speech phoneme recognition apparatus, including: the extraction unit is used for extracting a plurality of first voice features which are in one-to-one correspondence with a plurality of voice frames from the plurality of voice frames which are sequenced according to the time sequence; a first determining unit, configured to determine a plurality of key speech features from the plurality of first speech features, where a probability that each key speech feature corresponds to a phoneme in the set of phonemes is greater than or equal to a target probability threshold; a second determining unit, configured to determine a speech feature set corresponding to each key speech feature, where each speech feature set includes the corresponding key speech feature and one or more speech features adjacent to the corresponding key speech feature in the plurality of first speech features; the fusion unit is used for respectively performing feature fusion on the voice features in each voice feature set to obtain a plurality of fusion voice features, wherein each voice feature set corresponds to one fusion voice feature; and the recognition unit is used for respectively recognizing the phonemes corresponding to each fused voice feature in the phoneme set.

According to a further aspect of the embodiments of the present invention, there is also provided a storage medium having a computer program stored therein, wherein the computer program is configured to perform the above method when executed.

According to another aspect of the embodiments of the present invention, there is also provided an electronic apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method by the computer program.

In the embodiment of the invention, the key voice characteristics are determined according to the voice characteristics of the voice frame; determining a voice feature set corresponding to each key voice feature, wherein each voice feature set comprises the corresponding key voice feature and one or more voice features adjacent to the corresponding key voice feature in the plurality of first voice features; performing feature fusion on the voice features in each voice feature set to obtain a plurality of fused voice features; and respectively identifying phonemes corresponding to each fused speech feature in the phoneme set, and determining a speech feature segment (speech feature set) by using the key speech feature on the basis of determining the key speech feature based on the frame-level feature coding, so that more accurate segment (unit) level features can be extracted, the accuracy of the recognition result can be improved, and the technical problem of low accuracy of the recognition result of the speech phoneme recognition method in the related art is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:

FIG. 1 is a diagram illustrating an application environment of a speech phoneme recognition method according to an embodiment of the present invention;

FIG. 2 is a flow chart illustrating an alternative method of speech phoneme recognition according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of an alternative method of speech phoneme recognition according to embodiments of the present invention;

FIG. 4 is a schematic diagram of an alternative method of speech phoneme recognition according to embodiments of the present invention;

FIG. 5 is a schematic representation of an alternative CTC model according to embodiments of the present invention;

FIG. 6 is a schematic diagram of yet another alternative method of speech phoneme recognition in accordance with embodiments of the present invention;

FIG. 7 is a schematic diagram of yet another alternative method of speech phoneme recognition in accordance with embodiments of the present invention;

FIG. 8 is a schematic diagram of an alternative attention model in accordance with embodiments of the invention;

FIG. 9 is a schematic diagram of yet another alternative method of speech phoneme recognition in accordance with embodiments of the present invention;

FIG. 10 is a schematic illustration of voice data according to an embodiment of the present invention;

FIG. 11 is a schematic diagram of yet another alternative method of speech phoneme recognition in accordance with embodiments of the present invention;

FIG. 12 is a schematic diagram of yet another alternative method of speech phoneme recognition in accordance with embodiments of the present invention;

FIG. 13 is a schematic diagram of yet another alternative method of speech phoneme recognition in accordance with embodiments of the present invention;

FIG. 14 is a block diagram of an alternative speech phoneme recognition apparatus according to an embodiment of the present invention;

fig. 15 is a schematic structural diagram of an alternative electronic device according to an embodiment of the invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to an aspect of an embodiment of the present invention, there is provided a speech phoneme recognition method. Alternatively, the above-described speech phoneme recognition method may be applied, but not limited, to the application environment as shown in fig. 1. As shown in fig. 1, the above-described speech phoneme recognition method involves interaction between the terminal device 102 and the server 106 through the network 104.

The terminal device 102 may collect or obtain a plurality of speech frames ordered in time sequence from other devices, and the plurality of speech frames are sent to the server 106 through the network 104. The terminal device 102 may also collect or obtain target voice data from other devices and send the target voice data to the server 106 via the network 104, and the server 106 obtains a plurality of voice frames from the target voice data.

After obtaining the plurality of voice frames, the server 106 may extract a plurality of first voice features corresponding to the plurality of voice frames one to one from the plurality of voice frames; determining a plurality of key voice features from the plurality of first voice features, wherein the probability that each key voice feature corresponds to one phoneme in the phoneme set is greater than or equal to a target probability threshold; determining a voice feature set corresponding to each key voice feature, wherein each voice feature set comprises the corresponding key voice feature and one or more voice features adjacent to the corresponding key voice feature in the plurality of first voice features; respectively carrying out feature fusion on the voice features in each voice feature set to obtain a plurality of fused voice features, wherein each voice feature set corresponds to one fused voice feature; the phonemes corresponding to each fused speech feature are separately identified in the phoneme set.

Optionally, in this embodiment, the terminal device may include, but is not limited to, at least one of the following: mobile phones, tablet computers, desktop computers, and the like. The network may include, but is not limited to, at least one of: a wireless network, a wired network, wherein the wireless network comprises: bluetooth, WIFI and other networks that realize wireless communication, this wired network can include: local area networks, metropolitan area networks, wide area networks, and the like. The server may include, but is not limited to, at least one of: an apparatus for processing a target sequence model using a target neural network model. The above is only an example, and the present embodiment is not limited to this.

Optionally, in this embodiment, as an optional implementation manner, as shown in fig. 2, a flow of the speech phoneme recognition method may include the following steps:

s202, extracting a plurality of first voice features which are in one-to-one correspondence with a plurality of voice frames from the plurality of voice frames which are sequenced according to the time sequence;

s204, determining a plurality of key voice features from the first voice features, wherein the probability that each key voice feature corresponds to one phoneme in the phoneme set is greater than or equal to a target probability threshold;

s206, determining a voice feature set corresponding to each key voice feature, wherein each voice feature set comprises the corresponding key voice feature and one or more voice features adjacent to the corresponding key voice feature in the plurality of first voice features;

s208, respectively performing feature fusion on the voice features in each voice feature set to obtain a plurality of fused voice features, wherein each voice feature set corresponds to one fused voice feature;

s210, identifying phonemes corresponding to the fusion voice features in the phoneme set respectively.

Alternatively, the above-mentioned speech phoneme recognition method may be executed by the target server, and may be applied to, but not limited to, speech recognition, language translation and other tasks.

For example, taking language translation as an example, the plurality of speech frames are speech frames obtained from speech data to be translated corresponding to a first language (e.g., chinese). As shown in fig. 3, a plurality of first speech features may be extracted from a plurality of speech frames by the first module, a plurality of key speech features may be determined from the plurality of first speech features by the second module, a key speech feature identifier may be output to the third module, a speech feature set corresponding to each key speech feature may be determined by the third module, feature fusion may be performed on the speech features in each speech feature set, and a phoneme corresponding to each fused speech feature may be recognized in a phoneme set by the fourth module. After the phonemes are recognized, a word (or sentence) included in the speech data to be translated is determined from the recognized phonemes, and the included word (or sentence) is translated into a word (or sentence) in the second language.

According to the embodiment, on the basis of determining the key speech features based on the frame-level feature coding, the speech feature segments (speech feature sets) are determined by using the key speech features to extract more accurate segment (unit) level features and determine the phonemes corresponding to the speech feature segments, so that the technical problem that the speech phoneme recognition method in the related art is low in recognition result accuracy is solved, and the recognition result accuracy is improved.

The above-described speech recognition method is explained with reference to fig. 2.

In step S202, a plurality of first speech features corresponding one-to-one to a plurality of speech frames are extracted from the plurality of speech frames sorted in time order.

The plurality of speech frames may be speech frames obtained from the target speech data. The target speech data may be a speech segment of a target duration, for example, a 2s speech segment.

The target server may obtain the target speech data prior to obtaining the plurality of speech frames from the target speech data. The target voice data may be transmitted from the terminal to the target server via the network, or may be transmitted from the server storing the target voice data to the target server. The terminal may be a terminal that records the target voice data, a terminal that stores the target voice data, or another terminal that requests processing of the target voice data.

Optionally, in this embodiment, before extracting a plurality of first speech features corresponding to a plurality of speech frames one to one from the plurality of speech frames, the target speech data may be divided according to a predetermined time length to obtain a plurality of unit frames; and determining a plurality of voice frames from the plurality of unit frames according to the target period, wherein each voice frame comprises one or more unit frames.

After the target voice data is obtained, the multiple voice frames can be obtained from the target voice data in multiple ways: dividing target voice data into a plurality of unit frames; and sampling a plurality of voice frames from the plurality of unit frames, or combining the plurality of unit frames to obtain a plurality of voice frames.

The manner of dividing the target voice data into a plurality of unit frames may be: and dividing the target voice data according to a preset time length to obtain a plurality of unit frames. The predetermined time period may satisfy the following division condition: specific speech features can be identified. The predetermined time period may further satisfy the following division condition: the number of included speech features is less than or equal to 1. The predetermined time period may be set as needed, and may be 10ms, for example. By setting the preset time length, the voice characteristics can be ensured to be recognized, and missing recognition or wrong recognition caused by overlong time length can be avoided.

For example, for voice data with a length of 2s, the voice data may be divided into 200 unit frames according to a predetermined time duration of 10 ms.

After obtaining the plurality of unit frames, a plurality of speech frames may be determined from the plurality of unit frames according to the target period, where each speech frame includes one or more unit frames.

In order to reduce the complexity of the calculation and improve the efficiency of the speech phoneme recognition, a plurality of unit frames may be sampled or combined. For example, sampling may be performed such that one or more unit frames are extracted every N unit frames (the target period is N unit frames), thereby obtaining a plurality of speech frames. For another example, a plurality of speech frames may be obtained by combining unit frames in such a manner that every M unit frames form a group.

For example, for 200 unit frames obtained by dividing 2s of speech data by taking 10ms as a predetermined time, 100 speech frames may be obtained by extracting one unit frame every 2 unit frames, 50 speech frames may be obtained by extracting one unit frame every 4 unit frames, and 50 speech frames may be obtained by combining unit frames by taking 4 unit frames as a group.

According to the embodiment, the speech data is divided to obtain the unit frames, and the speech frames are obtained by sampling the unit frames, so that the calculation complexity of speech phoneme recognition can be reduced, and the efficiency of speech phoneme recognition can be improved.

After obtaining the plurality of speech frames, the target server may extract a plurality of first speech features from the plurality of speech frames, where the plurality of speech frames correspond to the plurality of first speech features one to one.

There are various ways to recognize speech features from speech frames, and for the existing speech feature extraction ways, as long as the extracted speech features can be used for speech phoneme recognition, they can be used in the speech phoneme recognition method in this embodiment.

In order to improve the effectiveness of the extracted voice features, a target neural network model can be adopted to extract the voice features.

Optionally, in this embodiment, extracting, from a plurality of speech frames sorted in time sequence, a plurality of first speech features that are in one-to-one correspondence with the plurality of speech frames may include: sequentially inputting each voice frame in the plurality of voice frames into a target neural network model, wherein the target neural network model is used for extracting a first voice feature corresponding to each voice frame; and acquiring a plurality of first voice features output by the target neural network model.

The target neural network model may be a frame-level Encoder model (i.e., an Encoder portion), may be a model of various types of deep neural networks, and may include, but is not limited to, at least one of the following: multilayer LSTM (Long Short-Term Memory), e.g., BilSTM (bidirectional LSTM), UniLSTM (derivative LSTM); a multi-layer convolutional network; FSMN (fed Sequential Memory Networks, feed forward type sequence Memory Network), TDNN (Time Delay Neural Network).

For example, as shown in fig. 4, each of the plurality of speech frames may be sequentially input into a CNN (convolutional Neural Networks), and the CNN extracts and outputs a first speech feature corresponding to each speech frame.

Through the embodiment, the neural network model is used for voice feature extraction, network model training can be carried out as required, and accuracy and effectiveness of voice feature extraction are improved.

In step S204, a plurality of key speech features are determined from the plurality of first speech features, wherein the probability that each key speech feature corresponds to a phoneme in the set of phonemes is greater than or equal to the target probability threshold.

For each extracted first speech feature, a probability that the first speech feature corresponds to each phoneme in the set of phonemes may be determined based on the extracted first speech feature.

The phoneme (phone) may be an element constituting each voice, and is a minimum language unit divided according to natural properties of a language. The analysis can be based on the pronunciation actions of syllables, one action constituting one phoneme. For Chinese, phonemes can be divided into vowels and consonants, e.g., Chinese syllables

There is one phoneme of the sound that is,

there are two phonemes and there are two phonemes,

there are three phonemes. In the case of phoneme recognition, tones in syllables (for example, yin-level, yang-level, up-level, and down-level) may be recognized or may not be recognized.

For each first speech feature, the sum of the probabilities corresponding to the phonemes in the set of phonemes may be 1 (normalization process). In all the first speech features, the probability that part of the first speech features corresponds to each phoneme in the phoneme set may not be determined due to limited information contained in the first speech features, and the first speech features can be ignored; the information represented by the portions of the first speech features that do not belong to the key speech features is ambiguous and the probabilities corresponding to each phoneme in the set of phonemes do not exceed a target probability threshold (e.g., 80%); the information represented by the partial first speech features is unambiguous and the probability corresponding to a phoneme in the set of phonemes exceeds a target probability threshold (determined as a probability of a phoneme greater than 80%), and these first speech features are determined as key speech features.

The determination of key speech features may be performed in a number of ways. The method may be used to determine the key speech feature as long as the method can determine the probability that the speech feature corresponds to each phoneme in the phoneme set according to the speech feature.

Optionally, in this embodiment, determining a plurality of key speech features from the plurality of first speech features may include: a plurality of peak locations are determined from the first plurality of speech features using a CTC model, wherein each peak location corresponds to a key speech feature.

The CTC model may be as shown in FIG. 5, with the CTC model containing an encoder, which encodes x₁…x_TSequentially input into the encoder, and the output (h) of the encoder is inverted using a Softmax function (normalized exponential function)^enc) Processing to obtain each input x (x)₁…x_T) For each y (y)₁…y_T) Probability of (P (y)₁|x)…P(y_T|x))。

CTC mainly solves the problem of correspondence between a tag sequence and an input sequence in a conventional RNN (Recurrent Neural Network, which is a type of Neural Network for processing sequence data) model. Adding a blank symbol blank in a label symbol set, then labeling by using RNN, and outputting the blank symbol when certain effective output cannot be judged; when it is sufficient to determine a valid cell, a valid symbol is output, so that the peak position of the valid symbol in label (label) can be obtained in CTC.

For example, as shown in fig. 6, after identifying a plurality of first speech features, the CNN may output a plurality of peak locations, each peak location corresponding to a key speech feature, using the CTC criterion, the peak locations being an identification of the key speech features.

Through the embodiment, the CTC model is adopted to position the key voice features, the boundaries of all phonemes are not required to be marked during model training, and convenience in model training and model use can be improved.

In step S206, a speech feature set corresponding to each key speech feature is determined, where each speech feature set includes the corresponding key speech feature and one or more adjacent speech features of the plurality of first speech features.

For each determined key speech feature, a set of speech features corresponding to each key speech feature may be determined. For the current key speech features, the speech feature set corresponding to the current key speech features comprises: the current key speech feature and one or more of the plurality of first speech features that are adjacent to the current key speech feature.

The set of speech features corresponding to each key speech feature may be determined in a number of ways. For example, the current key speech feature, one or more of the plurality of first speech features preceding and following the current speech feature may be determined as the set of speech features corresponding to the current key speech feature. For another example, the current key speech feature, one or more speech features of the plurality of first speech features that precede the current speech feature may be determined as the set of speech features corresponding to the current key speech feature. For another example, the current key speech feature, one or more speech features subsequent to the current speech feature in the plurality of first speech features may be determined as the set of speech features corresponding to the current key speech feature.

Optionally, in this embodiment, determining the speech feature set corresponding to each key speech feature may include: determining a second voice feature and a third voice feature corresponding to the current key voice feature in the plurality of key voice features, wherein the second voice feature is a first key voice feature which is in front of the current key voice feature and is not adjacent to the current key voice feature in the plurality of first voice features, and the third voice feature is a first key voice feature which is in back of the current key voice feature and is not adjacent to the current key voice feature in the plurality of first voice features; and determining a current voice feature set corresponding to the current key voice features, wherein the current voice feature set is a subset of a target voice feature set, and the target voice feature set comprises a second voice feature, a third voice feature and a first voice feature between the second voice feature and the third voice feature.

For a current speech feature of the plurality of key speech features, a first key speech feature (a second speech feature) before the current speech feature and not adjacent to the current key speech feature and a first key speech feature (a third speech feature) after the current speech feature and not adjacent to the current key speech feature may be determined, then the second speech feature, the third speech feature and a first speech feature between the second speech feature and the third speech feature may be determined as a target speech feature set, and then one or more speech features may be selected from the target speech feature set as a speech feature set corresponding to the current key speech feature.

It should be noted that, for the first key speech feature, the corresponding second speech feature is the first speech feature, and for the last key speech feature, the corresponding third speech feature is the last first speech feature.

For example, for 12 first speech features corresponding to 12 speech frames, the key speech features are: the 3 rd, 6 th, 7 th and 10 th first speech features. For the 1 st key speech feature, the corresponding target speech feature set is: 1 st-6 th first speech feature. For the 2 nd key speech feature, the corresponding target speech feature set is: 3 rd to 10 th first speech features. For the 3 rd key speech feature, the corresponding target speech feature set is: 3 rd to 10 th first speech features. For the 4 th key speech feature, the corresponding target speech feature set is: 7 th-12 th first speech features.

According to the embodiment, the second voice feature and the third voice feature corresponding to the current key voice feature are determined, and the target voice feature set corresponding to the current key voice feature is determined according to the second voice feature and the third voice feature, so that the voice feature set corresponding to the current key voice feature can be determined according to the target voice feature set, the influence between different key voice features can be avoided, and the accuracy of phoneme recognition is ensured.

In step S208, feature fusion is performed on the voice features in each voice feature set to obtain a plurality of fused voice features, where each voice feature set corresponds to one fused voice feature.

For a current voice feature set in the multiple voice feature sets, feature fusion can be performed on the voice features in the current voice feature set to obtain a fused voice feature corresponding to the current voice feature set.

Feature fusion may be performed in a variety of ways, for example, a weighted sum of the speech features of the current speech feature set may be used. The weights for each speech feature may be the same or different. For example, different speech features may be given different weights according to the distance between each speech feature of the current speech feature set and the current key speech feature, and the closer the distance to the current key speech feature, the greater the weight.

It should be noted that the distance between two speech features may be represented according to the distance between the speech frames corresponding to each speech feature, and the distance between two speech frames may be the starting position, the ending position, or the time difference between any same positions of the two speech frames.

Optionally, in this embodiment, the performing feature fusion on the speech features in each speech feature set respectively to obtain a plurality of fused speech features may include: and respectively inputting the voice features in each voice feature set into a target self-attention layer to obtain a plurality of fused voice features, wherein the target self-attention layer is used for carrying out weighted summation on the voice features in each voice feature set to obtain the fused voice features corresponding to each voice feature set.

The Self-Attention (Self-Attention) layer can be used for carrying out feature fusion on the voice features in each voice feature set, and the features at the unit length level are extracted to obtain fused voice features.

The self-attention model is a model that employs a self-attention mechanism. Unlike the standard attention mechanism: in standard attribution, the Query vector is related to the output label and is obtained by returning the output label to RNN; in self-attribute, its Query vector is generated by the encoder itself through transformation.

For example, as shown in fig. 7, for the self-attention layer, based on the plurality of peak positions output by the CTC and the plurality of first speech features output by the CNN, speech feature segments corresponding to the respective peak positions are determined, and a fused speech feature corresponding to the respective speech feature segments is output. For example, the speech feature set corresponding to the 1 st key speech feature is: 1 st-6 th first speech feature. The 1 st to 6 th first speech features are input into the self-attention layer, and the output of the self-attention layer is a fused speech feature corresponding to the 1 st key speech feature.

Through the embodiment, the segment level features are extracted by using the self-attention layer, so that the accuracy of speech feature fusion can be ensured, and the accuracy of speech phoneme recognition is further improved.

In step S210, phonemes corresponding to each of the fused speech features are respectively identified in the phoneme set.

After obtaining the plurality of fused speech features, a phoneme corresponding to each of the fused speech features may be obtained from the obtained plurality of fused speech features.

For a current fused speech feature of the multiple fused speech features, the probability that the current fused speech feature corresponds to each phoneme in the phoneme set can be obtained according to the current fused speech feature, and the phoneme corresponding to each fused speech feature is determined according to the probability that the current fused speech feature corresponds to each phoneme in the phoneme set.

Optionally, in this embodiment, the respectively identifying phonemes corresponding to each fused speech feature in the phoneme set may include: and sequentially inputting each fused voice feature into a decoder of the target attention model to obtain a phoneme corresponding to each fused voice feature, wherein the decoder is used for obtaining the current phoneme corresponding to the current fused voice feature at least according to the current input fused voice feature and a previous phoneme obtained by processing the previous voice feature of the current fused voice feature by using the decoder.

Attention is a Mechanism (Mechanism) for improving the effect of the RNN-based Encoder + Decoder model, commonly referred to as Attention Mechanism. The Attention Mechanism can be applied to many fields such as machine translation, voice recognition, Image annotation (Image Caption) and the like. The Attention gives the model the ability to distinguish, for example, in machine translation and speech recognition applications, each word in a sentence is given different weight, so that the learning of the neural network model becomes more flexible (soft), and meanwhile, the Attention can be used as an alignment relation to explain the alignment relation between translation input/output sentences and explain what knowledge the model learns.

The structure of the attention model may be as shown in fig. 8. Wherein x is₁…x_TIs the input of encoder, h^encIs the output of the encoder;

is the last output of the attention layer (the last input of the attention model is x)_u-1)，c_uOutput the current state of the attention layer (attention)This input of the model is x_u)，y_u-1For the purpose of attention to one of the outputs on the model,

for this output of the decoder, P (y)_u|y_u-1,…,y₀And x) is the output of the attention model at this time.

The phoneme corresponding to each fused speech feature may be determined using a Decoder network in a target Attention (Attention) model. The target Attention model may be a standard Attention model or an improved Attention model, and as long as a network model of phonemes corresponding to each of the fusion speech features can be obtained according to the input multiple fusion speech features, the target Attention model can be used for a process of determining phonemes corresponding to each of the fusion speech features.

For example, as shown in fig. 9, a plurality of fused speech features output from the attention layer may be input to a decoder of the attention model, and the decoder may determine a phoneme corresponding to the current fused speech feature from the input current fused speech feature and a phoneme corresponding to the previous fused speech feature.

According to the embodiment, the decoder using the attention model recognizes the phoneme corresponding to each fused speech feature, so that the accuracy of speech phoneme recognition can be improved.

After the phoneme corresponding to each fused speech feature is recognized in the phoneme set, a phoneme combination corresponding to the plurality of speech frames may be obtained according to the recognized plurality of phonemes.

Since the same phoneme may correspond to multiple speech frames, there may be a case where at least two key speech features among the multiple identified key speech features correspond to the same phoneme.

For example, as shown in fig. 10, for "hello", 5 phonemes "n", "i", "h", "a", "o" are included, corresponding to 12 speech frames, where "n" corresponds to 1-4 speech frames, "i" corresponds to 5-7 speech frames, "h" corresponds to 8-9 speech frames, "a" corresponds to 10-11 speech frames, and "h" corresponds to 12 speech frames. For "n", the identified key speech feature is the first speech feature corresponding to the 3 rd and 4 th speech frames, and for the other phonemes, there is only one identified key speech feature, so the combination of the final output phonemes corresponding to each fused speech feature is "nnihao".

Optionally, in this embodiment, after the phonemes corresponding to each of the fused speech features are respectively identified in the phoneme set, the phonemes corresponding to each of the fused speech features may be combined according to the language type to which the phoneme set belongs to obtain target display information, where the target display information is one or more syllables corresponding to the plurality of speech frames or one or more words corresponding to the plurality of speech frames; and outputting the target display information to a display device for display.

At the same time as the number of phonemes is identified, the individual syllables can be determined. The phoneme recognition results corresponding to the same phoneme may be combined according to rules of different language types to obtain one or more syllables, and one or more words corresponding to the obtained one or more syllables may be determined according to rules of different language types.

After one or more syllables or one or more words corresponding to the plurality of speech frames are obtained, the syllables or the words can be output to a display device (for example, a terminal device) in a target display information mode for display.

According to the embodiment, the plurality of recognized phonemes are determined to be one or more syllables or one or more words according to the language type to which the phoneme set belongs, and the one or more words are displayed through the display device, so that the phoneme recognition result can be clearly displayed, and the user experience is improved.

The above-described speech phoneme recognition method is explained below with reference to an alternative example. In this example, the deep convolutional neural network model is used to extract the first speech feature, the self-attention layer is used to perform feature fusion, and the standard attention model decoder is used to identify the phonemes corresponding to the fused speech features.

Two end-to-end modeling methods can be applied in acoustic model modeling: one is CTC; the other is Attention. The CTC model mainly only comprises an encoder, namely a frame level characteristic encoding module, has the advantages of simplicity and stability, and has the disadvantage of having a condition-independent hypothesis that the current output is only related to input characteristics and is not related to historical output. The Attention model has two main modules, the encoder and decoder, whose outputs are not only related to input features but also to historical outputs, more sophisticated than the CTC in the probabilistic model. Meanwhile, Attention can capture features of a longer range without being limited by previous and subsequent frames.

The combination of the two modeling modes can combine the two methods through a multi-task training framework, as shown in fig. 11, an encoder module is shared, and an interpolated loss function is optimized in training, wherein the loss function is shown as formula (1):

L_MTL＝λL_CTC+(1-λ)L_Attention (1)

wherein L is_MTLAs a function of the loss after combination, L_CTCAs a loss function of CTC, L_AttentionIs a loss function of the Attention model.

However, in a way of combining the two methods through a multitask training framework, the CTC and the Attention output unit sets must be the same, the Attention cannot utilize the unit range information given by the CTC, and the CTC and the Attention output one at a frame level and one at a unit level, so that a special processing fusion strategy is required.

The speech phoneme recognition method in the example is an acoustic modeling method, combines the existing CTC, ATTENTION and Self-orientation end-to-end modeling technologies, effectively utilizes the boundary ranges of a plurality of units before and after on the basis of the peak position given by a CTC model, firstly adopts a Self-orientation layer to extract more accurate unit level length characteristics, and further uses a standard encoder layer of the orientation, so that errors can be further repaired on the basis of the CTC, and the better recognition accuracy rate is achieved.

As shown in fig. 12, the modeling system corresponding to the speech phoneme recognition method in this example can be divided into the following four modules: a first module, a frame-level encoder model; a module II, a pronunciation unit boundary and position discrimination module; a third module, a segment (unit) level feature encoder module; and a module IV, a decoder (output unit judgment) module.

For the frame-level encoder model, various types of deep neural network models may be employed, such as multilayer LSTM, multilayer convolutional network, FSMN, or TDNN network. For the phonetic unit boundary and position discrimination module, CTC criterion may be used and phonetic unit peak position may be output. For the segment (unit) level feature encoder module, a Self-attribute layer can be adopted, and a Self-attribute network is used for extracting the features of the unit length level in the range covering the left unit and the right unit. For the pronunciation unit judgment output module, a Decoder network in a standard extension model can be adopted.

The pronunciation unit set of module two and the output unit set of module four may be different, for example, the pronunciation unit set uses context-dependent phone (context-dependent phone) and the output unit set uses syllables (syllabe).

As shown in fig. 13, the encoder output layer is the frame-level encoder model output, where the dark circles represent the peaks of the effective label under the CTC criterion; the self-attention layer extracts higher-level features in a certain cell boundary range from left to right (a cell range from left to right in the figure) through an unsupervised self-attention mechanism; on the basis of the segment (unit) level features extracted by the self-attribute layer, a decoder of the standard attribute is further adopted to judge the final output unit.

By the example, segment (unit) level characteristics are extracted by using unit range information given by the CTC through a self-attribute layer, and the self-attribute layer is introduced between the CTC and the attribute, so that the output of the attribute is independent of the output of the original CTC, the model can repair insertion and deletion errors introduced in the CTC model, and finally the output of the model is unified through a Decoder layer of the attribute, a fusion strategy with the CTC is not required to be considered, and the convenience of processing is improved.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.

According to another aspect of the embodiments of the present invention, there is also provided a speech phoneme recognition apparatus for implementing the speech phoneme recognition method, as shown in fig. 14, the apparatus including:

(1) an extracting unit 1402, configured to extract a plurality of first speech features that are one-to-one corresponding to a plurality of speech frames from the plurality of speech frames that are sorted according to the time sequence;

(2) a first determining unit 1404, configured to determine a plurality of key speech features from the plurality of first speech features, wherein a probability that each key speech feature corresponds to a phoneme in the set of phonemes is greater than or equal to a target probability threshold;

(3) a second determining unit 1406, configured to determine a speech feature set corresponding to each key speech feature, where each speech feature set includes the corresponding key speech feature and one or more speech features adjacent to the corresponding key speech feature in the plurality of first speech features;

(4) a fusion unit 1408, configured to perform feature fusion on the voice features in each voice feature set to obtain multiple fused voice features, where each voice feature set corresponds to one fused voice feature;

(5) a recognition unit 1410, configured to recognize a phoneme corresponding to each fused speech feature in the phoneme set respectively.

Alternatively, the speech phoneme recognition device may be executed by a target server, and may be applied to, but not limited to, speech recognition, language translation and other tasks.

Alternatively, the extracting unit 1402 may be configured to perform the step S202, the first determining unit 1404 may be configured to perform the step S204, the second determining unit 1406 may be configured to perform the step S206, the fusing unit 1408 may be configured to perform the step S208, and the identifying unit 1410 may be configured to perform the step S210.

As an alternative embodiment, the above apparatus further comprises:

(1) the dividing unit is used for dividing the target voice data according to a preset time length before extracting a plurality of first voice features which are in one-to-one correspondence with a plurality of voice frames from the plurality of voice frames which are sequenced according to the time sequence to obtain a plurality of unit frames;

(2) and the determining unit is used for determining a plurality of voice frames from the plurality of unit frames according to the target period, wherein each voice frame comprises one or more unit frames.

As an alternative embodiment, the extraction unit 1402 includes:

(1) the first input module is used for sequentially inputting each voice frame in the plurality of voice frames into a target neural network model, wherein the target neural network model is used for extracting a first voice feature corresponding to each voice frame;

(2) the acquisition module is used for acquiring a plurality of first voice features output by the target neural network model.

As an alternative embodiment, the first determining unit 1404 includes:

a first determination module to determine a plurality of peak locations from the plurality of first speech features using a connected time series classification (CTC) model, wherein each peak location corresponds to a key speech feature.

As an alternative embodiment, the second determining unit 1406 includes:

(1) the second determining module is used for determining a second voice feature and a third voice feature which correspond to the current key voice feature in the plurality of key voice features, wherein the second voice feature is a first key voice feature which is in front of the current key voice feature and is not adjacent to the current key voice feature in the plurality of first voice features, and the third voice feature is a first key voice feature which is in back of the current key voice feature and is not adjacent to the current key voice feature in the plurality of first voice features;

(2) and the third determining module is used for determining a current voice feature set corresponding to the current key voice feature, wherein the current voice feature set is a subset of the target voice feature set, and the target voice feature set comprises the second voice feature, the third voice feature and the first voice feature between the second voice feature and the third voice feature.

As an alternative embodiment, the fusion unit 1408 includes:

(1) and the input module is used for respectively inputting the voice features in each voice feature set into the target self-attention layer to obtain a plurality of fused voice features, wherein the target self-attention layer is used for weighting and summing the voice features in each voice feature set to obtain the fused voice features corresponding to each voice feature set.

As an alternative embodiment, the recognition unit 1410 includes:

(1) and the second input module is used for sequentially inputting each fused voice feature into a decoder of the target attention model to obtain a phoneme corresponding to each fused voice feature, wherein the decoder is used for obtaining the current phoneme corresponding to the current fused voice feature at least according to the current input fused voice feature and the previous phoneme obtained by processing the previous voice feature of the current fused voice feature by using the decoder.

As an alternative embodiment, the above apparatus further comprises:

(1) the combination unit is used for respectively identifying phonemes corresponding to each fused voice feature in the phoneme set, and then combining the phonemes corresponding to each fused voice feature according to the language type to which the phoneme set belongs to obtain target display information, wherein the target display information is one or more syllables corresponding to the voice frames or one or more words corresponding to the voice frames;

(2) and the output unit is used for outputting the target display information to the display equipment for display.

According to a further aspect of embodiments of the present invention, there is also provided a storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the above-mentioned method embodiments when executed.

Alternatively, in the present embodiment, the storage medium may be configured to store a computer program for executing the steps of:

s1, extracting a plurality of first voice features corresponding to a plurality of voice frames one by one from the plurality of voice frames sequenced according to the time sequence;

s2, determining a plurality of key voice features from the first voice features, wherein the probability that each key voice feature corresponds to one phoneme in the phoneme set is larger than or equal to a target probability threshold;

s3, determining a voice feature set corresponding to each key voice feature, wherein each voice feature set comprises the corresponding key voice feature and one or more voice features adjacent to the corresponding key voice feature in the plurality of first voice features;

s4, respectively performing feature fusion on the voice features in each voice feature set to obtain a plurality of fused voice features, wherein each voice feature set corresponds to one fused voice feature;

at S5, phonemes corresponding to each of the fused speech features are identified from the phoneme set, respectively.

Alternatively, in this embodiment, a person skilled in the art may understand that all or part of the steps in the methods of the foregoing embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.

According to still another aspect of the embodiments of the present invention, there is also provided an electronic device for implementing the above-mentioned speech phoneme recognition method, as shown in fig. 15, the electronic device including: a processor 1502, a memory 1504, a transmission 1506, and the like. The memory has stored therein a computer program, and the processor is arranged to execute the steps of any of the above method embodiments by means of the computer program.

Optionally, in this embodiment, the electronic apparatus may be located in at least one network device of a plurality of network devices of a computer network.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:

Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 15 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 15 is a diagram illustrating a structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 15, or have a different configuration than shown in FIG. 15.

The memory 1504 may be used for storing software programs and modules, such as program instructions/modules corresponding to the speech phoneme recognition method and apparatus in the embodiments of the present invention, and the processor 1502 executes various functional applications and speech phoneme recognition by running the software programs and modules stored in the memory 1504, so as to implement the speech phoneme recognition method. The memory 1504 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1504 can further include memory located remotely from the processor 1502, which can be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission device 1506 is used for receiving or transmitting data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 1506 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices to communicate with the internet or a local area Network. In one example, the transmission device 1506 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be substantially or partially implemented in the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, and including instructions for causing one or more computer devices (which may be personal computers, servers, or network devices) to execute all or part of the steps of the method according to the embodiments of the present invention.

In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, a division of a unit is merely a division of a logic function, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that it is obvious to those skilled in the art that various modifications and improvements can be made without departing from the principle of the present invention, and these modifications and improvements should also be considered as the protection scope of the present invention.

Claims

1. A method for speech phoneme recognition, comprising:

extracting a plurality of first voice features which are in one-to-one correspondence with a plurality of voice frames from the plurality of voice frames which are sequenced according to the time sequence;

locating a plurality of key speech features from the plurality of first speech features, wherein a probability that each of the key speech features corresponds to a phoneme in the set of phonemes is greater than or equal to a target probability threshold;

determining a voice feature set corresponding to each key voice feature through an unsupervised self-attention mechanism, wherein each voice feature set is a feature set within a certain boundary range around the key voice feature and comprises the corresponding key voice feature and one or more voice features adjacent to the corresponding key voice feature in the plurality of first voice features;

respectively carrying out feature fusion on the voice features in each voice feature set to obtain a plurality of fused voice features, wherein each voice feature set corresponds to one fused voice feature;

and respectively identifying phonemes corresponding to each of the fused speech features in the phoneme set.

2. The method of claim 1, wherein before extracting the first speech features corresponding to the speech frames one-to-one from the speech frames ordered in time sequence, the method further comprises:

dividing target voice data according to a preset time length to obtain a plurality of unit frames;

and determining the plurality of voice frames from the plurality of unit frames according to the target period, wherein each voice frame comprises one or more unit frames.

3. The method of claim 1, wherein extracting the first speech features corresponding to the speech frames one-to-one from the speech frames sorted in time sequence comprises:

inputting each voice frame in the plurality of voice frames to a target neural network model in sequence, wherein the target neural network model is used for extracting the first voice feature corresponding to each voice frame;

obtaining the plurality of first speech features output by the target neural network model.

4. The method of claim 1, wherein locating the plurality of key speech features from the plurality of first speech features comprises:

determining a plurality of peak locations from the plurality of first speech features using a Connected Temporal Classification (CTC) model, wherein each peak location corresponds to one of the key speech features.

5. The method of claim 1, wherein determining the set of speech features corresponding to each of the key speech features comprises:

determining a second voice feature and a third voice feature corresponding to a current key voice feature in the plurality of key voice features, wherein the second voice feature is a first key voice feature which is before the current key voice feature and is not adjacent to the current key voice feature in the plurality of first voice features, and the third voice feature is a first key voice feature which is after the current key voice feature and is not adjacent to the current key voice feature in the plurality of first voice features;

determining a current voice feature set corresponding to the current key voice feature, wherein the current voice feature set is a subset of a target voice feature set, and the target voice feature set comprises the second voice feature, the third voice feature and the first voice feature between the second voice feature and the third voice feature.

6. The method according to claim 1, wherein the feature fusion of the speech features in each speech feature set respectively to obtain the plurality of fused speech features comprises:

and respectively inputting the voice features in each voice feature set into a target self-attention layer to obtain the multiple fused voice features, wherein the target self-attention layer is used for performing weighted summation on the voice features in each voice feature set to obtain the fused voice features corresponding to each voice feature set.

7. The method of claim 1, wherein identifying a phoneme in the phone set corresponding to each of the fused speech features comprises:

and sequentially inputting each fused voice feature into a decoder of a target attention model to obtain a phoneme corresponding to each fused voice feature, wherein the decoder is used for obtaining a current phoneme corresponding to the current fused voice feature at least according to the current input fused voice feature and a previous phoneme obtained by processing a previous voice feature of the current fused voice feature by using the decoder.

8. The method of any of claims 1-7, wherein after identifying a phoneme in the set of phonemes that corresponds to each of the fused speech features, respectively, the method further comprises:

combining phonemes corresponding to each of the fusion voice features according to a language type to which the phoneme set belongs to obtain target display information, wherein the target display information is one or more syllables corresponding to the plurality of voice frames or one or more words corresponding to the plurality of voice frames;

and outputting the target display information to display equipment for displaying.

9. A speech phoneme recognition apparatus, comprising:

the device comprises an extraction unit, a comparison unit and a comparison unit, wherein the extraction unit is used for extracting a plurality of first voice features which are in one-to-one correspondence with a plurality of voice frames from the plurality of voice frames which are sequenced according to the time sequence;

a first determining unit, configured to locate a plurality of key speech features from the plurality of first speech features, where a probability that each of the key speech features corresponds to a phoneme in the set of phonemes is greater than or equal to a target probability threshold;

a second determining unit, configured to determine, through an unsupervised self-attention mechanism, a speech feature set corresponding to each of the key speech features, where each of the speech feature sets is a feature set within a certain boundary range around the key speech feature, and includes the corresponding key speech feature and one or more speech features of the first speech features that are adjacent to the corresponding key speech feature;

the fusion unit is used for respectively performing feature fusion on the voice features in each voice feature set to obtain a plurality of fusion voice features, wherein each voice feature set corresponds to one fusion voice feature;

and the identifying unit is used for respectively identifying the phoneme corresponding to each fused voice feature in the phoneme set.

10. The apparatus of claim 9, wherein the first determining unit comprises:

a first determination module configured to determine a plurality of peak locations from the plurality of first speech features using a connected time series classification (CTC) model, wherein each peak location corresponds to one of the key speech features.

11. The apparatus according to claim 9, wherein the second determining unit comprises:

a second determining module, configured to determine a second speech feature and a third speech feature corresponding to a current key speech feature in the multiple key speech features, where the second speech feature is a first key speech feature that is before the current key speech feature and is not adjacent to the current key speech feature in the multiple first speech features, and the third speech feature is a first key speech feature that is after the current key speech feature and is not adjacent to the current key speech feature in the multiple first speech features;

a third determining module, configured to determine a current speech feature set corresponding to the current key speech feature, where the current speech feature set is a subset of a target speech feature set, and the target speech feature set includes the second speech feature, the third speech feature, and the first speech feature between the second speech feature and the third speech feature.

12. The apparatus of claim 9, wherein the fusion unit comprises:

and the input module is used for respectively inputting the voice features in each voice feature set into a target self-attention layer to obtain the multiple fused voice features, wherein the target self-attention layer is used for performing weighted summation on the voice features in each voice feature set to obtain the fused voice features corresponding to each voice feature set.

13. The apparatus of any one of claims 9 to 12, further comprising:

a combining unit, configured to, after a phoneme corresponding to each of the fused speech features is identified in the phoneme set, combine the phoneme corresponding to each of the fused speech features according to a language type to which the phoneme set belongs, so as to obtain target display information, where the target display information is one or more syllables corresponding to the speech frames or one or more words corresponding to the speech frames;

and the output unit is used for outputting the target display information to display equipment for displaying.

14. A storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the method of any of claims 1 to 8 when executed.

15. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 8 by means of the computer program.