DE10253868B3 - Test and reference pattern synchronization method e.g. for speech recognition system, has test pattern potential synchronization points associated with reference synchronization points - Google Patents

Abstract

The synchronization method has the synchronization effected via predetermined or automatically determined synchronization points within the test and reference patterns, selected vectors of the reference pattern used as reference synchronization points and corresponding vectors of a test pattern identified as potential synchronization points, associated with one or more of the reference synchronization points, with automatic synchronization via the associated synchronization points when given synchronization criteria are fulfilled. Also included are Independent claims for the following: (a) a processor device for synchronization of test and reference patterns, a computer program product with a memory medium for storing a synchronization program for test and reference patterns; (b) a computer-readable memory medium storing a program for synchronization of test and reference patterns; (c) a databank for storing speech pattern synchronization fragments

Description

The invention relates to a method and an arrangement for the synchronization of test and reference patterns and a corresponding computer program product and a corresponding one computer-readable storage medium, which can be used in particular are for finding language-based keywords or notes.

Methods for comparing speech patterns are already used in word-spotting applications, speech recognition systems or, for example, in keyword search methods, such as in the patent application DE 100 54 583 A1 described, for use.

Out DE 195 10 095 A1 a method for segmenting voiced sounds in speech signals is known. For each word, striking points in the time signal are set as synchronization times and used in training for the search and definition of segment boundaries. Out DE 694 23 588 T2 a speech recognition device is known which serves to compensate for environmental differences between an input language and a reference pattern. Furthermore, WO 01/31835 A1 discloses a voice synchronization device which synchronizes a test and a reference signal.

In a typical, conventional system for comparing speech patterns, the acoustic speech signals are converted into analog, electrical signals by means of a microphone. The analog signals are sampled and digitized. The values obtained in this way are divided into a sequence of equally large time segments of typically 10 to 20 ms. 1 and 2 show the waveform in the upper part of the picture 101 . 201 one speech pattern each of the word "Jupiter", spoken by the same speaker. The upper scale 102 . 202 specifies the numbers of the time segments. In this case, the length of a time segment is 20 ms. The lower half of the picture shows the corresponding spectrogram in both figures 103 . 203 , The lower scale in each case 104 . 204 indicates the time. The language parameters are determined for each segment. Depending on the type of analysis, these parameters can e.g. B. represent the spectral energy, the LPC coefficients, the cepstrum coefficients or the like. The speech parameters of each time segment i are summarized in a feature vector a (i). The speech pattern is then available as a sequence A of I feature vectors a (1), a (2), ... a (I). The reference patterns are also present as sequences B (k) of feature vectors b (1, k), b (2, k), ..., b (J _k , k). The index k denotes the kth reference pattern, the associated sequence is B (k). J _k is the number of vectors of the kth reference pattern. Since the same words can also be spoken at different speeds, accordingly the sequences A and B (k) can be of different lengths, a time adjustment must take place between the test pattern and the reference pattern. This is usually done based on the time segments. The optimal assignment of the time indices i of the test pattern to the time indices j of the different reference patterns k is sought. Dynamic time warping (DTW) is required. A widespread method is "dynamic programming". The starting points (word beginnings: i = 1, j = 1) and the end points (word ends: i = I, j = J _k ) are fixed. A local distance measure d (i, j , k) between two feature vectors a (i) and b (j, k) determines the distance between the vectors of the two segments i and j. The greater the distance value, the more unequal the two language segments.

hierbei ist a_n(i) das n-te Element der N Elemente des Merkmalsvektors a(i); b_n(j,k) das n-te Element des Merkmalsvektors b(j,k).A variety of distance functions have been proposed in the literature. A classic distance measure is the squared Euclidean distance:

here a _n (i) is the nth element of the N elements of the feature vector a (i); b _n (j, k) the nth element of the feature vector b (j, k).

In the simplest form of time adjustment, all distance values d (i, j, k), i = 1 to I, j = 1 to J are determined. Then, starting from the starting point i = 1, j = 1, the possible paths are tried out. The sum D (1, k) of the distance values d (i, j, k) of the time segment pairs (i, j) passed through is formed for each path. The path with the smallest sum D _min ( l , k) corresponds to the optimal time adaptation between test pattern and reference pattern k. In some applications, D _min (l, k) is then standardized, e.g. B. by dividing D _min (l, k) by the number of segments passed or by the number of segments of the reference word. 3 illustrates the timing of the two speech patterns 1 and 2 , One of the patterns contains 59 segments, the other 94. Small distance values are shown lighter, larger values darker. The small squares indicate the optimal path 301 , For better orientation is in the diagrams 302 . 303 the course of the total energy of the individual segments is shown below and to the right of the matrix.

Depending on the application, e.g. B. is looking for the first reference pattern k from a set of reference patterns, in which the sum of the optimal path D _min (k) is less than a threshold value. In other applications, the reference pattern k with the smallest D _min (k) of all reference patterns of a given set is searched for. In still other applications, the best n hits, i.e. H. which requires n reference samples with the smallest sums D _min (k).

The calculation of all distance values d (i, j, k) and trying out all possible paths requires a great deal of computing effort. The number of possible paths and thus the search space are therefore restricted in real systems. A first limitation is that at least one index, i or j, or both at the same time, is increased in a path. This prevents a backward movement. Another possible restriction concerns the step size of the indices i or j for each individual step (e.g. to 1), another restriction limits the number of successive steps parallel to an axis. In still other applications, the search is limited to the diagonal and a predetermined number of time segments on both sides of the diagonal. A variety of other restrictions have been proposed and used over time. Nevertheless, the computational effort is considerable, especially in the case that there are many reference patterns. 4 shows an example of the same time adjustment with one of the common methods like 3 but with limited search space.

Another problem is the definition of word boundaries. Current methods determine the word limits for individually spoken words based on the energy content of several successive segments. The beginning and end of a word are due to the low energy content of some word beginnings 105 . 205 or word ends 106 . 206 ( 1 and 2 ) with disturbing background noise (in 1 and 2 there are no significant background noises) can only be inadequately determined with these algorithms. The problems arise even more if the beginning and end of the word are to be found in fluent language. Here the individual words often merge into one another without a recognizable pause.

To reduce the computing effort it is still an advantage if the number of to be compared Reference samples can be limited by a pre-selection.

The object of the invention is therefore in the mentioned Disadvantages of the known solution to fix and in particular a method and an arrangement for Synchronization of test and reference patterns as well as a corresponding one Computer program product and a corresponding computer-readable storage medium To provide, which the computing effort in the time adjustment to reduce. Another object of the invention is to be precise determining the word boundaries of individually spoken words; and one opposite the prior art improved solution for finding the word boundaries of Keywords in fluent Creating language.

According to the invention, this object is achieved by the features in the characterizing part of claims 1, 35 and 41 to 43 in Interaction with the features in the generic term. Appropriate configurations the invention are contained in the subclaims.

• – ein Teil der Merkmalsvektoren b(j) der Referenzmuster als Referenz-SP ausgezeichnet wird,
• – die Merkmalsvektoren a(i) eines Testmusters nach vorgebbaren Merkmalen durchsucht und diejenigen Merkmalsvektoren als potentielle SP gekennzeichnet werden, welche mindestens eins der vorgegebenen Merkmale aufweisen,
• – wenigstens ein Teil der ermittelten potentiellen SP gemäß vorgebbaren Regeln einem oder mehreren Referenz-SP zugeordnet wird und
• – bei Erfüllung vorgebbarer Kriterien die Synchronisation von Test- und Referenzmuster über einander zugeordnete SP automatisch hergestellt wird.

A particular advantage of the method for the synchronization of test and reference patterns, in particular for the synchronization of speech patterns, wherein test and reference patterns each as a sequence A = (a (1), a (2), ..., a (I)) or . B = (b (1), b (2), ..., b (J)) of (associated with time segments) feature vectors a (i) (i = 1, 2, ..., I) or b ( j) (j = 1, 2, ..., J), there is a reduction in the computing effort for the time adjustment by the synchronization being carried out via synchronization points (SP) which are predetermined and / or automatically determined in the test and reference patterns , in which

• Part of the feature vectors b (j) of the reference pattern is marked as a reference SP,
• The feature vectors a (i) of a test pattern are searched for predefinable features and those feature vectors are identified as potential SPs which have at least one of the predefined features,
• - At least part of the potential SP determined is assigned to one or more reference SP according to predefinable rules and
• - If predefined criteria are met, the synchronization of test and reference patterns is automatically established via mutually assigned SP.

• – ein Teil der Merkmalsvektoren b(j) der Referenzmuster als Referenz-SP ausgezeichnet wird,
• – die Merkmalsvektoren a(i) eines Testmusters nach vorgebbaren Merkmalen durchsucht und diejenigen Merkmalsvektoren als potentielle SP gekennzeichnet werden, welche mindestens eins der vorgegebenen Merkmale aufweisen,
• – wenigstens ein Teil der ermittelten potentiellen SP gemäß vorgebbaren Regeln einem oder mehreren Referenz-SP zugeordnet wird und
• – bei Erfüllung vorgebbarer Kriterien die Synchronisation von Test- und Referenzmuster über einander zugeordnete SP automatisch hergestellt wird.

An arrangement for the synchronization of test and reference patterns is advantageously set up in such a way that it comprises at least one processor which is (are) set up in such a way that a method for the synchronization of test and reference patterns, in particular for the synchronization of speech patterns, wherein Test and reference samples each as a sequence A = (a (1), a (2), ..., a (I)) or B = (b (1), b (2), ..., b ( J)) of (associated with time segments) feature vectors a (i) (i = 1, 2, ..., I) or b (j) (j = 1, 2, ..., J) are present, can be carried out, wherein the synchronization takes place via synchronization points (SP) specified and / or automatically determined in the test and reference patterns, wherein

• Part of the feature vectors b (j) of the reference pattern is marked as a reference SP,
• The feature vectors a (i) of a test pattern are searched for predefinable features and those feature vectors are identified as potential SPs which have at least one of the predefined features,
• - At least part of the potential SP determined is assigned to one or more reference SP according to predefinable rules and
• - If predefined criteria are met, the synchronization of test and reference patterns is automatically established via mutually assigned SP.

• – ein Teil der Merkmalsvektoren b(j) der Referenzmuster als Referenz-SP ausgezeichnet wird,
• – die Merkmalsvektoren a(i) eines Testmusters nach vorgebbaren Merkmalen durchsucht und diejenigen Merkmalsvektoren als potentielle SP gekennzeichnet werden, welche mindestens eins der vorgegebenen Merkmale aufweisen,
• – wenigstens ein Teil der ermittelten potentiellen SP gemäß vorgebbaren Regeln einem oder mehreren Referenz-SP zugeordnet wird und
• – bei Erfüllung vorgebbarer Kriterien die Synchronisation von Test- und Referenzmuster über einander zugeordnete SP automatisch hergestellt wird.

A computer program product for the synchronization of test and reference patterns comprises a computer-readable storage medium on which a program is stored which enables a computer, after it has been loaded into the memory of the computer, a method for the synchronization of test and reference patterns, in particular for Synchronization of Sprachmu stern, whereby test and reference patterns each as a sequence A = (a (1), a (2), ..., a (I)) or B = (b (1), b (2), ..., b (J)) of (associated with time segments) feature vectors a (i) (i = 1, 2, ..., I) or b (j) (j = 1, 2, ..., J ) are to be carried out, the synchronization taking place via synchronization points (SP) predetermined and / or automatically determined in the test and reference patterns, wherein

• Part of the feature vectors b (j) of the reference pattern is marked as a reference SP,
• The feature vectors a (i) of a test pattern are searched for predefinable features and those feature vectors are identified as potential SPs which have at least one of the predefined features,
• - At least part of the potential SP determined is assigned to one or more reference SP according to predefinable rules and
• - If predefined criteria are met, the synchronization of test and reference patterns is automatically established via mutually assigned SP.

• – ein Teil der Merkmalsvektoren b(j) der Referenzmuster als Referenz-SP ausgezeichnet wird,
• – die Merkmalsvektoren a(i) eines Testmusters nach vorgebbaren Merkmalen durchsucht und diejenigen Merkmalsvektoren als potentielle SP gekennzeichnet werden, welche mindestens eins der vorgegebenen Merkmale aufweisen,
• – wenigstens ein Teil der ermittelten potentiellen SP gemäß vorgebbaren Regeln einem oder mehreren Referenz-SP zugeordnet wird und
• – bei Erfüllung vorgebbarer Kriterien die Synchronisation von Test- und Referenzmuster über einander zugeordnete SP automatisch hergestellt wird.

In order to carry out a synchronization of test and reference patterns, a computer-readable storage medium is advantageously used, on which a program is stored which enables a computer, after it has been loaded into the memory of the computer, a method for the synchronization of test and reference patterns , in particular for the synchronization of speech patterns, whereby test and reference patterns each as a sequence A = (a (1), a (2), ..., a (I)) or B = (b (1), b (2 ), ..., b (J)) of (associated with time segments) feature vectors a (i) (i = 1, 2, ..., I) or b (j) (j = 1, 2, ... , J) are to be carried out, the synchronization taking place via synchronization points (SP) predetermined and / or automatically determined in the test and reference patterns, wherein

• Part of the feature vectors b (j) of the reference pattern is marked as a reference SP,
• The feature vectors a (i) of a test pattern are searched for predefinable features and those feature vectors are identified as potential SPs which have at least one of the predefined features,
• - At least part of the potential SP determined is assigned to one or more reference SP according to predefinable rules and
• - If predefined criteria are met, the synchronization of test and reference patterns is automatically established via mutually assigned SP.

• – Differenz der Energie von aufeinanderfolgenden Zeitsegmenten und/oder
• – Differenz der Energie bestimmter Frequenzbänder von aufeinanderfolgenden Zeitsegmenten und/oder
• – Änderung der Anzahl der Nulldurchgänge des Sprachsignals in aufeinanderfolgenden Zeitsegmenten und/oder
• – anhand von Cepstrum-, LPC- und/oder PARLOR-Koeffizienten und/oder anhand der Ableitungen dieser Koeffizienten.

A preferred embodiment of the invention provides that potential SPs are determined using the following features:

• - Difference in energy from successive time segments and / or
• - Difference in the energy of certain frequency bands from successive time segments and / or
• - Change the number of zero crossings of the speech signal in successive time segments and / or
• - on the basis of cepstrum, LPC and / or PARLOR coefficients and / or on the derivatives of these coefficients.

Advantageously, for the determination potential SP frequency bands weighted differently.

In another preferred embodiment the invention provides that the predefinable rules for the assignment potential SP to reference SP for at least part of the potential SP determined an analysis additional, preferably arranged in the temporal environment of the potential SP Prescribe feature vectors of the test pattern and an assignment from potential SP to reference SP depending on the results of these Analysis is done.

Another embodiment uses one of the known smoothing functions when calculating the differences or derivatives listed above.

Another embodiment of the method according to the invention stipulates that the to determine potential SP analyzed features with each other combined and / or with mathematical functions, such as in particular the logarithm function become.

It turns out to be an advantage if the synchronization the determination of a degree of similarity (score) of reference and test samples.

It also proves to be an advantage if stop consonants and / or explosive sounds are characteristic feature vectors serving as reference synchronization points.

Another preferred embodiment the invention provides that after synchronization has taken place, the determined reference pattern or patterns and / or the test pattern (s) are output, made available or passed to other applications or a dynamic time adjustment (DTW, Dynamic Time Warping) of test and reference pattern (s) or an analysis of the test pattern is carried out by a hidden Markov model (HMM) and then the or the determined reference pattern and / or the test pattern (s) output, made available or handed over to other applications become. The output can be acoustic and / or visual.

It is also an advantage if the number of possible times taken into account in the dynamic time adjustment Paths and thus the search space can be restricted by in one Path at least one index, i or j, to one at each step predeterminable value increased and / or both indexes simultaneously in each step in a path increased by predeterminable values and / or number of successive steps in parallel limited to one axis and / or the search for the optimal path on the diagonal and a predetermined number of time segments on either side of the diagonal is limited and / or the dynamic time adjustment only for the intervals executed between the SP becomes. To inaccuracies in determining the timing of synchronization points, it is advantageous if at dynamic time adjustment for intervals between SP the Search space is expanded by adding time segments from the search space temporal environment of the SP can be added.

For the implementation of the method according to the invention it proves to be advantageous if a dynamic time adjustment (DTW) using “Dynamic Programming "(DP) or with the help of the Viterbi algorithm or hidden Markov models using the SP to synchronize the DP or the HMM carried out becomes.

• – Cepstrum-Koeffizienten und/oder
• – LPC-Koeffizienten (Linear Predictive Coding) und/oder
• – PARLOR-Koeffizienten und/oder
• – LAR-Koeffizienten und/oder
• – LSP-Koeffizienten und/oder
• – LSF-Koeffizienten und/oder
• – Spektralenergieverteilung und/oder
• – MEL-Spektrum und/oder
• – Nulldurchgangsrate (zero crossing rate) und/oder
• – Mel- oder Bark-Transformationen der vorgenannten Koeffizienten und/oder
• – zeitliche Ableitungen der vorgenenannten Koeffizienten und/oder ihrer Mel- oder Bark-Transformationen und/oder
• – Kombinationen dieser Koeffizienten und/oder Parameter in geglätteter und ungeglätteter Form.

It also proves to be an advantage if the following parameters are determined in the method, in particular for calculating distance functions:

• - Cepstrum coefficients and / or
• - LPC coefficients (Linear Predictive Coding) and / or
• - PARLOR coefficients and / or
• - LAR coefficients and / or
• - LSP coefficients and / or
• - LSF coefficients and / or
• - spectral energy distribution and / or
• - MEL spectrum and / or
• - zero crossing rate and / or
• - Mel or Bark transformations of the aforementioned coefficients and / or
• - Time derivatives of the aforementioned coefficients and / or their Mel or Bark transformations and / or
• - Combinations of these coefficients and / or parameters in smoothed and unsmoothed form.

For an automatic determination of word boundaries of a speech pattern, it is advantageous if the path search for dynamic time adjustment takes place from the first SP of a test pattern beginning at the beginning of the word (backwards) and / or from the last SP of a test pattern beginning at the end of the word (forward). A particularly favorable procedure when determining the word boundaries consists in the fact that when the word boundaries of a speech pattern are automatically determined, the path search is terminated in the time segments for which no assignments of time segments can be found for which the distance value d (i, j) and a predeterminable threshold value D _S fulfills the condition d (i, j) <D _S , or if the distance value d (i, j) exceeds the threshold value D _S in a predeterminable number of successive time segments and the time segments at which the search was terminated was marked as a word boundary. The threshold value D _S is advantageously specified as a function of the application, preferably taking into account background noise. Here again, it is particularly simple if the threshold value D _S is determined by determining the values for d () in speech pauses. The quality of the method can also be improved by the automatic determination of word boundaries of a speech pattern with several of these reference speech patterns in the presence of a plurality of reference speech patterns representing the same word.

This is further improved Procedure if reference SP of at least one class of synchronization fragments (SF) are assigned, an SF comprising feature vectors which in the temporal environment of a reference SP of a given reference pattern are arranged. A classification of SP and / or synchronization fragments is advantageously carried out such that SP and / or synchronization fragments with increasing energy of a first class and SP and / or synchronization fragments be assigned to a second class of SP with falling energy.

It is also advantageous if reference patterns provide a list of data structures (templates) is arranged, which includes information on the reference SP of the reference pattern, in particular information on past assignments of SP from test patterns to reference SP of the reference pattern and / or on the degree of similarity (score) of the reference and test patterns. Depending on the design of the procedure, a list may initially be empty.

The invention is described below of an embodiment explained in more detail. The associated drawing demonstrate:

1 and 2 Waveforms with associated spectrograms of two speech patterns of the word "Jupiter" spoken by the same speaker;

3 Illustration of the time adjustment of the two speech patterns 1 and 2 ;

4 Illustration of a time adjustment with limited search space;

5 Illustration of a time adjustment with search space restricted by synchronization points;

6 Illustration of a time adjustment in which the search space limitation is additionally restricted by means of synchronization points using conventional methods;

7 Examples of structures used to determine classes of synchronization fragments;

8th Example of a pseudo code that can be used for the preselection of reference words;

9 Example for the extension of the template structure, which allows an assessment of the similarity of reference word and test pattern;

10 Illustration of a dynamic time adjustment (DTW) with overlap of the time segments;

11 Illustration of the process steps in determining the class belonging to synchronization points;

12 exemplary structure of a device for pattern comparison;

13 Illustration of an exemplary processing of the signals in a digital signal processor (DSP).

Possible embodiments of the method according to the invention are used by methods described below a device also described below realized.

To ensure synchronization between test and to be able to produce reference samples are in the speech samples Searched for times at which the time segments of the test pattern and of the reference pattern to be compared can be synchronized. This are referred to below as synchronization points (SP). As SP are great Speech energy changes which occur in a short time interval, or abrupt spectral changes, z. B. Energy changes with stop consonants or explosive sounds. In German these are u. a. the consonants b, p, d, t, g and k.

Each test pattern is made after possible SP searched. For example, the absolute value of the difference of the energy content of two successive time segments. exceeds this a certain threshold, for example 1/3 of the mean Energy content of e.g. B. twenty surrounding time segments applies he as a possible Synchronization point. The energy content of is also suitable chosen Frequency bands, or other methods that extract abrupt energetic or spectral changes can. A test or reference pattern can contain multiple SP. The SP the reference patterns are determined beforehand and together with their Feature vectors stored. The comparison using “Dynamic Programming " subsequently limited to the intervals between the SP.

Instead of "dynamic programming" can also Hidden Markov models for time adjustment in all variants of the im The following described exemplary method according to the invention be used.

In 1 and 2 possible synchronization points are shown: 110 . 111 and 112 in 1 such as 210 . 211 . 212 and 213 in 2 , 5 shows like in 3 , the DTW on the speech samples 1 and 2 , but with a search space restricted by synchronization points. In addition to the search space limitations using SP, the search space can be restricted using conventional methods. 6 shows such an additional limitation (limitation as in 4 ) with the same speech patterns.

The path search between the beginning of the word and the first synchronization point can be done in reverse. It starts at the first synchronization point and looks for a path with the restrictions regarding step size, etc. known from the forward search, back to the beginning of the word. Since the beginning of the word of the reference pattern is fixed, the search can be stopped when the most favorable path from the first synchronization point back to the beginning of the word b (1, k) of the reference pattern has been found. This solves the problem of determining the beginning of the word in the test pattern. Likewise, the search (forward) of the optimal path in the last part of the test pattern (last synchronization point to the end of the word) is terminated when the optimal path to the last vector b (J _k , k) of the respective reference pattern has been found. This solves the problem of determining the end of the word.

In the next step, the language parameters of the immediate environment of the synchronization points are used to preselect the reference samples to be compared. Classes of synchronization fragments (SF) around the SP are used for this. In a preferred application, an SF is a short sequence of feature vectors with a predetermined number of vectors. See also 11 , The SF begin at increasing synchronization points 902 . 903 with the vector of the SP. At SP with falling energy 901 the feature vectors immediately before the SP together with the vector at the SP form the SF 911 , The SF 911 . 912 . 913 be with the SF representatives 930 of specified SF classes 921 . 922 . 923 compared.

When searching, after determining the SP 901 . 902 respectively. 903 the associated SF 911 . 912 respectively. 913 compiled and their class membership determined. A lot of SF objects belong to each SF class 931 , Each SF object belongs to exactly one SF class. It contains a pointer to an RP object 941 . 942 (RP = Reference Pattern). There is an RP object for each reference word. The pointers of the SF objects 931 an SF class 921 . 922 . 923 point to the RP objects 941 . 942 the reference words which contain synchronization fragments of this class. After the class of a fragment has been determined, a structure is created for each reference word which is referenced in the said list, if it does not already exist and suitably, which is subsequently used as a template 951 referred to as. This newly created template 951 is included in the list of active templates. Each reference word can have such a list. In the templates 951 It is noted which SP of the reference pattern found in the currently examined test pattern matching SF. If an SF matching the last SP of a reference pattern has been found in the test pattern, or if a certain time has passed since a previous SF matched, the template is used 951 removed from the list of active templates and sent for further processing or

output: So z. B. stencils 951 , in which all SP are marked as found, then a time adjustment is carried out using "dynamic programming" (DP). Depending on the application, the DP can also choose between the test pattern and reference word for templates 951 , in which not all, but only a predeterminable number of SPs, are carried out. The execution of a DP can also depend on the relationship between the matching SPs found and the number of SPs in the reference word. Another extension of the method determines the "similarity" (score) of the synchronization fragments assigned to one another for each comparison of reference and test pattern SF. A structure which takes into account the "similarity" (d [NmOfSP]) is exemplified in 9 played. These "similarities" can then be used to determine a ranking order among the possible reference words. The ranking order in turn determines which reference patterns are compared first by means of DP. This method means that the DTW between reference word and test pattern does not have to be carried out for all reference words , ie the reference words are preselected.

The DTW search can, for example start with the reference words that have the highest number of hits, the best relationship from hits to the number of synchronization points of the reference pattern, or the best mean similarity the synchronization fragments (SF) of the reference pattern to the synchronization fragments of the test pattern. The mean similarity can e.g. B. as a sum of similarities of the individual SF for each class divided by the number of Synchronization points can be defined. If a reference word has several Synchronization points, so is the comparison between reference patterns and search pattern started by DTW when a synchronization point of the test pattern the last synchronization point of the reference pattern was assigned. additional the comparison can be started if since the last assignment a synchronization point of test pattern and reference pattern more time segments have passed than at maximum stretch or Upsetting of the samples by means of DTW is possible.

To limit the computing load through the computationally intensive DTW calculations, e.g. B. only the DTW comparisons with the greatest similarity of the SF are carried out, or the DTW calculations are carried out in the order of the mean similarity of the SF, and abort as soon as the first reference pattern k has been found, at which D _min (k) falls below a predetermined threshold.

In other applications, the Synchronization points, analogous to their use for synchronization of two patterns in the “Dynamic Programming", can be used for synchronization in hidden Markov models.

The synchronization described above with the help of the SP, actually represents the first step of pattern recognition In some applications, this first synchronization step is sufficient however, already to determine the reference word that matches the test pattern. A comparison using DP or the use of a hidden Markov model is with them Applications no longer necessary. This applies in particular to applications with a small number of reference words.

The creation of templates 951 In the manner described, the backward search in the first part of the word after the beginning of the word, and the termination of the search when the most favorable path to the beginning or end of the word of the reference pattern has been found leads to the fact that words are also found in fluent language if there are no clear pauses between individual words.

In the embodiment explained above the special case was explained that an SF was assigned to exactly one SF class can be that for each reference word has exactly one RP object, as well as a possible form of the data structures. For other applications can however, variations in the method may be advantageous, e.g. B. an SF class can have multiple representatives or each SF object can have its own representative or each SF object can have several classes with a specific one Probability can be assigned (fuzzy clustering).

Detailed description an exemplary arrangement according to the invention and the synchronization steps an exemplary embodiment:

12 shows an example of the structure of a device according to the invention. The acoustic speech signals are from a microphone 401 recorded and converted into electrical signals. An amplifier 402 amplifies the signal. A bandpass filter 403 suppresses any DC component, as well as frequencies below 300 Hz and above 3 kHz. This corresponds to the bandwidth limits in many telephone networks. Other limitations are also possible. Then the signal is in an analog-to-digital converter 404 sampled. The sampling rate is 8,000 samples / s with a resolution of 13 bits. This sampling rate is also used in many digital telephone systems. Other sampling rates and resolutions are possible. The signals are then sent from a digital signal processor (DSP) 405 further processed. The results are evaluated in a processor 406 with RAM 407 and read-only memory (ROM) 408 , The results are depending on the application from the processor 406 or from the DSP 405 output 409 , In still other applications, the input and / or output takes place remotely in a separate device.

13 shows the processing of the signals in the DSP 405 in a preferred embodiment. First be in a framer 421 the samples are divided into time segments (English frames) of 20 ms (corresponding to 160 samples). Other lengths of the time segments are possible. Versions with overlapping time segments are also possible. The signals are then passed through a high pass filter 422 guided. In particular, any DC components are filtered out. Then the individual values of a segment are multiplied by a window function. This reduces the effects of dividing the signal into time segments on the calculations of the speech parameters. The Hamming window function is used for this in the preferred application 423 used. Other functions such as Hanning, Kaiser, etc. are possible, as is a window function. The calculation of the filter 422 and the window function 423 can also be done in one step.

The language parameters of the language segments are then determined. In the preferred implementation, cepstrum coefficients are used. These are formed by first using a DFT (Discrete Fourier Transform) 424 of the time segment is calculated. Then the logarithm of the DFT results 425 calculated. An inverse discrete Fourier transformation results from these values 426 won the ceptrum coefficients. Other methods for calculating the cepstrum coefficients can also be used, for example the calculation using the autocorrelation coefficients and / or the LPC coefficients.

Instead of the cepstrum coefficients can other language parameters are used, such as. B. the LPC coefficients (LPC = Linear Predictive Coding), PARCOR coefficients, energy spectrum or MEL-scaled Spectrum, MEL-Cepstrum coefficients, number of zero crossings, as well comparable parameters.

In the preferred implementation, the ten first cepstrum coefficients, their first derivative (differences c _j (i) - c _j (i - 1) of the j-th cepstrum coefficients of the i-th and (i-1) -th pointing segment ), the total energy and its first derivatives (calculated in the unit for forming the first derivative 428 ) used as elements of the feature vector of a time segment. The use of other parameter sets and their derivations is also possible according to the invention. Other methods, e.g. B. the known methods with smoothing functions can be used.

The search for synchronization points in the SP detector 429 is based on the time segments. In a simple version, the difference Δ _{i of} the energy of the segments i and i-1 is formed. The mean energy e _{i of} the surrounding twenty segments is determined. For this purpose, the energy of the segments i - 10 to i + 9 is added and by the number of segments, in this case 20 , divided. To determine the average energy of the first segments of a recording, the average energy is formed from correspondingly fewer elements, ie to determine the average energy of the second segment, the sum of the energy values of segment 1 to 11 formed and divided by 11, when determining the average energy of the second element we use the energy of the segments 1 to 12 summed and divided by 12, etc. With this approach, the first segment of a record cannot have a synchronization point. A synchronization point is present when the absolute value of the ratio Δ _i / e _i exceeds a certain threshold. In the preferred implementation, a threshold of 1/3 is used, ie a synchronization on point exists if | Δ _i / e _i | > 1/3. Other threshold values also provide useful results. Likewise, instead of the difference in the total energy of two adjacent segments, the difference in the energy of these segments in certain frequency bands (e.g. 300 to 1,000 Hz) can be used. Individual frequency bands can be weighted differently. Furthermore, the change in the number of zero crossings of the speech signal in two adjacent time segments, as well as the cepstrum coefficients, the LPC coefficients, the PARCOR coefficients and / or their derivatives, or the sum or weighted sum (or other functions, such as, for E.g. the sum of the log values of listed coefficients) from combinations of these parameters can be used to determine the SP. In another embodiment of this invention, the synchronization points are not determined on the basis of a fixed segment division. Instead, the average energy is continuously determined from the samples (moving average). The moving average is determined over a short and a longer time window. If the ratio of the absolute value of the change in the mean value of the short time window to the mean value of the long time window exceeds a predetermined threshold, there is a synchronization point.

Then be in unity 430 for the determination of synchronization fragments, suitable synchronization fragments are compiled, their class determined if necessary, and comparisons between the test pattern and one or more reference patterns 432 using DTW 431 calculated. These steps can be done in the DSP unit 405 or in the processor unit 406 of the device. They are described in more detail in the following sections.

The embodiments described above are examples of special arrangements with which the method according to the invention can be carried out. They are by no means a conclusive presentation. Rather, arrangements that do without individual components, such as a bandpass filter, may prove practical in certain cases 403 or an amplifier 402 , The processor can also 406 the tasks of the DSP 405 take over, or vice versa. New developments in the field of microelectronics can also prove to be useful in further embodiments of the arrangement according to the invention, which also implement the method according to the invention.

calculation the DTW with synchronization points

In the simplest version of the method according to the invention take place after the calculation of the language parameters of the time segments and determining possible ones Synchronization points the comparisons of the test pattern with the reference patterns using "Dynamic Programming ". The synchronization points of the reference patterns are available (reference SP). They are usually together with the feature vectors of the reference pattern stored.

First is the number of synchronization points of the test pattern with the number of reference synchronization points of the respective reference pattern compared. If they are the same, then become the SP in the order of their appearance in the two Patterns associated with each other, d. H. the first reference SP of the reference pattern becomes the first SP of the test pattern, the second reference SP of the reference pattern assigned to the second SP of the test pattern, etc. To determine the similarity the two patterns will then using the "Dynamic Programming "the Path with the smallest sum of the distances of the individual feature vectors certainly. The DTW is carried out by individual DTW between the synchronization points or between the first SP and the beginning of the word and the last SP and the end of the word. The sum of the smallest sums of the individual sections gives the similarity of test and reference word.

In principle, the search between two synchronization points in the direction of the time axes, or backwards. Especially for the first period, from the beginning of the word to the first synchronization point is, due to the often inaccurate determination of the beginning of the word, one Reverse search from Advantage (see the explanations above). The reverse search is done exactly like the classic time adjustment, only with the reverse Sign of the individual steps. It starts at a synchronization point. At each step, at least one index (i or j) is then lowered. For limitation of the search space can the same restrictions in terms of step size etc. are used as in the forward search.

The simplest version is in the case that the number of synchronization points in test pattern and does not match in the reference pattern, the "dynamic Time adjustment "after traditional way, without synchronization through synchronization points, respectively. The same procedure is followed if there are no synchronization points being found.

Distinction of two types of SP

A possible extension of the method is to differentiate between two types of synchronization points: points with increasing speech energy and points with falling speech energy (typical for explosive sounds or stop consonants). Before comparing two samples by a DP process follows the assignment of the synchronization points of the two patterns as described above. It is then compared whether all pairs of synchronization points are of the same type, ie represent points of increasing or decreasing energy values. If at least one pair is not the same, the comparison by means of DP between the two patterns is either not carried out (ie the two patterns are assumed to be unequal), or the DP is carried out, the non-matching synchronization points not being used for synchronization. 5 shows an example: the (third) SP 212 out 2 is not used because in 1 no suitable SP was found. The assignment of the SP of the reference words and the test pattern cannot be clearly determined with this procedure. Either the possible combinations can be tried out, or the time intervals between the reference SP of the reference word and the time intervals between the SP of the test pattern can be used for an assignment.

synchronization fragments

A better assignment of the SP of reference and test patterns enables a further modification of the method: in order to subdivide the synchronization points into several classes, a synchronization fragment is assigned to each synchronization point. Synchronization fragments consist of a sequence of vectors with speech features of successive time segments. In the preferred implementation, the sequence consists of four vectors, other lengths can also lead to good results. In the case of synchronization points, which are characterized by a drop in voice energy, in the preferred implementation the feature vectors of the three time segments lying before the synchronization point together with the feature vector of the synchronization point form the said sequence. In the case of synchronization points, which are characterized by an increase in the speech energy, the vector of the time segment of the synchronization point forms this sequence together with the feature vectors of the three following time segments. Other methods of selecting the vectors are also possible. For example, regardless of whether the synchronization point is at a time of rising or falling speech energy, the sequence can consist of the feature vectors of the two before, the on and the two time segments lying behind the synchronization point. After the synchronization fragments of a test pattern are determined, their class affiliation is determined. For this purpose, each synchronization fragment of the test pattern is compared with a representative of each class. The comparison is made by calculating the sum of the distances of the feature vectors. In the preferred implementation, the same function d (i, j, k) is used as the distance measure in the case of time adjustment by means of "dynamic programming". It is also possible according to the invention to use other distance functions, for example functions which are particularly meaningful are in the range of phonetic transitions. Similarly, can be used for synchronization fragments with increasing energy content other distance functions than in synchronization fragments with decreasing energy content. A synchronization fragment of the class C _i is assigned, wherein the distance to the synchronization fragment of the representatives of this class is the smallest. Instead of the sum of the distances of the feature vectors with a fixed assignment of the vectors of the fragments of a reference word to the vectors of the fragments of the test pattern, a DTW comparison between the beginning and end of the two vector sequences can be used this is necessary if the sequences consist of a variable number of vectors. So z. B. fragments from the sequence of all vectors between two SP, or half of the vectors between two SP can be used to classify. In this variant of the method, sequences of vectors which are earlier than the SP are compared backwards from the SP (see above) by means of "dynamic programming", the backward comparison being ended when the cheapest path to the first vector leads to the Reference word belonging sequence is found.

overlaps at the synchronization points

Another variant of the method, which can be used in combination with all of the methods described above for determining the SP assignment, extends the search space by the SP. This eliminates problems caused by inaccuracies in determining the SP's temporal position. The search space is expanded in that the two search spaces (in front of and behind the SP) overlap by a few time segments. This overlap can occur along one or both axes at the same time. 10 shows a DTW with overlap of time segments 310 to the matching SP with the from 1 and 2 known speech patterns.

One way to execute the DP in case of overlap is to use the search area as in 5 or 6 limit and then immediately add a few fields to the SP by adding individual fields, for example by adding a field (n, m + 1) and (n + 1, m) to the SP (n, m). When searching backwards, you can proceed as follows: Suppose the SP is at (n, m), the overlap points have been defined with (n - 1, m) and (n + 1, m). Then the optimal path is backwards - starting from (n, m) - searched. The accumulated distance D (i, j) (ie the sum of the distance values d along the path up to and including (i, j)) of each field traversed by the optimal path is stored. Then the “backward search” of the optimal paths and the other overlap points is started. If one of these paths (P _b ) crosses an optimal path P _a (previously found) (another starting point), the search for the corresponding path can be terminated. The further optimal path P _b (towards the beginning) corresponds to the previously found residual section of P _a . The total distance for P _b can be calculated from the stored accumulated distances of P _a and the distance values of P _b accumulated up to the crossing point.

In addition to the algorithm described alternatively to search for the proposed overlap Variations of known approaches can be applied.

preselection of reference words

The method for preselecting reference words using classes of synchronization fragments uses the in 7 structures listed as examples.

The feature vector structure 701 contains a vector with the speech parameters of a time segment. The constant NumberOfParams indicates the number of parameters used. In the preferred implementation it is twenty two: ten cepstrum coefficients, ten first derivatives of the cepstrum coefficients, the energy and the first derivative of the energy. feature vector is used to display the time segments of both the test pattern and the reference pattern.

The sync fragment structure 702 contains the data of a synchronization fragment at a synchronization point. The pointer RefPattern points to the reference word in which the synchronization fragment occurs. In type it is specified whether it is an SP with increasing speech energy (START) or an SP with falling energy (STOP). PositionInPattern specifies the position of the synchronization fragment in the reference pattern. The number of synchronization fragments is counted from the beginning of the reference pattern. NumberOfContextVectors specifies the number of vectors that are used to classify the synchronization fragment. The preferred implementation uses four vectors.

For each class of synchronization fragments, an object of the structure fragment class 704 created. In addition to specifying the type of synchronization fragment (START, STOP) in type, it contains a field of vectors B [NumberOfContextVectors]. These vectors form the reference pattern of the class. The Set * Fragments contains pointers to all sync fragment objects that belong to the corresponding class.

For each reference pattern there is an object of the structure reference pattern 705 created. NumberOfFrames specifies how many time segments (frames) the pattern consists of. NumberOfSP indicates how many synchronization points the pattern contains. The template-list pointer points to the beginning of a chain of templates 951 , The field of vectors B [NumberOfFrames] contains the feature vectors of the reference pattern. The entries in the TimeToEnd [NumberOfSP] field indicate for each SP the time between the corresponding SP and the end of the reference pattern.

When the test pattern is searched for synchronization fragments, an object of the structure template is found for each reference word which has a synchronization fragment of the class of a found synchronization fragment 706 , if none already exists or the existing ones are not suitable, created. These objects are the templates described above 951 , They are managed in lists. Each reference word can have such a list. In the data structure reference pattern belonging to the reference word 705 points template list to the first object of the respective list. In each template object, the next pointer points to the next object in the list. The PreviousSyncPoint variable specifies which synchronization fragment of the template 951 last agreed during the ongoing search for synchronization fragments, ie to what extent the comparison of reference word and test pattern has progressed. timeout indicates the time at which the object is removed from the list at the latest.

Using the pseudo code in 8th the preselection of reference words will now be explained.

The FirstSyncPoint (TestPattern) function call 810 searches for the first synchronization point in test patterns, determines the class belonging to the associated synchronization fragment and provides a pointer to the data structure of the class to which the synchronization fragment belongs. The following while loop ( 812 to 894 ) is run through once for each synchronization fragment in test patterns, where c always points to the class of the respective synchronization fragment.

For each class of synchronization fragments 704 (see. 7 ) belongs to a set of objects with the structure sync fragment 702 (see. 7 ). Each of these objects has, among other things, a pointer to a reference word (RefPattern), in which a synchronization fragment of this class occurs, and a variable (PositionInPattern), which indicates the position of the synchronization fragment within the relevant reference word. If a reference word contains several synchronization fragments of the same class, the set * fragments contains, in addition to the sync fragment objects for other reference words, a corresponding number of sync fragment objects projects 702 for the reference word. The for loop 814 to 890 is run through once for each element of the * fragment set. The fragment _i pointer points to a different sync fragment object each time it is run. In 816 the pointer w _i to the reference pattern object 705 of the reference word belonging to * fragment _i , in 818 the position of the synchronization fragment in the reference word is assigned to the variable PosInPat _i . A list of templates (objects of the structure template 706 (see. 7 .)) belong. In 820 the pointer p is set to the beginning of the list of templates of the current reference word. If (query in 822 ) the reference word does not yet have a list of templates, is in 826 creates a new object for a template of the reference word. The new object will be in 828 first (and currently only) entered in the list of templates of the reference word. In 830 the position of the current fragment (in the reference word) is assigned to the variable PreviousSyncPoint. This variable serves to ensure the order of the synchronization fragments in the respective template. In 831 a point in time (timeout) is determined up to which the pattern belonging to the reference word of the template can be stretched to the maximum. The calculation is based on the current time at the SP (time). The constant MaxSprF specifies the factor by which the remaining time (SP to end of word) can be extended. In the preferred implementation, MaxSprF = 3 is set.

If the list with templates of the reference word was not empty (query 822 , else branch 832 ), the while loop ( 834 to 868 ) run through once for each object in the list. In a template, PreviousSyncPoint specifies the position of the last matching synchronization fragment. If (query 836 and 840 ) the position of the new fragment in * fragment _i lies behind the PreviousSyncPoint, but between the last matching synchronization point PreviousSyncPoint and the current synchronization point but there are one or more further synchronization points in the reference word (PosInPat ≠ PreviousSyncPoint + 1), in 846 and 848 creates a duplicate of the template. The duplicate is in line 847 inserted in a temporary list of NewTemplates. The structure and management of the NewTemplates list is obvious and therefore not shown. In 854 the template's PreviousSyncPoint is set to the position of the current synchronization fragment and in 855 the value for timeout is determined anew based on the new synchronization fragment. If the current synchronization point in the reference word is not directly behind the previously entered synchronization point, there are now two templates; one that entered the new synchronization fragment (previousSyncPoint and timeout changed) and one without the new synchronization fragment. This method prevents a pattern that is rated as SP in the test pattern, but not in the corresponding part of the reference word, from confusing the recognition of the synchronization fragment sequence.

If (query 836 , else branch 856 ) the position of the new fragment is in front of or on PreviousSyncPoint, this reference does not fit into the current template. In 858 to 863 a new template is created. Their variable PreviousSyncPoint is in line 862 set to the position of the synchronization fragment of the current pattern fragment. In 863 the timeout (as in 831 ) certainly.

After all active templates of a reference word in the loop 836 to 868 were processed in 874 any newly created templates in the NewTemplates list are added to the template list. The NewTemplates list is then empty again.

In the following for loop 876 to 888 all templates of the reference word are searched according to two criteria: timeout exceeded or the last synchronization point of the reference word reached. If one of the two criteria is met, the reference word with the test pattern is entered by the executeDTW (p, test pattern) procedure call 884 compared. The determination of the "finished" templates and the subsequent DTW comparison of reference word and test pattern can also take place at a different point in the overall process. It was only inserted at this point to illustrate how the preselection of reference words works.

The NextSyncPoint (TestPattern) function 892 searches for the next SP in the test pattern, determines the class belonging to the associated synchronization fragment and provides a pointer to the data structure of the class to which the synchronization fragment belongs.

9 shows an extension of the template structure. When creating a new template, the number of synchronization points of the corresponding reference word is specified as parameter NmOfSP. The structure now also contains a flag for each SP in the field SPoint [], one element each in the fields RPframe [], TPframe [] and d []. The elements of the SPoint [] field are initialized with the value FALSE. When a suitable synchronization fragment pair is found on the i-th SP, the corresponding element SPoint [i] is set to TRUE, the elements of the fields RPframe [i] and TPframe [i] are the numbers of the time segment of the SP in the reference word or in Test pattern assigned, and the distance of the two synchronization fragments is stored in d [i]. The fields SyPoint [], RPframe [] and TPframe [] are used in the subsequent DTW comparison to subdivide the search space. The sum of the distances in d [] gives a first statement about the similarity of the reference word and test pattern. They can be used to further narrow down the reference words to be compared using DTW.

Formation of SF classes

The synchronization fragment classes can with the known methods for class formation and class assignment can be formed from a set of speech patterns. In the simplest Version, synchronization points are searched for on existing patterns and the associated Sync fragments compiled. Then both in the Literature as “dynamic clustering "as also the "hierarchical known clustering Procedures are used.

Be in a modified form the classes are made up of several speech patterns of a word. First all potential SF of all language patterns of a word are localized, then the SF of the different speech patterns are assigned to each other. The SF is assigned based on its position in the word, and due to the energy curve at the associated SP. The assigned to each other SF then form a class. SF that cannot be clearly assigned ignored. This means that each word has its own own classes. This can be a comparatively good one Preselect reference words (when searching later). The higher Effort in determining the SF class (when searching later) can be determined by a appropriate design of the search procedures are reduced, e.g. B. by search trees. In a second step you can the many (word-specific) classes with the known methods to cross-word Classes are merged.

Determination of the beginning of the word, Formation of the borders of the reference pattern

In some applications, such as that in the patent application DE 100 54 583 A1 described methods, over time a series of speech patterns are known to represent the same word. In other applications, such patterns are present from the beginning, e.g. B. by audition in training. The problem with these applications can be to determine the beginning or end of the respective word in the speech pattern. Patterns with known word boundaries can e.g. B. are required as a reference pattern.

To determine the word boundaries, the synchronization points in the speech patterns are first determined. In the case of speech patterns which are known to represent the same words, the procedure is then as follows: the synchronization points of two speech patterns are assigned to one another. This can be done on the basis of their sequence and (additionally) by classifying the synchronization fragments of the SP. The beginning of a word is then determined by "backward search" starting from the first synchronization point, the end of the word by "forward search" starting from the last synchronization point. The search is carried out by means of "dynamic programming", ie the time segments of the two patterns are assigned to one another in such a way that a path with the smallest possible distance values between the feature vectors of the respectively assigned segments is created. The search is terminated for those time segments to which there are no assignments of time segments with distance values d (i, j) <D _S. The threshold value D _S depends on the application (background noise, etc.) and can be determined, for example, by determining the values of d () during speech pauses Alternatively, the search can be terminated if d (i, j) exceeds the threshold value in a certain number n _s of m successive segments. The time segments at which the search is terminated form the word boundaries in the corresponding speech pattern. if there are several speech patterns of a word, be repeated with other pairings of speech patterns r are not taken into account. Outliers are comparisons in which the word boundaries differ significantly from those determined by other pairings. In addition, the known methods, e.g. B. Determining the beginning and end of words using criteria relating to total energy or energy content in individual frequency bands.

The invention is not limited to the embodiments shown here. Rather, it is possible by combining and modifying the means and features mentioned further versions to realize without leaving the scope of the invention. In particular can instead of the "similarities" or the distance functions, also analogous to "probabilities", such as the "probability for the match of two segments ", be used. Furthermore, in the present description Time segments from 10-20 ms used as an example. It is also possible according to the invention, other lengths of To use time segments. In particular, based on individual Samples (the length of one Time segment then corresponds to a test value), the calculation of the speech characteristics then sliding using of the surrounding samples. Likewise, the individual time segments different lengths to have. So z. B. each time segment represent a phone.

101 201

waveform of the word "Jupiter"

102 202

scale of the time segments

103 203

spectrogram of the word "Jupiter"

104 204

timescale

105 205

energy content at the beginning of the word

106 206

energy content at the end of the word

110 111, 112

synchronization points

210 211, 212, 213

synchronization points

301

optimal path

302 303

diagrams the total energy of speech patterns

310

overlap of time segments

401

medium for data entry, microphone

402

amplifier

403

Bandpass filter

404

Analog to digital converter, A / D converter

405

digital Signal processor (DSP)

406

Processor, CPU

407

R.A.M.

408

ROME

409

medium for data output

421

unit to divide the samples into

Time segments, Framer

422

High Pass Filter

423

unit to multiply the segment values

With a window function

424

unit to carry out a discreet

Fourier transform (DFT)

425

unit for logarithmization of the DFT

Results

426

unit to carry out an inverse DFT

427

unit for energy calculation

428

unit to form the first derivative

429

unit to determine

synchronization points

430

unit to determine

synchronization fragments

431

unit to carry out a dynamic

time adjustment (DTW)

432

Storage means, preferably for storage

of reference patterns

701

feature vector structure

702

sync fragment structure

704

fragment class structure

705

reference pattern-structure

706

template structure

810 ... 894

Line numbers of the pseudo code

901

synchronization point with falling energy

902 903

synchronization points with increasing

energy

906

Advanced template structure

911, 912, 913

 synchronization fragments

921, 922, 923

 Synchronization fragment classes

930

Synchronization fragment Representative

931

Synchronization fragment object

941, 942

 Reference pattern object RP object

951

template

Claims (43)

Method for the synchronization of test and reference patterns, in particular for the synchronization of speech patterns, with test and reference patterns each as a sequence A = (a (1), a (2), ..., a (I)) or B = ( b (1), b (2), ..., b (J)) of feature vectors (assigned to time segments) a (i) (i = 1, 2, ..., I) or b (j) (j = 1, 2, ..., J) are present, characterized in that the synchronization takes place via synchronization points (SP) which are predetermined and / or automatically determined in the test and reference patterns, with - a part of the feature vectors b (j) of the reference patterns is marked as a reference SP, - the feature vectors a (i) of a test pattern are searched for predeterminable features and those feature vectors are identified as potential SP that have at least one of the predefined features, - at least part of the potential SP determined according to predefinable rules or is assigned to several reference SP and - if predefined criteria are met the synchronization of test and reference patterns is automatically established via mutually assigned SP. Method according to claim 1, characterized, that the Potential SP is determined using the following features: - difference the energy of successive time segments and / or - difference the energy of certain frequency bands of successive time segments and / or - Modification the number of zero crossings of the speech signal in successive time segments and / or - based of cepstrum, LPC and / or PARCOR coefficients and / or based on the derivatives of these coefficients. A method according to claim 2, characterized in that for Determination of potential SP frequency bands weighted differently become. Method according to one of the preceding claims, characterized characterized that the predefinable rules for the assignment of potential SP to reference SP for at least part of the potential SP determined an analysis of additional, preferably temporally Environment of the potential SP arranged feature vectors of the test pattern prescribe and assign potential SP to reference SP dependent on of the results of this analysis. Method according to one of the preceding claims, characterized characterized that the to determine potential SP analyzed features with each other combined and / or with mathematical functions, such as in particular the logarithm function become. Method according to one of the preceding claims, characterized characterized that for Determining potential SP the characteristics of at least two directly successive time segments are analyzed. Method according to one of the preceding claims, characterized characterized that the Synchronization determining a degree of similarity (score) of reference and test patterns. Method according to one of the preceding claims, characterized characterized that a Average of characteristics analyzed to determine potential SP continuously from within a predefinable time window lying samples is determined (moving average). Method according to one of the preceding claims, characterized characterized that stop consonants and / or feature vectors characterizing explosive sounds as reference synchronization points serve. Method according to one of the preceding claims, characterized in that to determine potential SP the absolute value of the ratio Δ _i / e _{i is} evaluated, wherein - Δ _{i is} the difference in the energy of the time segments i and i-1 and - e _i denote the mean energy of a predeterminable number of time segments surrounding the time segment i, and a time segment i is characterized as SP in that | Δ _i / e _i | exceeds a predeterminable threshold value. Method according to one of the preceding claims, thereby characterized in that to synchronization - the or the determined reference pattern and / or the test pattern (s) output, made available or handed over to other applications be or - one dynamic time adjustment (DTW, Dynamic Time Warping) of test and Reference pattern (s) or an analysis of the test pattern by a hidden Markov model (HMM) carried out and then the determined reference pattern (s) and / or the test pattern (s) output, made available or handed over to other applications become. A method according to claim 11, characterized in that the Output takes place acoustically and / or visually. Method according to one of the preceding claims, thereby characterized in that a dynamic time adjustment between a reference and a test pattern takes place when - for all reference SP of the reference pattern or - for a definable Number of reference SP of the reference pattern an assignment to SP of the test sample has been produced or - a predeterminable ratio between the total number of reference SP of the reference pattern and the number of Reference SP of the reference pattern for which an assignment of SP a test pattern has been reached, reached or exceeded. A method according to claim 13, characterized, that the number of possible considered in the dynamic time adjustment Paths and thus the search space can be restricted by - in one Path at least one index, i or j, to one at each step predeterminable value increased will and / or - in a path for each step both indices at the same time Predeterminable values increased be and / or - Number successive steps parallel to an axis and or - the Find the optimal path on the diagonal and a given one Number of time segments on both sides of the diagonal is limited and or - the dynamic time adjustment only for the intervals between the SP is executed. Method according to one of claims 13 or 14, characterized in that that at dynamic time adjustment for intervals between SP the Search space is expanded by adding time segments from the search space temporal environment of the SP can be added. Method according to one of claims 13 to 15, characterized in that that a Dynamic time adjustment (DTW) using "Dynamic Programming" (DP) or with the help of the Viterbi algorithm or using hidden Markov models carried out becomes. Method according to one of claims 13 to 16, characterized in that that the Path search for dynamic time adjustment from the first SP of one Test pattern beginning at the beginning of the word (backwards) and / or from the last SP a test pattern starting at the end of the word (forward). A method according to claim 17, characterized in that in the case of an automatic determination of word boundaries of a speech pattern, the path search for the time segments is terminated, - in which no assignments of time segments can be found for which the distance value d (i, j) and a predefinable threshold value D _S fulfill the condition d (i, j) <D _S , or - if the distance value d (i, j) in a predeterminable number of successive time segments the smolder lenwert D _S exceeds, and the time segments at which the search was terminated are marked as a word boundary. A method according to claim 18, characterized in that the threshold value D _{S is} predetermined as a function of the application, preferably taking background noise into account. Method according to Claim 18 or 19, characterized in that the threshold value D _S is determined by determining the values for d () in speech pauses. Method according to one of claims 18 to 20, characterized in that that at Presence of multiple reference speech patterns representing the same word the automatic determination of word boundaries of a speech pattern is performed with several of these reference speech patterns. Method according to one of the preceding claims, thereby characterized in that at the method, in particular for calculating distance functions, the following parameters are determined: - Cepstrum coefficients and / or - LPC coefficients (Linear Predictive Coding) and / or - PARLOR coefficients and / or - LAR coefficients and or - LSP coefficients and or - LSF coefficients and or - spectral energy distribution and or - MEL spectrum and or - zero crossing rate (zero crossing rate) and / or - Mel or Bark transformations the aforementioned coefficients and / or - Time derivatives of the aforementioned coefficients and / or their Mel or Bark transformations and / or - combinations these coefficients and / or parameters in smoothed and unsmoothed form. Method according to one of the preceding claims, characterized characterized that reference SP at least be assigned to a class of synchronization fragments (SF), an SF comprises feature vectors which are in temporal surroundings a reference SP of a predetermined reference pattern are. A method according to claim 23, characterized in that a Classification of SP and / or synchronization fragments takes place such that SP and / or synchronization fragments with increasing energy first class and SP and / or synchronization fragments with falling Energy can be assigned to a second class of SP. Method according to one of claims 23 or 24, characterized in that that synchronization fragments of immediately successive feature vectors of a speech pattern be formed. Method according to one of claims 23 to 25, thereby characterized in that one SP associated with increasing energy synchronization fragments are formed by feature vectors that are temporal in the pattern are arranged after the SP and / or an SP with falling Energy-associated synchronization fragments of feature vectors are formed, which are arranged in the pattern before the SP are. Method according to one of claims 23 to 26, characterized in that that the membership an SP to a class of synchronization fragments by dynamic Time adjustment takes place. Method according to one of claims 23 to 27, characterized in that - to determine the affiliation of an SP to a class of synchronization fragments - distance functions are used which are particularly meaningful in the area of phonetic transitions, or - other synchronization functions are used for synchronization fragments with increasing energy content than with synchronization fragments with falling energy content. Method according to one of the preceding claims, characterized characterized that after Determine the SP of a test pattern by comparing the number of these SP with the number of reference SPs of at least some of the reference patterns done and in agreement the number an assignment between SP and reference SP in temporal Order of their appearance in the respective language pattern becomes. A method according to claim 29, characterized in that after assignment of the SP and reference SP a test of the SP pairs this is followed by whether both SP of a pair belong to the same class of Heard synchronization fragments. Method according to one of the preceding claims, thereby characterized in that reference patterns a list of data structures (templates) is assigned which Information on the reference SP of the reference pattern, in particular information - about done Assignments of SP from test patterns to reference SP of the reference pattern and or - on the Degree of similarity (score) of reference and test samples includes. Method according to one of the preceding claims, characterized characterized that after assignment of an SP of a test pattern to a reference SP the SP is included in the set of reference SP. Method according to one of the preceding claims, characterized characterized that after synchronization of a test and reference pattern the test pattern is included in the set of reference samples. Method according to one of claims 11 to 33, thereby characterized in that the execution the DTW starts with the reference pattern, - which is the highest number of assigned SP or - which is the best ratio of Assignments to the number of reference SP of the reference pattern has or - For which the best mean similarity the synchronization fragments of the reference pattern to the synchronization fragments of the test pattern was determined, the mean similarity as the sum of the similarities of the individual fragments for each class divided by the Number of reference SP of the reference pattern is defined. Arrangement with at least one processor that is (are) set up in such a way that a method for synchronization of test and reference patterns, especially for synchronization of speech patterns, with test and reference patterns each as a sequence A = (a (1), a (2), ..., a (I)) or B = (b (1), b (2), ..., b (J)) of (associated with time segments) feature vectors a (i) (i = 1, 2, ..., I) or b (j) (j = 1, 2, ..., J) are present, can be carried out, being the synchronization over in the test and Reference patterns predefined and / or automatically determined synchronization points (SP) takes place where - on Part of the feature vectors b (j) of the reference pattern as a reference SP is awarded - the Feature vectors a (i) of a test pattern according to predefinable features searched and those feature vectors marked as potential SP which have at least one of the specified characteristics, - at least Part of the potential SP determined according to predefinable rules is assigned to one or more reference SPs and - if it can be specified Criteria the synchronization of test and reference patterns over each other assigned SP is automatically created. Arrangement according to claim 35, characterized by means for data input ( 401 ) and edition ( 409 ), at least one amplifier ( 402 ), at least one bandpass filter ( 403 ), at least one analog-to-digital converter ( 404 ), at least one digital signal processor (DSP) ( 405 ) at least one processor ( 406 ) with RAM ( 407 ) and read-only memory (ROM) ( 408 ), data inputs and outputs of these units being connected to one another in such a way that signal transmission from the (one) means for data input ( 401 ) via amplifier ( 402 ), Bandpass filter ( 403 ) and analog-digital converter ( 404 ) to the (a) digital signal processor (DSP) ( 405 ) is feasible, RAM ( 407 ) and read-only memory (ROM) ( 408 ) with the processor (s) ( 406 ), the processor (s) ( 406 ) with the digital signal processor (s) (DSP) ( 405 ) and at least one processor ( 406 ) and / or digital signal processor (DSP) ( 405 ) with means for data output ( 409 ) are connected by means of data exchange. Arrangement according to one of claims 35 or 36, characterized in that the arrangement comprises means for data exchange with external data processing devices and / or for acoustic and / or visual data output ( 409 ) having. Arrangement according to one of Claims 36 or 37, characterized in that the digital signal processor (DSP) ( 405 ) and / or processor ( 406 ) - at least one framer ( 421 ), - at least one unit for calculating language parameters of the language segments, - at least one unit for determining the synchronization points ( 429 ), - at least one unit for compiling the synchronization fragments ( 430 ), - at least one unit for performing a dynamic time adjustment (DTW, Dynamic Time Warping) ( 431 ), - storage means ( 432 ), preferably for storing reference patterns, with data inputs and outputs of these units being connected to one another in such a way that signal transmission from the (a) framer ( 421 ) via (the) unit (s) to calculate language parameters of the language segments, (the) unit (s) to determine the synchronization points ( 429 ) and (the) unit (s) for compiling the synchronization fragments ( 430 ) to the unit (s) for performing a dynamic time adjustment (DTW, Dynamic Time Warping) ( 431 ) is feasible, the (the) storage means ( 432 ) via means of data exchange with the unit (s) for carrying out a DTW ( 431 ) and the data input (s) of the framer (s) with data inputs from the digital signal processor (DSP) ( 405 ) and / or processor ( 406 ) and the data output (s) of the unit (s) for performing a DTW ( 431 ) with data outputs from digital signal processor (DSP) ( 405 ) and / or processor ( 406 ) are connected. Arrangement according to claim 38, characterized, that a Unit for calculating language parameters of the language segments - Medium to calculate the cepstrum coefficients and / or - Medium to calculate the LPC coefficients (Linear Predictive Coding) and or - Medium to calculate the PARLOR coefficients and / or - Medium to calculate the LAR coefficients and / or - Medium to calculate the LSP coefficients and / or - Medium to calculate the LSF coefficients and / or - Medium to calculate the spectral energy distribution and / or - Medium to calculate the MEL spectrum and / or - Means for calculating the Zero crossing rate and / or - Medium to calculate the Mel or Bark transformations of the aforementioned Coefficients and / or - Medium to calculate the time derivatives of the aforementioned coefficients and / or their Mel or Bark transformations and / or - Medium for combinations of these coefficients and / or parameters in smoothed and unsmoothed Form includes. Arrangement according to claim 39, characterized in that a means for calculating the cepstrum coefficients - at least one unit for performing a discrete Fourier transformation (DFT) ( 424 ), - at least one unit for logarithmizing the DFT results ( 425 ), - at least one unit for performing an inverse DFT ( 426 ), - at least one unit to form the first derivative ( 428 ), with data inputs and outputs of these units being connected to one another in such a way that a signal transmission from the unit (s) for performing a discrete Fourier transform (DFT) ( 424 ) via (the) unit (s) for logarithmizing the DFT results ( 425 ), (the) units) for performing an inverse DFT ( 426 ) to the unit (s) to form the first derivative ( 428 ) can be carried out, the data input of the unit for calculating speech parameters of the speech segments from the data input (s) of the unit (s) for performing a discrete Fourier transformation (DFT) ( 424 ) and the data output of the unit for the calculation of language parameters of the speech segments from the data output (data outputs) of the units) to form the first derivative ( 428 ) are formed. Computer program product that is a computer readable Storage medium includes on which a program is stored that enables a computer to after it has been loaded into the computer's memory Methods for the synchronization of test and reference patterns, in particular for the synchronization of speech patterns, whereby test and reference patterns each as a sequence A = (a (1), a (2), ..., a (I)) or B = (b (1), b (2), ..., b (J)) of (associated with time segments) feature vectors a (i) (i = 1, 2, ..., I) or b (j) (j = 1, 2, ..., J) are present, whereby the synchronization over predefined and / or automatic in the test and reference samples determined synchronization points (SP) takes place, whereby - a part of the feature vectors b (j) of the reference pattern as a reference SP becomes, - the Characteristic vectors a (i) of a test pattern according to predefinable characteristics searched and those feature vectors marked as potential SP which have at least one of the specified characteristics, - at least Part of the potential SP determined according to predefinable rules is assigned to one or more reference SPs and - if it can be specified Criteria the synchronization of test and reference patterns over each other assigned SP is automatically created. Computer-readable storage medium on which a program is saved, which allows a computer after it is in memory of the computer has been loaded, a method for synchronizing Test and reference samples, especially for the synchronization of Speech patterns, with test and reference patterns each as a sequence A = (a (1), a (2), ..., a (I)) or B = (b (1), b (2), ..., b (J)) of (time segments assigned) feature vectors a (i) (i = 1, 2, ..., I) or b (j) (j = 1, 2, ..., J) are to be carried out, with the synchronization via in the Test and reference samples predefined and / or automatically determined Synchronization points (SP) takes place, whereby - part of the feature vectors b (j) the reference pattern is designated as a reference SP, - the feature vectors a (i) a test pattern is searched for predeterminable features and those feature vectors are identified as potential SP, which have at least one of the specified characteristics, - at least Part of the potential SP determined according to predefinable rules is assigned to one or more reference SPs and - if it can be specified Criteria the synchronization of test and reference patterns over each other assigned SP is automatically created. Database containing information such as, in particular, lists of similarities of speech patterns or synchronization fragments and / or class details to include synchronization fragments, which by a method according to a of claims 1 to 34 were won.