KR20020052196A - Pattern matching method and apparatus - Google Patents

Abstract

According to the present invention, a system is provided for matching two or more sequences of phonemes that may be generated from text or speech. Preferably, a dynamic programming matching technique is used that has constraints that are weighted by phoneme confusion score, phoneme insertion score and phoneme deletion score where scoring of the dynamic programming path is appropriate, depending on whether the two sequences are generated from text or speech do.

Description

[0001] PATTERN MATCHING METHOD AND APPARATUS [0002]

BACKGROUND OF THE INVENTION A database of information is known and there exists a problem of a method of locating and retrieving desired information quickly and efficiently from a database. Existing database search tools use typed keywords to allow users to search the database. While this is fast and efficient, this type of search is not appropriate for a variety of databases such as video or audio databases.

A video and audio database is annotated with phonetic transcription of speech content in audio and video files so that subsequent retrieval can be done by comparing the phoneme notes in the database with the voice recordings of the user's input query This has been done recently. A technique proposed to match a sequence of phonemes first defines a set of characteristics in each taken query as overlapping fixed-size fragments from a phoneme string, then identifies the frequency of occurrence of the features in both the query and annotation, Finally, a measure of the similarity between query and annotation is determined using the cosine measure of frequency. The advantage of this kind of phoneme comparison technique is that it can cope with situations where the word sequence of the query does not exactly match the word sequence of the annotation. However, we experience the problem that errors tend to occur, especially when the query and annotation are spoken at different rates and there is a partial deletion of the word from the query, but not from the annotation, or vice versa.

The present invention relates to an apparatus and method for matching sequences of phonemes or the like. The present invention may be used to search a database of data files having associated phonetic annotations in response to a user ' s input query. The input query may be a voiced or typed query.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a schematic block diagram illustrating a user terminal that allows an annotation of a data file to be generated from a typing or voice input from a user.

2 is a schematic diagram of phoneme and word lattice annotation data generated from a typing input by a user to annotate a data file;

3 is a schematic diagram of phoneme and word grid annotation data generated from speech input by a user to annotate a data file;

4 is a schematic block diagram of a user's terminal that allows a user to retrieve information from a database by typing or voice query;

5A is a flow chart illustrating some of the flow control of the user terminal shown in FIG. 4;

FIG. 5B is a flow chart showing a part of the flow control of the user terminal shown in FIG. 4; FIG.

6A is a schematic diagram showing a basic statistical model estimated to generate both a query and an annotation;

6B shows a first sequence of phonemes representing the typing input, a second sequence of phonemes representing the user's speech input, and the possibility of phoneme insertions and deletions from the user ' s voice input to the typing input Fig.

6C shows a query and annotation of a third sequence of phonemes representing a canonical sequence of phonemes corresponding to the phonemes corresponding respectively to the first and second sequences of phonemes respectively representing speech input; A schematic diagram showing a basic statistical model presumed to have been generated.

Figure 7 is a schematic diagram of a search space formed by a sequence of phoneme phonemes and a sequence of query phonemes with a start null node and an end null node;

8 is a two-dimensional plot showing a plurality of lattice points corresponding to possible matches between the annotated phoneme and the query phoneme, with the horizontal axis being provided to the phoneme of the annotation and the vertical axis being provided to the phoneme of the query;

9A is a schematic diagram illustrating dynamic programming constraints employed in a dynamic programming matching process when an annotation is a typing input and a query is generated from a speech input.

9B is a schematic diagram illustrating dynamic programming constraints employed in a dynamic programming matching process when the query is a typing input and when the annotation is a voice input.

10 is a schematic diagram showing deletion and decoding probabilities stored in phonemes;

11 is a schematic diagram illustrating dynamic programming constraints employed in a dynamic programming matching process when annotations and queries are both speech input;

12 is a flow chart illustrating the main processing steps performed in the dynamic programming matching process;

13 is a flow chart illustrating the main processing steps employed to initiate a dynamic programming process by propagating from a null starting node to all possible starting points;

14 is a flow chart showing the main processing steps employed in propagating the dynamic programming path from the starting point to all possible ending points;

15 is a flow chart showing the main processing steps employed in propagating the path from an end point to all possible null end nodes;

16A is a flow diagram illustrating some of the processing steps performed to propagate a path using dynamic programming constraints.

16B is a flow chart illustrating the remaining processing steps involved in propagating a path using dynamic programming constraints.

17 is a flow chart showing the processing steps involved in determining a transition score for propagating a path from a starting point to an ending point;

18A is a flow chart illustrating some of the processing steps employed to calculate score for erasure and decoding of annotation and query phonemes;

FIG. 18B is a flow chart showing the remaining processing steps employed in calculating score for erasure and decoding of annotation and query phonemes; FIG.

19 is a schematic diagram showing a search space formed by two sequences of a query phoneme and one sequence of annotated phonemes together with a start null node and an end null node;

20 is a flow chart illustrating the main processing steps employed to initiate a dynamic programming process by propagating from a null starting node to all possible starting points;

Figure 21 is a flow chart illustrating the main processing steps employed to propagate a dynamic programming path from a starting point to all possible ending points;

22 is a flow chart showing the main processing steps employed in propagating the path from an end point to a null end node;

23 is a flow chart illustrating the processing steps performed to propagate a path using dynamic programming constraints.

24 is a flow chart illustrating the processing steps involved in determining a transition score for propagating a path from a starting point to an ending point;

25A is a flow chart illustrating a first portion of a processing step employed to calculate a score for annotation and query phoneme deletion and decoding.

25B is a flow chart showing a second part of the processing steps employed to calculate scores for erasure and decoding of annotation and query phonemes;

Figure 25c is a flow chart showing a third part of the processing steps employed to calculate the score for annotation and query phoneme deletion and decoding;

FIG. 25D is a flow chart showing a fourth portion of processing steps employed to calculate score for erasure and decoding of annotation and query phonemes; FIG.

Figure 25E is a flow chart illustrating the remaining processing steps employed to calculate scores for erasure and decoding of annotations and query phonemes;

26A is a schematic diagram illustrating an alternative embodiment employing a different technique for aligning each annotation and query;

Figure 26B is a plot showing how the dynamic programming score varies according to a comparison of an annotation and a query in the embodiment shown in Figure 26A.

Figure 27 is a schematic block diagram illustrating a form of an alternate user terminal operable to retrieve a data file from a database located in a remote server in response to an input voice query;

28 is a schematic block diagram illustrating another user terminal that enables a user to retrieve data from a database located within a remote server in response to an input voice query;

It is an object of the present invention to provide an alternative system for retrieving a database.

According to one aspect, the present invention provides an apparatus comprising: means for receiving a characteristic of a first sequence and a characteristic of a second sequence; Means for aligning characteristics of the second sequence and characteristics of the first sequence to form a plurality of aligned property pairs; Means for comparing characteristics of each ordered property pair to produce a comparison score representing similarity between the ordered property pairs; Characterized in that it comprises means for combining a comparison score for all ordered characteristic pairs to provide a measure of the similarity between the characteristics of the first sequence and the characteristics of the second sequence, Comparing the plurality of characteristics of the first sequence with the characteristics of the first sequence of the aligned pair against each aligned pair to determine a corresponding plurality of intermediate comparison scores representing the similarity between the characteristics of the first sequence and the respective characteristics from the set First comparing means for comparing the first comparison result with the second comparison result; Comparing each of the plurality of characteristics from the set with the characteristics of the second sequence of the aligned pairs for each ordered pair to determine a further correspondence indicating a similarity between the characteristics of the second sequence and the respective characteristics from the set Second comparison means for providing a plurality of intermediate comparison scores; And means for calculating a comparison score for the aligned pair by combining a plurality of intermediate comparison scores. Such a system has an advantage considering the change in both the characteristics of the first sequence and the characteristics of the second sequence due to the misrecognition of the characteristics by the recognition system.

According to another aspect, the present invention provides an apparatus comprising: means for retrieving a database of a plurality of information entries each comprising a sequence of voice characteristics to identify the information to be retrieved, and receiving an input query comprising a sequence of voice characteristics; Means for comparing a query sequence of voice characteristics with a database sequence of each voice characteristic to provide a set of comparison results; An apparatus comprising means for identifying information to be retrieved from a database using a comparison result, the comparison means having a plurality of different comparison modes of operation, the method comprising the steps of: (i) determining whether a query sequence of speech characteristics is generated from the audio signal, Created; And (ii) means for determining whether a current database sequence of speech characteristics has been generated from an audio signal or from text and outputting a determination result; And means for selecting an operation mode of the comparison means for the current database sequence according to the determination result. Preferably, when the determining means determines that both the input query and the commentary are generated from the speech, the comparing means operates like the above-described apparatus.

According to another aspect, the invention provides an apparatus for searching a database comprising a plurality of information entries each having an associated annotation that includes a sequence of voice annotation properties to identify the information to be retrieved, the apparatus comprising: a plurality of audio renditions means for receiving a rendition; Means for transforming each rendition of the input query into a sequence of voice query characteristics representing the voice in the rendition; Means for comparing a voice annotation characteristic of each annotation with a voice quality characteristic of each rendition to provide a set of comparison results; Means for combining the comparison results obtained by comparing the voice annotation characteristics of the same annotation with the voice query characteristics of each rendition to provide a measure of similarity between the input query and the annotation for each annotation; And means for identifying information to be retrieved from the database using similarity measures provided by the combining means for all annotations.

According to another aspect of the present invention, there is provided an apparatus comprising: means for receiving first and second sequences of query characteristics, each representing a rendition of an input query; Means for receiving a sequence of annotation properties; Means for aligning the annotation properties of the annotation and the quality characteristics of each rendition to form a plurality of ordered property groups each comprising a query property and an annotation property from each rendition; Means for comparing characteristics of each ordered property group to generate a comparison score that indicates a similarity between properties of the sorted group; Means for combining the comparison scores for all ordered property groups to provide a measure of similarity between the input query and the renditions of the annotation, wherein the comparing means comprises means for comparing Comparing a plurality of characteristics and characteristics of the first query sequence of the sorted group against each sorted group to determine a corresponding plurality of intermediate comparison scores representing similarities between the characteristics of the first query sequence and respective characteristics from the set A first characteristic comparator for providing a first input signal; Comparing each of the plurality of characteristics from the set and the characteristics of the second query sequence of the sorted group against each of the sorted groups to determine the similarity between the characteristics of the second query sequence and the respective characteristics from the set A second characteristic comparator for providing a corresponding plurality of intermediate comparison scores of the second characteristic comparator; Comparing each of the plurality of properties from the set with the annotation properties of the sorted group for each ordered group to determine an additional corresponding plurality of intermediate comparison scores representing similarity between the annotation properties and respective characteristics from the set A third characteristic comparator for providing the second characteristic comparator; And means for calculating a comparison score for the sorted group by combining a plurality of intermediate comparison scores.

Now, an exemplary embodiment of the present invention will be described with reference to FIGS. 1 to 28. FIG.

Although embodiments of the present invention may be practiced using dedicated hardware circuitry, the embodiments described below may be implemented in a personal computer, a workstation, a photocopier, a facsimile machine, a personal digital assistant (PDA) And is implemented in computer software or code operating in conjunction with processing hardware.

Comment the data file

1 shows a form of a user terminal 59 that allows a user to input a typing or voice annotation via the keyboard 3 and microphone 7 to annotate a data file 91 stored in the database 29. [ . In the above embodiment, the data file 91 includes a two-dimensional image generated by a camera, for example. The user terminal 59 allows the user 39 to annotate the 2D image with an appropriate annotation that can then be used to fetch the 2D image from the database 29. In this embodiment, the typing input is converted into phonemes (such as phonemes) and word grid annotation data that are passed to the control unit 55 by the voice recording unit 75. Fig. 2 shows the form of the phoneme and word grid annotation data generated for the typing input " picture of the Taj Mahal ". As shown in Figure 2, phonemes and word grids are acyclic directed graphs with a single entry point and a single entry point, which represent different parses of the user's input. As shown, the voice recording unit 75 identifies a number of different possible phonetic strings corresponding to the typing input from an internal phonetic dictionary (not shown).

Similarly, the speech input is converted into phonemes (such as phonemes) and word grid annotation data that are passed to the control unit 55 by an automatic speech recognition unit 51. The automatic speech recognition unit 51 generates (i) a phoneme lattice for input utterance; (Ii) Next, identify the words in the phonetic lattice; (Iii) Finally, these two are combined to generate such phoneme and word grid annotation data. Fig. 3 shows the form of phonemic and word grid annotation data generated for the input utterance " picture of the Taj Mahal ". As shown, the automatic speech recognition unit identifies a number of different possible phoneme strings corresponding to this input pronunciation. As is known in the speech recognition arts, these different possibilities may have their own weightings, which are generated by the speech recognition unit 51 and represent the output reliability of the speech recognition unit. However, in the above embodiment, the weight of such a phoneme is not performed. As shown in Fig. 3, the words that the automatic speech recognition unit 51 identifies in the phoneme lattice are incorporated into the phoneme lattice data structure. As shown for the example sentence provided above, the automatic speech recognition unit 51 includes the words "picture", "of", "off", "the", "other" quot ;, " ah ", " hal ", " ha "

As shown in FIG. 3, the phoneme and word grid generated by the automatic speech recognition unit 51 are acyclic graphs having a single entry point and a single entry point. This represents a different syntax analysis of the user's input annotation. Each word does not need to be replaced with a single substitute, and one word can be replaced with two or more words or phonemes, and the entire structure can form a substitute for one or more words or phonemes, It is not a word sequence. Thus, the data density in the phoneme and word grid annotation data is maintained linearly throughout the annotation data, rather than growing exponentially as in the case of a system that basically produces an N-best word list for the audio annotation input.

In the above embodiment, the annotation data generated by the automatic voice recognition unit 51 or the voice recording unit 75 has the following general form.

Header

- flag Whether words, phonemes, or mixtures

- a time index associated with the block location of the annotation data in the memory up to a point in time

- the set of words used (ie, the dictionary)

- Phoneme set used

- the language the vocabulary belongs to

- Phoneme probability data

Block (i) i = 0, 1, 2, .....

Node N _j j = 0, 1, 2, .....

- the time offset of the node from the beginning of the block

- phoneme link (k) k = 0, 1, 2, .....

- offset for N _j (N _k is the node that links K extend) node N _j = _k N

Phoneme associated with link (k)

- Word link (l) l = 0, 1, 2, .....

- an offset for a node N _j = N _i - N _j [N _j is the node from which the link (l) extends)

Words related to link (l)

Since not all the data files in the database contain the combined phoneme and word grid annotation data discussed above, a flag is provided identifying whether the annotation data is word annotation data, phoneme annotation data, or mixed, A different search strategy is used to retrieve this annotation data.

In the above embodiment, the annotation data is divided into blocks of the node so that the search can be skipped in the middle of annotation data for a given search. Thus, the header includes a time index associated with the block location of the annotation data in the memory up to a predetermined time offset between the start time and the time corresponding to the beginning of the block.

The header also includes data defining a set of words to be used (i.e., dictionaries), a set of phonemes to be used, their probability, and the language to which the vocabulary belongs. The header may also include details of the automatic speech recognition system used to generate annotation data and any appropriate settings used during the generation of this annotation data.

Next, a block of annotation data follows the header, and includes a time offset of the node from the beginning of the block, a phoneme link that links the node to another node by phonemes, and a word link ≪ / RTI > for each node of the block. Each phoneme link and word link identifies the phonemes and words associated with the link. They also identify the offset for the current node. For example, the node N ₅₀ is linked to the node N ₅₅ by a phoneme link, and the offset to the node N ₅₀ is 5. As will be appreciated by those skilled in the art, the use of such an offset indication allows the partitioning of consecutive annotation data into separate blocks.

In embodiments in which the automatic speech recognition unit outputs weights indicative of the reliability of the speech recognition unit outputs, these weights or reliability scores are included in the data structure. In particular, a reliability score is provided for each node, indicating reliability to arrive at the node, and each phoneme and word link includes a transition score according to a weight given to the corresponding phoneme or word. These weights are then used to control retrieval and retrieval of data files by discarding such matching with a low reliability score.

In response to the user's input, the control unit 55 loads the appropriate 2D file from the database 29 and adds the phoneme and word annotation data generated in the data file 91. The increased data file is then returned to the database 29. In this annotation formation step, the control unit 55 is operable to display the 2D image on the display 57 so that the user can assure that the annotation data is combined with the correct data file 91. [

The use of such phonemic and word grid annotation data, as will be described in more detail below, allows quick and efficient retrieval of the database 29 to identify and recall the desired 2D image data files stored therein. This can be accomplished by first searching the database 29 using word data and performing an additional search using more powerful phoneme data if such a search fails to provide the requested data file. As one of ordinary skill in the speech recognition arts appreciates, the use of phonemic data is more powerful because phonemes are dictionary-independent and allow the system to cope with vocabulary words such as names, places, and foreign words. In addition, the use of phonemic data allows the user to recall data files located in the database 29 when the original annotations have been entered by speech and the original automatic speech recognition system did not understand the words of the commentary entered, It can be oriented.

Import data file

4 is a block diagram showing the form of the user terminal 59 used in the above embodiment to retrieve the annotated 2D image from the database 29. [ Such a user terminal 59 may be, for example, a personal computer, a hand-held device, or the like. As shown, in this embodiment, the user terminal 59 comprises a database 29 of annotated 2D images, an automatic speech recognition unit 51, a voice recording unit 75, a keyboard 3, a microphone 7 ), A search engine 53, a control unit 55, and a display 57. [ In operation, the user inputs one of a voice query via the microphone 7 or a typing query via the keyboard 3, which queries the automatic speech recognition unit 51 or voice recording RTI ID = 0.0 > 75 < / RTI > In addition, such data may take the form of phonemes and word lattices, but this is not necessary. These phonemes and word data are then input to control unit 55, which is operable to initiate an appropriate search of database 29 using search engine 53. [ The result of the search generated by the search engine 53 is then resent to the control unit 55 which analyzes the search results and generates and displays appropriate display data to the user via the display 57. [

Figures 5A and 5B are flow charts illustrating how the user terminal 59 operates in this embodiment. In step s1, the user terminal 59 is in an idle state and waits for an input query from the user 39. [ Upon reception of the input query, the phoneme and word data for this input query are generated in step s3 by the automatic speech recognition unit 51 or the voice recording unit 75. [ Next, in step s5, the control unit 55 instructs the search engine 53 to perform a search of the database 29 using the word data generated from the input query. The word search employed in the above embodiment is the same as that used in the art for the typed word search, and will not be described in detail here. In step s7, if the control unit 55 identifies a match for the user's input query from the search result, the search result is output from the user via the display 57. [

In this embodiment, the user terminal 59 allows the user to consider the search results and waits for the user's confirmation of whether the results match the information requested by the user. If the search results match, the process proceeds from step s11 to the end of the process, and the user terminal 59 returns to the stop state and waits for the next input query. However, if the user indicates that the search result does not match the desired information (for example, by entering an appropriate voice command), processing proceeds from step s11 to step s13 and the search engine 53 searches the database 29 for phoneme search . However, in the above embodiment, the phoneme search performed in step s13 is not for the entire database 29, because it may take several hours depending on the size of the database.

Instead, the phoneme search performed at s13 uses the result of the word search performed at step s5 to identify one or more parts of the database that match the user's input query. For example, a phoneme search of some of the annotations around the identified word (s) is performed if the query includes three words and the word search identifies one or two of the query words in the annotation. The method in which the phoneme search performed in s13 is performed in the above embodiment will be described in detail later.

After the phoneme search is performed, the control unit 55 identifies at step s15 whether or not a match has been found. If the match is found, the process proceeds to step s17 and the control unit 55 causes the search result to be displayed on the display 57 to the user. Again, the system waits for the user's confirmation that the search results match the desired information. If the result is correct, the process passes from step s19 to end, and the user terminal 59 returns to the stop state and waits for the next input query. However, if the user indicates that the search result does not match the desired information, the process proceeds from step s19 to step s21, where the control unit 55 displays on the display 57 whether the phoneme search should be performed for the entire database 29. [ Lt; RTI ID = 0.0 > user < / RTI > In response to this query, if the user indicates that such a search should be performed, the process proceeds to step s23, and the search engine performs a phoneme search for the entire database 29. [

Upon completion of this search, the control unit 55 identifies in step s25 whether the user's input query has been found. If a match is found, the process proceeds to step s27, and the control unit 55 causes the search result to be displayed to the user on the display 57. [ If the search result is correct, the process proceeds from step s29 to the end of the process, and the user terminal 59 returns to the stop state and waits for the next input query. Conversely, if the user indicates that the search result does not match the desired information, the process is passed to step s31 and the control unit 55 informs the user via display 57 whether the user wants to redefine or correct the search query Query. If the user wishes to redefine or correct the search query, the process returns to step s3 and the user's subsequent input query is processed in a similar manner. If the search is not redefined or corrected, the search result and the user's initial input query are discarded, and the user terminal 59 returns to the idle state and waits for the next input query.

The manner in which the search is performed in the embodiment by the user terminal 59 has been presented as a general description. Now, with a brief description of the underlined search strategy, a more detailed description of how the search engine 53 performs the phoneme search will be presented.

Loading information as a classification problem

In a classical classification scenario, the test data is classified as one of the K classes. This is done using information about other data for which the class is known. The classification problem assumes that there is a " class " random variable that can take a number from 1 to K. [ Next, the optimal classification is found by identifying the classes in which the test data most likely belongs. It is assumed that training data is generated by an N generative process that results in n _k data of class k. ^{_{Here, Σ K k = 1 n k}} = N. vector _{(n 1, n 2, ...} , n k) when the by n, by the training data in the D and shown by the test data in the x, classic classification The problem is to determine the number of k that maximizes the probability of:

The second term in the numerator is the prior probability for the class giving a larger weight to the more frequently occurring class. In the context of information retrieval, each class has single training data (i.e. annotation data). Thus, for information retrieval, the second term on the right side of the above expressions can be ignored. Similarly, P (x | D) is the same for each class, normalizing only the molecules, so denominators can also be ignored. Finally, the order of the class can be evaluated by evaluating P (x | d _k ) for all classes by evaluating only the order of the first term in the numerator of the expression for the class. Here, d _k is training data for class k.

In the above embodiment, the test data (x) represents the input query, the training data for class k (i.e., d _k ) represents the kth annotation, and the base It is assumed that a statistical model (M) exists. In the general case, this model has a state sequence (S _q and S _a ) and an output distribution (C) through the model for all three unknowns, the model structure (m) . In this case, the output distribution is known because it embodies the characteristics of a speech recognition system that generates a phoneme string from the input speech. As will be described later, this can be achieved by applying a large database of speech known to speech recognition systems, hereinafter referred to as confusion statistics. Therefore, by introducing the state sequence and model as the probability equation (and using the variable q for the input query and the variable a for the annotation), the following equation is computed:

This can be developed using the Bayesian method as follows:

While the above expression may seem complex, the sum over the set of state sequences s _q and s _a may be performed using standard dynamic programming algorithms. Also, the last term in both the numerator and the denominator seems to be the same, and the term [P (s | m, c)] of the state sequence can be ignored since it can be assumed that each state sequence seems to be the same. In addition, by estimating that the base model structure is a reference sequence of phonemes having a length approximately equal to the query to which the insert is applied, summations over different models can be eliminated even if they are replaced by merging over all possible phonemes, The basic sequence of phonemes in the model is not known. Thus, by ignoring the sum of the state sequences, the terms evaluated within the dynamic program algorithm are,

And in the denominator (that is, the normalization term)

. Where N _p is the total number of negative numbers known to the system, and a _i , q _j, and p _r are the annotation phoneme, query phoneme, and model phoneme, respectively, corresponding to the existing DP lattice points to be evaluated. As can be seen from the comparison of equations (4) and (5), the probability term calculated in the denominator is also calculated in the numerator. Thus, all of these terms can be accumulated during the same dynamic programming sequence. Considering the probability of being determined in more detail, P (q _j | p _r , c) is the probability of decoding the reference phoneme p _r as the query phoneme q _j , assuming confusion statistics; P (a _i | p _r , c) is the probability of decoding the reference phoneme p _r as the annotation phoneme a _i , assuming confusion statistics; P (p _r | c) is the probability of decoding the reference phoneme p _r unconditionally assuming confusion statistics.

In addition to the above, additional terms must be calculated to handle the insertion or deletion of queries or annotations for the model at each point in the dynamic programming computation. As will be appreciated by those skilled in the art, the insertion or deletion in the query is independent of insertion or deletion in the annotation, and vice versa. Therefore, these additional terms are handled separately. In addition, insertions and deletions in the annotations for the model should be considered for the normalization term given in equation (5) above.

It will be understood by those skilled in the art from the description of FIGS. 4 and 5 that in this embodiment both the annotation phoneme data and the query phoneme data may be obtained from either text or speech. Thus, there are four situations to consider:

I) the annotations and queries are all generated from text;

Ii) a situation where the annotation is generated from text and the query is generated from speech;

Iii) the annotation is generated from speech and the query is generated from text; And

Iv) Situations in which queries and annotations are all generated from speech.

The first situation is a simple case where the compression / expansion of the annotation or query and the comparison between annotation and query can not be time to be performed by a simple boolean comparison of the sequence of each phoneme.

In the second situation, the annotation is considered correct, and the dynamic programming alignment enables the insertion and deletion of phonemes in the query to find the best alignment between these two. To illustrate this case, FIG. 6B shows the sequence a ₀ , a ₁ , a ₂ ... of the annotation phoneme when the annotation phoneme is generated from the text. And a sequence of query phonemes q ₀ , q ₁ , q ₂ ... Lt; / RTI > As shown by the dotted arrows, the annotation phoneme a ₀ is aligned with the query phoneme q ₀ , the annotation phoneme a ₁ is aligned with the query phoneme q ₂ , the annotation phoneme a ₂ is aligned with the query phoneme q ₃ , a ₃ is aligned with the query phoneme q _3, and the phoneme a ₄ is aligned with the query phoneme q ₄ . For each of these alignments, the dynamic programming sequence computes the terms given in equations (4) and (5). However, in this case, these equations are simplified because the reference sequence of the model phoneme is known (these are annotated phonemes). In particular, the normalization term is one reason why the annotation is the model and the numerator is P (q _i | a _j , c). In addition to these decoding terms, the dynamic programming routines may be used for phonemes inserted in queries for annotations [such as query phoneme q ₁ ], and for query phoneme q ₃ matched with the two annotation a ₂ and a ₃ ] Calculate the related insertion and deletion probabilities for the phoneme deleted from the query for the annotation.

The third situation mentioned above is similar to the second situation except that the sequence of the query phonemes is taken correctly and the dynamic programming alignment is capable of inserting and deleting phonemes in the annotation to the query. However, in this situation, the equations (1) to (5) can not be used because the query is known. Thus, in this situation, equation (1) can be reconstructed as follows:

As with the corresponding term in Equation (1) above, the second term in the numerator and denominator can all be ignored. In Equation (6), the first term of the molecule can be developed in a manner similar to the way in which the first term in the numerator of Equation (1) is developed. However, in this situation, with the query taken to be a model, the normalization term computed during the dynamic programming routine is simplified to one, and the molecule is simplified to P (a _i | q _j , c). As in the second situation discussed above, the dynamic programming routine calculates the related insertion and deletion probabilities for the phoneme inserted in the annotation for the query and for the phoneme deleted in the annotation for the query.

Finally, in the fourth situation, when annotations and queries are all generated from speech, the sequence of phoneme data may all have insertions and deletions to the unknown reference sequence of the model phoneme that actually represents the spoken text. This is the sequence of the annotation phoneme a _i , a _{i + 1} , a _{i + 2} ... And a phoneme sequence of the query _{q j, q j + 1,} q j + 2 ... And a sequence of phonemes p _n , p _{n + 1} , p _{n + 2} ... indicating the reference sequence of the phonemes actually all spoken by the query and annotation. Lt; RTI ID = 0.0 > 6C. ≪ / RTI > As shown in Fig. 6C, in this case, the dynamic programming alignment technique is applied to the reference sequence of model phonemes (represented by phoneme a _{i + 3} and q _{j + 1} aligned with two phonemes in the reference sequence of all phonemes ) The insertion of phonemes in both annotations and queries (as indicated by the inserted phoneme a _{i + 3} and q _{j + 1} ) as well as deletion of phonemes from both annotations and queries should be allowed.

Those skilled in the art will appreciate that by introducing a model sequence of phonemes into the above calculation, the algorithm is more flexible for pronunciation variation in both queries and annotations.

In the above, the present embodiment has been generally described as a method of performing information retrieval by matching a sequence of a phoneme in a database and a sequence of a query phoneme in a database. To better understand the operation of the present embodiment, the standard dynamic programming algorithm will now be briefly described and then the specific algorithm used in this embodiment will be described in more detail.

DP Search Overview

As is known to those skilled in the art, dynamic programming is a technique that can be used to find optimal alignment between feature sequences that are phonemes in this embodiment. This is done by simultaneously propagating a number of dynamic programming paths each representing a possible match between the sequence of the annotation phoneme and the sequence of querying phonemes. All paths start at the start null node where the annotation and query begin, and propagate until they reach the end null node, which is the end of the annotation and query. Figures 7 and 8 schematically illustrate the matching performed and the path propagation performed. In particular, Figure 7 shows a Cartesian plot in which a horizontal axis is provided for annotations and a vertical axis is provided for queries. A start null node (? _S ) is provided in the upper left corner and an end null node (? _S ) is provided in the lower right corner. As shown in Fig. 8, the phoneme of the annotation is provided along the horizontal axis, and the phonemes of the query are provided along the vertical axis. Figure 8 also shows a number of lattice points each representing a possible alignment between phonemes in the annotation and ques- tions of the query. The lattice point 21 represents a possible alignment between the annotation phoneme a ₃ and the query phoneme q ₁ . Figure 8 also shows three possible matches between the sequence of phonemes representing the annotation and the sequence of phonemes representing the query and propagates from the start null node ( _s ) to the end null node ( _s ) through the lattice points . Referring again to the above equations (2) and (3), these dynamic programming paths represent the different state sequences s _q and s _a discussed above.

As represented by the different lengths of the horizontal and vertical axes shown in FIG. 7, the input query need not include all the words of the annotation. For example, if the annotation is "a picture of Taj Mahal", the user can simply search the database 29 for that picture by entering the query "Taj Mahal". In this situation, the optimal alignment path passes along the upper horizontal axis until the query begins to match the annotation. It then begins to traverse through the grid points to the lower horizontal axis and ends at the end node. This is shown in FIG. 7 as path 23. However, those skilled in the art will recognize that the words of the query must be in the same order as they appear in the annotation, otherwise the dynamic programming alignment will not work.

To determine similarities between a sequence of annotated phonemes and a sequence of query phonemes, the dynamic programming process maintains a score for each of the propagating dynamic programming paths, where the score follows the overall similarity of the phonemes aligned along the path. To limit the number of deletions and insertions of phonemes when sequences are matched, the dynamic programming process places certain constraints on the direction in which the dynamic programming path can propagate. Those skilled in the art will recognize that such dynamic programming constraints will differ from the four situations discussed above.

DP Pharmaceutical

Both annotations and queries are text.

When both the query phonemic data and the annotation phoneme data are generated from text, the dynamic programming alignment is transformed into a boolean match between two phoneme sequences.

Note is text and query is speech.

In the case where the annotated phoneme data is generated from text and the query phonemic data is generated from speech, there may be no phoneme deletion and insertion in the annotation, but there may be phonemic deletion and insertion in the query related to the annotation. 9A shows the dynamic programming constraints used in this embodiment when an annotation is generated from text and a query is generated from speech. As shown, if the dynamic programming path ends at a lattice point (i, j) that represents an alignment between the algebraic phoneme _ai and the query phoneme _qj , then the dynamic programming path is the lattice points (i + 1, j) i + 1, j + 1) and (i + 1, j + 2). Propagation to point (i + 1, j) represents a case where there is phoneme deletion from a verbal query compared to a typed annotation. Propagation to point (i + 1, j + 1) represents a case where there is a simple decoding between the next phoneme in the annotation and the next phoneme in the query. Propagation to point (i + 1, j + 2) has the insertion of phoneme q _{j + 1 in} the verbal query compared to the typed annotation, and there is decoding between the phoneme a _{i + 1} and the query phoneme q _{j + 2} .

The commentary is voice and the query is text.

If the annotation is generated from speech and the query is generated from text, there may be no phonemic insertions or deletions from the query, but insertions and deletions from the annotations about the query may exist. Figure 9B shows the dynamic programming constraints used in this embodiment when an annotation is generated from speech and a query is generated from the text. As shown, if the dynamic programming path ends at a lattice point (i, j) that represents an alignment between the algebraic phoneme _ai and the query phoneme _qj , then the dynamic programming path is the lattice points (i, j + i + 1, j + 1), and (i + 2, j + 1). Propagation to point (i, j + 1) represents the case of phoneme deletion from a verbal query compared to the typed annotation. Propagation to point (i + 1, j + 1) represents a case where there is a simple decoding between the next phoneme in the annotation and the next phoneme in the query. Propagation to the point (i + 2, j + 1) has the insertion of the phoneme a _{i + 1 in} the verbal query compared to the typed annotation, and there is decoding between the annotation phoneme a _{i + 2} and the query phoneme q _{j + 1} .

Zhuhai is the voice and the query is the voice.

If both annotations and queries are generated from speech, phonemes can be inserted and deleted from each of the queries about annotations and remainders. Figure 11 shows the dynamic programming constraints used in this embodiment when annotation phonemes and query phonemes are generated from speech. In particular, if the dynamic programming path ends at a lattice point (i, j) that represents an alignment between the annotation phoneme _ai and the query phoneme _qj , then the dynamic programming path is the lattice points (i + 1, j) i, j + 1, j + 1, j + 1, j + 1, j + 1, +1, j + 2), and (i, j + 3). Thus, these propagations enable the insertion and deletion of phonemes in both the query and annotation of model phonemes corresponding to the actually spoken text.

The start and end of DP constraints

In this embodiment, the dynamic programming sorting operation causes the dynamic programming path to start and end at any commentary phonemes. As a result, even if the query words need to be identical to the order in which they appear in the commentary, the query does not need to include all the words of the commentary.

DP score propagation

As described above, the dynamic programming process maintains a score for each of the dynamic programming paths, which depends on the similarity of the phonemes aligned along the path. Therefore, when a path terminated at a point (i, j) propagates to three different points, the dynamic programming process begins by traversing each " cost " , Which is stored in the store (SCORE (i, j)) associated with that point. Those skilled in the art will recognize that the cost includes the above-described insertion probabilities, deletion probabilities, and decoding probabilities. In particular, when there is an insertion, the cumulative score is multiplied by the probability of inserting the given phoneme. When there is an erasure, the cumulative score is multiplied by the probability of erasing the phoneme. When there is decoding, the cumulative score is multiplied by the probability of decoding two phonemes.

To be able to calculate these probabilities, the system stores the probabilities for all possible phoneme combinations. In this embodiment, deletion of phonemes in annotations or queries is handled in a similar manner as for decoding. This is done simply by treating the deletion as another phoneme. Thus, if there are 43 phonemes known in the system, the system will store 1892 (1892 = 43 x 44) decoding / erasure probabilities, one for each possible phoneme decoding and erasure. This shows possible phoneme decodings that can be stored for phoneme / ax /, and is shown in Fig. 10 which includes the phoneme erased as one of the probabilities. Those skilled in the art will appreciate that since there are no other probabilities, all decoding probabilities for a given phoneme must be summed together. In addition to these decoding / erasure probabilities, the system stores 43 insertion probabilities, one for each possible phoneme insert. As described below, these probabilities are determined prior to the training data.

To illustrate score propagation, a number of examples can be considered. For the path propagated from the point (i, j) to the point (i + 1, j + 2), the phoneme q _{j + 1} is inserted in relation to the annotation and the query phoneme q _{j +2} is decoded to the annotation phoneme _{ai + 1} . Therefore, the score propagated to the point (i + 1, j + 2) is as follows.

Where _{PI (q j + 1 | C} ) is the probability of the inserted phoneme q _{j + 1} of the shoe query P (q _{j + 2} | a _{i + 1,} C) are annotation phoneme a to _{i + 1} query phoneme q _{j + 2} < / RTI >

Annotation and the query time all be propagated to the case that is generated from the voice, point to point (i + 2, j + 1 ) from the (i, j), annotation phoneme a _{i + 1} is inserted to the query annotation phoneme a _{i + 2} and the query phoneme q _{j + 1} . Therefore, the score propagated to the point (i + 2, j + 1) is given by the following equation (8).

Those skilled in the art will appreciate that, during such path propagation, several paths coincide at the same lattice point. In this embodiment, the scores associated with matching paths are simply added together. Alternatively, comparisons between scores can be made and the path with the best score continues and other paths can be ignored. However, this is not important in this embodiment because the dynamic programming process is only interested in finding a score that represents the similarity between the phonemic data of the query and the phonemic data of the annotation. I am not interested in finding out what the best alignment is between the two.

If the query and annotation are generated from speech, then all the paths are propagated to the end node ( _e ) and the overall score for the similarity between the query and the current annotation is determined, the system uses the normalization condition that was accumulated during the DP process Normalize the score. Next, the system compares the query with the following annotation in a similar manner. If the query has matched all annotations, the normalized scores for the annotation are ranked and based on this stratification, the system outputs the user annotation most similar to the input query to the user.

Detailed description of DP search

The manner in which a dynamic programming search is performed when matching a sequence of querying phonemes with a sequence of commented phonemes will be described in more detail. Referring to Fig. 12, in step s101, the system initializes dynamic programming scores. Next, at step s103, the system propagates the paths from the null starting node ( _s ) to all possible starting points. Next, at step s105, the system propagates the dynamic programming paths from all start points to all possible end points using the dynamic programming constraints discussed above. Finally, at step s107, the system propagates the paths terminating at the end points to the null end node ( _e ).

FIG. 13 shows in more detail the processing steps involved in step s103 when propagating dynamic programming paths to all possible starting points defined by dynamic programming constraints from null starting node ( _s ). One of the constraints is that the dynamic programming path can start with any annotation phonemes and other constraints, which define the number of hops that are allowed in the sequence of query phonemes depending on whether the query is text or speech. In particular, when the query is generated from text, the starting points include points (i, 0) where i = 0 for the first row of lattice points in the search space, i. E., Nann-1, (I, 0), (i, 1), (i, 2) and (i, 3) with i = 0 for the first four rows of grid points in the search space, .

A method which can be performed with reference to the steps shown in Fig. 13 will be described. As shown, in step s111, the system determines whether the input query is a text query or not. If the query is text, the processing proceeds to step s113 where the system sets the value of the variable mx to 1, which defines the maximum number of " hops " allowed in the sequence of phonemes of the query when the query is text. (I, 0) associated with points (i, 0) with i = 0 for Nann-1, and a transition score for passing from the null start node to the lattice point (i, To steps s115, s117, and s119 that are operable to start the dynamic programming path at each of the lattice points in the first row of the search space. When the query is text, the processing in step s103 shown in Fig. 12 is ended, and then the processing proceeds to step s105.

If the system determines in step s111 that the query is not from text and is generated from verbal input, then the system determines that mx is set to mxhops, which is a constraint with a value having a value that is at least equal to the maximum number of " hops & The process proceeds to step S121. As shown in FIGS. 9 and 10, in the case where the query is negative, the path can jump to the query phoneme which is at most three phonemes following the sequence of query phonemes. Thus, in the present embodiment, when there are four or more phonemes in the query, mxhops has a value of 4 and the variable mx is set to 4; otherwise, mx is set equal to the number of phonemes in the query. The processing then includes steps s123, s125, s127, s127, s122, s123, s124, s122, s124, s122, s129, and s131. When the query is generated from the verbal input, the processing ends at step s103 shown in Fig. 12, and then the processing goes to step s105.

In this embodiment, the system propagates dynamic programming paths from start points to end points in step s105 by processing lattice points within the search space per row with a technique such as raster. In step s105, the system compares the annotation loop pointer i with the number of phonemes Nann in the annotation. Initially, the annotation phoneme loop point (i) is set to zero and the processing will proceed to step s153 where a similar comparison is made to the query phoneme loop pointer j with respect to the total number of phonemes (Nquery) in the query. Initially, loop pointer j is also set to zero and processing proceeds to step s155 where the system propagates the path terminating at point (i, j) using the dynamic programming constraints discussed above. The manner in which the system propagates the paths in step s155 will be described in more detail below. After step s155, the loop pointer j is incremented by one in step s157 and the processing returns to step s153. If this process is looped through all of the phonemes in the query (thus processing the current row of lattice points), processing proceeds to step s159 where the query phoneme loop pointer j is reset to zero and the annotation loop pointer I is incremented by one . Processing then returns to step s151 where a similar process is performed for the next column of lattice points. If the last row of the lattice points is processed, the processing proceeds to step s161 where the annotation loop pointer i is reset to 0, and the processing in step s105 shown in Fig. 12 ends.

Fig. 15 shows in more detail the processing steps included in step s107 shown in Fig. 12 when the paths at the end points propagate to the end null node _e . In propagating from the start null node ( _s ), the lattice points that are " end points " are defined by dynamic programming constraints, depending on whether the query is text or speech. Further, in this embodiment, the dynamic programming constraints cause the dynamic programming paths to depart from the annotation at any point along the sequence of annotation phonemes. Thus, if the question text, the system with i = 0 in the point (i, Nquery-1) dynamic programming paths end to end at the last row of the board nodes (ø _e) for the grid point, that is Nann-1 Propagate. (I, Nquery-4), (i, Nquery-3), (i, Nquery-2) where i = 0 for the lattice points, Nann-1, , And (i, Nquery-1) to propagate to the end null node ( _e ).

As shown in Fig. 15, the process starts in step s171 in which the system determines whether the query is text or not. If it is text, the processing proceeds to step s173 where the query phoneme loop pointer j is set to Nquery-1. Next, the processing proceeds to step s175 in which the annotated phoneme loop pointer i is compared with the number of phonemes Nann in the annotation. Initially, the annotation loop pointer i is set to 0 and processing will proceed to step s117 where the system computes a transition score from the point (i, Nquery-1) to the null end node ( _e ). Next, this transition score is compared to the cumulative score for the path ending at point (i, Nquery-1) stored in SCORE (i, Nquery-1). As described above, in this embodiment, the system uses log probabilities for transition and cumulative scores to eliminate the need to perform multiplications and avoid the use of high floating point precision. Therefore, at step s179, the system adds the cumulative score for the path terminating at point (i, Nquery-1) to the transition score calculated at step s117 and the result is copied to the temporary store TEMPENDSCORE.

As described above, if two or more dynamic programming paths are encountered at the same point, the cumulative scores for each of the paths are added together. Therefore, since log probabilities are being used, the scores associated with matching paths are effectively transformed back into probabilities again, and are re-transformed into the next log probabilities that are added. In this embodiment, such an operation is referred to as " log addition ". This is a well known technique and is described, for example, in Kluwer Academic Publisher, 1989 by Lee, Kai-Fu, entitled " Development of an Automatic Speech Recognition System (Sphinx) Are described on pages 28 and 29 of the published book.

Since the propagation path from point (i, Nquery-1) to the null-termination node will meet with other dynamic programming paths, the system performs log addition of the TEMPENDSCORE and score stored in the end node (ENDSCORE) and stores the result in ENDSCORE do. Processing then returns to step S175 where a similar process is performed for the next lattice point in the last row of lattice points. If all the grid points in the final row are processed in the same manner as described above, the processing performed in step s107 shown in Fig. 12 ends.

If the query is determined to be non-text in step S171, processing proceeds to step s185 where the query phoneme loop pointer j is set to the number of phonemes in the query - mxhops, Nquery-4. Next, the processing proceeds to step s187 where the system checks to see if the annotation loop pointer i is less than the number of phonemes (Nann) in the annotation. Initially, the annotation loop pointer i is set to zero and processing proceeds to step s189 where the system checks to see if the query phoneme loop pointer j is less than the number of phonemes in the query (Nquery). Next, processing proceeds to step s191 in which the system calculates a transition score from the lattice point (i, j) to the null end node ( _e ). Next, at step s193, this transition score is added to the cumulative score for the path that ends at point (i, j), and the result is copied to the temporary score TEMPENDSCORE. Next, processing performs the log addition of TEMPENDSCORE and ENDSCORE, and the result is stored in ENDSCORE. Next, the processing proceeds to step s197 where the query phoneme loop pointer j is increased by 1, and the processing returns to step s189. Next, the processing steps are repeated until the query phoneme loop pointer j is incremented to become equal to the number of phonemes in the query (Nquery). Next, the processing proceeds to step s199 where the query phoneme loop pointer j is set to Nquery-4 and the annotation loop pointer i is incremented by one. Next, the processing returns to S187. The processing steps are repeated until all the grid points in the last four rows of the search space have been processed in the same manner, and then the processing performed in step s107 shown in FIG. 12 ends.

spread

In step s155 shown in Fig. 14, the system propagates the path terminating at the lattice point (i, j) using the dynamic programming constraints discussed above. 16 is a flow chart illustrating the processing steps involved in performing the propagation step. As shown, in step s211, the system sets the values of the two variables mxi and mxj and initializes the annotated phoneme loop pointer i2 and the query phoneme loop pointer j2. Loop pointers i2 and j2 are provided to loop through all lattice points propagated by the path terminating at point (i, j), and variables mxi and mxj take i2 and j2 only those values allowed by dynamic programming constraints . In particular, if mxi is less than or equal to the number of phonemes in the annotation, it is set to i + mxhops, otherwise mxi is set to the number of phonemes (Nann) in the annotation. Similarly, if mxj is less than or equal to the annotation of the annotation, it is set to j + mxhops, otherwise mxj is set to the number of phonemes in the query (Nquer). Finally, in step s211, the system initializes the annotated phoneme loop pointer i2 to be equal to the current value of the annotated loop pointer i, and the query phoneme loop pointer j2 to be equal to the current value of the query phoneme loop pointer j.

The dynamic programming constraints used by the system depend on whether the annotation is text or speech and whether the query is text or speech, so the next step is to determine how annotations and queries were generated. This is done by decision blocks s213, s215, and s217. If both annotations and queries are generated from speech, the dynamic programming path terminating at the lattice point (i, j) may propagate to the other points shown in Fig. 11, and the process steps s219 to s235 may be such that the path Lt; / RTI > Specifically, in step s219, the system compares the annotation loop pointer i2 with the variable mxi. Since the annotation loop pointer i2 is set to i and mxi is set to i + 1, in step s211, the processing will proceed to step s221 where a similar comparison is made to the query phoneme loop pointer j2. Processing then proceeds to step s233, which ensures that the path does not stay at the same lattice point (i, j), since initially i2 is equal to i and j2 is equal to j. Thus, the processing will proceed to step s225 where the query phoneme loop pointer j2 is incremented by one.

The processing returns to step s221 where the incremented value of j2 is compared to mxj. If j2 is less than mxj, the processing returns to step s223 and proceeds to step s227, which is operable to prevent the hops along both the note phonemes and the query phonemes from becoming too large. This is done by ensuring that the path propagates only if i2 + j2 is less than i + j + mxhops. This ensures that only the points of the triangular set shown in FIG. 11 are processed. If this condition is satisfied, then processing proceeds to step s229 where the system calculates a transition score TRANSCORE from the lattice point (i, j) to the lattice point (i2, j2). This processing then proceeds to step s231 where the system adds the transition score determined in step s229 to the cumulative score stored for point (i, j) and copies it to the temporary store TEMPSCORE. As described above, in this embodiment, if two or more dynamic programming paths meet at the same lattice point, the cumulative scores associated with each of the paths are summed together. Thus, at step s233, the system performs a log sum of cumulative scores already stored for TEMPSCORE and point (i2, j2) and the result is stored in the score (i2, j2). Next, the processing returns to step s225 to increment the query phoneme loop pointer j2 by one, and the processing returns to step s221. If the query phoneme loop pointer j2 has reached the value of mxj, the processing proceeds to step s235 where the query phoneme loop pointer j2 is reset to the initial value j and the annotation loop pointer i2 is incremented by one. Next, processing proceeds to step s219 where processing is resumed for the next column of points shown in Fig. If the path has propagated from point (i, j) to all other points shown in Fig. 11, the processing ends.

If the decision blocks s213 and s215 determine that the annotation is text and the query is negative, then processing proceeds to steps s241 through s251 (i, j) operating to propagate the path ending at point (i, j) . Specifically, in step s241, the system determines whether the annotation loop pointer i indicates the final phoneme in the annotation. If so, there are no more phonemes in the annotation and the processing ends. If the annotation loop pointer i is smaller than Nann-1, processing proceeds to step s243 where the query phoneme loop pointer j2 is compared to mxj. Initially, j2 will be less than mxj and therefore processing proceeds to step s245 where the system calculates a transition score TRANSCORE from point (i, j) to point (i + 1, j2). Next, this transition score is added to the cumulative score associated with the path terminating at point (i, j) and the result is copied to the temporary score TEMPSCORE. Next, at step s249, the system performs a log addition of cumulative scores associated with TEMPSCORE and point (i + 1, j2) and stores the result in SCORE (i + 1, j2) , < / RTI > j2) are combined. Next, the processing proceeds to step s251 where the query phoneme loop pointer j2 is incremented by one, and then the processing returns to step s243. If the path terminating at point (i, j) is propagated to other points shown in FIG. 9A, j2 will be equal to mxj and the propagation of the path terminating at point (i, j) will end.

If the decision blocks s213 and s217 are negative and the query is determined to be text, the steps shown in Fig. 16B, which are operable to propagate the path ending at point (i, j) to the other points shown in Fig. S255 to s265. This is done in step s255 by first checking whether the query phoneme loop pointer j indicates the final phoneme in the sequence of phonemes representing the query. Otherwise, the processing proceeds to step s257 where the annotation loop pointer i2 is compared to mxi. Initially, if i2 has a value of i and the phoneme i is not at the end of the sequence of phonemes representing the annotation, processing may be performed using a transition score for moving from point (i, j) to point (i2, j + And proceeds to step s259 in which it is calculated. Processing then proceeds to step s261 where the transition score is added to the cumulative score for the path ending at point (i, j) and the result is copied to a temporary score (TEMPSCORE). Next, the processing proceeds to step s263 where the log addition of cumulative scores already stored for TEMPSCORE and point (i2, j + 1) is performed and the result is stored in SCORE (i2, j + 1). Processing then proceeds to step s265 where the annotation loop pointer i2 is incremented by one and processing returns to s257. These processing steps are repeated until the next ending path at point (i, j) propagates to each of the other points shown in Figure 9b. At this time, the propagation of the path at the point (i, j) is completed and the processing ends.

Finally, if the decision blocks s213 and s215 determine that both the annotation and the query are text, then processing proceeds to the point (i + j) if the additional annotation phoneme and the additional query phoneme exist, 1, j + 1) shown in Fig. 16B. Specifically, in step s271, the system checks whether the annotation loop pointer i indicates the final annotation phoneme. Otherwise, the processing proceeds to step s273 where a similar check is made for the query phoneme loop pointer j related to the sequence of querying phonemes. If there are no more annotated phonemes or no more query phonemes, the processing ends. However, if there are more annotated phonemes and query phonemes, processing proceeds to step S275 where the system calculates a transition score from point (i, j) to point (i + 1, j + 1). In step s277, this transition score is summed with the cumulative score stored for point (i, j) and stored in a temporary score (TEMPSCORE). Processing then proceeds to step s279 where the system performs a log change of the accumulated score already stored for TEMPSCORE and point (i + 1, j + 1) and the result is copied to SCORE (i + 1, j + 1) Go ahead. Those skilled in the art will appreciate that the dynamic programming constraints may be used in the present embodiment because the path is to start at any phoneme in the sequence of phonemes representing the annotation and thus the point (i + 1, j + 1) Lt; RTI ID = 0.0 > s277 < / RTI > After step s279, the propagation of the point (i, j) is completed and the processing ends.

Transition Score

In steps s103, s105, and s107 shown in Fig. 12, the dynamic programming paths are propagated and a transition score from one point to another during its propagation is calculated using steps s127, s117, s177, s191, s229, s245, s259 , And s275. In these steps, the system calculates appropriate insertion probabilities, deletion probabilities, and decoding probabilities associated with the start and end points of the transition. The method of this embodiment will be described with reference to Figs. 17 and 18. Fig.

In particular, FIG. 17 shows a flow chart illustrating typical processing steps involved in calculating a transition score for a path propagating from a lattice point (i, j) to a lattice point (i2, j2). In step s 291, the system calculates a score (the log of the probabilities PI () discussed above) for inserting the embedded phoneme (s) for each annotation phoneme inserted between the point (i, j) And stores it in the appropriate store INSERTSCORE. Processing then proceeds to step s 293 where the system performs a similar calculation for each query phoneme inserted between point (i, j) and point (i 2, j 2) and adds this to INSERTSCORE. However, if (i, j) is the start null node (ø _s ) or (i2, j2) is the end null node ( _øe ), the system calculates the insertion probabilities for any inserted query phonemes, Does not calculate insertion probabilities for any inserted annotation phonemes (since there is no penalty for starting or ending a path in any annotation phonemes). As discussed above, since the calculated scores are log based on probabilities, the addition of the scores of INSERTSCORE corresponds to the multiplication of corresponding insertion probabilities. Processing then calculates the scores for any deletions and / or any decoding as the system propagates from point (i, j) to point (i2, j2) and these scores are added to and stored in the appropriate store DELSCORE The flow advances to step s295. Processing then proceeds to step s297 where the system adds INSERTSCORE and DELSCORE and copies the result to TRANSCORE.

The processing involved in step S295 for determining erasure and / or decoding scores when propagating from point (i, j) to point (i2, j2) will be described in more detail with reference to FIG. Since deletion and decoding depend on whether an annotation is generated from text and whether the query is generated from text, decision blocks s301, s303, and s305 determine whether the annotation is text or speech and whether the query is text or speech. If these decision blocks determine that both the annotation and the query are text, there is no deletion and the decoding of the two phonemes is performed by mismatching in step s307. If the annotation phoneme a _i2 is equal to the query phoneme q _j2 , processing proceeds to step s309 where TRANSCORE is set to log [1] (i.e., 0) and processing then ends. However, the annotation phoneme a _i2 not identical to the query phoneme q _j2, the processing advances to step 311 and TRANSCORE is set to a very large negative system approximation (approximation) of the log [0] and then the processing is ended.

If decision blocks s301 and s305 determine that the annotation is speech and the query is text, the transition scores are determined using the simplified form of Equation 4 discussed above. In this case, processing proceeds from step s303 to step s313 where the system determines whether annotated loop pointer i2 is equal to annotation loop pointer i. If they are the same, this means that the path has propagated from point (i, j) to point (i, j + 1). Therefore, the query phoneme q _{j + 1} has been removed from the sequence of annotation phonemes associated with the sequence of query phonemes. Accordingly, in step s317, the system log probability (that is, log P (ø | q _{j + 1,} C)) to remove the phoneme q _{j + 1} is copied to the DELSCORE and processing is terminated. (I + 1, j + 1), (i + 2, j + 1) ) Or (i + 3, j + 1). In this case, there is no deletion, and there is only insertion and decoding between the annotation phoneme a _i2 and the query phoneme q _{j + 1} . Thus, in step s315, the system log probability _{i.e., logP (a i2 | q j} + 1, C)] to decode the query phoneme q _{j + 1} as the annotation phoneme a _i2 Copy to DELSCORE and processing is terminated.

If the decision blocks s301 and s305 determine that the annotation is text and the query is negative, the transition score is determined using the other simplified form of Equation 4 discussed above. In this case, the processing proceeds from step s305 to step s319 in which the system determines whether the query phoneme loop pointer j2 is the same as the query phoneme loop pointer j. If they are the same, the system calculates the transition score from point (i, j) to point (i + 1, j). In this case, the annotation phoneme a _{i + 1} has been deleted from the sequence of query phonemes associated with the sequence of annotation phonemes. Therefore, in step s321, the system determines the log probability (i.e., log P (? | A _{i + 1} , C)) that deletes the annotation phoneme a _{i + 1} and copies it to DELSCORE. If the system determines in step s319 that the query phoneme loop pointer j2 is not the same as the query phoneme loop pointer j, the system extracts points (i + 1, j + 1), (i + , j + 2) or (i + 1, j + 3). In this case, there is no erasure, and there is only insertion and decoding between the annotation phoneme a _{i + 1} and the query phoneme q _j2 . Therefore, in step s323, the system determines and copies the log probability (i.e., log P (q _j2 | a _{i + 1} , C)) decoding the annotation phoneme a _{i + 1} as the query phoneme q _j2 to DELSCORE And terminates.

If decision blocks s301 and s303 determine that both annotations and queries are generated from speech, the transition scores are determined using Equation (4) above. In this case, processing passes from step s303 to step s325 where the system determines whether annotation loop pointer i2 is equal to annotation loop pointer i. If so, processing proceeds to step s327 where the phoneme loop pointer r is initialized to 1. The phoneme loop pointer r is used to loop through each possible phoneme known to the system during each computation of Equation (4) above. Next, processing proceeds to step s329 of comparing the phoneme pointer r with the number of known phonemes (Nphonemes) (43 in this embodiment) known to the system. Initially, r is set to 1 in step s327 and processing therefore determines the log probability (i.e., log P (p _r | C)) of the phoneme p _r that the system is generating and copies it to the temporary score TEMPDELSCORE s331 . (I, j + 1), (i, j + 2), or (i, j + 1) if the annotation loop pointer i2 is the same as the annotation phoneme i, 3). Therefore, there are phonemes in the query that are not in the annotation. Thus, in step s333, the system adds to the TEMPDELSCORE the log probability (i.e., log P (? | P _r , C)) that deletes the phoneme p _r from the annotation. Processing proceeds to step s335 of adding to the TEMPDEL SCORE the log probability (i.e., log P (q _j2 | p _r , C)) at which the system decodes the phoneme p _r as the querying phoneme q _j2 . Processing then proceeds to step s337 where the log addition of TEMPDELSCORE and DELSCORE is performed and the result is stored in DELSCORE. Processing then proceeds to step s339 where the phoneme loop pointer r is incremented by one and then processing returns to step s329 where similar processing is performed for the next phoneme known to the system. If this calculation has been performed for each of the 43 phonemes known to the system, the processing ends.

In step s325, if the system determines that i2 and i are not the same, processing proceeds to step s341 where the system determines whether the query phoneme loop pointer j2 is equal to the query phoneme loop pointer j. If so, the processing proceeds to step s343 where the phoneme loop pointer r is initialized to 1. Next, processing proceeds to step s345 where the phoneme loop pointer r is compared to the total number of phonemes (Nphonemes) known to the system. Initially, r is set to 1 in step s343, and thus processing proceeds to step s347 where the log probability of the resulting phoneme p _r is determined and copied to the temporary store TEMPDELSCORE. Processing then proceeds to step s349 where the system determines the log probability that the phoneme p _r is decoded as phoneme a _i2 and adds it to TEMPDELSCORE. If the query phoneme loop pointer j2 is the same as the query phoneme loop pointer j, the system returns the path ending at point (i, j) to points (i + 1, j), (i + , j). Therefore, there are phonemes in the annotation that are not in the query. Thus, at step s351, the system determines the log probability to delete the phoneme p _r from the query and adds it to TEMPDEL SCORE. Processing then proceeds to step s353 where the system performs the log addition of TEMPDELSCORE and DELSCORE and stores the result in DELSCORE. The phoneme loop pointer r is increased by 1 in step s355 and the processing returns to step s345. If the processing steps s347 to s353 have been performed for all phonemes known to the system, the processing ends.

In step s341, if the system determines that the query phoneme loop pointer j2 is not the same as the query phoneme loop pointer j, processing proceeds to step s357 where the phoneme loop pointer r is initialized to one. Processing then proceeds to step s359 where the system compares the phoneme counter r to the number of known phonemes (Nphonemes) in the system. Initially, r is set to 1 in step s357, so processing proceeds to step s361 where the system determines the log probability of the phoneme p _r that the system is generating and copies it to the temporary score TEMPDELSCORE. If the query phoneme loop pointer j2 is not the same as the query phoneme loop pointer j2, the system computes the paths ending at point (i, j) as points (i + 1, j + , And (i + 2, j + 1). Therefore, there is no erasure, only insertion and decoding. Therefore, the processing proceeds to step s363 in which the log probability to decode the phoneme p _r as the note phoneme a _i2 is added to TEMP DELSCORE. Next, a log probability of decoding the phoneme p _r as the query phoneme q _j2 is determined, and the process proceeds to step s365 where TEMPDELSCORE is added. Next, in step s367, the system performs a log addition of TEMPDELSCORE and DELSCORE and stores the result in DELSCORE. Then, the phoneme counter r is incremented by 1 in step s369 and the processing returns to step s359. If the processing steps s361 through s367 have been performed for all phonemes known to the system, the processing ends.

Normalization

The above description of the dynamic programming process has been dealt with only in the molecular part of the above equation (3). Therefore, after the input query is matched with the sequence of note phonemes in the database, the score for the match (stored in ENDSCORE) should be normalized by the normalization term defined by the denominator of equation (3). As described above, the calculation of the denominator term is performed at the same time as the calculation of the numerator, i. E., The dynamic programming routine described above. This is because the terms required for the denominator are all computed on the molecule, as can be seen from the comparison of the denominator to the denominator. However, it should be noted that when an annotation or query is generated from text, no normalization is performed. In this embodiment, normalization is performed so that longer annotations are not given more weight than shorter annotations, and annotations containing common phonemes are not weighted more than annotations containing non-common phonemes. This is done by normalizing the score by the conditions depending on how well the annotation matches the base model.

Training

In the above example, the system used 1892 decoding / erasure probabilities (referred to above as confusion statistics) and 43 insertion probabilities used to score the dynamic programming path in the phoneme matching operation. In the present embodiment, these probabilities are predetermined in a training session and stored in a memory (not shown). Specifically, during such a training session, the speech recognition system is used to provide phoneme decoding of speech in two ways. In the first method, the speech recognition system is provided with pronounced voice and actual words. Thus, the speech recognition apparatus can generate the reference phoneme sequence of pronounced words to use the information to obtain an ideal decoding of the speech. Next, the speech recognition system is used to decode (hereinafter, referred to as pre-decoding) the same speech but without recognizing the actual word to be pronounced. The phoneme sequence generated from the pre-decoding will be different from the reference phoneme sequence in the following method.

i) The pre-decoding may be a mistake to insert phonemes into decoding that does not exist in the reference sequence, or alternatively, omit the phonemes of decoding present in the reference sequence.

ii) One phoneme can be confused with another.

iii) Although the speech recognition system completely decodes the speech, the pre-decoding may be different from the reference decoding due to the difference between the pronunciation of the speech and the reference pronunciation. For example, in a conversation voice, the words "and" (with the reference forms / ae / / n / / d / and / ax / n / d /) are often / ax / / n / .

Therefore, if various pronunciations are decoded in its reference form and its pre-decoded form, then the dynamic programming method can be used to sort by two. This provides a count of the decoded phoneme d when the phoneme is p. From these training results, the decoding, erasure and insertion probabilities can be approximated in the following way.

The probability that a phoneme d is inserted is given as follows.

Where I _d is the number of times the speech recognition system inserted the phoneme d and n _o ^d is the total number of decoded phonemes inserted relative to the reference sequence.

The probability of decoding phoneme p as phoneme d is provided as follows.

Where C _dp is the number of times the automatic speech recognition system decoded d when n _p was p and the number of times that the automatic speech recognition system decoded something (including erasure) when n _p was p.

When phoneme p is decoded, the probability that it has not decoded anything (that is, erased) is provided by

Where O _p is the number of times that the automatic recognition system did not decode when p was decoded and n _p is the same as above.

Second Embodiment

In the first embodiment, a single input query was compared to a number of stored comments. In this embodiment, the two input voice queries are compared with the stored annotations. This embodiment is suitable for applications where input queries are performed in a loud environment or where high accuracy is required. This is obviously not appropriate for situations where certain queries are text because they duplicate other queries. Therefore, the system can handle the following two points.

(i) all of the input queries are generated from speech and the annotation is generated from speech.

(ii) all of the input queries are generated from the speech and the annotation is generated from the text.

This embodiment uses a dynamic programming algorithm similar to that used in the first embodiment, except that the two queries are applied simultaneously to match the annotation. 19 illustrates a three-dimensional coordinate plot in which one dimension is provided for each of the two queries and the remaining dimension is provided for the annotation. Figure 19 shows a three dimensional grid of points processed by the dynamic programming algorithm of this embodiment. The algorithm uses the same transition scores, dynamic programming constraints, and confusion statistics (i.e., phoneme probability) that were used in the first embodiment to propagate and score each of the paths through the three-dimensional network of lattice points in the plot shown in FIG. ).

This three-dimensional dynamic programming process will be described in more detail. It will be appreciated by those skilled in the art that the three-dimensional dynamic programming algorithm is essentially the same as the two-dimensional dynamic programming algorithm used in the first embodiment except that it adds some additional control loops to account for the additional questions. To 25 and 13 to 18, respectively. The 3D dynamic programming algorithm compares the two queries with the annotations following all the steps shown in Fig. Figure 20 shows in more detail the processing steps included in step s103 when propagating dynamic programming paths from all possible starting points defined by dynamic programming constraints from a null starting node ( _s ). In this regard, the constraints are that the dynamic programming path starts at any one of the annotation phonemes and the path will start at any one of the first four phonemes in any query. Therefore, referring to Fig. 20, in step s401, the system sets the values of the variables mxj and mxk to the same mxhops as the constraint used in the first embodiment. Therefore, in this embodiment, if each input query includes four or more phonemes, mxj and mxk are all set equal to four. Otherwise, mxj and / or mxk are set equal to the number of phonemes in the corresponding query. The processing then proceeds to steps s403 through s417 which are operable to start dynamic programming paths at points (i, j, k) with Nann-1, j = 0 through 3 and i = 0 for k = . This ends processing at step s103 shown in Fig. 12 and then processing proceeds to step s105 where the dynamic programming paths are propagated to end points.

As in the first embodiment, in the present embodiment, the system propagates dynamic programming paths from start points to end points by processing points in a search space, such as a raster. The control algorithm used to control the raster processing operation is shown in FIG. As can be seen by comparing FIGS. 21 and 14, the control algorithm has the same general form as the control algorithm used in the first embodiment. The only difference is in the provision of query blocks s421, s423, and s425, which are needed to process more complex propagation steps s419 and additional points caused by the second input query. For a better understanding of how the control algorithm shown in FIG. 21 operates, the reader refers to the description given in FIG. 14 above.

Fig. 22 shows the processing steps used in step s107 shown in Fig. 12 in this embodiment in more detail when propagating the paths at the end points to the end null node _e . As can be seen by comparing FIGS. 22 and 15, the processing steps included in step s107 in this embodiment are similar to the corresponding steps used in the first embodiment. The difference is a more complex transition score calculation block s443 and additional blocks s439, s441, and s449 and variable k required to process additional grid points due to the second query. Therefore, in order to understand the processing included in steps s 431 to s 449, reference is made to the description of FIG.

Fig. 23 shows the processing steps included in the propagation step s419 shown in Fig. Fig. 16 shows a corresponding flow chart for the two-dimensional embodiments described above. As can be seen from the comparison of Figures 23 and 16, the main differences between the two embodiments are the processing blocks s451, s453, s455, and s457, which are required to process additional lattice points due to the second query, , And additional variables (mxk and k2). Also, Figure 23 is somewhat simpler because there are two main branches for which both queries should be negative, one for when the annotation is text, and the other for when the annotation is negative. Reference is made to the description of FIG. 16 to better understand the processing steps involved in the flow chart shown in FIG.

Figure 24 is a flow chart illustrating the processing steps involved in calculating the transition score when the dynamic programming path is propagated from point (i, j, k) to point (i2, j2, k2) during the processing steps of Figure 23 . Figure 17 shows a corresponding flow chart for the two-dimensional embodiments described above. As can be seen by comparing Figs. 24 and 17, the main difference between this embodiment and the first embodiment is the additional process step s461 for calculating insertion probabilities for phonemes inserted in the second query. Therefore, to better understand the processing steps included in the flowchart shown in FIG. 24, reference is made to the description of FIG.

The processing steps included in step s463 of Fig. 24 for determining erasure and / or decoding scores when propagating from point (i, j, k) to point (i2, j2, k2) will be. Since possible deletions and decodings depend on whether the annotation is generated from text or speech, decision block s501 determines whether the annotation is text or speech. If an annotation is generated from the text, the phoneme loop pointer i2 should be indicated by the annotation phoneme _{ai + 1} . The processing then proceeds to steps s503, s505, and s507 that are operable to determine whether any phoneme deletes in the first and second queries related to the annotation are present. If present, j2 and / or k2 will be equal to j or k, respectively.

If - j2 is not equal to j and k2 is not equal to k, there is no deletion in queries related to the annotation and the processing has the log probability to decode the annotated phoneme _{ai + 1} as the first query phoneme q _j2 to DELSCORE The process advances to step s509. Next, processing proceeds to step s511, the log probability of decoding a phoneme annotation a _{i + 1} as the second query phoneme q _k2 is added to the DELSCORE.

- If the system determines that j2 is k2 is equal to k not equal to the j processing annotation phoneme probability of deleting the a _{i + 1} is determined and copied into the DELSCORE annotation phoneme a _{i + 1,} the first query phoneme q _j2 Lt; RTI ID = 0.0 > s513 < / RTI > and s515, respectively.

If the system determines that j2 is equal to j and k2 is equal to k, the processing determines the log probability that the system deletes the annotation phoneme _{ai + 1} from both the first and second qualities and passes the result to DELSCORE The process proceeds to storing steps s517 and s519 and stores the result in DELSCORE.

- the system is j2 is not, the same as j and k2 are not equal to k, the processing for each copy and the log probability of the annotation phoneme a _{i + 1} to DELSCORE and decode annotation phoneme a _{i + 1} as the second query phoneme q _k2 Proceed to steps s521 and s523 operable to add the log probability to DELSCORE.

If the system determines in step s501 that the annotation is generated from speech, the system determines whether there is any phoneme deletion from the annotation or two queries by comparing i2, j2, and k2, respectively, with i, j, s525 to s537). As shown in FIGS. 25B-25E, there are eight major branches that are operative to determine appropriate decoding and erasure probabilities for eight possible situations when an annotation is generated from speech. Since the processing performed in each situation is very similar, only one of r's situations will be explained.

Specifically, at steps 525, 527 and s531, if there is no deletion from the two queries (since j2? J and k2? K) and there is no deletion from the annotation (because i2 = i) the process proceeds to step s541 where r is initialized to 1. The phoneme loop pointer r is used to loop through each possible phoneme known to the system during the calculation of the mathematical expression similar to Equation 4 above in the first embodiment. Next, processing proceeds to step s543 of comparing the phoneme pointer to the number of phonemes (Nphonemes) known in the system (43 in this embodiment). Initially, r is set to 1 in step s541. Thus, processing proceeds to step s545 where the system determines the log probability of the phoneme p _r that is generated and copies it to the temporary score TEMPDELSCORE. Processing then proceeds to step s547 where the system determines the log probability of erasing the phoneme p _r in the annotation and adds it to TEMPDELSCORE. Processing then proceeds to step s549 of determining the log probability to decode the phoneme p _{r as the} first quality phoneme q ¹ _j2 and adding it to TEMPDELSCORE. Processing then proceeds to step s551 where the system determines the log probability that the phoneme p _r is decoded as the second query phoneme q ² _k2 and adds it to TEMPDELSCORE. Processing then proceeds to step s553 where the system performs the log addition of TEMPDELSCORE and DELSCORE and stores the result in DELSCORE. Processing then proceeds to step s555 where the phoneme pointer r is incremented by one. Processing then returns to step s543 where similar processing is performed for the next phoneme known to the system. If these calculations were performed for each of the 43 phonemes known to the system, the processing ends.

As can be seen from the processing steps performed in Fig. 25 and the steps performed in Fig. 18, the terms computed within the dynamic programming algorithm for decoding and deleting are similar to Equation 4, but the addition to the second query Probability terms. Specifically, it is as follows.

Since the two queries are conditionally independent of each other, this is expected.

After all the dynamic programming paths have been propagated to the terminating node? _E , the overall score for the alignment is normalized to the same normalization term (given in Equation 5 above) as that calculated in the first embodiment. This is because the normalization term is only dependent on the similarity of the annotation to the model. If the two queries are matched for all annotations, the normalization score for the annotation is layered, and based on this stratification, the system outputs the annotations or annotations closest to the input queries to the user.

In the second embodiment described above, the two input queries were compared to stored annotations. Those skilled in the art will appreciate that the algorithm can be adapted to any number of input queries. As evidenced for the case of two queries, the additional query addition simply includes an addition of the number of loops in the algorithm to account for the additional query. However, in embodiments in which three or more input queries are compared to saved annotations, it may be necessary to use dynamic programming routines employing cuts to meet constant speed or memory constraints. In this case, rather than summing all probabilities together in all paths, only the best score for the path that they meet is propagated and path scoring ends.

Other Embodiments

Those skilled in the art will recognize that the techniques described above for matching a sequence of phonemes with other sequences of phonemes may be applied to applications other than data retrieval. Also, those skilled in the art will appreciate that while the system described above uses phonemes and phonemes within the word grid, other phoneme-like units such as syllables or katakana (Japanese alphabets) may be used.

Those skilled in the art will recognize that the above description of dynamic programming matching and alignment of two sequences of phonemes has been provided in an exemplary manner only and various modifications may be made. For example, while employing a raster scanning technique to propagate paths through lattice points, other techniques can be employed to propagate the path through lattice points. It will also be appreciated by those skilled in the art that other dynamic programming constraints than those discussed above may be used for matching process control.

In the above example, the annotation is generally longer than the query and the dynamic programming algorithm aligned the query and the entire annotation. In another embodiment, the sorting algorithm may compare the query and the annotation by stepping the query over the annotation from start to end, and at each step, compare the query with a portion of the annotation of approximately the same size as the query do. In this embodiment, at each step, the query is aligned with the corresponding portion of the annotation using a dynamic programming technique similar to that described above. This technique is shown in FIG. 26A with a result plot showing how the dynamic programming score for the alignment between the query and the current annotation changes as the query is stepped over the annotation shown in FIG. 26B. The peaks in the plot shown in Figure 26B represent part of the annotation that best matches the query. Next, the annotation most similar to the query can be determined by comparing the peak DP score obtained during the query comparison with the respective annotations.

In the above example, the photograph was annotated using phoneme and word grid annotation data. Those skilled in the art will appreciate that such phoneme and word grid data may be used to comment on many different types of data files. For example, this kind of annotation data can be used in medical applications to annotate 3D video, such as patient x-rays, e.g., NMR scans, ultrasound scans, and the like.

In the above embodiment, a speech recognition system for generating a sequence of phonemes from input speech signals has been used. Those skilled in the art will appreciate that other types of speech recognition systems may be used that generate an output word or sequence of word lattices that can be decomposed into corresponding columns of phonemes as an alternative, for example, to simulate a recognizer that generates phoneme strings ≪ / RTI >

In this embodiment, the insertion, deletion and decoding probabilities have been calculated from the confusion statistics for the speech recognition system using the maximum likelihood estimates of probabilities. Those skilled in the art will recognize that other techniques, such as maximum entropy techniques, can be used to estimate these probabilities. Details of suitable maximum entropy techniques have been described by John Skilling and published by the Institute of Clinical Academic Publishers Pp. 45 to 52 of the title entitled " Maximum Entropy and Bayesian Methods, " published by Kluwer Academic publishers, the contents of which are incorporated herein by reference.

In the above embodiment, the database 29 and the automatic speech recognition device 51 are located in the user terminal 59. [ Those skilled in the art will recognize that this is not necessary. Figure 27 shows an example in which database 29 and search engine 53 are located at remote server 60 and user terminal 59 is connected to network interface devices 67 and 69 and data network 68 (such as the Internet) 29 in accordance with an embodiment of the present invention. In this embodiment, the user terminal 59 can only receive voice queries from the microphone 7. These queries are converted into phoneme and word data by the automatic speech recognition device 51. [ This data is then passed through the data network 68 to the control device 55 which controls the transfer of data to the search engine 53 located in the remote server 60. Next, the search engine 53 performs the search in a manner similar to that in which the search was performed in the above embodiment. Next, the result of the search is sent back from the search engine 53 to the control device 55 via the data network 68. Next, the control device 55 displays appropriate data on the display 57 so as to be viewed by the user 39 in consideration of the retrieval result received from the network again. In addition to locating the database 29 and the search engine 53 in the remote server 60, it is also possible to place the automatic speech recognition device 51 within the remote server 60. This embodiment is shown in Fig. As shown, in this embodiment, an input voice query from a user is transmitted over the input line 61 to a voice encoding device 73 operable to encode the voice for efficient transmission over the data network 68 . The encoded data is then transmitted via network 68 to a control device 55 which transmits the data to the remote server 60, the data being processed by the automatic speech recognition device 51. The phonemes and word data generated by the speech recognition device 51 for the input query are passed to the search engine 53 for use in the database 29 search. The search results generated by the search engine 53 are transmitted to the user terminal 59 via the network interface 69 and the network 68 again. The retrieved results received from the remote server are transmitted to the control device 55 via the network interface device 67 where the control device 55 analyzes the results and displays 57 on the display for viewing by the user 39. [ Lt; / RTI >

In a similar manner, the user terminal 59 has only a typed input from the user and has a search engine and a database located within the remote server. In this embodiment, the voice recording device 75 may likewise be located in the remote server 60. [

In the above embodiments, the dynamic programming algorithm was used to align the sequence of query phonemes with the phonemes. Those skilled in the art will recognize that any alignment technique may be used. For example, a technique unique to identifying all possible alignments may be used. However, dynamic programming is preferably implemented using standard processing hardware.

The manner in which two or more reference sequences of phonemes are compared using multiple programming techniques has been described above. However, as shown in Figs. 2 and 3, the annotations are preferably stored as grids. Those skilled in the art will recognize that the phoneme sequence defined by the grids must be " flattened " with a single sequence of branchless phonemes in order to allow the comparison technique to operate using gratings. A unique approach to doing this is to identify all possible different phoneme sequences defined by the lattice and compare each with each of the query sequences. However, this is not preferred because the common portions of the grid will not be matched multiple times with each query sequence. Thus, the lattice is well flattened by sequentially labeling each phoneme in the lattice according to available time stamp information for each phoneme therein. Next, during dynamic programming alignment, different dynamic programming constraints are used at each DP lattice point to ensure that the path propagates according to the lattice structure.

The table below shows the DP constraints used for some of the phoneme grids shown in FIG. In particular, the first column shows the phoneme numbers (p _i to p ₉ ) assigned to the respective phonemes of the lattice. The middle column corresponds to the actual phoneme in the lattice. The last column shows the phonemes for each phoneme that can be propagated by the path ending at that phoneme at the point of dynamic programming. Although not shown, the middle column will also include the details of the node to which the phoneme is connected and the corresponding phoneme is linked.

Phoneme number phoneme Dynamic programming constraints p ₁ / p / p ₁ ; p ₂ ; p ₃ ; p ₄ p ₂ / ih / p ₂ ; p ₃ ; p ₄ ; p ₅ p ₃ / k / p ₆ p ₃ ; p ₄ ; p ₅ ; p ₇ p ₈ p ₄ / ch / p ₆ ; p ₁₀ p ₄ ; p ₅ ; p ₇ ; p ₉ p ₈ ; p ₁₁ p ₅ / ax / p ₆ ; p ₁₀ ; p ₁₂ p ₅ ; p ₇ ; p ₉ ; p ₁₂ p ₈ ; p ₁₁ ; p ₁₃ p ₁₄ p ₆ / ax / p ₆ ; p ₁₀ ; p ₁₂ ; p ₁₅ p ₁₆ p ₇ / ao / p ₇ ; p ₉ ; p ₁₂ ; p ₁₅ p ₁₆ p ₈ / ah / p ₈ ; p ₁₁ ; p ₁₃ ; p ₁₈ p ₁₄ ; p ₁₇ p ₉ / f / p ₉ ; p ₁₂ ; p ₁₅ ; p ₁₈ p ₁₆ ; p ₁₈

For example, if the dynamic programming path ends at time aligned phoneme p ₄ , the dynamic programming path may remain at phoneme p ₄ or propagate to any phoneme of time-aligned phoneme p ₅ to p ₁₁ . As shown in the diagram, the phonemes at which the path is likely to extend at some point are not consecutively arranged within the time aligned phoneme sequence. For example, in a dynamic programming path which ends at time aligned phoneme p _6, the path may proceed stay in the phoneme or a phoneme p _10, p _12, p _15, or ₁₆ p. In this manner, efficient dynamic programming matching between the input query and the annotation lattice can be achieved by sequentially numbering the phonemes in the lattice and varying the dynamic programming constraints used in dependence on the lattice. Also, those skilled in the art will appreciate that when the input query generates a grid, this can be planarized in a similar manner, so that dynamic programming constraints can be adjusted accordingly.

In the above example, the same phoneme confusion probabilities were used for both annotation and query. Those skilled in the art will recognize that different phoneme confusion probabilities should be used for annotations and phonemes if different recognition systems are used to generate them. These confusion probabilities depend on the recognition system used to generate the phoneme sequence.

In this embodiment, when the annotation or query is generated from the text, it is assumed that the reference sequence of the phonemes corresponding to the typed text is correct. This assumes that spelling of the typed word or words can not be present if wrong or incorrectly typed. Thus, in other embodiments, confusion probabilities may also be used for typed queries and / or comments. That is, equations (4) and (12) can also be used when the annotation or query is both text. The confusion probabilities used may include either spelling wrong, incorrect typing, or both. Those skilled in the art will recognize that the confusion probability for mis-typing will depend on the type of keyboard being used. In particular, the confusion probabilities of typing error words will depend on the layout of the keyboard. For example, if the letter "d" is typed, the keys around the letter "d" key will have a high typographical probability and the keys farther from the "d" key will have typing error probability. As noted above, these typing error probabilities can be used or replaced with confusion probabilities for mis-spelling of words. This spelling error probability can be resolved by analyzing typed documents from a number of different users and monitoring the type of spelling errors that often occur. These spelling error probabilities can also be considered as recording errors caused by mis-keying. In such an embodiment, the dynamic programming constraints used allow insertion and / or deletion at the typing input. For example, the constraints shown in Fig. 11 can be used.

Another choice is that the text is input via a keyboard that assigns one or more characters to each key (such as the keyboard of a mobile phone), where the user can repeatedly < RTI ID = 0.0 > repeatedly Should be pressed. In this embodiment, the confusion probability has a higher typing error confusion probability than the character assigned the same key as the input character is associated with other keys. This is because a typing error has occurred because a key has not been pressed the correct number of times to input a desired character by someone who has transmitted a text message using a mobile phone.

In the above embodiments, the controller calculates the decoding scores for each transition using Equation (4) or (12) above. Instead of summing all the possible phonemes known to the system according to these equations, the control device identifies the unknown phoneme _r maximizing the probability terms in the sum and decodes the corresponding phonemes of the annotation and query Probability can be arranged to use. However, this is undesirable because the phoneme p _r contains additional calculations that maximize the probability term in the sum.

In the first embodiment described above, during the dynamic programming algorithm, Equation 4 was calculated for each ordered pair of phonemes. In the calculation of equation (4), the phoneme and the query phoneme were compared with each phoneme known to the system. It will be appreciated by those skilled in the art that for a given phoneme and query phoneme pair, the multiple probabilities given in equation (4) are close to or equal to zero. Thus, in other embodiments, the annotation and query phoneme pairs can be compared only to a subset of all known phonemes, where the subset is determined ahead of the confusion statistics. To implement this embodiment, the annotation phoneme and the query phoneme can be used to address a lookup table that identifies the model phonemes that are compared with the annotation and query phonemes using Equation (4).

In the above embodiments, the characteristics of the aligned and matched annotations and queries are the units of speech representation. Those skilled in the art will appreciate that the techniques described above may be used in other applications where the nature of the query and annotation may be confused by the inaccuracy of the recognition system that generated the feature sequences. For example, the techniques may be used in an optical character or handwriting recognition system where the recognition system is likely to fail.

While a number of embodiments and variations have been described above, those skilled in the art will recognize that many other embodiments and modifications may be made.

Claims (123)

In the characteristic comparison device, Means for receiving characteristics of a first sequence and characteristics of a second sequence; Means for aligning a characteristic of the first sequence with a characteristic of the second sequence to form a plurality of aligned characteristic pairs; Means for comparing characteristics of each of the aligned feature pairs to generate a comparison score that indicates similarities between the aligned feature pairs; And Means for combining a comparison score for all aligned feature pairs to provide a measure of the similarity between the characteristics of the first sequence and the characteristics of the second sequence / RTI > Wherein the comparison means comprises: Comparing the characteristics of the first sequence of each of the plurality of features taken from the predetermined feature set and the characteristics of the first sequence of the aligned pair against each of the aligned pairs to determine a similarity between the characteristics of the first sequence and the respective characteristics from the set First comparing means for providing a corresponding plurality of intermediate comparison scores indicating; Comparing the characteristics of each of the plurality of characteristics from the set and the characteristics of the second sequence of the aligned pair against each of the aligned pairs to determine the similarity between the characteristics of the second sequence and the respective characteristics from the set, Second comparison means for providing a corresponding plurality of intermediate comparison scores of the second comparison means; And Means for calculating the comparison score for the aligned pair by combining the plurality of intermediate comparison scores, And the characteristic comparison device. The characteristic comparison device of claim 1, wherein the first and second comparison means are operable to compare, respectively, the characteristics of the first sequence and the characteristics of the second sequence with the characteristics of each of the predetermined feature sets. 3. The method of claim 1 or 2, wherein the comparing means is adapted to generate a comparison score for an ordered set of characteristics that indicates a probability of confusing the characteristics of the second sequence of ordered pairs into the characteristics of the first sequence of ordered pairs Operable characteristic comparison device. 4. The apparatus of claim 3, wherein the first and second comparison means are operable to provide an intermediate comparison score indicative of a probability of confusing the corresponding property taken from the predetermined property set into the aligned property pair. 5. The method of claim 4, wherein the computing means is configured to: (i) multiply the intermediate score obtained when comparing the characteristics of the first sequence of the ordered pair and the characteristics of the second sequence to the same characteristic from the set, To provide a comparison score, and (ii) to add the multiplied median score of the result to calculate the comparison score for the aligned pair. 6. The method of claim 5, wherein the characteristics of each of the predetermined feature sets have a certain probability that will occur in a sequence of characteristics, and the calculation means calculates a probability of occurrence of each characteristic from the set used to generate the multiplied intermediate comparison score To each of the multiplied intermediate comparison scores. 7. The apparatus of claim 6, wherein the calculation means is operable to calculate: q _j and a _i are the characteristics of the first sequence and the characteristics of the second sequence, respectively, of the aligned pair; P (q _j | p _r ) is a probability to confuse the set characteristic p _r with the characteristic q _j of the first sequence; P (a _i | p _r ) is the probability that the set characteristic p _r will be confused with the characteristic a _i of the second sequence; P (p _r ) is a probability of a set characteristic p _r occurring in a sequence of characteristics. 8. The apparatus of claim 7, wherein the confusion probabilities for the characteristics of the first sequence and for the characteristics of the second sequence are predetermined and are used to generate each of the first and second sequences. 9. The method according to any one of claims 5 to 8, wherein the intermediate score represents a log probability, and the calculation means is operable to perform the multiplication by adding each intermediate score, And perform an addition of the multiplied scores. 10. The apparatus of claim 9, wherein the combining means is operable to add a comparison score for all aligned pairs to determine the similarity measure. 11. A method according to any one of the preceding claims, wherein the alignment means is operable to identify a characteristic deletion and insertion in a characteristic of the first sequence and in a characteristic of a second sequence, And to generate the comparison score for the feature pairs aligned according to the feature deletion and insertion identified by the alignment means occurring in the vicinity of the pair. 12. Apparatus according to any one of the preceding claims, wherein the means for sorting comprises dynamic programming means for sorting the characteristics of the first sequence and the characteristics of the second sequence using dynamic programming techniques. 13. The apparatus according to claim 12, wherein the dynamic programming means is operable to step-by-step determine a plurality of possible arrangements between the characteristics of the first sequence and the characteristics of the second sequence, And to determine a comparison score for each possible pair of aligned properties. 14. The apparatus of claim 13, wherein the comparing means is operable to generate the comparison score during a phased determination of the possible alignment. 15. The apparatus according to any one of claims 12 to 14, wherein the dynamic programming means is operable to determine an optimum alignment between the characteristics of the first sequence and the characteristics of the second sequence, Lt; RTI ID = 0.0 > 1, < / RTI > 15. The characteristic comparison device according to claim 13 or 14, wherein the combining means is operable to provide the similarity measure by combining all comparison scores for all possible aligned feature pairs. 17. The method according to any one of claims 1 to 16, wherein each characteristic in the characteristic of the first sequence and the characteristic of the second sequence belongs to the set of predetermined characteristics, To provide said intermediate score using predetermined data correlating the characteristics of said characteristic points. 18. The apparatus of claim 17, wherein the predetermined data used by the first comparing means is in accordance with a system used to generate the characteristics of the first sequence, and wherein the predetermined data used by the second comparing means Is used to generate the characteristics of the second sequence different from the predetermined data used by the second sequence. 19. The apparatus of claim 17 or 18, wherein the predetermined or respective predetermined data comprises a probability of confusing the characteristic for each characteristic of the characteristic set with each of the other characteristics of the characteristic set. 20. The apparatus of claim 19, wherein the predetermined or respective predetermined data further comprises a probability of inserting a characteristic into a sequence of characteristics for each characteristic of the set of characteristics. 21. Apparatus according to any one of claims 19 to 20, wherein the predetermined or respective predetermined data further comprises a probability of deleting the characteristic from the sequence of characteristics for each characteristic of the set of characteristics. 22. Apparatus according to any one of the preceding claims, wherein the characteristics of the first sequence and the characteristics of the second sequence represent a temporal sequential signal. 23. The apparatus of any one of claims 1 to 22, wherein the characteristics of the first sequence and the characteristics of the second sequence are audio signals. 24. The apparatus of claim 23, wherein the characteristics of the first sequence and the characteristics of the second sequence represent text and / or speech. 25. The apparatus of claim 24, wherein each characteristic is a sub-word unit of text or speech. The characteristic comparison apparatus according to claim 25, wherein each characteristic is a phoneme. 27. The method of any one of claims 1 to 26, wherein the characteristics of the first sequence include a plurality of sub-word units generated from a typed input, and wherein the first comparing means comprises a mis-typing probability And / or to provide the intermediate comparison score using a mis-spelling probability. 28. The method of any one of claims 1 to 27, wherein the characteristics of the second sequence include a sequence of sub-word units generated from a spoken input, and wherein the second comparison means uses a recognition error probability A characteristic comparison device operable to provide a median score. 29. The method according to any one of claims 1 to 28, The receiving means being operable to receive characteristics of three or more sequences, The alignment means being operable to align a characteristic of each characteristic of the received sequence to form a plurality of aligned characteristic groups, Wherein the comparing means is operable to compare the characteristics of each ordered property group to generate a comparison score representing the similarity between the sorted property groups, Wherein the combining means is operable to combine a comparison score for all ordered property groups to provide a measure of similarity between properties of the three or more sequences. 30. The apparatus of claim 29, wherein the alignment means is operable to simultaneously align the sequence of characteristics with respect to each other. 31. The method according to any one of claims 1 to 30, The receiving means being operable to receive a plurality of characteristics of a second sequence, Wherein the alignment means is operable to align the characteristics of the first sequence with the characteristics of each second sequence to form a plurality of aligned property pairs for each alignment, Wherein the combining means is operable to combine a comparison score for each alignment to provide a respective measure of similarity between the characteristics of the first sequence and the characteristics of the plurality of second sequences. 32. The apparatus of claim 31, further comprising means for comparing a plurality of similarity measurements output by the combining means, and means for outputting a signal representative of characteristics of a second sequence most similar to the characteristics of the first sequence Device. The characteristic comparison device according to claim 31 or 32, wherein the combining means includes normalization means for normalizing each of the similarity measurement values. 34. The feature comparison device of claim 33, wherein the normalization means is operable to normalize each similarity measure by dividing each similarity measure by a respective normalization score that varies with a length of a characteristic of the corresponding second sequence. 35. The apparatus of claim 34, wherein each normalization score varies according to a property sequence of characteristics of the corresponding second sequence. 36. The characteristic comparison device according to claim 34 or 35, wherein each normalization score is changed with a corresponding intermediate comparison score calculated by the second comparison means. 37. Apparatus according to any one of claims 33 to 36, wherein the means for sorting comprises dynamic programming means for aligning the characteristics of the first sequence and the characteristics of the second sequence using dynamic programming techniques, And to calculate each normalization score during the stepwise computation of the possible alignment by the dynamic programming means. 38. The apparatus of claim 37, wherein the normalization means is operable to calculate the following equation for each possible aligned property pair, P (a _i | p _r ) is a probability that the set property p _r is confused with the characteristic a _i of the second sequence, and P (p _r ) is the probability of the set property p _r occurring in the sequence of characteristics. 39. The feature comparison device of claim 38, wherein the normalization means is operable to calculate the respective normalization by multiplying normalized terms that are calculated for each aligned feature pair. An apparatus for searching a database that includes a plurality of information entries, each having an associated annotation that identifies information to be retrieved and includes a sequence of annotation properties, Means for receiving a plurality of renditions of an input query; Means for transforming each rendition of the input query into a sequence of query characteristics representing the rendition; Means for comparing the query property of each rendition to the annotation property of each annotation to provide a comparison result set; Means for combining the comparison results obtained by comparing the query properties of each rendition with the annotation properties of the same annotation to provide a measure of similarity between the input query and the annotation for each annotation; And Means for identifying said information to be retrieved from said database using similarity measures provided by said combining means for all annotations; And a database search device. 41. The apparatus of claim 40, wherein the comparing means is operable to compare the query characteristics of each rendition simultaneously with the annotation characteristics of the current annotation. 42. The method according to claim 40 or 41, Wherein the comparison means comprises: Means for aligning the query properties of each rendition with the annotation properties of the current annotation to form a plurality of ordered property groups each comprising a query property and an annotation property from each rendition; And A feature comparator that compares the characteristics of each ordered feature group to produce a comparison score that indicates similarity between the groups of aligned features Lt; / RTI > Wherein the combining means is operable to combine a comparison score for all ordered property groups for a current annotation to provide a measure of similarity between the input query and the current annotation. 43. The apparatus of claim 42, wherein the feature comparator compares the group characteristics with each of a plurality of respective characteristics taken from a set of predetermined properties for each feature of the aligned group, Each characteristic comparison means providing a corresponding plurality of intermediate comparison scores indicative of the similarity of the plurality of comparison scores; And means for calculating the comparison score for the sorted group by combining the plurality of intermediate comparison scores generated by the respective feature comparison means. 44. The apparatus of any one of claims 40 to 43, wherein the sequence of voice annoying properties for some or all of the annotations is generated from an audio annotation signal. 45. The apparatus of any of claims 40 to 44, wherein the sequence of voice annotation properties for some or all of the annotations is generated from a text annotation. 46. The database search apparatus according to any one of claims 40 to 45, wherein the conversion means includes a speech recognition system. 47. The apparatus of any one of claims 40-46, wherein the at least one information entry is an associated comment. An apparatus for searching a database comprising a plurality of information entries, each having an associated annotation that identifies information to be retrieved and comprises a sequence of characteristics, Means for receiving an input query comprising a sequence of characteristics; 39. Apparatus according to any one of claims 1 to 39, for comparing a query sequence of characteristics to a characteristic of each annotation to provide a comparison result set. And Means for identifying said information to be retrieved from said database using said comparison result And a database search device. An apparatus for searching a database that includes a plurality of information entries, each having an associated annotation that identifies information to be retrieved and comprises a sequence of voice properties, Means for receiving an input query comprising a sequence of voice characteristics; Means for comparing the query sequence of the speech characteristics with the speech characteristics of each annotation to provide a comparison result set; And Means for identifying said information to be retrieved from said database using said comparison result / RTI > The comparison means comprising a plurality of different comparison modes of operation, The device (I) means for determining whether the query sequence of the speech characteristics is an audio signal or a text, (ii) whether the sequence of speech characteristics of the current annotation is an audio signal or a text, and outputting a determination result ; And And means for selecting an operation mode of the comparison means for the current annotation according to the determination result Further comprising: a database searching unit for searching for a database of the database. 50. The apparatus of claim 49, wherein when said determining means determines that both said input query and said current annotation are generated from speech, And to select an operation mode to operate the database. 51. The apparatus of any of claims 48 to 50, wherein the at least one information entry is an associated comment. In the characteristic comparison device, Means for receiving first and second sequences of query properties, each representing a rendition of an input query; Means for receiving a sequence of annotation properties; Means for aligning the query properties of each rendition with the annotation properties of the annotation to form a plurality of ordered property groups each comprising a query property and an annotation property from each rendition; Means for comparing characteristics of each ordered property group to produce a comparison score that indicates a similarity between properties of the sorted group; And Means for combining a comparison score for the characteristics of all ordered groups to provide a measure of similarity between the input query and the renditions of the annotation; / RTI > Wherein the comparison means comprises: Comparing the characteristics of the first query sequence of the sorted group with each of the plurality of characteristics taken from the predetermined feature set and for each sorted group to determine the difference between the characteristics of the first query sequence and the respective characteristics from the set A first characteristic comparator that provides a corresponding plurality of intermediate comparison scores indicative of similarity; Comparing the characteristics of the second query sequence of the sorted group to each of the plurality of characteristics from the set for each sorted group to determine a similarity between the characteristics of the second query sequence and respective characteristics from the set A second characteristic comparator for providing an additional corresponding plurality of intermediate comparison scores representing the intermediate comparison score; For each sorted group, an annotation property of the sorted group to each of the plurality of properties from the set to determine an additional corresponding number of properties representing the similarity between the annotation property and the respective property from the set A third characteristic comparator for providing an intermediate comparison score; And Means for calculating the comparison score for the sorted group by combining the plurality of intermediate comparison scores, And the characteristic comparison device. An apparatus for searching a database comprising a plurality of information entries, each having an associated annotation that identifies information to be retrieved and comprises a sequence of voice annotation properties, Means for receiving a plurality of renditions of a verbal input query; Means for transforming each rendition of the input query into a sequence of voice query characteristics representing the voice in the rendition; Means for comparing a voice annotation characteristic of each annotation with a voice query characteristic of each rendition to provide a measure of similarity between the input query and each annotation; And Means for identifying said information to be retrieved from said database using similarity measures provided by said combining means for all said annotations; / RTI > The comparison means comprising a plurality of different comparison modes of operation, The apparatus comprises: Means for determining whether a sequence of speech characteristics of the current annotation is generated from an audio signal or from text and outputting a determination result; And And means for selecting an operation mode of the comparison means for the current annotation according to the determination result Further comprising: a database searching unit for searching for a database of the database. In the characteristic comparison method, Receiving characteristics of the first sequence and characteristics of the second sequence; Aligning the characteristics of the first sequence with the characteristics of the second sequence to form a plurality of aligned property pairs; Comparing each ordered property pair to generate a comparison score that indicates similarity between the ordered property pairs; And Combining the comparison scores for all aligned feature pairs to provide a measure of the similarity between the characteristics of the first sequence and the characteristics of the second sequence Lt; / RTI > Wherein the comparing comprises: Comparing the characteristics of each of the plurality of features taken from the predetermined feature set for each aligned pair with the characteristics of the first sequence of the aligned pairs to determine a similarity between the characteristics of the first sequence and the respective characteristics from the set A first comparison step of providing a corresponding plurality of intermediate comparison scores; For each aligned pair, comparing the characteristics of each of the plurality of characteristics from the set with the characteristics of the second sequence of the aligned pairs to determine the similarity between the characteristics of the second sequence and the respective characteristics from the set. A second comparison step of providing a corresponding plurality of intermediate comparison scores of the comparison result; And Calculating the comparison score for the aligned pair by combining the plurality of intermediate comparison scores Wherein the characteristic comparison method comprises the steps of: 55. The method of claim 54, wherein the first and second comparison steps compare respective characteristics of the predetermined feature set with characteristics of the first sequence and characteristics of the second sequence. 54. The method of claim 54 or 55, wherein the comparing step comprises comparing scores of the ordered pairs of characteristics indicating a probability of confusing the characteristics of the second sequence of the ordered pairs with the characteristics of the first sequence of the aligned pairs How to compare characteristics to generate. 57. The method of claim 56, wherein the first and second comparison steps provide an intermediate comparison score that indicates a probability of confusing a corresponding property taken from the predetermined property set into an ordered property pair. 58. The method of claim 57, wherein the step of calculating comprises: (i) multiplying the intermediate score obtained when comparing the characteristics of the first sequence of the ordered pair and the characteristics of the second sequence to the same characteristic from the set, Providing a comparison score, and (ii) adding the intermediate scores of the result multiplied to calculate a comparison score for the ordered pair. 59. The computer-readable medium of claim 58, wherein each characteristic of the predetermined feature set has a certain probability that will occur in a property sequence, and wherein the calculating step further comprises calculating each of the multiplied intermediate comparison scores using the set And the probability of each occurrence of the characteristic from the second characteristic. 60. The method of claim 59, wherein the calculating step calculates the following equation, q _j and a _i are the characteristics of the first sequence and the characteristics of the second sequence, respectively, of the aligned pair; P (q _j | p _r ) is a probability to confuse the set characteristic p _r with the characteristic q _j of the first sequence; P (a _i | p _r ) is the probability that the set characteristic p _r will be confused with the characteristic a _i of the second sequence; P (p _r ) is a probability of a set characteristic p _r occurring in a sequence of characteristics. 61. The method of claim 60, wherein the confusion probabilities for the characteristics of the first sequence and the characteristics of the second sequence are predetermined and are used to generate each of the first and second sequences. 62. The method of any one of claims 58 to 61, wherein the intermediate score represents a log probability, the calculating step performs the multiplication by adding each intermediate score, and performing the log addition calculation to calculate the multiplied score < RTI ID = And the addition of the characteristic values is performed. 63. The method of claim 62, wherein the combining step adds a comparison score for all aligned pairs to determine the similarity measure. 63. The method of any one of claims 54 to 63, wherein the aligning step identifies a feature deletion and insertion in the characteristics of the first sequence and the features of the second sequence, And generating a comparison score for the property pairs sorted in accordance with the property deletion and insertion identified by the sorting step occurring in the query. 65. The method of any of claims 54 to 64, wherein the aligning step uses dynamic programming techniques to align the properties of the first sequence and the properties of the second sequence. 66. The method of claim 65, wherein the step of arranging stepwise determines a plurality of possible alignments between the characteristics of the first sequence and the characteristics of the second sequence, A characteristic comparison method for determining a comparison score for each. 67. The method of claim 66, wherein the comparing step generates the comparison score during a staged determination of the possible alignment. 67. The method of any one of claims 65 to 67, wherein said aligning step determines an optimal alignment between the characteristics of said first sequence and the characteristics of a second sequence, And providing the similarity measure by combining the comparison scores only for the comparison score. 67. The method of claim 67 or 68, wherein the combining step provides the similarity measure by combining all comparison scores for all possible aligned feature pairs. 71. The method according to any one of claims 54 to 69, wherein each characteristic in the characteristic of the first sequence and the characteristic of the second sequence belongs to the set of predetermined characteristics, ≪ / RTI > wherein the intermediate score is provided using predetermined data relating characteristics of the set to each other. 70. The method of claim 70, wherein the predetermined data used in the first comparison step is in accordance with a system used to generate a characteristic of the first sequence, and the predetermined data used in the second comparison step is used in the first comparison step Wherein the second sequence is different from the predetermined data and used to generate the characteristics of the second sequence. 72. The method of claim 70 or claim 71, wherein the predetermined or respective predetermined data comprises a probability to confuse the property with each of the other properties of the property set for each property of the property set. 73. The method of claim 72, wherein the predetermined or respective predetermined data further comprises the probability of inserting a characteristic into the sequence of characteristics for each characteristic of the set of characteristics. 74. The method of claim 72 or 73, wherein the predetermined or respective predetermined data further comprises the probability of deleting the property from the sequence of properties for each property of the feature set. 74. The method of any one of claims 54 to 74, wherein the characteristics of the first sequence and the characteristics of the second sequence are time sequential signals. The method according to any one of claims 54 to 75, wherein the characteristics of the first sequence and the characteristics of the second sequence are audio signals. 77. The method of claim 76, wherein the characteristics of the first sequence and the characteristics of the second sequence are speech. 77. The method of claim 77, wherein each characteristic is a lower word unit of speech. 82. The method of claim 78, wherein each characteristic is a phoneme. 80. The method of any one of claims 54 to 79, wherein the characteristics of the first sequence include a plurality of sub-word units, and wherein the first comparing step uses a typing error and / or a spelling error probability to calculate an intermediate comparison score Wherein the characteristic comparison method comprises: 79. A method according to any one of claims 54 to 80, wherein the characteristics of the second sequence include a sub-word unit sequence generated from a verbal input, and the second comparison step uses a recognition error probability to provide a median score . 83. The method according to any one of claims 54 to 81, Wherein the receiving step receives the characteristics of three or more sequences, Wherein the aligning step aligns the characteristics of each of the characteristics of the received sequence to form a plurality of aligned property groups, Wherein the comparing step compares the characteristics of each ordered property group to generate a comparison score representing the similarity between the sorted property groups, The combining step combines the comparison scores for all ordered property groups to provide a measure of the similarity between the properties of the three or more sequences How to compare characteristics. 83. The method of claim 82, wherein the aligning step simultaneously aligns the sequence of characteristics with respect to each other. 83. The method according to any one of claims 54 to 83, Wherein the receiving step receives a plurality of characteristics of the second sequence, Wherein the aligning step aligns the characteristics of the first sequence with the characteristics of each second sequence to form a plurality of aligned feature pairs for each alignment, Wherein the combining step combines a comparison score for each alignment to provide a respective measure of similarity between the characteristics of the first sequence and the characteristics of the plurality of second sequences. 85. The method of claim 84, further comprising: comparing the plurality of similarity measurements output by the combining means; and outputting a signal representative of a characteristic of a second sequence most similar to the characteristics of the first sequence Comparison method. 84. The method of claim 84 or 85, wherein the combining step includes a normalizing step of normalizing each of the similarity measures. 90. The method of claim 86, wherein the normalizing step normalizes each similarity measure by dividing each similarity measure by a respective normalization score that varies with a length of a characteristic of the corresponding second sequence. 89. The method of claim 87, wherein each normalization score varies according to a sequence of characteristics of a characteristic of the corresponding second sequence. The characteristic comparison method according to claim 87 or 88, wherein each normalization score is changed with a corresponding intermediate comparison score calculated by the second comparison step. 90. A method according to any one of claims 86-89, wherein the step of arranging stepwise determines a plurality of possible alignments between the characteristics of the first sequence and the characteristics of the second sequence, Wherein the normalization step computes a respective normalization score during the stepwise computation of the possible alignment by the aligning step. 91. The method of claim 90 wherein the normalizing step calculates the following equation for each possible aligned property pair, P (a _i | p _r ) is the probability of confusion of the set property p _r with the property a _i of the second sequence, and P (p _r ) is the probability of the set property p _r occurring in the sequence of properties. 92. The method of claim 91, wherein the normalizing step calculates each normalization by multiplying normalized terms that are calculated for each ordered property pair. CLAIMS 1. A method for retrieving a database comprising a plurality of information entries each having an associated annotation identifying the information to be retrieved and comprising a sequence of annotation properties, Receiving a plurality of renditions of an input query; Transforming each rendition of the input query into a sequence of query characteristics representing the rendition; Comparing the query characteristics of each rendition to the annotation properties of each annotation to provide a comparison result set; Combining the comparison results obtained by comparing the query properties of each rendition to the annotation properties of the same annotation to provide a measure of similarity between the input query and the annotation for each annotation; And Identifying the information to be retrieved from the database using similarity measures provided by the combining step for all annotations / RTI > The method of claim 93, wherein the comparing step simultaneously compares the query characteristics of each rendition with the annotation characteristics of the current annotation. 95. The method of claim 93 or 94, Wherein the comparing comprises: Aligning the query characteristics of each rendition with the annotation properties of the current annotation to form a plurality of sorted feature groups each including a query property and an annotation property from each rendition; And Using a property comparator that compares the characteristics of each ordered property group to produce a comparison score that indicates a similarity between properties of the sorted group Lt; / RTI > Wherein the combining step combines a comparison score for all ordered property groups for the current annotation to provide a measure of the similarity between the input query and the current annotation. 95. A method according to any one of claims 93 to 95, wherein the sequence of the respective query characteristics and the sequence of the annotation properties represent audio signals. 96. The method of claim 96, wherein the sequence of each query characteristic and the sequence of annotative properties are voices. 97. The method of claim 97, wherein each characteristic is a lower word unit of speech. 97. The method of claim 98, wherein each characteristic is a phoneme. 100. The method of any one of claims 93 to 99, wherein the sequence of voice annotation properties for some or all of the annotations is generated from an audio annotation signal or a text annotation. In the characteristic comparison method, Receiving characteristics of the first sequence and characteristics of the second sequence; Aligning characteristics of the second sequence with characteristics of the first sequence; Comparing each aligned property pair to generate a comparison score for the aligned property pair; And Combining the comparison scores for all aligned feature pairs to provide a measure of the similarity between the characteristics of the first sequence and the characteristics of the second sequence Lt; / RTI > Wherein the comparing comprises: A first comparison step of comparing each of a plurality of possible characteristics with an ordered characteristic of the first sequence and providing a corresponding plurality of intermediate comparison scores; And A second comparison step of comparing each of the plurality of possible characteristics with the aligned characteristics of the second sequence to provide an additional corresponding plurality of intermediate comparison scores Lt; / RTI > Wherein combining the plurality of intermediate comparison scores provides the comparison score for the aligned pair. CLAIMS 1. A method for retrieving a database comprising a plurality of information entries each having an associated annotation identifying the information to be retrieved and comprising a sequence of properties, Receiving an input query comprising a sequence of characteristics; Using the method according to any one of claims 54 to 101 for comparing the query sequence of the characteristics and characteristics of each annotation to provide a comparison result set; And Identifying the information to be retrieved from the database using the comparison result / RTI > CLAIMS What is claimed is: 1. A method for searching a database comprising a plurality of information entries each having an annotation identifying information to be retrieved and comprising a sequence of voice properties, Receiving an input query comprising a sequence of speech characteristics; Comparing the query sequence of the speech characteristics with the speech characteristics of each annotation to provide a comparison result set; And Identifying the information to be retrieved from the database using the comparison result Lt; / RTI > The comparing step may use a plurality of different comparison techniques to perform the comparison, The method comprises: (I) determining whether the query sequence of the speech characteristic is an audio signal or a text, (ii) determining whether the sequence of the speech characteristic of the current annotation is an audio signal or a text, and outputting the determination result; And Selecting a technique to be used to perform the comparing step for the current annotation according to the determination result Further comprising the steps of: 111. The method of claim 103, wherein when the determining step determines that the input query and the current annotation are both generated from a voice, the selecting step comprises searching a database for performing the method according to any one of claims 54 to 101. & Way. CLAIMS 1. A method for retrieving a database comprising a plurality of information entries each having an associated annotation identifying the information to be retrieved and comprising a sequence of annotation properties, Receiving a plurality of renditions of an input query; Transforming each rendition of the input query into a sequence of query characteristics representing the rendition; Comparing the annotation properties of each annotation with the query properties of each rendition to provide a comparison result set; Combining the comparison results obtained by comparing the annotation properties of the same annotation with the query characteristics of each rendition to provide a measure of similarity between the input query and the annotation for each annotation; And Identifying said information to be retrieved from said database using said similarity measure provided by said combining step for all said annotations / RTI > 111. The database searching method according to claim 105, wherein the comparing step simultaneously compares the query characteristics of each rendition with the annotation characteristics of the current annotation. 107. The method of claim 105 or 106, Wherein the comparing comprises: Aligning the query characteristics of each rendition with the annotation properties of the current annotation to form a plurality of sorted feature groups each including a query property and an annotation property from each rendition; And Using a property comparator that compares the characteristics of each ordered property group to generate a comparison score that indicates a similarity between the groups of aligned properties Lt; / RTI > Wherein the combining step combines a comparison score for all ordered property groups for the current annotation to provide a measure of the similarity between the input query and the current annotation. 107. The apparatus of claim 107, wherein the feature comparator compares a respective feature of the group with a predetermined feature set to provide a corresponding plurality of intermediate comparison scores that indicate a similarity between the group feature and each feature from the set And calculating the comparison score for the sorted group by combining the corresponding generated plurality of intermediate comparison scores. 109. The database search method according to any one of claims 105 to 108, wherein the sequence of the respective query characteristics and the sequence of the annotative properties represent a time sequential signal. 109. The method of any one of claims 105 to 109, wherein the sequence of the respective query characteristics and the sequence of the annotation properties represent audio signals. 112. The method of claim 110, wherein the sequence of each query characteristic and the sequence of the annotative characteristic are voices. 111. The method of claim 111, wherein each characteristic is a lower word unit of speech. The method of claim 112, wherein each characteristic is a phoneme. 115. The method of any one of claims 105 to 113, wherein the sequence of voice annotation properties for some or all of the annotations is generated from an audio annotation signal. 115. The method of any one of claims 105 to 113, wherein the sequence of voice annotation properties for some or all of the annotations is generated from a text annotation. 115. The method of any one of claims 105 to 115, wherein the converting step uses a speech recognition system. 114. The method of any one of claims 105 to 116, wherein the at least one information entry is an associated comment. In the characteristic comparison method, The method comprising: receiving first and second sequences of query characteristics each representing a rendition of an input query; Receiving a sequence of annotation properties; Aligning the query properties of each rendition with the annotation properties of the annotation to form a plurality of ordered property groups each comprising a query property and an annotation property from each rendition; Comparing characteristics of each ordered property group to generate a comparison score that indicates similarities between properties of the sorted group; And Combining the comparison scores for the characteristics of all ordered groups to provide a measure of similarity between the input query and the renditions of the annotation; Lt; / RTI > Wherein the comparing comprises: Comparing the characteristics of the first query sequence of the sorted group with each of the plurality of characteristics taken from the set of predetermined characteristics for each sorted group to determine a difference between the characteristics of the first query sequence and the respective characteristics from the set Providing a corresponding plurality of intermediate comparison scores indicative of similarity; Comparing the characteristics of each of the plurality of characteristics from the set and the characteristics of the second query sequence of the sorted group for each sorted group to determine a similarity between the characteristics of the second query sequence and the respective characteristics from the set, Providing a further corresponding plurality of intermediate comparison scores representing a plurality of intermediate comparison scores; For each sorted group, comparing each of the plurality of properties from the set with the annotation properties of the ordered group to determine an additional corresponding plurality of annotations representing the similarity between the annotation property and the respective property from the set Providing an intermediate comparison score; And Calculating the comparison score for the sorted group by combining a plurality of intermediate comparison scores Wherein the characteristic comparison method comprises the steps of: CLAIMS What is claimed is: 1. A method for retrieving a database comprising a plurality of information entries each having an associated annotation identifying the information to be retrieved and comprising a sequence of voice annotation properties, Receiving a plurality of renditions of a verbal input query; Converting each rendition of the input query into a sequence of voice query characteristics representing the voice in the rendition; Comparing the voice annotation characteristics of each annotation with the voice query characteristics of each rendition to provide a measure of similarity between the input query and each annotation; And Identifying the information to be retrieved from the database using the similarity measure provided by the combining step for all annotations Lt; / RTI > Wherein the comparing comprises a plurality of different comparison modes of operation, The method comprises: Determining whether a sequence of speech characteristics of the current annotation is generated from an audio signal or from text and outputting a determination result; And Selecting an operation mode of the comparison step for the current annotation according to the determination result Further comprising the steps of: 119. The method of any one of claims 102 to 119, wherein the at least one information entry is an associated annotation. 120. The method according to any one of claims 54 to 120, wherein the method steps are performed in the claimed order. 121. A storage medium storing a processor capable of executing instructions that control a processor to perform the method according to any one of claims 54 to 121. 121. A processor capable of executing instructions to control a processor to perform the method of any one of claims 54 to 121.