JP5210440B2 - Method, program and apparatus for high speed speech retrieval - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a high-speed voice retrieval method and apparatus. <P>SOLUTION: A large voice database in a multiprocessor system is retrieved to specify a target voice clip. The large voice database is divided into a plurality of smaller groups, and the plurality of small groups are dynamically scheduled for a plurality of available processors within the system. The processor divides each group into a plurality of smaller segments, extracts a voice feature from the segment and uses a common component Gaussian mixed model (CCGMM) to model the segment, thereby processing the plurality of scheduled groups in parallel. One processor further extracts a voice feature from the target voice clip and uses the CCGMM to model the extracted voice feature. On the basis of a KL distance between the target voice clip and each of the segments, it is determined whether the segment matches the target voice clip. <P>COPYRIGHT: (C)2012,JPO&amp;INPIT

Description

  The present disclosure relates generally to signal processing and multimedia applications. More specifically, but not exclusively, it relates to a fast voice search and voice fingerprint method and apparatus.

  Voice search (eg, searching for a large audio stream for an audio clip and performed even if the audio stream is corrupted / distorted) has many uses, for example, broadcast music / Examples include commercial analysis, copyright management on the Internet, or identifying metadata for uncategorized audio clips. A typical voice search system is designed for serial, single processor systems. Such a search system usually takes a long time to search for a target audio clip in a large audio stream. However, speech search systems are often required to operate efficiently on large speech databases, for example, must search large databases in a very short time (eg, in near real time). In addition, a part or all of the voice database may be distorted, damaged, and / or compressed. For this reason, the voice search system is sufficiently robust to identify the voice segment even if the same voice segment as the target voice clip is distorted, damaged and / or compressed. There is a need. Accordingly, there is a need for a voice search system that can quickly and robustly search for target voice clips from a large voice database.

  The features and advantages of the disclosed subject matter will become apparent from the following detailed description of the subject matter.

1 is a diagram illustrating an example of a computing system in which robust and parallel voice searches can be performed based on a voice search module. FIG.

FIG. 6 illustrates another example of a computing system in which robust and parallel voice searches can be performed based on a voice search module.

FIG. 10 illustrates yet another example of a computing system in which robust and parallel voice searches can be performed based on a voice search module.

It is a block diagram which shows an example of the voice search module which performs robust voice search.

It is a figure which shows the operation example of the robust voice search module shown in FIG.

It is a block diagram which shows an example of the voice search module which performs robust and parallel voice search in a multiprocessor system.

FIG. 3 illustrates a method for dividing a large speech database into small groups to perform robust and parallel speech searches in a multiprocessor system. FIG. 3 illustrates a method for dividing a large speech database into small groups to perform robust and parallel speech searches in a multiprocessor system. FIG. 3 illustrates a method for dividing a large speech database into small groups to perform robust and parallel speech searches in a multiprocessor system.

FIG. 4 is a pseudo code illustrating an example of a process for performing robust and parallel speech retrieval in a multiprocessor system.

  According to embodiments of the presently disclosed subject matter, a robust and parallel search method may be used to search a large audio stream or large audio database in a multiprocessor system for a target audio clip. A large speech database may be divided into a plurality of small groups. These small groups may be dynamically scheduled to be repaired by the processors or processing cores available in the multiprocessor system. The processor or processing core may process the scheduled groups in parallel. Such parallel processing involves dividing each group into smaller segments, extracting speech features from the segments, and modeling the segments using a Common Component Gaussian Mixture Model (CCCGMM). Made by. The length of these segments may be the same as the length of the target audio clip. Prior to processing any group, one processor or processing core may extract audio features from the target audio clip and model them using CCGMM. A Cullback liber (KL) distance or a KL maximum distance between the target audio clip model and each segment of the group may be further calculated. If the distance is less than or equal to the predetermined value, the corresponding segment is identified as the target audio clip.

  If the distance exceeds a predetermined value, the processor or processing core may omit any number of segments and continue searching for the target audio clip. When the processor or processing core finishes searching for one group, a new group to be processed is given to search for the target audio clip and the search is performed for all groups. The size of the group can be determined so as to reduce load impedance and computation duplication. Furthermore, input / output (I / O) can be optimized to improve the efficiency of parallel processing of voice groups performed by multiple processors or processing cores.

  In this specification, the expression "one embodiment" or "an embodiment of the disclosed subject matter" means that a particular feature, structure, or characteristic described in connection with the embodiment is at least a disclosed subject matter. It is included in one embodiment. Thus, although the phrase “one embodiment” is used many times in this specification, all do not necessarily refer to the same embodiment.

  FIG. 1 is a diagram illustrating an example of a computing system 100 in which robust and parallel voice searches can be performed based on a voice search module 120. Computing system 100 includes one or more processors 110 coupled to a system interconnect 115. The processor 110 may have a plurality of processing cores or a number of processing cores (for convenience of explanation, the expression “a plurality of cores” will hereinafter mean both a plurality of processing cores and a number of processing cores). The processor 110 may include a speech search module 120 that performs robust and parallel speech searches using multiple cores. The voice search module may include a plurality of components such as a division mechanism, a schedule, and a plurality of voice search units (more detailed description will be described later with reference to FIGS. 4 to 6). One or more components included in the voice search module may be arranged in one core, and the other components may be arranged in another core.

  The speech search module may first divide a large speech database into multiple small groups, or may divide a large speech stream into smaller substreams that are partially overlapping. Subsequently, one core processes the audio clip to be searched (“target audio clip”) to build a model of the target audio clip. Meanwhile, the voice search module dynamically schedules voice small groups / substreams for a plurality of cores. The multiple cores divide each group / substream into multiple segments to build a model for each audio segment. This is done in parallel. The size of each segment may be equal to the size of the target audio clip. Model each audio segment and target audio clip using a Gaussian mixture model (GMM) containing multiple Gaussian components common to all audio segments including both the target audio clip and the audio database / stream. It may be converted. Once the audio segment model has been constructed, the Cullback-Liver (KL) distance or the KL maximum distance between the segment model and the target audio clip model may be calculated. If the distance is less than or equal to a predetermined value, the audio segment can be identified as the target audio clip. The search process may continue until all voice groups / substreams have been processed.

  The computing system 100 may further comprise a chipset 130 that is coupled to the system interconnect 115. The chipset 130 may have one or more integrated circuit packages or chips. The chipset 130 may have one or more device interfaces 135 that support data transfer with other components 160 of the computing system 100. Other components 160 are, for example, BIOS firmware, a keyboard, a mouse, a storage device, a network interface, and the like. Chipset 130 may be coupled to a peripheral device interconnect (PCI) bus 170. The chipset 130 may have a PCI bridge 145 that provides an interface to the PCI bus 170. The PCI bridge 145 may provide a data path between the processor 110 and other components 160 and peripheral devices. The peripheral devices are, for example, an audio device 180 and a disk drive 190. Although not shown in FIG. 1, other devices may also be coupled to the PCI bus 170.

  The chipset 130 may also have a memory controller 125 coupled to the main memory 150. Main memory 150 may store sequences of instructions and data executed by multiple cores of processor 110 or any other device in the system. The memory controller 125 may access the main memory 150 in response to memory transactions associated with multiple cores of the processor 110 and other devices within the computing system 100. According to one embodiment, the memory controller 125 may be located in the processor 110 or other circuitry. Main memory 150 may include various memory devices that provide addressable storage locations from which memory controller 125 reads and writes data. Main memory 150 has one or more different types of memory devices, such as dynamic random access memory (DRAM) devices, synchronous DRAM (SDRAM) devices, double data rate (DDR) SDRAM devices, or other memory devices. Also good.

  FIG. 2 is a diagram illustrating another example computing system 200 in which robust and parallel voice searches can be performed using the voice search module 240. System 200 may include multiple processors, such as processor 0 220A. One or more processors in system 200 may have many cores. The system 200 may include a voice search module 240 that performs robust and parallel voice searches with multiple cores. The voice search module may include a plurality of components such as a division mechanism, a schedule, and a plurality of voice search units (more detailed description will be described later with reference to FIGS. 4 to 6). One or more components included in the voice search module may be arranged in one core, and the other components may be arranged in another core. The processors in system 200 may be connected to each other by system interconnect 210. The system interconnect 210 may be a front side bus (FSB). Each processor may be connected to an input / output (I / O) device and memory 230 via the system interconnect. All cores may receive audio data from memory 230.

  FIG. 3 is a diagram illustrating another example computing system 300 that may perform robust and parallel voice searches using the voice search module 340. In system 300, system interconnect 310 that connects multiple processors (eg, 320A, 320B, 320C, and 320D) is a link-based point-to-point connection. Each processor may be connected to the system interconnect via a link hub (eg, 330A, 330B, 330C, and 330D). According to some embodiments, the link hub may be co-located with the memory controller, which regulates traffic for system memory. One or more processors may include many cores. The system 300 may include a speech search module 340 that performs robust and parallel speech searches with multiple cores. The voice search module may include a plurality of components such as a division mechanism, a schedule, and a plurality of voice search units (more detailed description will be described later with reference to FIGS. 4 to 6). One or more components included in the voice search module may be arranged in one core, and the other components may be arranged in another core. Each processor / core in system 300 may be connected to a shared memory (not shown in FIG. 3) via a system interconnect. All cores may receive audio data from shared memory.

  2 and 3, the speech search module (ie, 240 and 340) may first divide a large speech database into multiple small groups, or smaller sub-parts that partially overlap a large speech stream. It may be divided into streams. Subsequently, one core processes the audio clip to be searched (“target audio clip”) to build a model of the target audio clip. Meanwhile, the voice search module dynamically schedules voice small groups / substreams for a plurality of cores. The multiple cores divide each group / substream into multiple segments to build a model for each audio segment. This is done in parallel. The size of each segment may be equal to the size of the target audio clip. Model each audio segment and target audio clip using a multiple Gaussian mixture model (GMM) common to all audio segments including both the target audio clip and the audio database / stream You may do that. Once the audio segment model has been constructed, the Cullback-Liver (KL) distance or the KL maximum distance between the segment model and the target audio clip model may be calculated. If the distance is less than or equal to a predetermined value, the audio segment can be identified as the target audio clip. The search process may continue until all voice groups / substreams have been processed.

  FIG. 4 is a block diagram illustrating an example of a voice search module 400 that performs a robust voice search. The voice search module 400 includes a feature extraction unit 410, a modeling mechanism 420, and a determination unit 430. The feature extraction unit 410 may receive an input audio stream (for example, a target audio clip, a substream of a large audio stream, etc.) and extract audio features from the input audio stream. When the input audio stream is an audio stream to be searched to identify the target audio clip, the feature extraction unit applies a sliding window to the audio stream to overlap the audio streams with each other. It may be divided into segments. The window is the same length as the target audio clip. Each segment of the input audio stream (the target audio stream has only one segment) is further divided into a plurality of frames. Each frame may have the same length and overlap with an adjacent frame. For example, according to one embodiment, the frame length may be 20 milliseconds and the overlap between frames may be 10 milliseconds. A feature vector may be extracted for each frame. Each frame may include features such as Fourier coefficients, mel frequency cepstrum coefficients, spectral flatness, and the mean, variance, and other derivative coefficients of these parameters. A feature vector sequence is formed by the feature vectors of all frames of the speech segment.

  The reason why two adjacent segments overlap is to reduce the possibility of missing a target audio clip between two adjacent segments. The longer the overlap, the less likely it will be missed. According to one embodiment, the length of the overlap may be set equal to the length of the frame minus the length of the segment, so as not to miss any matches. However, the calculation increases as the overlapping portion becomes longer. For this reason, it is necessary to balance between the calculation load and the possibility of oversight (for example, the length of the overlapping portion is equal to or less than half the length of the segment). In any case, in the case of a feature vector of a frame that overlaps between two segments, the extraction needs to be performed only once.

  The modeling mechanism 420 may construct a speech segment model based on the feature vector sequence extracted by the feature extraction unit 410. Depending on the model used, the modeling mechanism estimates the model parameters. According to one embodiment, speech segments may be modeled based on a common component Gaussian mixture model (“CCGMM”). The CCGMM includes multiple Gaussian components that are common across all segments. For each segment, the modeling mechanism estimates a series of blend weight values for the common Gaussian component. According to another embodiment, speech segments may be modeled based on other models (eg, hidden Markov models). According to one embodiment, only the target audio clip may be modeled and the feature vector sequence of the audio segment may be used as is to determine whether the audio segment is substantially identical to the target audio clip.

  The determination unit 430 may determine whether the audio segments included in the input audio stream are sufficiently similar and the audio segment can be identified as a copy of the target audio clip. For this reason, the determination unit may derive the similarity measure by comparing the model of the audio segment and the model of the target audio clip. According to one embodiment, the similarity measure may be a distance calculated between these two models. According to another embodiment, the similarity measure may be the probability that the audio segment model and the target audio clip model are identical. According to yet another embodiment, the similarity measure may be obtained by comparing the feature vector sequence of the audio segment with the model of the target audio clip. For example, when modeling a target audio clip based on a Hidden Markov Model (HMM), the likelihood score between the audio segment and the target audio clip is calculated based on the feature vector sequence of the audio segment and the HMM of the target audio clip. A Viterbi-based algorithm may be used for calculation.

  Based on the value of the similarity measure, the determination unit may determine whether the audio segment can be identified as the target audio clip. For example, if the value of the similarity measure is less than or equal to a predetermined threshold (eg, the similarity measure is the distance between the audio segment model and the target audio clip), the audio segment is substantially the same as the target audio clip. Can be identified. Similarly, if the value of the similarity measure is greater than or equal to a predetermined threshold (eg, the similarity measure is a likelihood score that the audio segment is approximately the same as the target audio clip), the audio segment is approximately the same as the target audio clip. It can be specified to be identical. On the other hand, if the similarity measure shows that the audio segment is significantly different from the target audio clip, an arbitrary number of segments immediately after the audio segment may be omitted. The number of segments that are actually omitted depends on the value of the similarity measure and / or the data based on the experiment. If the similarity measure shows that the current segment is very different from the target audio clip, it is not possible to miss the target audio clip by omitting any number of subsequent segments. This is because the similarity measure is recognized as continuity when moving from one segment to the next as a result of the windows used to divide the input audio stream into segments that gradually slide forward.

ＧＭＭ

は、Ｍ個のガウス成分を含み、

は成分重みで、

は平均で、

は共分散であり、ｉ＝１、２・・・、Ｍであって、ｋはセグメントｋを表し、Ｎ（）はガウス分布を表す。ターゲット音声クリップについては、セグメントは１つのみであるので、セグメントを特定するためのｋは利用する必要はない。しかし、入力音声ストリーム５２０については、セグメントは通常複数あるので、異なるセグメントについてＧＭＭを特定するのが望ましい。 FIG. 5 is a diagram illustrating an operation example of the robust speech search module illustrated in FIG. The target audio clip 510 is given to the feature extraction unit and divided into a plurality of frames. The feature extraction unit then generates a feature vector sequence (540) with the feature vectors for each frame in block 530A. Since the feature vector may include one or more parameters, it may be an x-dimensional vector (where x ≧ 1). In block 570A, the feature vector sequence 540 may be modeled using a GMM as follows.

GMM

Contains M Gaussian components,

Is the component weight,

Is on average,

Is the covariance, i = 1, 2,..., M, k represents the segment k, and N () represents a Gaussian distribution. For the target audio clip, since there is only one segment, it is not necessary to use k for specifying the segment. However, since there are usually multiple segments for the input audio stream 520, it is desirable to identify GMMs for different segments.

および分散

は音声セグメントが変わっても同じである、と仮定される。このため、式（１）は以下のように変形される。

各音声セグメントについて、１セットの重み付け

のみを、共通ガウス成分について推定する必要がある。Ｔ個の特徴ベクトル、

を持つセグメントｋの特徴ベクトルシーケンスの場合、重みは以下のように推定され得る。

ここで、

はｉ番目またはｊ番目のセグメントについての普遍的な重みであって、いくつかのサンプル音声ファイルに基づいて実験により得られるとしてもよいし、乱数値によって初期化されるとしてもよい。 In the example shown in FIG. 5, the Cullback-Liver (KL) distance or the KL maximum distance is used as the similarity measure. In order to simplify the calculation of the KL maximum distance, the GMM used for all speech segments has a common Gaussian component common group, that is, an average of the i-th Gaussian component.

And distributed

Is assumed to be the same even if the speech segment changes. For this reason, Formula (1) is deform | transformed as follows.

One set of weights for each audio segment

Only need to be estimated for the common Gaussian component. T feature vectors,

For a feature vector sequence of segment k with, the weights can be estimated as follows:

here,

Is a universal weight for the i th or j th segment, which may be obtained experimentally based on several sample audio files, or may be initialized with a random value.

  The input audio stream 520 searched to identify the target audio clip 510 may be provided to the feature extraction unit. In block 530B, the feature extraction unit divides the input audio stream into a plurality of segments that partially overlap each other. The feature extraction unit further divides each segment into a plurality of partially overlapping frames, and extracts a feature vector for each frame. Block 560 shows the feature vector sequence of the input audio stream 520 and shows the audio stream being divided into segments that partially overlap each other. For example, a window having the same size as the length of the target audio clip may be applied to the input audio stream 520. For illustration purposes, a window is shown for the feature vector sequence of the target audio clip to obtain segment 560A, but in this case there is only one segment, so there is usually no need to apply a window to the target audio clip. . Applying the shifting window to the input audio stream results in a plurality of partially overlapping segments such as 560B and 560C. The amount of window shift is time τ from segment 560B to segment 560C, where τ is smaller than the window size.

  Each speech segment is modeled using CCGMM. For example, segment 560B is modeled at block 570B and segment 560C is modeled at block 570C. The model of each segment of the input audio stream 520 and the model of the target audio clip 510 have different combinations of weights but have a common Gaussian component. According to one embodiment, feature vectors may be extracted frame by frame from the entire input audio stream to generate a long feature vector sequence corresponding to the entire input audio stream. Subsequently, a window having a length of N × FL (where N is a positive integer and FL is the length of the frame) is applied to the long feature vector sequence. A plurality of feature vectors in the window constitute one feature vector of one speech segment, and this feature vector is used to construct a CCGMM. The window is shifted forward by time τ.

このようにして算出されたＫＬ最大距離が所定のしきい値未満の場合、音声クリップが検出されたとみなされるとしてもよい。入力音声ストリーム５２０に対して適用されるウィンドウが時間方向で前方向にシフトされていくと、距離は通常、１つのタイムステップから次のタイムステップまでの間で一定の継続性を示す。つまり、距離が大きすぎると、現在のセグメントの直後に続く１以上のセグメントがターゲット音声クリップに一致する可能性は低い。このため、距離の値によっては、同一音声ストリーム／サブストリーム内の直後の所定数のセグメントについては検索を省略するとしてもよい。 In order to determine whether the segment is substantially the same as the target audio clip, the KL maximum distance between the model of the segment and the model of the target audio clip may be calculated as follows.

If the KL maximum distance calculated in this way is less than a predetermined threshold value, it may be considered that an audio clip has been detected. As the window applied to the input audio stream 520 is shifted forward in the time direction, the distance typically exhibits a constant continuity from one time step to the next. That is, if the distance is too large, it is unlikely that one or more segments immediately following the current segment will match the target audio clip. Therefore, depending on the distance value, the search may be omitted for a predetermined number of segments immediately after the same audio stream / substream.

以上でなければならない。ここで、

はターゲット音声クリップ内のフレーム総数である。 FIG. 6 is a block diagram illustrating an example of a speech search module 600 that performs robust and parallel speech searches in a multiprocessor system. The voice search module 600 includes a dividing mechanism 610, a scheduler 620, an I / O optimization unit 630, and a plurality of voice search units (for example, 640A,..., 640N). The splitting mechanism 610 may split the large audio stream into multiple smaller substreams and / or split the large audio database into multiple small groups. 7A, 7B, and 7C illustrate a method for dividing a large speech database into small groups to perform robust and parallel speech searches in a multiprocessor system. FIG. 7A is a diagram illustrating an example of a database that includes a single large audio stream 710. The splitting mechanism may split the audio stream 710 into multiple smaller substreams 712, 714, and 716. Here, each substream constitutes one group. The lengths of the substreams may be different from each other, but are usually uniform in length to simplify processing. Each substream overlaps with the immediately following substream so that accurate detection of the target audio clip is not overlooked, and overlap between two adjacent substreams (eg, 712 and 714, 714 and 716) Part,

It must be more than that. here,

Is the total number of frames in the target audio clip.

  FIG. 7B illustrates another example database that includes multiple relatively small audio streams (eg, 720, 725, 730, 735, and 740). According to one embodiment, the splitting mechanism 610 may split the database into a plurality of small groups such that each group contains only one audio stream. According to another embodiment, as shown in FIG. 7B, the splitting mechanism includes a database, some groups each include only one audio stream, and other groups each include a plurality of small audio streams. It may be divided into a plurality of small groups. FIG. 7C illustrates yet another example database that includes multiple relatively small audio streams (eg, 750, 755, and 760) and a large audio stream (eg, 770). The splitting mechanism may split a relatively small audio stream into multiple groups such that each group contains only one audio stream, or some groups contain only one audio stream (eg, 750) Other groups may be divided into multiple groups so as to include multiple small audio streams (eg, 755 and 760 may be in the same group). For large audio streams such as 770, the splitting mechanism may split into multiple smaller substreams (eg, 772 and 774) that partially overlap each other. Here, according to the method shown in FIG. 7A, each substream may constitute one group.

  The splitting mechanism also reduces computational load (when large audio streams are split into multiple smaller substreams that partially overlap each other) and load imbalance in parallel processing by multiple processors, Divide a large speech database into multiple groups of appropriate size. As the group size decreases, the overlap of operations can increase, while as the group size increases, the load imbalance can become significant. According to one embodiment, the size of the group may be about 25 times the size of the target audio clip.

  Returning to FIG. 6, the scheduler 620 dynamically schedules multiple groups of a large database for multiple processors in a multiprocessor system so that each processor has one group to be processed at a time. You may do that. The scheduler periodically checks whether or not a plurality of processors of the system can be used, assigns an audio group to each of the available processors, and executes processing and search for a target audio clip. When another processor becomes available thereafter, the scheduler may assign a group to this processor. The scheduler also assigns a voice group that has not yet been searched to the processor immediately after the processor finishes searching for the previously assigned group, regardless of whether other processors have finished the search process. . In fact, even if the group size is the same, the number of segments that skip the search process may vary from segment to segment, so the time required to search for the same target audio clip varies from processor to processor. May be different. By using dynamic scheduling as described above, load imbalance can be effectively reduced.

  The I / O optimization unit 630 may optimize I / O traffic on a system interconnect (for example, a system bus connecting a system processor and a shared system memory). While the data range of each processor is defined, the I / O optimization unit may initially determine that the entire search target speech database is not loaded from the disk to the memory. Further, the I / O optimization unit may read only a part at a time when each processor reads the allocated segment received from the memory. By optimizing the I / O traffic, the I / O optimization unit may reduce I / O contention, overlap I / O processing and computation, and contribute to improving computation efficiency. As a result, the scalability of voice search can be greatly improved.

  The voice search module 600 further includes a plurality of voice search units 640A to 640N. Each voice search unit (eg, 640A) is arranged in one processor and processes a group assigned to the processor to search for a target voice clip. Similar to the voice search module 400 illustrated in FIG. 4, the voice search unit includes a feature extraction unit (for example, 410), a modeling mechanism (for example, 420), and a determination unit (for example, 430). Each voice search unit performs a continuous active search of a voice group assigned to itself for specifying a target voice clip. This is based on CCGMM as shown in equations (1) to (4) by dividing the audio stream of the audio group into a plurality of segments that partially overlap each other and extracting a feature vector sequence for each segment. This is done by modeling each segment. Here, the length of the segment is the same as the length of the target audio clip. Further, the CCGMM for the target voice clip used by all voice search units may be estimated once by one of the voice search units. Each voice search unit calculates the KL maximum distance between the model of each segment and the model of the target voice clip. Based on this KL maximum distance, the voice search unit may determine whether or not a target voice clip is detected. Further, each voice search unit may omit a plurality of segments following the current segment when the KL maximum distance of the current segment is larger than the threshold value.

  FIG. 8 is a pseudo code illustrating an example of a process 800 for performing robust and parallel speech searches in a multiprocessor system. At line 802, the voice search module may be initialized. For example, the target audio clip file and the audio database file may be opened and the global parameters may be initialized. At line 804, the large audio database may be divided into NG small groups as illustrated in FIGS. 7A, 7B, and 7C. At line 806, a model (eg, CCGMM) may be built for the target audio clip. At line 808, NG voice groups may be dynamically scheduled to available processors to initiate parallel processing of the scheduled groups. Line 808 utilizes one instruction to set up a parallel implementation, and other parallel implementation instructions may also be used.

  Lines 810 to 846 show how the processors of the multiprocessor system process each of the NG groups in parallel and how to search to identify the target. Note that for convenience of explanation, the processing from line 812 to line 846 is illustrated as an iteration from the first group to the last group. In practice, when multiple processors are available, multiple groups are processed in parallel by these available processors. At line 814, some or all of the multiple audio streams in each group are further divided into NS segments that partially overlap each other if these streams are temporally longer than the target audio clip. It is good. Line 816 initiates an iterative process for each segment of the group, as indicated by lines 818-832. At line 820, a feature vector sequence (for each frame) may be extracted from the segment. At line 822, a model (eg, a CCGMM as shown in equations (1) through (3)) may be built for the segment. In line 824, a distance between the segment model and the target audio clip model (for example, a KL maximum distance as shown in Expression (4)) may be calculated. In line 826, it may be determined whether the segment matches the target audio clip based on the distance calculated in line 824 and a predetermined threshold # 1. If the distance is less than threshold # 1, the segment matches the target audio clip. Whether the search for a predetermined number of subsequent segments (eg, M segments) within the same audio stream / substream is to be omitted in line 828, whether the distance calculated in line 824 and a predetermined threshold # 2 It may be determined based on the above. If the distance is greater than threshold value # 2, the search for M segments may be omitted. According to one embodiment, the number of segments to omit may vary depending on the distance value. At line 830, the search results (eg, the index or start time of matching segments in each group) may be stored in an array local to the processor that processes the group. At line 842, search results obtained from all processors and stored in the local array may be summarized and output to the user.

  The robust and parallel search strategy outlined in FIG. 8 along with other techniques such as I / O optimization greatly improves the speed of searching for target audio clips in a large audio database in a multiprocessor system. obtain. According to one experiment, the speed of searching for a 15-second target audio clip in a 27-hour audio stream is to continuously search for the same audio stream in the same target audio clip in a 16-way Unisys system. It is known that it is 11 times faster than the case.

  According to one embodiment, a modified search strategy may be used. Using this strategy, a temporary model (eg, CCGMM) is built for the first K (K ≧ 1) frames of the target audio clip, and a complete model is built for the entire target audio clip. Good. Thus, first, a temporary model (eg, CCGMM) can be constructed for the first K (K ≧ 1) frames of the speech segment. In the active search, the temporary model of the first K frames of each audio segment is first compared with the temporary model of the first K frames of the target audio clip to generate a temporary similarity measure. If the preliminary similarity measure shows that these two temporary models are very similar, a complete model is constructed for the entire audio segment and compared against the complete model for the entire target audio clip. Otherwise, a complete model is not built for the audio segment, and a temporary model is first built for the first K frames and then compared with the temporary model and the temporary model of the target audio clip. You may search for segments. Such a modified search strategy can further reduce the computation load.

  While example embodiments of the disclosed subject matter have been described with reference to block diagrams and flowcharts shown in FIGS. 1-8, those skilled in the art can implement the disclosed subject matter in many other ways. Easy to understand. For example, the execution order of the blocks in the flowchart may be changed, and / or a part of the blocks in the block diagram / flow chart described may be changed, deleted, or combined.

  In the foregoing description, the disclosed subject matter has been described in various aspects. In the description, specific numerical values, systems, and configurations are described for the purpose of fully explaining the subject matter. However, it will be apparent to those skilled in the art, upon reference to the disclosure, that the subject matter may be practiced without such specific details. In other instances, well-known features, components or modules have been omitted, simplified, combined or divided in order to avoid obscuring the disclosed subject matter.

  Various embodiments of the disclosed subject matter can be implemented in hardware, firmware, software, or combinations thereof. Also, various embodiments of the disclosed subject matter may be described by referring to or associating with program code. Program code is, for example, design simulation, emulation and manufacturing instructions, functions, procedures, data structures, logic, application programs, design representations or formats, when accessed by a machine, the machine performs a task, Define an abstract data type or low-level hardware context and generate a result.

  For simulation purposes, the program code represents the hardware using a hardware description language or another functional description language that essentially provides a model of how the designed hardware works. Also good. The program code may be assembly or machine language, or data that can be compiled and / or interpreted. It is also common in the art to recognize software as performing an action or producing a result in one form or another. Such an expression is merely intended to briefly describe the execution of program code by a processing system that causes a processor to perform an operation or generate a result.

  The program code may be stored in, for example, volatile and / or nonvolatile memory. Volatile and / or nonvolatile memory can be a storage device and / or an associated machine-readable or machine-accessible medium. The machine-readable or machine-accessible medium may be solid state memory, hard drive, floppy disk, optical storage, tape, flash memory, memory stick, digital video disk, digital versatile disk (DVD), etc., and machine accessible It may be a more unusual medium such as a biological state storage. A machine-readable medium may have any mechanism for storing, transmitting, or receiving information in a form readable by a machine, such as an electrical form of a propagated signal or carrier wave encoding program code, It may have a tangible medium that passes through optical, acoustic, or other forms, such as an antenna, optical fiber, communication interface, and the like. The program code may be transmitted in the form of a packet, serial data, parallel data, a propagation signal, etc., or may be used in a compressed or encrypted form.

  The program code is a mobile or stationary computer, personal digital assistant (PC) comprising a processor, volatile and / or non-volatile memory readable by the processor, at least one input device and / or one or more output devices. (PDA), set-top boxes, mobile phones and pagers, and other electronic devices may be implemented in programs that run on programmable machines. The program code may be applied to data input using an input device, and output information may be generated by executing the above-described embodiment. The output information may be applied to one or more output devices. One skilled in the art can appreciate that embodiments of the disclosed subject matter can be implemented by various computer system configurations. Such computer system configurations include multiprocessor or multicore processor systems, minicomputers, mainframe computers, pervasive or miniature computers or processors that can be embedded in virtually any device. Embodiments of the disclosed subject matter can also be practiced in distributed computing environments where it is a remote processing device that is linked through a communications network to perform a task.

  Although the processes are described as being executed sequentially, some processes may actually be executed in parallel, simultaneously, and / or in a distributed environment, and the program code may be single processor. Stored locally and / or remotely for access by type or multiprocessor machines. Also, according to some embodiments, the order of processing may be rearranged without departing from the spirit of the disclosed subject matter. The program code may be used by the embedded controller or may be used in connection with the embedded controller.

  Although the disclosed subject matter has been described with reference to example embodiments, this description should not be construed as limiting the invention. Various modifications of the example embodiments and other embodiments of the subject matter will be apparent to those skilled in the art and are intended to be within the scope of the disclosed subject matter.

Claims (18)

A method of searching an audio database and identifying a target audio clip in a multiprocessor system,
Dividing the voice database into a plurality of groups;
Building a model for the target audio clip;
Dynamically scheduling the plurality of groups for a plurality of processors of the multiprocessor system;
Processing the plurality of scheduled groups in parallel using the plurality of processors to retrieve the target audio clip; and
Processing the plurality of scheduled groups in parallel includes:
Dividing each of the plurality of scheduled groups into a plurality of segments;
A temporary model is constructed for the first partial frame included in the target audio clip, and a temporary model is constructed for the first partial frame included in at least one of the segments. Obtaining a provisional similarity measure for the provisional model;
If the temporary similarity measure indicates that both the temporary models are similar , build a model for each of the segments and determine a similarity measure between the model of the target audio clip and the model of each of the segments ; Determining that the segment matches the target audio clip based on the similarity measure;
Omitting processing of the immediately following segment based on the similarity measure;
Having a method. The step of omitting the processing of the segment immediately after the
Omitting processing of a number of the segments determined according to the value of the similarity measure,
The method of claim 1. Building a model for the target audio clip comprises:
Extracting a feature vector sequence from the target audio clip; and modeling the feature vector sequence based on a Gaussian mixture model (GMM) including a plurality of Gaussian components;
Building a model for each said segment consists of:
3. For each of the segments, including extracting a feature vector sequence of the segment and modeling the feature vector sequence based on a Gaussian mixture model (GMM) including a plurality of Gaussian components. The method described. The method according to any one of claims 1 to 3, wherein the plurality of divided segments have portions that partially overlap each other. The step of dividing the speech database determines a size for each of the plurality of groups so as to reduce a load imbalance and an amount of operations overlapping between the plurality of groups in the parallel processing of the plurality of groups. The method according to any one of claims 1 to 4, further comprising: Based on the similarity measure, determining that the segment matches the target audio clip comprises:
The method according to any one of claims 1 to 5, comprising determining that the segment matches the target audio clip if the similarity measure is greater than or equal to a predetermined threshold. When executed by a computer , the computer
Dividing the speech database into groups,
Building a model for the target audio clip;
Dynamically scheduling the plurality of groups for a plurality of processors of a multiprocessor system;
Processing the plurality of scheduled groups in parallel using the plurality of processors to retrieve the target audio clip; and
Processing the plurality of scheduled groups in parallel includes:
Dividing each of the plurality of scheduled groups into a plurality of segments;
A temporary model is constructed for the first partial frame included in the target audio clip, and a temporary model is constructed for the first partial frame included in at least one of the segments. Obtaining a provisional similarity measure for the provisional model;
If the temporary similarity measure indicates that both the temporary models are similar , build a model for each of the segments and determine a similarity measure between the model of the target audio clip and the model of each of the segments ; Determining that the segment matches the target audio clip based on the similarity measure;
Omitting processing of the immediately following segment based on the similarity measure;
To execute,
Program . The step of omitting the processing of the segment immediately after the
Omitting processing of a number of the segments determined according to the value of the similarity measure,
The program according to claim 7. Building a model for the target audio clip comprises:
Extracting a feature vector sequence from the target audio clip; and modeling the feature vector sequence based on a Gaussian mixture model (GMM) including a plurality of Gaussian components;
Building a model for each said segment consists of:
The method according to claim 7 or 8, comprising, for each of the segments, extracting a feature vector sequence of the segment and modeling the feature vector sequence based on a Gaussian mixture model (GMM) including a plurality of Gauss components. The listed program . The program according to claim 7, wherein the plurality of segments have portions that partially overlap each other. The step of dividing the speech database determines a size for each of the plurality of groups so as to reduce a load imbalance and an amount of operations overlapping between the plurality of groups in the parallel processing of the plurality of groups. The program according to any one of claims 7 to 10, further comprising: Based on the similarity measure, determining that the segment matches the target audio clip comprises:
The program according to any one of claims 7 to 11, further comprising: determining that the segment matches the target audio clip if the similarity measure is equal to or greater than a predetermined threshold. A memory for receiving voice data from the voice database;
A device comprising: a plurality of processor cores connected to the memory; and a target audio clip is identified by searching the audio database,
Dividing the voice database into a plurality of groups;
Build a model for the target audio clip,
Dynamically scheduling the plurality of groups for the plurality of processor cores;
Each of the processor cores divides each of the scheduled groups into a plurality of segments,
A temporary model is constructed for the first partial frame included in the target audio clip, and a temporary model is constructed for the first partial frame included in at least one of the segments. Find a temporary similarity measure for the temporary model,
If the temporary similarity measure indicates that both the temporary models are similar, each processor core builds a model for each of the segments, and each processor core causes the model of the target audio clip and each of the Determining a similarity measure of the segment with the model, and determining, by each of the processor cores, that the segment matches the target audio clip based on the similarity measure;
Each processor core omits the processing of the immediately following segment based on the similarity measure,
The plurality of divided segments have portions that partially overlap each other. Each processor omits processing of the segment immediately after a number determined according to the value of the similarity measure.
The apparatus of claim 13. Building a model for the target audio clip by extracting a feature vector sequence from the target audio clip and modeling the feature vector sequence based on a Gaussian mixture model (GMM) including a plurality of Gaussian components;
For each said segment, construct a model for each said segment by extracting the feature vector sequence of said segment and modeling said feature vector sequence based on a Gaussian mixture model (GMM) comprising a plurality of Gaussian components To
15. Apparatus according to claim 13 or 14. In the parallel processing of the plurality of groups, the size is determined for each of the plurality of groups and the speech database is determined so as to reduce load imbalance and the amount of computation overlapping between the plurality of groups. The apparatus according to any one of claims 13 to 15, which is divided into a plurality of groups of sizes. The apparatus according to any one of claims 13 to 16, wherein the segment is determined to match the target audio clip if the similarity measure is greater than or equal to a predetermined threshold. The apparatus according to claim 13, wherein the plurality of processor cores are included in a plurality of processors.

Images