US20020029144A1 - Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data - Google Patents
Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data Download PDFInfo
- Publication number
- US20020029144A1 US20020029144A1 US09/976,023 US97602301A US2002029144A1 US 20020029144 A1 US20020029144 A1 US 20020029144A1 US 97602301 A US97602301 A US 97602301A US 2002029144 A1 US2002029144 A1 US 2002029144A1
- Authority
- US
- United States
- Prior art keywords
- target speaker
- model
- score
- block
- frames
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L17/00—Speaker identification or verification
Definitions
- This invention is directed to a multimedia search apparatus and methods for searching multimedia content using speaker detection to segment the multimedia content.
- GMMs Gaussian Mixture Models
- This method uses Gaussian Mixture Models to verify the identity of a speaker such as when conducting financial transactions.
- the above-described speaker identification and verification method assumes that only one speaker is the source of the audio input for all samples. Thus, this method is only practical for identifying a single speaker. Therefore, there is a need for new technology to provide more reliable speaker detection when more than one speaker may be present in multimedia information.
- This invention provides multimedia search apparatus and methods for searching multimedia content using speaker detection to segment the multimedia content.
- the multimedia search apparatus and methods may aid in browsing multimedia content and may be used in conjunction with known browsing techniques such as word spotting, topic spotting, image classification, and the like.
- the multimedia search apparatus receives a search request from a user device.
- the search request includes information regarding the target speaker for which the search is to be conducted.
- the multimedia search apparatus retrieves the multimedia content from a multimedia database.
- the multimedia search apparatus retrieves Gaussian Mixture Models (GMMs) from a Gaussian Mixture Model storage device, corresponding to the target speaker and background data. Based on the retrieved Gaussian Mixture Models, the multimedia search device searches the multimedia data of the multimedia content and segments the multimedia data. The segments are identified by determining an average normalized score for blocks of frames of the multimedia data and determining if the average normalized score exceeds one or more predetermined thresholds. If the average normalized score exceeds the one or more thresholds, the frame may be part of a target speaker segment. If the normalized score falls below one or more of the thresholds, the frame may be considered to be in a background segment.
- GMMs Gaussian Mixture Models
- the segments may be provided to the user device as results of the search. Accordingly, the user device may choose from the identified multimedia content and multimedia segments for playback.
- FIG. 1 is an exemplary block diagram of a system according to this invention.
- FIG. 2 is an exemplary functional block diagram of the multimedia search device of FIG. 1;
- FIG. 3 is an exemplary data flow illustrating the scoring and segmenting method according to this invention.
- FIG. 4 is a example of search results obtained from the multimedia search device according to this invention.
- FIG. 5 is a flowchart outlining a first exemplary embodiment operation of the advertisement service provider according to this invention.
- FIG. 1 is an exemplary block diagram of a system 10 according to this invention.
- the system 10 includes a user device 100 , a local access provider (LAP) 105 , a network 110 , a multimedia search device 120 , and a multimedia database 130 .
- LAP local access provider
- the user device 100 may be any type device that is capable of sending and receiving communication signals.
- the user device 100 may be a computer, a terminal, a wired or wireless telephone, a WebTVTM terminal, a personal digital assistant (PDA), and the like.
- PDA personal digital assistant
- the user device 100 sends and receives communication signals to and from the network 110 via the local access provider 105 .
- the local access provider 105 may be any type of access provider that provides communication access to the network 110 .
- the local access provider 105 may be an Internet service provider (ISP), a network server, a local exchange carrier (LEC), a private branch exchange (PBX), and the like.
- the user device 100 may “log-onto” the local access provider 105 in order to gain access to the network 110 by, for example, dialing an access number of the local access provider 105 user ID and/or password.
- the network 110 may be any type of network that is capable of sending and receiving communication signals.
- the network 110 may be a data network, such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), and the like.
- the network 110 may also be a telecommunications network, such as a local telephone network, a long distance telephone network, a cellular telephone network, a satellite communications network, and the like.
- the network 110 may include more than one network and may include a plurality of different types of networks.
- the network 110 may include a plurality of data networks, a plurality of telecommunications networks, a combination of data and telecommunications networks, and the like.
- the multimedia search device 120 is in communication with the network 110 and has access to the multimedia database 130 .
- the multimedia database 130 stores multimedia content in the form of, for example, multimedia data files.
- the multimedia data files may include audio data, video data, textual data, and the like.
- the multimedia search device 120 When the multimedia search device 120 receives a search request from the user device 100 , the multimedia search device 120 searches the audio data of the multimedia content stored in the multimedia database 130 in order to identify which of the multimedia content files and their corresponding segments contain a particular target speaker. Based on the results of the search of the audio data, information regarding the multimedia content files identified as having target speaker segments and those segments of the multimedia content in which the target speaker is present may be provided to the user device 100 as results of the multimedia search.
- the search request may be entered by a user of the user device 100 via a user interface (not shown) associated with the user device 100 .
- the user interface may be, for example, a telephone keypad, a keyboard, a pointing device, a speech recognition system, and the like.
- the search request includes information regarding the target speaker for which the search is to be conducted and may also include other search criteria such as keywords, topics, image search criteria, and the like.
- the multimedia search device 120 makes use of models of the target speaker, background audio data, and the like, when searching multimedia content for the presence of a target speaker.
- the models may include Gaussian Mixture Models, vector quantization codebook models, hidden Markov models, and the like.
- any type of model that is suitable for text independent speaker detection may be used without departing from the spirit and scope of the present invention.
- the description of the exemplary embodiments will assume that the models being utilized by the multimedia search device 120 are Gaussian Mixture Models.
- the multimedia search device 120 makes use of Gaussian Mixture Models (GMMs) to model the target speaker, background speakers and other background audio data.
- GMMs Gaussian Mixture Models
- the background speakers and other background audio information will hereafter be collectively referred to as background data.
- Gaussian Mixture Models are described, for example, in Reynolds, incorporated above.
- the multimedia search device 120 stores a Gaussian Mixture Model for each possible target speaker and a set of background Gaussian Mixture Models for the background data.
- the Gaussian Mixture Model for each possible target speaker and the Gaussian Mixture Models for the background data are determined from training data, as described hereinafter.
- the multimedia search device 120 uses the target speaker and background data Gaussian Mixture Models to generate an average normalized score for a block of frames of the audio data of the multimedia content and to determine if the average normalized score indicates that the target speaker may be present in the block of frames.
- the multimedia search device 120 may identify starting points and ending points of segments in the multimedia content where the target speaker is present by determining if the average normalized score for a particular block of frames meets one or more predetermined threshold conditions. Based on the determined starting and ending points of the audio data, the segments of the multimedia content that have the target speaker present may be provided to the user device 100 .
- FIG. 2 is a functional block diagram of an exemplary embodiment of the multimedia search device 120 of FIG. 1.
- the multimedia search device 120 includes a controller 201 , a network interface 202 , a multimedia database interface 203 , a Gaussian Mixture Model storage device 204 , and a memory 205 .
- the above elements are in communication with one another via the control/signal bus 206 .
- the search request is received by the controller 201 via the network interface 202 .
- the search request identifies the target speaker for which the search is to be conducted.
- the search request may be entered using a user interface (not shown) and includes information regarding the target speaker for which the search is to be conducted.
- the target speaker may be selected from a list, a target speaker identifier may input via the user interface, a user may speak a name of the target speaker and have the name translated by a speech recognition system into a corresponding target speaker identifier, and the like. Any manner of identifying the target speaker for which the search is to be conducted may be used without departing from the spirit and scope of the invention.
- the controller 201 retrieves a Gaussian Mixture Model for the identified target speaker and Gaussian Mixture Models for the background data from the Gaussian Mixture Model storage device 204 .
- the background data Gaussian Mixture Models are preferably customized for the multimedia content that is to be searched. However, generalized background data Gaussian Mixture Models may be used for a plurality of multimedia content.
- the controller 201 searches the audio data of the multimedia content using the Gaussian Mixture Model for the target speaker and Gaussian Mixture Models for the background data.
- the searching of the audio data includes determining an average normalized score for a block of frames of the audio data and determining whether the average normalized score is above one or more predetermined thresholds.
- the average normalized score is above the one or more predetermined thresholds, it is determined that the target speaker is present in the block of audio data frames. Otherwise, it is determined that the speaker is not present in the block of audio data frames.
- the start and end of segments of the audio data may be determined based on when the average normalized scores are above one or more predetermined thresholds and when they fall below one or more thresholds, respectively.
- the start point may be defined as the point in which the average normalized score first exceeds the threshold.
- the end point may be defined as the point where the average normalized score for a block of frames again falls below the threshold.
- those segments where the target speaker is present may be sent to the user device 100 as results of the multimedia search in addition to simply identifying those multimedia content files having a target speaker segment.
- These segments may be identified to the user device 100 , for example by identifying via time stamps the starting and ending points of each segment, or they may be broadcast or played to the user device 100 .
- the user of the user device 100 may have indicated which of the segments are to be broadcast by selecting the segments via the user interface (not shown) once the segments are identified by the multimedia search device 120 .
- An audio sample of multimedia content may be represented by a time sequence of N feature vectors x 1 , x 2 , . . . , x t , . . . , x N , where t is the time index of the frame.
- the feature vectors represent the short time spectrum of the audio sample and may be obtained, for example, by converting the audio data into 12 th order cepstral coefficients by carrying out a discrete cosine transformation on the output of 31 mel frequency spaced filters.
- the analysis windows may be, for example, 20 msecs in duration spaced every 10 msec through each audio data file.
- cepstral feature vectors may be augmented, for example, by 12 delta-cepstral feature vectors determined over 5 frame windows.
- the delta-cepstral feature vectors represent dynamic measurements of the short time spectrum. All audio data frames having an energy below a predetermined energy threshold may be omitted.
- the target speaker Gaussian Mixture Model will be designated ⁇ T and the background Gaussian Mixture Models will be designated ⁇ B1 , ⁇ B2 , . . . , ⁇ Bk , for example.
- a score for example a log-likelihood score, for each frame t of the feature vector sequence with respect to the target Gaussian Mixture Model and the background Gaussian Mixture Models may be determined using the following relations:
- s t ( ⁇ T ) is the score for the frame t with respect to the target Gaussian Mixture Model
- s t ( ⁇ Bk ) is the score for the frame t with respect to the background Gaussian Mixture Models
- ⁇ Bk ) is the Gaussian mixture probability density function
- k is the background Gaussian Mixture Model number.
- Successive frame scores may be averaged over blocks of M frames shifted every L frames through the sample. For example, a block of 100 frames, each frame being 10 msecs, may be shifted 20 frames through the audio data. In this way, there is an overlap in the frames being used for determining average frame scores.
- a likelihood ratio determination between the target and background block scores produces a normalized score S j :
- the normalized score represents a difference between the score for the target speaker model and a function of the scores for the background models.
- the function of the score for the background models is the maximum score value, however, other functions may be used without departing from the spirit and scope of the present invention.
- the normalized scores are used by the controller 201 to detect the start and end points of the segments in which the target speaker is present.
- An exemplary fragment of program code in the C program language which performs the start and end point detection is provided in Appendix I.
- the exemplary fragment of program code utilizes three thresholds to identify and verify the start and end of target speaker segments, as described in more detail hereafter.
- FIG. 3 is an exemplary data flow diagram illustrating the above outlined method for scoring and detecting start and end points of the audio data segments.
- the data flow of FIG. 3 is illustrative of the above method when only a single background model is utilized. However, as discussed above, generally multiple background models may be used.
- the feature vectors 301 of the audio data of the multimedia content are input and the log likelihood scores 302 , 303 for each frame are determined based on the target speaker Gaussian Mixture Model 304 and the background Gaussian Mixture Models 305 .
- the frame scores are then block averaged 306 , 307 and the block average scores are normalized 308 .
- the normalized scores are then used to detect the target segments 309 , i.e., the segments of the audio data in which the target speaker is present. These target segments are then output to the user device 100 .
- the Gaussian Mixture Models for the target speakers and the background data may be determined through an experimental training method. For example, pre-labeled training data may be used from which feature vectors are compiled into the Gaussian Mixture Models, as described, in the incorporated Reynolds publication, for example.
- FIG. 4 is an example of results of the multimedia search device 120 when the multimedia search device 120 utilizes three thresholds to identify and verify the start and end points of target speaker segments.
- the solid vertical lines designate the segment boundaries identified by the multimedia search device 120 and the dashed vertical lines designate the actual segment boundaries of the multimedia content for a particular target speaker.
- a possible target segment start point is detected when the normalized score exceeds the threshold ⁇ 0 , which in the case shown in FIG. 4 is at approximately 11,150 csecs.
- the possible end point of a target segment is detected when the normalized score falls below the threshold ⁇ 0 , which in the case shown in FIG. 4 is at approximately 13,100 csecs.
- two thresholds ⁇ 1 and ⁇ 2 may be used to provide a verification of the segment boundary determinations.
- the thresholds ⁇ 1 and ⁇ 2 may be arbitrarily determined or may be determined based on empirical data; however, ⁇ 1 and ⁇ 2 should be chosen with the following constraint ⁇ 2 ⁇ 0 ⁇ 1 .
- the normalized block score for a subsequent block of frames falls below ⁇ 2
- the end point of the segment is detected as the point where the normalized block score fell below threshold ⁇ 0 . If the normalized block score of a subsequent block of frames does not exceed threshold ⁇ 1 , then the start and end of the target speaker segment is not verified and a target speaker segment is not identified by the multimedia search device 120 .
- Appendix I includes an example section of program code that implements this threshold approach.
- “cand” represents a target speaker segment candidate flag
- “score[block]” represents a score for a block of frames of audio data
- “tentstart” represents a tentative start of a target speaker segment
- “tentend” represents a tentative end of a target speaker segment.
- the values th0, th1 and th2 correspond to threshold values ⁇ 0 , ⁇ 1 and ⁇ 2 , respectively.
- the second and third threshold values, th1 and th2 serve as a check on the first threshold, th0, to increase the likelihood that detected tentative start and end points of a target speaker segment are in fact the actual start and end points of the target speaker segment.
- the present invention may set criteria such as requiring the normalized block score to be above the threshold ⁇ 0 for a predetermined amount of time, to be above the threshold value ⁇ 1 or below threshold value ⁇ 2 for a predetermined period of time, and the like.
- criteria such as requiring the normalized block score to be above the threshold ⁇ 0 for a predetermined amount of time, to be above the threshold value ⁇ 1 or below threshold value ⁇ 2 for a predetermined period of time, and the like.
- any type of criteria may be sent to increase the reliability of the detection of the start and end points of the target speaker segments.
- FIG. 5 is an exemplary flowchart outlining a method of searching multimedia content according to the present invention.
- the exemplary flowchart of FIG. 5 is a generalized description of the steps performed by the method according to the present invention. The details of how each step in the flowchart is performed have been previously described or may be apparent to those of ordinary skill in the art. Various modifications and additions to the flowchart of FIG. 5, as may be apparent to those of ordinary skill in the art in view of this disclosure, may be made without departing from the spirit and scope of the invention.
- step 501 the controller 201 receives a search request from the user device 100 and goes to step 502 .
- step 502 the controller 201 retrieves the multimedia content, identified by the search request, from the multimedia database 130 .
- step 503 the controller 201 retrieves the Gaussian Mixture Models associated with the target speaker, identified in the search request, and background data.
- step 504 the controller 201 determines normalized scores for audio data frames of the multimedia content and averages these normalized scores over a block of frames to obtain a block score, and goes to step 505 .
- step 505 the controller 201 determines whether a possible, or tentative, start of a target speaker segment has been previously identified, such as in a previous iteration of the flowchart. If so, control goes to step 508 ; otherwise, control continues to step 506 .
- step 506 the controller 201 determines if the block score obtained in step 504 is such that a possible start of a target speaker segment is identified. If so, control goes to step 507 ; otherwise control skips to step 515 . This determination may be made based on preset threshold values in the manner described above.
- step 507 the controller 201 identifies a possible target speaker segment start and goes to step 515 .
- step 515 the controller 201 determines if there are more blocks to be processed. If so, control returns to step 504 ; otherwise, control goes to step 516 and ends.
- step 505 if a possible start of a target speaker segment has been identified, control goes to step 508 .
- step 508 the controller 201 determines if the block score obtained in step 504 indicates that the block of frames is still within a valid target speaker segment. If so, control goes to step 509 ; otherwise, control goes to step 511 .
- step 509 the controller 201 determines if the block score from step 504 identifies a possible target segment end. If so, control goes to step 510 ; otherwise control goes to step 515 .
- step 510 the controller 201 identifies a possible target speaker segment end and goes to step 515 .
- step 508 if the block score from step 504 indicates that the block of frames is not within a valid target speaker segment, control goes to step 511 .
- step 511 the controller 201 determines if a possible target speaker segment end has been identified. If so, control goes to step 512 ; otherwise, control goes to step 514 where the possible target speaker segment start is reset and control goes to step 515 .
- step 512 the controller 201 registers the target speaker segment start and end to mark the target speaker segment as a confirmed target speaker segment. Then in step 513 , the controller 201 resets the possible target speaker segment start and end so that a new target speaker segment may be identified if further blocks are to be processed.
- multimedia search device 120 may be distributed throughout the network 110 , for example.
- the multimedia search device 120 and user device 100 may be combined into a single stand-alone device.
- the multimedia search device 120 would not need the network interface 202 .
- the multimedia database 130 may be a floppy disk, hard disk, CD-ROM, ZIP disk, and the like, and the multimedia database interface 203 may be a corresponding drive.
- the method of this invention is preferably implemented on a programmed processor.
- the multimedia search device 120 can also be implemented on a general purpose or special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an ASIC or other integrated circuit, a hardware electronic or logic circuit such as a discrete element circuit, a programmable logic device such as a PLD, PLA, FPGA or PAL, or the like.
- any device capable of implementing the flowchart shown in FIG. 5 can be used to implement the multimedia search device 120 functions of this invention.
Abstract
Description
- This nonprovisional application claims the benefit of U.S. provisional application No. 60/096,372 entitled “Speaker Detection in Broadcast Speech Databases” filed on Aug. 13, 1998. The provisional application and all references cited therein are hereby incorporated by reference.
- 1. Field of Invention
- This invention is directed to a multimedia search apparatus and methods for searching multimedia content using speaker detection to segment the multimedia content.
- 2. Description of Related Art
- In one known method for speaker identification and verification, Gaussian Mixture Models (GMMs) are used to model the spectral shapes of the speaker's voice. This method is described in “Robust Text-Independent Speaker Identification Using Gaussian Mixture Speaker Models,” Douglas A. Reynolds,IEEE Transactions on Speech and Audio Processing, vol. 3, no. 1, January 1995 (Reynolds), which is incorporated herein by reference. This method uses Gaussian Mixture Models to verify the identity of a speaker such as when conducting financial transactions. However, the above-described speaker identification and verification method assumes that only one speaker is the source of the audio input for all samples. Thus, this method is only practical for identifying a single speaker. Therefore, there is a need for new technology to provide more reliable speaker detection when more than one speaker may be present in multimedia information.
- This invention provides multimedia search apparatus and methods for searching multimedia content using speaker detection to segment the multimedia content. The multimedia search apparatus and methods may aid in browsing multimedia content and may be used in conjunction with known browsing techniques such as word spotting, topic spotting, image classification, and the like.
- The multimedia search apparatus receives a search request from a user device. The search request includes information regarding the target speaker for which the search is to be conducted. Based on the search request, the multimedia search apparatus retrieves the multimedia content from a multimedia database.
- In one embodiment of the invention, the multimedia search apparatus retrieves Gaussian Mixture Models (GMMs) from a Gaussian Mixture Model storage device, corresponding to the target speaker and background data. Based on the retrieved Gaussian Mixture Models, the multimedia search device searches the multimedia data of the multimedia content and segments the multimedia data. The segments are identified by determining an average normalized score for blocks of frames of the multimedia data and determining if the average normalized score exceeds one or more predetermined thresholds. If the average normalized score exceeds the one or more thresholds, the frame may be part of a target speaker segment. If the normalized score falls below one or more of the thresholds, the frame may be considered to be in a background segment.
- Once the segments are identified by the multimedia search device, the segments may be provided to the user device as results of the search. Accordingly, the user device may choose from the identified multimedia content and multimedia segments for playback.
- The preferred embodiments of the present invention will be described with reference to the following drawings, wherein like numerals designate like elements, and wherein:
- FIG. 1 is an exemplary block diagram of a system according to this invention;
- FIG. 2 is an exemplary functional block diagram of the multimedia search device of FIG. 1;
- FIG. 3 is an exemplary data flow illustrating the scoring and segmenting method according to this invention;
- FIG. 4 is a example of search results obtained from the multimedia search device according to this invention; and
- FIG. 5 is a flowchart outlining a first exemplary embodiment operation of the advertisement service provider according to this invention.
- FIG. 1 is an exemplary block diagram of a
system 10 according to this invention. As shown in FIG. 1, thesystem 10 includes auser device 100, a local access provider (LAP) 105, anetwork 110, amultimedia search device 120, and amultimedia database 130. - The
user device 100 may be any type device that is capable of sending and receiving communication signals. For example, theuser device 100 may be a computer, a terminal, a wired or wireless telephone, a WebTV™ terminal, a personal digital assistant (PDA), and the like. For simplicity of the following description of the exemplary embodiments, it is assumed that theuser device 100 is a computer. Theuser device 100 sends and receives communication signals to and from thenetwork 110 via thelocal access provider 105. - The
local access provider 105 may be any type of access provider that provides communication access to thenetwork 110. For example, thelocal access provider 105 may be an Internet service provider (ISP), a network server, a local exchange carrier (LEC), a private branch exchange (PBX), and the like. Theuser device 100, for example, may “log-onto” thelocal access provider 105 in order to gain access to thenetwork 110 by, for example, dialing an access number of thelocal access provider 105 user ID and/or password. - The
network 110 may be any type of network that is capable of sending and receiving communication signals. For example, thenetwork 110 may be a data network, such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), and the like. Thenetwork 110 may also be a telecommunications network, such as a local telephone network, a long distance telephone network, a cellular telephone network, a satellite communications network, and the like. Furthermore, thenetwork 110 may include more than one network and may include a plurality of different types of networks. Thus, thenetwork 110 may include a plurality of data networks, a plurality of telecommunications networks, a combination of data and telecommunications networks, and the like. - The
multimedia search device 120 is in communication with thenetwork 110 and has access to themultimedia database 130. Themultimedia database 130 stores multimedia content in the form of, for example, multimedia data files. The multimedia data files may include audio data, video data, textual data, and the like. - When the
multimedia search device 120 receives a search request from theuser device 100, themultimedia search device 120 searches the audio data of the multimedia content stored in themultimedia database 130 in order to identify which of the multimedia content files and their corresponding segments contain a particular target speaker. Based on the results of the search of the audio data, information regarding the multimedia content files identified as having target speaker segments and those segments of the multimedia content in which the target speaker is present may be provided to theuser device 100 as results of the multimedia search. - The search request may be entered by a user of the
user device 100 via a user interface (not shown) associated with theuser device 100. The user interface may be, for example, a telephone keypad, a keyboard, a pointing device, a speech recognition system, and the like. The search request includes information regarding the target speaker for which the search is to be conducted and may also include other search criteria such as keywords, topics, image search criteria, and the like. - The
multimedia search device 120 makes use of models of the target speaker, background audio data, and the like, when searching multimedia content for the presence of a target speaker. The models may include Gaussian Mixture Models, vector quantization codebook models, hidden Markov models, and the like. In short, any type of model that is suitable for text independent speaker detection may be used without departing from the spirit and scope of the present invention. For purposes of illustration, the description of the exemplary embodiments will assume that the models being utilized by themultimedia search device 120 are Gaussian Mixture Models. - Thus, in the following exemplary embodiments, the
multimedia search device 120 makes use of Gaussian Mixture Models (GMMs) to model the target speaker, background speakers and other background audio data. The background speakers and other background audio information will hereafter be collectively referred to as background data. Gaussian Mixture Models are described, for example, in Reynolds, incorporated above. - The
multimedia search device 120 stores a Gaussian Mixture Model for each possible target speaker and a set of background Gaussian Mixture Models for the background data. The Gaussian Mixture Model for each possible target speaker and the Gaussian Mixture Models for the background data are determined from training data, as described hereinafter. - The
multimedia search device 120 uses the target speaker and background data Gaussian Mixture Models to generate an average normalized score for a block of frames of the audio data of the multimedia content and to determine if the average normalized score indicates that the target speaker may be present in the block of frames. Themultimedia search device 120 may identify starting points and ending points of segments in the multimedia content where the target speaker is present by determining if the average normalized score for a particular block of frames meets one or more predetermined threshold conditions. Based on the determined starting and ending points of the audio data, the segments of the multimedia content that have the target speaker present may be provided to theuser device 100. - FIG. 2 is a functional block diagram of an exemplary embodiment of the
multimedia search device 120 of FIG. 1. As shown in FIG. 2, themultimedia search device 120 includes acontroller 201, anetwork interface 202, amultimedia database interface 203, a Gaussian MixtureModel storage device 204, and amemory 205. The above elements are in communication with one another via the control/signal bus 206. - When a user of a
user device 100 sends a search request to themultimedia search device 120, the search request is received by thecontroller 201 via thenetwork interface 202. The search request identifies the target speaker for which the search is to be conducted. As described above, the search request may be entered using a user interface (not shown) and includes information regarding the target speaker for which the search is to be conducted. For example, the target speaker may be selected from a list, a target speaker identifier may input via the user interface, a user may speak a name of the target speaker and have the name translated by a speech recognition system into a corresponding target speaker identifier, and the like. Any manner of identifying the target speaker for which the search is to be conducted may be used without departing from the spirit and scope of the invention. - Once the
controller 201 receives the search request, thecontroller 201 retrieves a Gaussian Mixture Model for the identified target speaker and Gaussian Mixture Models for the background data from the Gaussian MixtureModel storage device 204. The background data Gaussian Mixture Models are preferably customized for the multimedia content that is to be searched. However, generalized background data Gaussian Mixture Models may be used for a plurality of multimedia content. - The
controller 201 then searches the audio data of the multimedia content using the Gaussian Mixture Model for the target speaker and Gaussian Mixture Models for the background data. The searching of the audio data includes determining an average normalized score for a block of frames of the audio data and determining whether the average normalized score is above one or more predetermined thresholds. - If the average normalized score is above the one or more predetermined thresholds, it is determined that the target speaker is present in the block of audio data frames. Otherwise, it is determined that the speaker is not present in the block of audio data frames.
- The start and end of segments of the audio data may be determined based on when the average normalized scores are above one or more predetermined thresholds and when they fall below one or more thresholds, respectively. The start point may be defined as the point in which the average normalized score first exceeds the threshold. The end point may be defined as the point where the average normalized score for a block of frames again falls below the threshold.
- Once the audio data is divided into segments in which the target speaker is present and segments in which the target speaker is not present, those segments where the target speaker is present may be sent to the
user device 100 as results of the multimedia search in addition to simply identifying those multimedia content files having a target speaker segment. These segments may be identified to theuser device 100, for example by identifying via time stamps the starting and ending points of each segment, or they may be broadcast or played to theuser device 100. Additionally, the user of theuser device 100 may have indicated which of the segments are to be broadcast by selecting the segments via the user interface (not shown) once the segments are identified by themultimedia search device 120. - Next, the method of scoring frames and blocks of frames of the audio data will be described. An audio sample of multimedia content, for example, may be represented by a time sequence of N feature vectors x1, x2, . . . , xt, . . . , xN, where t is the time index of the frame. The feature vectors represent the short time spectrum of the audio sample and may be obtained, for example, by converting the audio data into 12th order cepstral coefficients by carrying out a discrete cosine transformation on the output of 31 mel frequency spaced filters. The analysis windows may be, for example, 20 msecs in duration spaced every 10 msec through each audio data file. These cepstral feature vectors may be augmented, for example, by 12 delta-cepstral feature vectors determined over 5 frame windows. The delta-cepstral feature vectors represent dynamic measurements of the short time spectrum. All audio data frames having an energy below a predetermined energy threshold may be omitted.
- The target speaker Gaussian Mixture Model will be designated λT and the background Gaussian Mixture Models will be designated λB1, λB2, . . . , λBk, for example. Using these representations, a score, for example a log-likelihood score, for each frame t of the feature vector sequence with respect to the target Gaussian Mixture Model and the background Gaussian Mixture Models may be determined using the following relations:
- s t(λT)=log p(x t|λT) (1)
- s t(λBk)=log p(x t|λBk), k=1, 2, . . . , K (2)
-
- A likelihood ratio determination between the target and background block scores produces a normalized score Sj:
- S j(λT; λB1, . . . , λBK)=S j(λT)−maxS j(λBk) (4)
- The normalized score represents a difference between the score for the target speaker model and a function of the scores for the background models. In Equation 4, the function of the score for the background models is the maximum score value, however, other functions may be used without departing from the spirit and scope of the present invention.
- The normalized scores are used by the
controller 201 to detect the start and end points of the segments in which the target speaker is present. An exemplary fragment of program code in the C program language which performs the start and end point detection is provided in Appendix I. The exemplary fragment of program code utilizes three thresholds to identify and verify the start and end of target speaker segments, as described in more detail hereafter. - FIG. 3 is an exemplary data flow diagram illustrating the above outlined method for scoring and detecting start and end points of the audio data segments. The data flow of FIG. 3 is illustrative of the above method when only a single background model is utilized. However, as discussed above, generally multiple background models may be used.
- As shown in FIG. 3, the
feature vectors 301 of the audio data of the multimedia content are input and the log likelihood scores 302, 303 for each frame are determined based on the target speakerGaussian Mixture Model 304 and the backgroundGaussian Mixture Models 305. The frame scores are then block averaged 306, 307 and the block average scores are normalized 308. The normalized scores are then used to detect thetarget segments 309, i.e., the segments of the audio data in which the target speaker is present. These target segments are then output to theuser device 100. - The Gaussian Mixture Models for the target speakers and the background data may be determined through an experimental training method. For example, pre-labeled training data may be used from which feature vectors are compiled into the Gaussian Mixture Models, as described, in the incorporated Reynolds publication, for example.
- FIG. 4 is an example of results of the
multimedia search device 120 when themultimedia search device 120 utilizes three thresholds to identify and verify the start and end points of target speaker segments. The solid vertical lines designate the segment boundaries identified by themultimedia search device 120 and the dashed vertical lines designate the actual segment boundaries of the multimedia content for a particular target speaker. As shown in FIG. 4, a possible target segment start point is detected when the normalized score exceeds the threshold θ0, which in the case shown in FIG. 4 is at approximately 11,150 csecs. The possible end point of a target segment is detected when the normalized score falls below the threshold θ0, which in the case shown in FIG. 4 is at approximately 13,100 csecs. - Rather than using only a single threshold θ0, however, two thresholds θ1 and θ2 may be used to provide a verification of the segment boundary determinations. The thresholds θ1 and θ2 may be arbitrarily determined or may be determined based on empirical data; however, θ1 and θ2 should be chosen with the following constraint θ2<θ0<θ1. Thus, after the threshold θ0 has been exceeded and a tentative start of a target speaker segment has been identified, if the normalized block score for a subsequent block of frames exceeds the threshold θ1, the start point of the segment is detected as being at the point where the threshold θ0 was exceeded. Likewise, after the threshold θ0 has been traversed and a tentative end of the target speaker segment has been identified, if the normalized block score for a subsequent block of frames falls below θ2, the end point of the segment is detected as the point where the normalized block score fell below threshold θ0. If the normalized block score of a subsequent block of frames does not exceed threshold θ1, then the start and end of the target speaker segment is not verified and a target speaker segment is not identified by the
multimedia search device 120. - As discussed above, Appendix I includes an example section of program code that implements this threshold approach. In Appendix I, “cand” represents a target speaker segment candidate flag, “score[block]” represents a score for a block of frames of audio data, “tentstart” represents a tentative start of a target speaker segment, and “tentend” represents a tentative end of a target speaker segment. The values th0, th1 and th2 correspond to threshold values θ0, θ1 and θ2, respectively.
- As shown in Appendix I, if cand is zero, i.e., the block of frames is not a target speaker segment candidate, and the score for the block of frames is above a first threshold, th0, then a tentative start of a target speaker segment is identified (tentstart=block) and cand is set to 1. If the block of frames is a candidate for a target speaker segment (cand=1), and the score for the block of frames is greater than a second threshold, th1, then a legitimate start of a target speaker segment is identified and cand is set to 2.
- If the block of frames is a candidate (cand=2) and the score is less than the first threshold, th0, then a tentative end of the target speaker segment is identified (tentend=block). If a legitimate start of a target speaker segment is identified (cand=2) and the score for a block of frames is less than a third threshold, th2, then the start of the target speaker segment is set to the tentative start and the end of the target speaker segment is set to the tentative end. In this way, the second and third threshold values, th1 and th2, serve as a check on the first threshold, th0, to increase the likelihood that detected tentative start and end points of a target speaker segment are in fact the actual start and end points of the target speaker segment.
- In addition to or in place of setting actual threshold values θ1 and θ2, the present invention may set criteria such as requiring the normalized block score to be above the threshold θ0 for a predetermined amount of time, to be above the threshold value θ1 or below threshold value θ2 for a predetermined period of time, and the like. In short, any type of criteria may be sent to increase the reliability of the detection of the start and end points of the target speaker segments.
- FIG. 5 is an exemplary flowchart outlining a method of searching multimedia content according to the present invention. The exemplary flowchart of FIG. 5 is a generalized description of the steps performed by the method according to the present invention. The details of how each step in the flowchart is performed have been previously described or may be apparent to those of ordinary skill in the art. Various modifications and additions to the flowchart of FIG. 5, as may be apparent to those of ordinary skill in the art in view of this disclosure, may be made without departing from the spirit and scope of the invention.
- As shown in FIG. 5, in
step 501, thecontroller 201 receives a search request from theuser device 100 and goes to step 502. Instep 502, thecontroller 201 retrieves the multimedia content, identified by the search request, from themultimedia database 130. Next, instep 503, thecontroller 201 retrieves the Gaussian Mixture Models associated with the target speaker, identified in the search request, and background data. - In
step 504, thecontroller 201 determines normalized scores for audio data frames of the multimedia content and averages these normalized scores over a block of frames to obtain a block score, and goes to step 505. Instep 505, thecontroller 201 determines whether a possible, or tentative, start of a target speaker segment has been previously identified, such as in a previous iteration of the flowchart. If so, control goes to step 508; otherwise, control continues to step 506. Instep 506, thecontroller 201 determines if the block score obtained instep 504 is such that a possible start of a target speaker segment is identified. If so, control goes to step 507; otherwise control skips to step 515. This determination may be made based on preset threshold values in the manner described above. - In
step 507, thecontroller 201 identifies a possible target speaker segment start and goes to step 515. Instep 515, thecontroller 201 determines if there are more blocks to be processed. If so, control returns to step 504; otherwise, control goes to step 516 and ends. - In
step 505, if a possible start of a target speaker segment has been identified, control goes to step 508. Instep 508, thecontroller 201 determines if the block score obtained instep 504 indicates that the block of frames is still within a valid target speaker segment. If so, control goes to step 509; otherwise, control goes to step 511. Instep 509, thecontroller 201 determines if the block score fromstep 504 identifies a possible target segment end. If so, control goes to step 510; otherwise control goes to step 515. Instep 510, thecontroller 201 identifies a possible target speaker segment end and goes to step 515. - In
step 508, if the block score fromstep 504 indicates that the block of frames is not within a valid target speaker segment, control goes to step 511. Instep 511, thecontroller 201 determines if a possible target speaker segment end has been identified. If so, control goes to step 512; otherwise, control goes to step 514 where the possible target speaker segment start is reset and control goes to step 515. Instep 512, thecontroller 201 registers the target speaker segment start and end to mark the target speaker segment as a confirmed target speaker segment. Then instep 513, thecontroller 201 resets the possible target speaker segment start and end so that a new target speaker segment may be identified if further blocks are to be processed. - While the above embodiments describe the
multimedia search device 120 as a separate device connected to thenetwork 100, the present invention is not limited to such a configuration. Themultimedia search device 120 may be distributed throughout thenetwork 110, for example. - Alternatively, the
multimedia search device 120 anduser device 100 may be combined into a single stand-alone device. In this case, themultimedia search device 120 would not need thenetwork interface 202. Furthermore, themultimedia database 130 may be a floppy disk, hard disk, CD-ROM, ZIP disk, and the like, and themultimedia database interface 203 may be a corresponding drive. - As shown in FIGS. 1 and 2, the method of this invention is preferably implemented on a programmed processor. However, the
multimedia search device 120 can also be implemented on a general purpose or special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an ASIC or other integrated circuit, a hardware electronic or logic circuit such as a discrete element circuit, a programmable logic device such as a PLD, PLA, FPGA or PAL, or the like. In general, any device capable of implementing the flowchart shown in FIG. 5 can be used to implement themultimedia search device 120 functions of this invention. - While this invention has been described with specific embodiments thereof, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, the preferred embodiments of the invention as set forth herein are intended to be illustrative, not limiting. Various changes may be made without departing from the spirit and scope of the invention.
Claims (34)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/976,023 US6405166B1 (en) | 1998-08-13 | 2001-10-15 | Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US9637298P | 1998-08-13 | 1998-08-13 | |
US09/353,192 US6317710B1 (en) | 1998-08-13 | 1999-07-14 | Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data |
US09/976,023 US6405166B1 (en) | 1998-08-13 | 2001-10-15 | Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/353,192 Continuation US6317710B1 (en) | 1998-08-13 | 1999-07-14 | Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data |
Publications (2)
Publication Number | Publication Date |
---|---|
US20020029144A1 true US20020029144A1 (en) | 2002-03-07 |
US6405166B1 US6405166B1 (en) | 2002-06-11 |
Family
ID=26791637
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/353,192 Expired - Lifetime US6317710B1 (en) | 1998-08-13 | 1999-07-14 | Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data |
US09/976,023 Expired - Lifetime US6405166B1 (en) | 1998-08-13 | 2001-10-15 | Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/353,192 Expired - Lifetime US6317710B1 (en) | 1998-08-13 | 1999-07-14 | Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data |
Country Status (1)
Country | Link |
---|---|
US (2) | US6317710B1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030097261A1 (en) * | 2001-11-22 | 2003-05-22 | Hyung-Bae Jeon | Speech detection apparatus under noise environment and method thereof |
US20090006102A1 (en) * | 2004-06-09 | 2009-01-01 | Canon Kabushiki Kaisha | Effective Audio Segmentation and Classification |
US20110202340A1 (en) * | 2008-10-29 | 2011-08-18 | Ariyaeeinia Aladdin M | Speaker verification |
US8959022B2 (en) | 2012-07-03 | 2015-02-17 | Motorola Solutions, Inc. | System for media correlation based on latent evidences of audio |
Families Citing this family (47)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7136870B1 (en) * | 1998-03-04 | 2006-11-14 | The Regents Of The University Of California | Method and apparatus for accessing and displaying multimedia content |
US6317710B1 (en) * | 1998-08-13 | 2001-11-13 | At&T Corp. | Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data |
US6714909B1 (en) * | 1998-08-13 | 2004-03-30 | At&T Corp. | System and method for automated multimedia content indexing and retrieval |
US6557042B1 (en) * | 1999-03-19 | 2003-04-29 | Microsoft Corporation | Multimedia summary generation employing user feedback |
US7293280B1 (en) | 1999-07-08 | 2007-11-06 | Microsoft Corporation | Skimming continuous multimedia content |
US7313808B1 (en) | 1999-07-08 | 2007-12-25 | Microsoft Corporation | Browsing continuous multimedia content |
US6868440B1 (en) * | 2000-02-04 | 2005-03-15 | Microsoft Corporation | Multi-level skimming of multimedia content using playlists |
US7263484B1 (en) | 2000-03-04 | 2007-08-28 | Georgia Tech Research Corporation | Phonetic searching |
CA2386303C (en) | 2001-05-14 | 2005-07-05 | At&T Corp. | Method for content-based non-linear control of multimedia playback |
JP2003216183A (en) * | 2002-01-25 | 2003-07-30 | Sony Corp | Method and device for retrieving information |
US7231351B1 (en) * | 2002-05-10 | 2007-06-12 | Nexidia, Inc. | Transcript alignment |
AU2003204588B2 (en) * | 2002-05-31 | 2006-02-23 | Canon Kabushiki Kaisha | Robust Detection and Classification of Objects in Audio Using Limited Training Data |
AUPS270902A0 (en) * | 2002-05-31 | 2002-06-20 | Canon Kabushiki Kaisha | Robust detection and classification of objects in audio using limited training data |
US6957226B2 (en) * | 2002-06-27 | 2005-10-18 | Microsoft Corporation | Searching multi-media databases using multi-media queries |
US7124149B2 (en) * | 2002-12-13 | 2006-10-17 | International Business Machines Corporation | Method and apparatus for content representation and retrieval in concept model space |
US7904296B2 (en) | 2003-07-23 | 2011-03-08 | Nexidia Inc. | Spoken word spotting queries |
US7421391B1 (en) | 2003-09-08 | 2008-09-02 | Zamtech Llc | System and method for voice-over asset management, search and presentation |
WO2005034395A2 (en) * | 2003-09-17 | 2005-04-14 | Nielsen Media Research, Inc. | Methods and apparatus to operate an audience metering device with voice commands |
KR100927731B1 (en) * | 2003-09-27 | 2009-11-18 | 한국전자통신연구원 | Package metadata and targeting and synchronization service provision system using it |
WO2005041109A2 (en) * | 2003-10-17 | 2005-05-06 | Nielsen Media Research, Inc. | Methods and apparatus for identifiying audio/video content using temporal signal characteristics |
FR2868587A1 (en) * | 2004-03-31 | 2005-10-07 | France Telecom | METHOD AND SYSTEM FOR RAPID CONVERSION OF A VOICE SIGNAL |
US8078463B2 (en) * | 2004-11-23 | 2011-12-13 | Nice Systems, Ltd. | Method and apparatus for speaker spotting |
US20060212595A1 (en) * | 2005-03-15 | 2006-09-21 | 1000 Oaks Hu Lian Technology Development (Beijing) Co., Ltd. | Method and computer-readable medium for associating sequence numbers with data blocks for distribution of data in a peer-to-peer network |
US20070078898A1 (en) * | 2005-09-30 | 2007-04-05 | Yahoo! Inc. | Server-based system and method for retrieving tagged portions of media files |
US20070078896A1 (en) * | 2005-09-30 | 2007-04-05 | Yahoo! Inc. | Identifying portions within media files with location tags |
US20070078883A1 (en) * | 2005-09-30 | 2007-04-05 | Yahoo! Inc. | Using location tags to render tagged portions of media files |
JP4241736B2 (en) * | 2006-01-19 | 2009-03-18 | 株式会社東芝 | Speech processing apparatus and method |
US7716048B2 (en) * | 2006-01-25 | 2010-05-11 | Nice Systems, Ltd. | Method and apparatus for segmentation of audio interactions |
US8200688B2 (en) * | 2006-03-07 | 2012-06-12 | Samsung Electronics Co., Ltd. | Method and system for facilitating information searching on electronic devices |
US8024189B2 (en) | 2006-06-22 | 2011-09-20 | Microsoft Corporation | Identification of people using multiple types of input |
JP4282704B2 (en) * | 2006-09-27 | 2009-06-24 | 株式会社東芝 | Voice section detection apparatus and program |
US9286385B2 (en) | 2007-04-25 | 2016-03-15 | Samsung Electronics Co., Ltd. | Method and system for providing access to information of potential interest to a user |
US8131550B2 (en) * | 2007-10-04 | 2012-03-06 | Nokia Corporation | Method, apparatus and computer program product for providing improved voice conversion |
CA2713355C (en) * | 2008-01-14 | 2014-05-06 | Algo Communication Products Ltd. | Methods and systems for searching audio records |
JP4950930B2 (en) * | 2008-04-03 | 2012-06-13 | 株式会社東芝 | Apparatus, method and program for determining voice / non-voice |
US7996422B2 (en) | 2008-07-22 | 2011-08-09 | At&T Intellectual Property L.L.P. | System and method for adaptive media playback based on destination |
US8990848B2 (en) | 2008-07-22 | 2015-03-24 | At&T Intellectual Property I, L.P. | System and method for temporally adaptive media playback |
US9124769B2 (en) | 2008-10-31 | 2015-09-01 | The Nielsen Company (Us), Llc | Methods and apparatus to verify presentation of media content |
US20100299131A1 (en) * | 2009-05-21 | 2010-11-25 | Nexidia Inc. | Transcript alignment |
US20100332225A1 (en) * | 2009-06-29 | 2010-12-30 | Nexidia Inc. | Transcript alignment |
US8442823B2 (en) | 2010-10-19 | 2013-05-14 | Motorola Solutions, Inc. | Methods for creating and searching a database of speakers |
US20120155663A1 (en) * | 2010-12-16 | 2012-06-21 | Nice Systems Ltd. | Fast speaker hunting in lawful interception systems |
US9996628B2 (en) * | 2012-06-29 | 2018-06-12 | Verisign, Inc. | Providing audio-activated resource access for user devices based on speaker voiceprint |
JP5743976B2 (en) * | 2012-08-10 | 2015-07-01 | 株式会社東芝 | Information processing apparatus, program, and information processing method |
KR102413692B1 (en) * | 2015-07-24 | 2022-06-27 | 삼성전자주식회사 | Apparatus and method for caculating acoustic score for speech recognition, speech recognition apparatus and method, and electronic device |
KR102192678B1 (en) | 2015-10-16 | 2020-12-17 | 삼성전자주식회사 | Apparatus and method for normalizing input data of acoustic model, speech recognition apparatus |
US10951859B2 (en) | 2018-05-30 | 2021-03-16 | Microsoft Technology Licensing, Llc | Videoconferencing device and method |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4773093A (en) * | 1984-12-31 | 1988-09-20 | Itt Defense Communications | Text-independent speaker recognition system and method based on acoustic segment matching |
US5548647A (en) * | 1987-04-03 | 1996-08-20 | Texas Instruments Incorporated | Fixed text speaker verification method and apparatus |
US5271088A (en) * | 1991-05-13 | 1993-12-14 | Itt Corporation | Automated sorting of voice messages through speaker spotting |
US5522012A (en) * | 1994-02-28 | 1996-05-28 | Rutgers University | Speaker identification and verification system |
US6317710B1 (en) * | 1998-08-13 | 2001-11-13 | At&T Corp. | Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data |
-
1999
- 1999-07-14 US US09/353,192 patent/US6317710B1/en not_active Expired - Lifetime
-
2001
- 2001-10-15 US US09/976,023 patent/US6405166B1/en not_active Expired - Lifetime
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030097261A1 (en) * | 2001-11-22 | 2003-05-22 | Hyung-Bae Jeon | Speech detection apparatus under noise environment and method thereof |
US20090006102A1 (en) * | 2004-06-09 | 2009-01-01 | Canon Kabushiki Kaisha | Effective Audio Segmentation and Classification |
US8838452B2 (en) * | 2004-06-09 | 2014-09-16 | Canon Kabushiki Kaisha | Effective audio segmentation and classification |
US20110202340A1 (en) * | 2008-10-29 | 2011-08-18 | Ariyaeeinia Aladdin M | Speaker verification |
US9343067B2 (en) * | 2008-10-29 | 2016-05-17 | British Telecommunications Public Limited Company | Speaker verification |
US8959022B2 (en) | 2012-07-03 | 2015-02-17 | Motorola Solutions, Inc. | System for media correlation based on latent evidences of audio |
Also Published As
Publication number | Publication date |
---|---|
US6317710B1 (en) | 2001-11-13 |
US6405166B1 (en) | 2002-06-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6405166B1 (en) | Multimedia search apparatus and method for searching multimedia content using speaker detection by audio data | |
EP1989701B1 (en) | Speaker authentication | |
US6223155B1 (en) | Method of independently creating and using a garbage model for improved rejection in a limited-training speaker-dependent speech recognition system | |
US8775174B2 (en) | Method for indexing multimedia information | |
US9230547B2 (en) | Metadata extraction of non-transcribed video and audio streams | |
US8793127B2 (en) | Method and apparatus for automatically determining speaker characteristics for speech-directed advertising or other enhancement of speech-controlled devices or services | |
US6421645B1 (en) | Methods and apparatus for concurrent speech recognition, speaker segmentation and speaker classification | |
US7263485B2 (en) | Robust detection and classification of objects in audio using limited training data | |
EP1385149B1 (en) | Normalisation of verification score in a device for speaker verification | |
US6345252B1 (en) | Methods and apparatus for retrieving audio information using content and speaker information | |
US6219407B1 (en) | Apparatus and method for improved digit recognition and caller identification in telephone mail messaging | |
US7245767B2 (en) | Method and apparatus for object identification, classification or verification | |
US20170323643A1 (en) | Method for Speaker Diarization | |
US4773093A (en) | Text-independent speaker recognition system and method based on acoustic segment matching | |
US20060212897A1 (en) | System and method for utilizing the content of audio/video files to select advertising content for display | |
US7181393B2 (en) | Method of real-time speaker change point detection, speaker tracking and speaker model construction | |
WO2017162053A1 (en) | Identity authentication method and device | |
JP2005532582A (en) | Method and apparatus for assigning acoustic classes to acoustic signals | |
CN112530408A (en) | Method, apparatus, electronic device, and medium for recognizing speech | |
Hansen et al. | Speaker height estimation from speech: Fusing spectral regression and statistical acoustic models | |
US20010043659A1 (en) | Signal detection method and apparatus, relevant program, and storage medium storing the program | |
US20040193415A1 (en) | Automated decision making using time-varying stream reliability prediction | |
Dharanipragada et al. | Audio-Indexing For Broadcast News. | |
CN110491375B (en) | Target language detection method and device | |
US7340398B2 (en) | Selective sampling for sound signal classification |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
FPAY | Fee payment |
Year of fee payment: 12 |
|
AS | Assignment |
Owner name: AT&T CORP., NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HUANG, QIAN;MAGRIN-CHAGNOLLEAU, IVAN;PARTHASARATHY, SARANGARAJAN;AND OTHERS;SIGNING DATES FROM 19990622 TO 19990713;REEL/FRAME:038959/0712 Owner name: AT&T PROPERTIES, LLC, NEVADA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AT&T CORP.;REEL/FRAME:038961/0317 Effective date: 20160204 Owner name: AT&T INTELLECTUAL PROPERTY II, L.P., GEORGIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AT&T PROPERTIES, LLC;REEL/FRAME:038961/0431 Effective date: 20160204 |
|
AS | Assignment |
Owner name: NUANCE COMMUNICATIONS, INC., MASSACHUSETTS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AT&T INTELLECTUAL PROPERTY II, L.P.;REEL/FRAME:041498/0316 Effective date: 20161214 |