US20030171936A1 - Method of segmenting an audio stream - Google Patents
Method of segmenting an audio stream Download PDFInfo
- Publication number
- US20030171936A1 US20030171936A1 US10/370,065 US37006503A US2003171936A1 US 20030171936 A1 US20030171936 A1 US 20030171936A1 US 37006503 A US37006503 A US 37006503A US 2003171936 A1 US2003171936 A1 US 2003171936A1
- Authority
- US
- United States
- Prior art keywords
- stage
- values
- sub
- modules
- segmentation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 230000011218 segmentation Effects 0.000 claims abstract description 48
- 238000004364 calculation method Methods 0.000 claims abstract description 23
- 238000004458 analytical method Methods 0.000 claims abstract description 21
- 239000013598 vector Substances 0.000 claims description 64
- 239000003550 marker Substances 0.000 claims description 25
- 238000009434 installation Methods 0.000 claims 4
- 239000006185 dispersion Substances 0.000 claims 2
- 230000005236 sound signal Effects 0.000 abstract description 7
- 230000004807 localization Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 6
- 230000003595 spectral effect Effects 0.000 description 6
- 238000001514 detection method Methods 0.000 description 3
- 238000001228 spectrum Methods 0.000 description 3
- 238000007619 statistical method Methods 0.000 description 3
- 238000013518 transcription Methods 0.000 description 3
- 230000035897 transcription Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000004907 flux Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 241001672694 Citrus reticulata Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 229910003460 diamond Inorganic materials 0.000 description 1
- 239000010432 diamond Substances 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000010183 spectrum analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/04—Segmentation; Word boundary detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/233—Processing of audio elementary streams
Definitions
- the present invention relates to a segmentation method of an audio data stream, which is broadcasted or recorded using some media, wherein this audio data stream is a sequence of digital samples, or may be transformed to the sequence of digital samples.
- the goal of such segmentation is the division of audio data stream into segments, which correspond to different physical sources of audio signal.
- parameters of background source(s) is not changed essentially in the framework of one segment.
- Audio and video recordings have become commonplace with the advent of consumer grade recording equipment. Unfortunately, both the audio and video streams provide few clues to assist in accessing the desired section of the record.
- indexing is provided by the table of contents at the front and the index at the end, which readers can browse to locate authors and references to authors.
- a similar indexing scheme would be useful in an audio stream, to help in location of sections where, for example, specific speakers were talking.
- the limited amount of data associated with most audio records does not provide enough information for confidently and easily access to desired points of interest. So, user has to peruse the contents of a record in sequential order to retrieve desired information.
- the indexation process consists of two sequential parts: segmentation and classification processes. Segmentation process implies division of the audio stream into homogeneous (in some sense) segments. Classification process implies the attributing of these segments by appropriate notes. Thus, segmentation process is the first and very important stage in the indexation process. To this problem, the basic notice in the given invention is given.
- Broadcast news data come to use in long unsegmented speech streams, which not only contain speech with various speakers, backgrounds, and channels, but also contain a lot of non-speech audio information. So it is necessary to chop the long stream into smaller segments. It is also important to make these smaller segments homogeneous (each segment contains the data from one source only), so that the non-speech information can be discarded, and those segments from the same or similar source can be clustered for speaker normalization and adaptation.
- An amplitude-based detector was used to break the input into chunks that are 20 to 30 seconds long.
- Balasubramanian et al. patent U.S.5,606,643, enables retrieval based on indexing an audio stream of a recording according to the speaker.
- the audio stream may be segmented into speaker events, and each segment labeled with the type of event, or speaker identity.
- the audio stream may be segregated into events according to speaker difference, with segments created by the same speaker identified or marked.
- Creating an index in an audio stream may enable a user to locate particular segments of the audio data. For example, this may enable a user to browse a recording to select audio segments corresponding to a specific speaker, or “fast-forward” through a recording to the next speaker.
- knowing the ordering of speakers can also provide content clues about the conversation, or about the context of the conversation.
- the ultimate goal of the segmentation is to produce a sequence of discrete segments with particular characteristics remaining constant within each one.
- the characteristics of choice depend on the overall structure of the indexation system.
- the present invention provides a segmentation procedure to chunk an input audio stream into segments having homogeneous acoustic characteristics.
- This audio stream is a sequence of digital samples, which are broadcasted or recorded using some media.
- An object of the invention is to provide a fast segmentation procedure with a relatively low numerical complexity.
- the segmentation procedure comprises three stages. These are: first-grade characteristic calculation, second-grade characteristic calculation, and decision-making.
- the stage of first-grade characteristic calculation is aimed for calculation of audio features vectors from the input audio stream. These features vectors define characteristics of audio signals.
- the stage of second-grade characteristic calculation forms sequence of statistic features vectors from the sequence of audio features vectors.
- the statistic features vectors define statistic features of the first-grade features.
- the stage of decision-making analyses variation of the second grade features and performs definition of the segments boundaries basing on that analysis.
- an essential aim of the invention is to provide the segmentation method, firstly, that can be used for a wide variety of applications, secondary, that the segmentation procedure may be industrial-scaled manufactured, based on the development of one relatively simple integrated circuit.
- FIG. 1 is a block diagram of a generalized audio processing system within which the present invention may be embodied
- FIG. 2 is a generalized flow diagram of the audio segmentation procedure
- FIG. 3 is a flow diagram in detail of the audio segmentation procedure
- FIG. 4 illustrates a flowchart of the sub-stage of initial segmentation
- FIG. 5 illustrates a flowchart of the sub-stage of accurate segmentation
- FIG. 6 shows improvement of the dividing markers positions
- FIG. 7 shows defining the homogeneous interval inside the segment.
- FIG. 1 is a block diagram of a generalized audio processing system 1 , within which the present invention may be embodied.
- an audio stream is provided from a source of audio data 2 , which may be provided by recorded broadcast, a recorded video with accompanying audio track, or other audio source.
- the audio data is sent to an audio processor 3 , which may be any well-known device such as a general purpose computer, configured according to the present invention.
- the audio processor outputs segments of the audio data 4 .
- FIG. 2 is a generalized flow diagram of an audio segmentation procedure 5 .
- Box 10 is the audio stream input, for example, broadcast news input.
- the step in box 20 is aimed for calculation of audio features vectors from the audio stream. These features vectors define characteristic features of the audio-stream.
- Next step 30 forms sequence of statistic features vectors from the sequence of audio features vectors. The statistic features vectors define statistic characteristic of the audio features vectors.
- variation of the statistic features vectors is analyzed and the definition of the segment boundaries basing on that analysis is performed.
- the proposed segmentation procedure is based on the analysis of audio signal statistical features variation.
- the output of the resulting index segmentation of the audio stream is performed at the step 50 .
- FIG. 3 is a flow diagram in detail of the audio segmentation procedure.
- an input sequence of the digital samples is divided into the sequence of short (e.g. 10-20 ms) not-overlapped frames 21 .
- the feature vectors are computed for each frame. This computation is performed using 10th order Linear Predictive Coding (LPC) analysis of the samples in possibly overlapped windows, which contain said frames.
- LPC Linear Predictive Coding
- K 1 ,K 2 First and the Second Reflection Coefficients
- the audio feature vectors (9 characteristics at all). These vectors have definite physical meaning and the dynamical range sufficient for the precise segmentation of the audio stream.
- the further work of the segmentation procedure is the statistical analysis of the obtained data. The calculation of the statistical characteristics is performed in non-overlapped second-grade windows, each of these windows consists of some predefined number of frames (e.g. 20-100 frames in one window). Thus, some number of vectors of the first-grade characteristics describes such a window.
- the division of the input sequence of the audio feature vectors is performed at the step 31 .
- the sequence of those vectors is transformed to the statistic feature vectors.
- - ⁇ i 1 M ⁇
- M is a number of frames in one window.
- FIG. 3 serves to give an overview of the method described by the invention.
- the sub-stage of initial segmentation 100 is performed in such a way that the dividing markers, which corresponds boundaries of segments, are determined with the accuracy corresponding to one second-grade window.
- the sub-stage of improvement of the segmentation precision 200 carried out by the previous step implies the correction of the position of each dividing marker with the accuracy corresponding to one frame and eliminating of false segments.
- the sub-stage of internal markers definition 300 implies the determination of a stationary interval inside each segment. The resulting sequence of the not intersected audio segments with their time boundaries is outputted at the step 50 .
- FIG. 4 illustrates a flowchart of the sub-stage of initial segmentation 100 of FIG. 3.
- the algorithm of this sub-stage parses four sequential entry vectors. The result of the analysis is the information, where the dividing marker is placed.
- [0054] are calculated for the first sub-vectors of the statistical features vectors 137 . If at least one of these values is greater than the corresponding predefined threshold 138 , the dividing marker is installed between the second-range windows 139 . In this case, another steps of this sub-stage does not performed and the next four vectors, first of which is the first vector after the installed dividing marker will be taken from the set of sequential statistical features vectors for analysis 148 .
- next four vectors first of which is the vector ⁇ right arrow over (V) ⁇ [k+1] will be taken from the set of sequential statistical features vectors for analysis 148 . If the dividing marker is taken at the step in diamond 147 , then the sub-stage of initial segmentation ends and the initial segmentation marker passes to the sub-stage of accurate segmentation.
- FIG. 5 illustrates a flowchart of the sub-stage of accurate segmentation 200 of FIG. 3.
- the new dividing marker ⁇ is placed into the position corresponded to this J between shaded rectangles on FIG. 6.
- the shift of vectors is performed from the position of the new marker ⁇ .
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
Description
- 1. Field of the Invention
- The present invention relates to a segmentation method of an audio data stream, which is broadcasted or recorded using some media, wherein this audio data stream is a sequence of digital samples, or may be transformed to the sequence of digital samples. The goal of such segmentation is the division of audio data stream into segments, which correspond to different physical sources of audio signal. In the case when some source(s) and some background source(s) emit audio signal, parameters of background source(s) is not changed essentially in the framework of one segment.
- 2. Description of the Related Art
- Audio and video recordings have become commonplace with the advent of consumer grade recording equipment. Unfortunately, both the audio and video streams provide few clues to assist in accessing the desired section of the record. In books, indexing is provided by the table of contents at the front and the index at the end, which readers can browse to locate authors and references to authors. A similar indexing scheme would be useful in an audio stream, to help in location of sections where, for example, specific speakers were talking. The limited amount of data associated with most audio records does not provide enough information for confidently and easily access to desired points of interest. So, user has to peruse the contents of a record in sequential order to retrieve desired information.
- As a solution of this problem, it is possible to use the automatic indexing system of audio events in the audio data stream. The indexation process consists of two sequential parts: segmentation and classification processes. Segmentation process implies division of the audio stream into homogeneous (in some sense) segments. Classification process implies the attributing of these segments by appropriate notes. Thus, segmentation process is the first and very important stage in the indexation process. To this problem, the basic notice in the given invention is given.
- As the basic audio events in the audio stream, it is accepted to consider speech, music and noise (that is non-speech and non-music). The basic notice in a world is given to the speech detection, segmentation and indexation in audio stream, such as broadcast news.
- Broadcast news data come to use in long unsegmented speech streams, which not only contain speech with various speakers, backgrounds, and channels, but also contain a lot of non-speech audio information. So it is necessary to chop the long stream into smaller segments. It is also important to make these smaller segments homogeneous (each segment contains the data from one source only), so that the non-speech information can be discarded, and those segments from the same or similar source can be clustered for speaker normalization and adaptation.
- Zhan et al., “Dragon Systems' 1997 Mandarin Broadcast News System”, Proceedings of the Broadcast News transcription and Understanding Workshop, Lansdowne, Va., pp. 25-27, 1998, produced the segments by looking for sufficiently long silence regions in the output of a coarse recognition pass. This method generated considerable multi-speaker segments, and no speaker change information was used in the segmentation.
- In the subsequent works, Wegmann et al., “Progress in Broadcast News Transcription at Dragon System”, Proceedings of ICASSP'99, Phoenix, Ariz., March, 1999, used the speaker change detection in the segmentation pass. The following is a procedure of their automatic segmentation:
- An amplitude-based detector was used to break the input into chunks that are 20 to 30 seconds long.
- These chunks were chopped into 2 to 30 seconds long, based on silences produced from a fast word recognizer.
- These segments were further refined using a speaker change detector.
- Balasubramanian et al., patent U.S.5,606,643, enables retrieval based on indexing an audio stream of a recording according to the speaker. In particular, the audio stream may be segmented into speaker events, and each segment labeled with the type of event, or speaker identity. When speech from individuals is intermixed, for example in conversational situations, the audio stream may be segregated into events according to speaker difference, with segments created by the same speaker identified or marked.
- Creating an index in an audio stream, either in real time or in post-processing, may enable a user to locate particular segments of the audio data. For example, this may enable a user to browse a recording to select audio segments corresponding to a specific speaker, or “fast-forward” through a recording to the next speaker. In addition, knowing the ordering of speakers can also provide content clues about the conversation, or about the context of the conversation.
- The ultimate goal of the segmentation is to produce a sequence of discrete segments with particular characteristics remaining constant within each one. The characteristics of choice depend on the overall structure of the indexation system.
- Saunders, “Real-Time Discrimination of Broadcast Speech/Music”, Proc. ICASSP 1996, pp. 993-996, has described a speech/music discriminator based on zero-crossings. Its application is for discrimination between advertisements and programs in radio broadcasts. Since it is intended to be incorporated in consumer radios, it is intended to be low cost and simple. It is mainly designed to detect the characteristics of speech, which are described as, limited bandwidth, alternate voiced and unvoiced sections, limited range of pitch, syllabic duration of vowels, energy variations between high and low levels. It is indirectly using the amplitude, pitch and periodicity estimate of the waveform to carry out the detection process since zero-crossings give an estimate of the dominant frequency in the waveform.
- Zue and Spina, “Automatic Transcription of General Audio Data: Preliminary Analyses”, Proc. ICSP 1996, pp. 594-597, use an average of the cepstral coefficients over a series of frames. This is shown to work well in distinguishing between speech and music when the speech is band-limited to 4 kHz and music to 16 kHz but less well when both signals occupied a 16 kHz bandwidth.
- Scheier and Slaney, “Construction and Evalution of a Robust Multifeature Speech/Music Discriminator”, Proc. ICASSP 1997, pp. 1331-1334, use a variety of features. These are: four hertz modulation energy, low energy, roll off of the spectrum, the variance of the roll off of the spectrum, the spectral centroid, variance of the spectral centroid, the spectral flux, variance of the spectral flux, the zero-crossing rate, variance of the zero-crossing rate, the cepstral residual, variance of the cepstral residual, pulse metric. The first two features are amplitude related. The next six features are derived from the fine spectrum of the input signal and therefore are related to the techniques described in the previous reference.
- Carey et al., “A Comparison of Features for Speech, Music Discrimination”, Proc. IEEE 1999, pp. 149-152, use a variety of features. There are: cepstral coefficients, delta cepstral coefficients, amplitude, delta amplitude, pitch, delta pitch, zero-crossing rate, delta zero-crossing rate. The pitch and cepstral coefficients encompass the fine and broad spectral features respectively. The zero-crossing parameters and the amplitude were believed worthy of investigation as a computationally inexpensive alternative to the other features.
- The present invention provides a segmentation procedure to chunk an input audio stream into segments having homogeneous acoustic characteristics. This audio stream is a sequence of digital samples, which are broadcasted or recorded using some media.
- An object of the invention is to provide a fast segmentation procedure with a relatively low numerical complexity.
- The segmentation procedure comprises three stages. These are: first-grade characteristic calculation, second-grade characteristic calculation, and decision-making. The stage of first-grade characteristic calculation is aimed for calculation of audio features vectors from the input audio stream. These features vectors define characteristics of audio signals. The stage of second-grade characteristic calculation forms sequence of statistic features vectors from the sequence of audio features vectors. The statistic features vectors define statistic features of the first-grade features. The stage of decision-making analyses variation of the second grade features and performs definition of the segments boundaries basing on that analysis.
- Thus, an essential aim of the invention is to provide the segmentation method, firstly, that can be used for a wide variety of applications, secondary, that the segmentation procedure may be industrial-scaled manufactured, based on the development of one relatively simple integrated circuit.
- Other aspects of the present invention can be seen upon review of the figure, the detailed description and the claims, which follow.
- The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate an embodiment of the invention and together with the description serve to explain the principle of the invention. In the drawings:
- FIG. 1 is a block diagram of a generalized audio processing system within which the present invention may be embodied;
- FIG. 2 is a generalized flow diagram of the audio segmentation procedure;
- FIG. 3 is a flow diagram in detail of the audio segmentation procedure;
- FIG. 4 illustrates a flowchart of the sub-stage of initial segmentation;
- FIG. 5 illustrates a flowchart of the sub-stage of accurate segmentation;
- FIG. 6 shows improvement of the dividing markers positions;
- FIG. 7 shows defining the homogeneous interval inside the segment.
- FIG. 1 is a block diagram of a generalized
audio processing system 1, within which the present invention may be embodied. Generally, an audio stream is provided from a source ofaudio data 2, which may be provided by recorded broadcast, a recorded video with accompanying audio track, or other audio source. The audio data is sent to anaudio processor 3, which may be any well-known device such as a general purpose computer, configured according to the present invention. The audio processor outputs segments of theaudio data 4. - FIG. 2 is a generalized flow diagram of an
audio segmentation procedure 5.Box 10 is the audio stream input, for example, broadcast news input. The step inbox 20 is aimed for calculation of audio features vectors from the audio stream. These features vectors define characteristic features of the audio-stream.Next step 30 forms sequence of statistic features vectors from the sequence of audio features vectors. The statistic features vectors define statistic characteristic of the audio features vectors. At thestep 40, variation of the statistic features vectors is analyzed and the definition of the segment boundaries basing on that analysis is performed. Thus, the proposed segmentation procedure is based on the analysis of audio signal statistical features variation. The output of the resulting index segmentation of the audio stream is performed at thestep 50. - FIG. 3 is a flow diagram in detail of the audio segmentation procedure. After the input of
audio stream data 10, an input sequence of the digital samples is divided into the sequence of short (e.g. 10-20 ms) not-overlapped frames 21. At thestep 25, the feature vectors are computed for each frame. This computation is performed using 10th order Linear Predictive Coding (LPC) analysis of the samples in possibly overlapped windows, which contain said frames. - Parameters of the autoregressive linear model, which is the foundation of LPC analysis are reliable and may be defined with relatively small computation complexity. The following parameters form coordinates of audio features vector:
- Λi,i=1.5—Formants Frequencies;
- K1,K2—First and the Second Reflection Coefficients;
- E0—Energy of the Prediction Error Coefficient;
- E1—Preemphasized Energy Ratio Coefficient.
- Parameters K1,K2, E0 are calculated simultaneously with LPC analysis, according to Marple, Jr. “Digital Spectral Analysis”, Prentice-Hall, Inc., Englewood Cliffs, N.J., 1987. After LPC analysis, 10 coefficients Line Spectral Pairs (LSP) are computed according to the patent U.S. Pat. No. 4,393,272 or ITU-T, Study Group 15 Contribution--Q. 12/15, Draft Recommendation G.729, Jun. 8, 1995, Version 5.0. Λi,i=1.5 Formant Frequencies are calculated as half of sum of the corresponding LSP coefficients. E1 is the ratio of the energies in the 6-dB preemrhasized first-order difference audio signal to the regular audio signal, according to Campbell et al. “Voiced/Unvoiced Classification of Speech with Applications to the U.S.
- Government LPC-10E Algorithm”, Proceedings ICASSP' 86, April, Tokyo, Japan, V.1, pp 473-476.
- As the result, there are the audio feature vectors (9 characteristics at all). These vectors have definite physical meaning and the dynamical range sufficient for the precise segmentation of the audio stream. The further work of the segmentation procedure is the statistical analysis of the obtained data. The calculation of the statistical characteristics is performed in non-overlapped second-grade windows, each of these windows consists of some predefined number of frames (e.g. 20-100 frames in one window). Thus, some number of vectors of the first-grade characteristics describes such a window. The division of the input sequence of the audio feature vectors is performed at the
step 31. At the step 35, the sequence of those vectors is transformed to the statistic feature vectors. -
-
- where M is a number of frames in one window.
- As the result, there are the statistic feature vectors (15 characteristics at all).
- The sub-stages of the decision-making40 will be discussed in more details below, but FIG. 3 serves to give an overview of the method described by the invention.
- The sub-stage of
initial segmentation 100 is performed in such a way that the dividing markers, which corresponds boundaries of segments, are determined with the accuracy corresponding to one second-grade window. The sub-stage of improvement of thesegmentation precision 200 carried out by the previous step implies the correction of the position of each dividing marker with the accuracy corresponding to one frame and eliminating of false segments. The sub-stage ofinternal markers definition 300 implies the determination of a stationary interval inside each segment. The resulting sequence of the not intersected audio segments with their time boundaries is outputted at thestep 50. - Sub-Stage of Initial Segmentation
- FIG. 4 illustrates a flowchart of the sub-stage of
initial segmentation 100 of FIG. 3. In this sub-stage, the statistical features vectors {right arrow over (V)}[k],k=1, . . . , K are analyzed. On each step, the algorithm of this sub-stage parses four sequential entry vectors. The result of the analysis is the information, where the dividing marker is placed. - Let {right arrow over (V)}[k],{right arrow over (V)}[k+1],{right arrow over (V)}[k+2],{right arrow over (V)}[k+3] four sequential statistical features vectors, which are taken136 from the set of sequential statistical features vectors.
- The differences Aj i=|Vi[k+j]−Vi[k+j+1]|, j=0,1,2,i=1, . . . , 10
- are calculated for the first sub-vectors of the
statistical features vectors 137. If at least one of these values is greater than the correspondingpredefined threshold 138, the dividing marker is installed between the second-range windows 139. In this case, another steps of this sub-stage does not performed and the next four vectors, first of which is the first vector after the installed dividing marker will be taken from the set of sequential statistical features vectors foranalysis 148. - Otherwise the differences Ai=|(Vi[k]+Vi[k+1])−(Vi[k+2]+Vi[k+3])|, i=11.15 are calculated 140 for the second sub-vectors of the statistical features vectors. These values are matched with the
predefined thresholds 141. The case when all of these values are smaller than the corresponding threshold values corresponds to the absence of the dividingmarker 142. In this case, the last steps of this sub-stage does not performed and the next four vectors, first of which is the vector {right arrow over (V)}[k+1] will be taken from the set of sequential statistical features vectors foranalysis 148. Otherwise the differences Aj i=|Vi[k+j]−Vi[k+j+1]|, i=11.15, j=0,1,2 are calculated 143 for the second sub-vectors of the statistical features vectors. If at least one of these values is greater than the correspondingpredefined thresholds 144 then the dividing marker is installed between the second-range windows 145. In this case, another steps of this sub-stage is not performed and the next four vectors, first of which is the first vector after the installed dividing marker will be taken from the set of sequentialstatistical features vectors 148. Otherwise the next four vectors, first of which is the vector {right arrow over (V)}[k+1] will be taken from the set of sequential statistical features vectors foranalysis 148. If the dividing marker is taken at the step indiamond 147, then the sub-stage of initial segmentation ends and the initial segmentation marker passes to the sub-stage of accurate segmentation. - Sub-Stage of Accurate Segmentation
- FIG. 5 illustrates a flowchart of the sub-stage of
accurate segmentation 200 of FIG. 3. The sense of this given stage operation consists in an improvement of dividing markers positions. It is achieved as a result of a precise statistical analysis of the sequence of LSP coefficients Λi,i=1, . . . , 5 close to each dividing marker (see FIG. 6). Let's consider an arbitrary dividing marker μ with some neighborhood, which consists of n frames, close to Formants Frequencies coefficients. At the step inbox 210, the difference is evaluated: -
- At the
step 230, the new dividing marker μ is placed into the position corresponded to this J between shaded rectangles on FIG. 6. At the step inbox 148 in FIG. 4, the shift of vectors is performed from the position of the new marker μ. - Sub-Stage of Internal Markers Definition
- The sub-stage of internal markers definition of the final segmentation analyses each segment with the purpose of the definition of two internal markers (μint,ηint) defining the most homogeneous interval inside the segment. It is made with the following purposes: the placed dividing markers separate two audio events of the different nature. These events, as a rule, smoothly transiting one to another and do not have drastic border. Therefore there is a time interval containing information about both the events. That may hamper their correct classification.
- As well as at the previous sub-stage, this task is solved by usage of a precise statistical analysis of a sequence of Formants Frequencies coefficients Λi,i=1, . . . , 5 close to each dividing marker. Let's consider an arbitrary segment, limited by markers μ and η, (so that η−μ=n+1 frames), and composed from Formants Frequencies coefficients (see FIG. 7).
-
-
- Then, the new markers μint and ηint are placed into the positions corresponded to these J1, J2 between shaded rectangles on FIG. 7.
- Thus, the process of segmentation is ended. As the result, the sequence of not intersected audio intervals with their time boundaries is obtained.
Claims (22)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020020009209A KR20030070179A (en) | 2002-02-21 | 2002-02-21 | Method of the audio stream segmantation |
KR2002/9209 | 2002-02-21 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20030171936A1 true US20030171936A1 (en) | 2003-09-11 |
US7346516B2 US7346516B2 (en) | 2008-03-18 |
Family
ID=29546250
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/370,065 Expired - Fee Related US7346516B2 (en) | 2002-02-21 | 2003-02-21 | Method of segmenting an audio stream |
Country Status (2)
Country | Link |
---|---|
US (1) | US7346516B2 (en) |
KR (1) | KR20030070179A (en) |
Cited By (120)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050126369A1 (en) * | 2003-12-12 | 2005-06-16 | Nokia Corporation | Automatic extraction of musical portions of an audio stream |
US20080236368A1 (en) * | 2007-03-26 | 2008-10-02 | Sanyo Electric Co., Ltd. | Recording or playback apparatus and musical piece detecting apparatus |
US20110161074A1 (en) * | 2009-12-29 | 2011-06-30 | Apple Inc. | Remote conferencing center |
US20140207456A1 (en) * | 2010-09-23 | 2014-07-24 | Waveform Communications, Llc | Waveform analysis of speech |
US8879761B2 (en) | 2011-11-22 | 2014-11-04 | Apple Inc. | Orientation-based audio |
US8892446B2 (en) | 2010-01-18 | 2014-11-18 | Apple Inc. | Service orchestration for intelligent automated assistant |
US8977584B2 (en) | 2010-01-25 | 2015-03-10 | Newvaluexchange Global Ai Llp | Apparatuses, methods and systems for a digital conversation management platform |
US9262612B2 (en) | 2011-03-21 | 2016-02-16 | Apple Inc. | Device access using voice authentication |
US9300784B2 (en) | 2013-06-13 | 2016-03-29 | Apple Inc. | System and method for emergency calls initiated by voice command |
US9330720B2 (en) | 2008-01-03 | 2016-05-03 | Apple Inc. | Methods and apparatus for altering audio output signals |
US9338493B2 (en) | 2014-06-30 | 2016-05-10 | Apple Inc. | Intelligent automated assistant for TV user interactions |
US9368114B2 (en) | 2013-03-14 | 2016-06-14 | Apple Inc. | Context-sensitive handling of interruptions |
US9430463B2 (en) | 2014-05-30 | 2016-08-30 | Apple Inc. | Exemplar-based natural language processing |
US9483461B2 (en) | 2012-03-06 | 2016-11-01 | Apple Inc. | Handling speech synthesis of content for multiple languages |
US9495129B2 (en) | 2012-06-29 | 2016-11-15 | Apple Inc. | Device, method, and user interface for voice-activated navigation and browsing of a document |
US9502031B2 (en) | 2014-05-27 | 2016-11-22 | Apple Inc. | Method for supporting dynamic grammars in WFST-based ASR |
US9535906B2 (en) | 2008-07-31 | 2017-01-03 | Apple Inc. | Mobile device having human language translation capability with positional feedback |
US9576574B2 (en) | 2012-09-10 | 2017-02-21 | Apple Inc. | Context-sensitive handling of interruptions by intelligent digital assistant |
US9582608B2 (en) | 2013-06-07 | 2017-02-28 | Apple Inc. | Unified ranking with entropy-weighted information for phrase-based semantic auto-completion |
US9620104B2 (en) | 2013-06-07 | 2017-04-11 | Apple Inc. | System and method for user-specified pronunciation of words for speech synthesis and recognition |
US9620105B2 (en) | 2014-05-15 | 2017-04-11 | Apple Inc. | Analyzing audio input for efficient speech and music recognition |
US9626955B2 (en) | 2008-04-05 | 2017-04-18 | Apple Inc. | Intelligent text-to-speech conversion |
US9633674B2 (en) | 2013-06-07 | 2017-04-25 | Apple Inc. | System and method for detecting errors in interactions with a voice-based digital assistant |
US9633004B2 (en) | 2014-05-30 | 2017-04-25 | Apple Inc. | Better resolution when referencing to concepts |
US9633660B2 (en) | 2010-02-25 | 2017-04-25 | Apple Inc. | User profiling for voice input processing |
US9646609B2 (en) | 2014-09-30 | 2017-05-09 | Apple Inc. | Caching apparatus for serving phonetic pronunciations |
US9646614B2 (en) | 2000-03-16 | 2017-05-09 | Apple Inc. | Fast, language-independent method for user authentication by voice |
US9668121B2 (en) | 2014-09-30 | 2017-05-30 | Apple Inc. | Social reminders |
US9697820B2 (en) | 2015-09-24 | 2017-07-04 | Apple Inc. | Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks |
US9697822B1 (en) | 2013-03-15 | 2017-07-04 | Apple Inc. | System and method for updating an adaptive speech recognition model |
US9711141B2 (en) | 2014-12-09 | 2017-07-18 | Apple Inc. | Disambiguating heteronyms in speech synthesis |
US9715875B2 (en) | 2014-05-30 | 2017-07-25 | Apple Inc. | Reducing the need for manual start/end-pointing and trigger phrases |
US9721566B2 (en) | 2015-03-08 | 2017-08-01 | Apple Inc. | Competing devices responding to voice triggers |
US9734193B2 (en) | 2014-05-30 | 2017-08-15 | Apple Inc. | Determining domain salience ranking from ambiguous words in natural speech |
US9760559B2 (en) | 2014-05-30 | 2017-09-12 | Apple Inc. | Predictive text input |
US9785630B2 (en) | 2014-05-30 | 2017-10-10 | Apple Inc. | Text prediction using combined word N-gram and unigram language models |
US9798393B2 (en) | 2011-08-29 | 2017-10-24 | Apple Inc. | Text correction processing |
US9818400B2 (en) | 2014-09-11 | 2017-11-14 | Apple Inc. | Method and apparatus for discovering trending terms in speech requests |
US9842101B2 (en) | 2014-05-30 | 2017-12-12 | Apple Inc. | Predictive conversion of language input |
US9842105B2 (en) | 2015-04-16 | 2017-12-12 | Apple Inc. | Parsimonious continuous-space phrase representations for natural language processing |
US9858925B2 (en) | 2009-06-05 | 2018-01-02 | Apple Inc. | Using context information to facilitate processing of commands in a virtual assistant |
US9865280B2 (en) | 2015-03-06 | 2018-01-09 | Apple Inc. | Structured dictation using intelligent automated assistants |
US9886953B2 (en) | 2015-03-08 | 2018-02-06 | Apple Inc. | Virtual assistant activation |
US9886432B2 (en) | 2014-09-30 | 2018-02-06 | Apple Inc. | Parsimonious handling of word inflection via categorical stem + suffix N-gram language models |
US9899019B2 (en) | 2015-03-18 | 2018-02-20 | Apple Inc. | Systems and methods for structured stem and suffix language models |
US9922642B2 (en) | 2013-03-15 | 2018-03-20 | Apple Inc. | Training an at least partial voice command system |
US9934775B2 (en) | 2016-05-26 | 2018-04-03 | Apple Inc. | Unit-selection text-to-speech synthesis based on predicted concatenation parameters |
US9953088B2 (en) | 2012-05-14 | 2018-04-24 | Apple Inc. | Crowd sourcing information to fulfill user requests |
US9959870B2 (en) | 2008-12-11 | 2018-05-01 | Apple Inc. | Speech recognition involving a mobile device |
US9966068B2 (en) | 2013-06-08 | 2018-05-08 | Apple Inc. | Interpreting and acting upon commands that involve sharing information with remote devices |
US9966065B2 (en) | 2014-05-30 | 2018-05-08 | Apple Inc. | Multi-command single utterance input method |
US9971774B2 (en) | 2012-09-19 | 2018-05-15 | Apple Inc. | Voice-based media searching |
US9972304B2 (en) | 2016-06-03 | 2018-05-15 | Apple Inc. | Privacy preserving distributed evaluation framework for embedded personalized systems |
US10043516B2 (en) | 2016-09-23 | 2018-08-07 | Apple Inc. | Intelligent automated assistant |
US10049663B2 (en) | 2016-06-08 | 2018-08-14 | Apple, Inc. | Intelligent automated assistant for media exploration |
US10049668B2 (en) | 2015-12-02 | 2018-08-14 | Apple Inc. | Applying neural network language models to weighted finite state transducers for automatic speech recognition |
US10057736B2 (en) | 2011-06-03 | 2018-08-21 | Apple Inc. | Active transport based notifications |
US10067938B2 (en) | 2016-06-10 | 2018-09-04 | Apple Inc. | Multilingual word prediction |
US10074360B2 (en) | 2014-09-30 | 2018-09-11 | Apple Inc. | Providing an indication of the suitability of speech recognition |
US10078631B2 (en) | 2014-05-30 | 2018-09-18 | Apple Inc. | Entropy-guided text prediction using combined word and character n-gram language models |
US10079014B2 (en) | 2012-06-08 | 2018-09-18 | Apple Inc. | Name recognition system |
US10083688B2 (en) | 2015-05-27 | 2018-09-25 | Apple Inc. | Device voice control for selecting a displayed affordance |
US10089072B2 (en) | 2016-06-11 | 2018-10-02 | Apple Inc. | Intelligent device arbitration and control |
US10101822B2 (en) | 2015-06-05 | 2018-10-16 | Apple Inc. | Language input correction |
US10127220B2 (en) | 2015-06-04 | 2018-11-13 | Apple Inc. | Language identification from short strings |
US10127911B2 (en) | 2014-09-30 | 2018-11-13 | Apple Inc. | Speaker identification and unsupervised speaker adaptation techniques |
US10134385B2 (en) | 2012-03-02 | 2018-11-20 | Apple Inc. | Systems and methods for name pronunciation |
US10170123B2 (en) | 2014-05-30 | 2019-01-01 | Apple Inc. | Intelligent assistant for home automation |
US10176167B2 (en) | 2013-06-09 | 2019-01-08 | Apple Inc. | System and method for inferring user intent from speech inputs |
US10186254B2 (en) | 2015-06-07 | 2019-01-22 | Apple Inc. | Context-based endpoint detection |
US10185542B2 (en) | 2013-06-09 | 2019-01-22 | Apple Inc. | Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant |
US10192552B2 (en) | 2016-06-10 | 2019-01-29 | Apple Inc. | Digital assistant providing whispered speech |
US10199051B2 (en) | 2013-02-07 | 2019-02-05 | Apple Inc. | Voice trigger for a digital assistant |
US10223066B2 (en) | 2015-12-23 | 2019-03-05 | Apple Inc. | Proactive assistance based on dialog communication between devices |
US10241644B2 (en) | 2011-06-03 | 2019-03-26 | Apple Inc. | Actionable reminder entries |
US10241752B2 (en) | 2011-09-30 | 2019-03-26 | Apple Inc. | Interface for a virtual digital assistant |
US10249300B2 (en) | 2016-06-06 | 2019-04-02 | Apple Inc. | Intelligent list reading |
US10255907B2 (en) | 2015-06-07 | 2019-04-09 | Apple Inc. | Automatic accent detection using acoustic models |
US10269345B2 (en) | 2016-06-11 | 2019-04-23 | Apple Inc. | Intelligent task discovery |
US10276170B2 (en) | 2010-01-18 | 2019-04-30 | Apple Inc. | Intelligent automated assistant |
US10283110B2 (en) | 2009-07-02 | 2019-05-07 | Apple Inc. | Methods and apparatuses for automatic speech recognition |
US10289433B2 (en) | 2014-05-30 | 2019-05-14 | Apple Inc. | Domain specific language for encoding assistant dialog |
US10297253B2 (en) | 2016-06-11 | 2019-05-21 | Apple Inc. | Application integration with a digital assistant |
US10318871B2 (en) | 2005-09-08 | 2019-06-11 | Apple Inc. | Method and apparatus for building an intelligent automated assistant |
US10356243B2 (en) | 2015-06-05 | 2019-07-16 | Apple Inc. | Virtual assistant aided communication with 3rd party service in a communication session |
US10354011B2 (en) | 2016-06-09 | 2019-07-16 | Apple Inc. | Intelligent automated assistant in a home environment |
US10366158B2 (en) | 2015-09-29 | 2019-07-30 | Apple Inc. | Efficient word encoding for recurrent neural network language models |
US10410637B2 (en) | 2017-05-12 | 2019-09-10 | Apple Inc. | User-specific acoustic models |
US10446141B2 (en) | 2014-08-28 | 2019-10-15 | Apple Inc. | Automatic speech recognition based on user feedback |
US10446143B2 (en) | 2016-03-14 | 2019-10-15 | Apple Inc. | Identification of voice inputs providing credentials |
US10482874B2 (en) | 2017-05-15 | 2019-11-19 | Apple Inc. | Hierarchical belief states for digital assistants |
US10490187B2 (en) | 2016-06-10 | 2019-11-26 | Apple Inc. | Digital assistant providing automated status report |
US10496753B2 (en) | 2010-01-18 | 2019-12-03 | Apple Inc. | Automatically adapting user interfaces for hands-free interaction |
US10509862B2 (en) | 2016-06-10 | 2019-12-17 | Apple Inc. | Dynamic phrase expansion of language input |
US10521466B2 (en) | 2016-06-11 | 2019-12-31 | Apple Inc. | Data driven natural language event detection and classification |
US10552013B2 (en) | 2014-12-02 | 2020-02-04 | Apple Inc. | Data detection |
US10553209B2 (en) | 2010-01-18 | 2020-02-04 | Apple Inc. | Systems and methods for hands-free notification summaries |
US10568032B2 (en) | 2007-04-03 | 2020-02-18 | Apple Inc. | Method and system for operating a multi-function portable electronic device using voice-activation |
US10567477B2 (en) | 2015-03-08 | 2020-02-18 | Apple Inc. | Virtual assistant continuity |
US10592095B2 (en) | 2014-05-23 | 2020-03-17 | Apple Inc. | Instantaneous speaking of content on touch devices |
US10593346B2 (en) | 2016-12-22 | 2020-03-17 | Apple Inc. | Rank-reduced token representation for automatic speech recognition |
US10659851B2 (en) | 2014-06-30 | 2020-05-19 | Apple Inc. | Real-time digital assistant knowledge updates |
US10671428B2 (en) | 2015-09-08 | 2020-06-02 | Apple Inc. | Distributed personal assistant |
US10679605B2 (en) | 2010-01-18 | 2020-06-09 | Apple Inc. | Hands-free list-reading by intelligent automated assistant |
US10691473B2 (en) | 2015-11-06 | 2020-06-23 | Apple Inc. | Intelligent automated assistant in a messaging environment |
US10705794B2 (en) | 2010-01-18 | 2020-07-07 | Apple Inc. | Automatically adapting user interfaces for hands-free interaction |
US10706373B2 (en) | 2011-06-03 | 2020-07-07 | Apple Inc. | Performing actions associated with task items that represent tasks to perform |
US10733993B2 (en) | 2016-06-10 | 2020-08-04 | Apple Inc. | Intelligent digital assistant in a multi-tasking environment |
US10747498B2 (en) | 2015-09-08 | 2020-08-18 | Apple Inc. | Zero latency digital assistant |
US10755703B2 (en) | 2017-05-11 | 2020-08-25 | Apple Inc. | Offline personal assistant |
US10762293B2 (en) | 2010-12-22 | 2020-09-01 | Apple Inc. | Using parts-of-speech tagging and named entity recognition for spelling correction |
US10791176B2 (en) | 2017-05-12 | 2020-09-29 | Apple Inc. | Synchronization and task delegation of a digital assistant |
US10789041B2 (en) | 2014-09-12 | 2020-09-29 | Apple Inc. | Dynamic thresholds for always listening speech trigger |
US10791216B2 (en) | 2013-08-06 | 2020-09-29 | Apple Inc. | Auto-activating smart responses based on activities from remote devices |
US10810274B2 (en) | 2017-05-15 | 2020-10-20 | Apple Inc. | Optimizing dialogue policy decisions for digital assistants using implicit feedback |
US11010550B2 (en) | 2015-09-29 | 2021-05-18 | Apple Inc. | Unified language modeling framework for word prediction, auto-completion and auto-correction |
US11025565B2 (en) | 2015-06-07 | 2021-06-01 | Apple Inc. | Personalized prediction of responses for instant messaging |
US11217255B2 (en) | 2017-05-16 | 2022-01-04 | Apple Inc. | Far-field extension for digital assistant services |
US11587559B2 (en) | 2015-09-30 | 2023-02-21 | Apple Inc. | Intelligent device identification |
US11894865B2 (en) * | 2013-11-07 | 2024-02-06 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods and devices for vector segmentation for coding |
Families Citing this family (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8838452B2 (en) * | 2004-06-09 | 2014-09-16 | Canon Kabushiki Kaisha | Effective audio segmentation and classification |
CA2536976A1 (en) * | 2006-02-20 | 2007-08-20 | Diaphonics, Inc. | Method and apparatus for detecting speaker change in a voice transaction |
JP4665836B2 (en) * | 2006-05-31 | 2011-04-06 | 日本ビクター株式会社 | Music classification device, music classification method, and music classification program |
US7680657B2 (en) * | 2006-08-15 | 2010-03-16 | Microsoft Corporation | Auto segmentation based partitioning and clustering approach to robust endpointing |
KR200449646Y1 (en) * | 2007-12-17 | 2010-07-28 | 비엔지 주식회사 | A prefabricated pillar for structure |
CN102044244B (en) * | 2009-10-15 | 2011-11-16 | 华为技术有限公司 | Signal classifying method and device |
GB2489489B (en) * | 2011-03-30 | 2013-08-21 | Toshiba Res Europ Ltd | A speech processing system and method |
US9535450B2 (en) | 2011-07-17 | 2017-01-03 | International Business Machines Corporation | Synchronization of data streams with associated metadata streams using smallest sum of absolute differences between time indices of data events and metadata events |
US8798996B2 (en) * | 2012-03-05 | 2014-08-05 | Coupons.Com Incorporated | Splitting term lists recognized from speech |
CN105632503B (en) * | 2014-10-28 | 2019-09-03 | 南宁富桂精密工业有限公司 | Information concealing method and system |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6567775B1 (en) * | 2000-04-26 | 2003-05-20 | International Business Machines Corporation | Fusion of audio and video based speaker identification for multimedia information access |
US6931373B1 (en) * | 2001-02-13 | 2005-08-16 | Hughes Electronics Corporation | Prototype waveform phase modeling for a frequency domain interpolative speech codec system |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS5853352B2 (en) | 1979-10-03 | 1983-11-29 | 日本電信電話株式会社 | speech synthesizer |
US5606643A (en) | 1994-04-12 | 1997-02-25 | Xerox Corporation | Real-time audio recording system for automatic speaker indexing |
US5655058A (en) * | 1994-04-12 | 1997-08-05 | Xerox Corporation | Segmentation of audio data for indexing of conversational speech for real-time or postprocessing applications |
KR19990035846U (en) * | 1998-02-10 | 1999-09-15 | 구자홍 | Position and posture adjuster of audio / control head for videocassette recorder |
US6185527B1 (en) | 1999-01-19 | 2001-02-06 | International Business Machines Corporation | System and method for automatic audio content analysis for word spotting, indexing, classification and retrieval |
US7120575B2 (en) * | 2000-04-08 | 2006-10-10 | International Business Machines Corporation | Method and system for the automatic segmentation of an audio stream into semantic or syntactic units |
-
2002
- 2002-02-21 KR KR1020020009209A patent/KR20030070179A/en not_active Application Discontinuation
-
2003
- 2003-02-21 US US10/370,065 patent/US7346516B2/en not_active Expired - Fee Related
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6567775B1 (en) * | 2000-04-26 | 2003-05-20 | International Business Machines Corporation | Fusion of audio and video based speaker identification for multimedia information access |
US6931373B1 (en) * | 2001-02-13 | 2005-08-16 | Hughes Electronics Corporation | Prototype waveform phase modeling for a frequency domain interpolative speech codec system |
Cited By (169)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9646614B2 (en) | 2000-03-16 | 2017-05-09 | Apple Inc. | Fast, language-independent method for user authentication by voice |
US20050126369A1 (en) * | 2003-12-12 | 2005-06-16 | Nokia Corporation | Automatic extraction of musical portions of an audio stream |
WO2005060337A3 (en) * | 2003-12-12 | 2006-09-21 | Nokia Corp | Automatic extraction of musical portions of an audio stream |
US7179980B2 (en) * | 2003-12-12 | 2007-02-20 | Nokia Corporation | Automatic extraction of musical portions of an audio stream |
KR100840745B1 (en) * | 2003-12-12 | 2008-06-23 | 노키아 코포레이션 | Method and Apparatus for selectively recording music portions of audio stream |
US10318871B2 (en) | 2005-09-08 | 2019-06-11 | Apple Inc. | Method and apparatus for building an intelligent automated assistant |
US9117447B2 (en) | 2006-09-08 | 2015-08-25 | Apple Inc. | Using event alert text as input to an automated assistant |
US8930191B2 (en) | 2006-09-08 | 2015-01-06 | Apple Inc. | Paraphrasing of user requests and results by automated digital assistant |
US8942986B2 (en) | 2006-09-08 | 2015-01-27 | Apple Inc. | Determining user intent based on ontologies of domains |
US20080236368A1 (en) * | 2007-03-26 | 2008-10-02 | Sanyo Electric Co., Ltd. | Recording or playback apparatus and musical piece detecting apparatus |
US7745714B2 (en) * | 2007-03-26 | 2010-06-29 | Sanyo Electric Co., Ltd. | Recording or playback apparatus and musical piece detecting apparatus |
US10568032B2 (en) | 2007-04-03 | 2020-02-18 | Apple Inc. | Method and system for operating a multi-function portable electronic device using voice-activation |
US9330720B2 (en) | 2008-01-03 | 2016-05-03 | Apple Inc. | Methods and apparatus for altering audio output signals |
US10381016B2 (en) | 2008-01-03 | 2019-08-13 | Apple Inc. | Methods and apparatus for altering audio output signals |
US9865248B2 (en) | 2008-04-05 | 2018-01-09 | Apple Inc. | Intelligent text-to-speech conversion |
US9626955B2 (en) | 2008-04-05 | 2017-04-18 | Apple Inc. | Intelligent text-to-speech conversion |
US10108612B2 (en) | 2008-07-31 | 2018-10-23 | Apple Inc. | Mobile device having human language translation capability with positional feedback |
US9535906B2 (en) | 2008-07-31 | 2017-01-03 | Apple Inc. | Mobile device having human language translation capability with positional feedback |
US9959870B2 (en) | 2008-12-11 | 2018-05-01 | Apple Inc. | Speech recognition involving a mobile device |
US9858925B2 (en) | 2009-06-05 | 2018-01-02 | Apple Inc. | Using context information to facilitate processing of commands in a virtual assistant |
US11080012B2 (en) | 2009-06-05 | 2021-08-03 | Apple Inc. | Interface for a virtual digital assistant |
US10475446B2 (en) | 2009-06-05 | 2019-11-12 | Apple Inc. | Using context information to facilitate processing of commands in a virtual assistant |
US10795541B2 (en) | 2009-06-05 | 2020-10-06 | Apple Inc. | Intelligent organization of tasks items |
US10283110B2 (en) | 2009-07-02 | 2019-05-07 | Apple Inc. | Methods and apparatuses for automatic speech recognition |
US8560309B2 (en) * | 2009-12-29 | 2013-10-15 | Apple Inc. | Remote conferencing center |
US20110161074A1 (en) * | 2009-12-29 | 2011-06-30 | Apple Inc. | Remote conferencing center |
US8903716B2 (en) | 2010-01-18 | 2014-12-02 | Apple Inc. | Personalized vocabulary for digital assistant |
US10276170B2 (en) | 2010-01-18 | 2019-04-30 | Apple Inc. | Intelligent automated assistant |
US10706841B2 (en) | 2010-01-18 | 2020-07-07 | Apple Inc. | Task flow identification based on user intent |
US12087308B2 (en) | 2010-01-18 | 2024-09-10 | Apple Inc. | Intelligent automated assistant |
US9548050B2 (en) | 2010-01-18 | 2017-01-17 | Apple Inc. | Intelligent automated assistant |
US10553209B2 (en) | 2010-01-18 | 2020-02-04 | Apple Inc. | Systems and methods for hands-free notification summaries |
US10496753B2 (en) | 2010-01-18 | 2019-12-03 | Apple Inc. | Automatically adapting user interfaces for hands-free interaction |
US10705794B2 (en) | 2010-01-18 | 2020-07-07 | Apple Inc. | Automatically adapting user interfaces for hands-free interaction |
US8892446B2 (en) | 2010-01-18 | 2014-11-18 | Apple Inc. | Service orchestration for intelligent automated assistant |
US11423886B2 (en) | 2010-01-18 | 2022-08-23 | Apple Inc. | Task flow identification based on user intent |
US10679605B2 (en) | 2010-01-18 | 2020-06-09 | Apple Inc. | Hands-free list-reading by intelligent automated assistant |
US9318108B2 (en) | 2010-01-18 | 2016-04-19 | Apple Inc. | Intelligent automated assistant |
US9424862B2 (en) | 2010-01-25 | 2016-08-23 | Newvaluexchange Ltd | Apparatuses, methods and systems for a digital conversation management platform |
US8977584B2 (en) | 2010-01-25 | 2015-03-10 | Newvaluexchange Global Ai Llp | Apparatuses, methods and systems for a digital conversation management platform |
US9424861B2 (en) | 2010-01-25 | 2016-08-23 | Newvaluexchange Ltd | Apparatuses, methods and systems for a digital conversation management platform |
US9431028B2 (en) | 2010-01-25 | 2016-08-30 | Newvaluexchange Ltd | Apparatuses, methods and systems for a digital conversation management platform |
US9633660B2 (en) | 2010-02-25 | 2017-04-25 | Apple Inc. | User profiling for voice input processing |
US10049675B2 (en) | 2010-02-25 | 2018-08-14 | Apple Inc. | User profiling for voice input processing |
US20140207456A1 (en) * | 2010-09-23 | 2014-07-24 | Waveform Communications, Llc | Waveform analysis of speech |
US10762293B2 (en) | 2010-12-22 | 2020-09-01 | Apple Inc. | Using parts-of-speech tagging and named entity recognition for spelling correction |
US10102359B2 (en) | 2011-03-21 | 2018-10-16 | Apple Inc. | Device access using voice authentication |
US9262612B2 (en) | 2011-03-21 | 2016-02-16 | Apple Inc. | Device access using voice authentication |
US10241644B2 (en) | 2011-06-03 | 2019-03-26 | Apple Inc. | Actionable reminder entries |
US10057736B2 (en) | 2011-06-03 | 2018-08-21 | Apple Inc. | Active transport based notifications |
US10706373B2 (en) | 2011-06-03 | 2020-07-07 | Apple Inc. | Performing actions associated with task items that represent tasks to perform |
US11120372B2 (en) | 2011-06-03 | 2021-09-14 | Apple Inc. | Performing actions associated with task items that represent tasks to perform |
US9798393B2 (en) | 2011-08-29 | 2017-10-24 | Apple Inc. | Text correction processing |
US10241752B2 (en) | 2011-09-30 | 2019-03-26 | Apple Inc. | Interface for a virtual digital assistant |
US10284951B2 (en) | 2011-11-22 | 2019-05-07 | Apple Inc. | Orientation-based audio |
US8879761B2 (en) | 2011-11-22 | 2014-11-04 | Apple Inc. | Orientation-based audio |
US10134385B2 (en) | 2012-03-02 | 2018-11-20 | Apple Inc. | Systems and methods for name pronunciation |
US9483461B2 (en) | 2012-03-06 | 2016-11-01 | Apple Inc. | Handling speech synthesis of content for multiple languages |
US9953088B2 (en) | 2012-05-14 | 2018-04-24 | Apple Inc. | Crowd sourcing information to fulfill user requests |
US10079014B2 (en) | 2012-06-08 | 2018-09-18 | Apple Inc. | Name recognition system |
US9495129B2 (en) | 2012-06-29 | 2016-11-15 | Apple Inc. | Device, method, and user interface for voice-activated navigation and browsing of a document |
US9576574B2 (en) | 2012-09-10 | 2017-02-21 | Apple Inc. | Context-sensitive handling of interruptions by intelligent digital assistant |
US9971774B2 (en) | 2012-09-19 | 2018-05-15 | Apple Inc. | Voice-based media searching |
US10199051B2 (en) | 2013-02-07 | 2019-02-05 | Apple Inc. | Voice trigger for a digital assistant |
US10978090B2 (en) | 2013-02-07 | 2021-04-13 | Apple Inc. | Voice trigger for a digital assistant |
US9368114B2 (en) | 2013-03-14 | 2016-06-14 | Apple Inc. | Context-sensitive handling of interruptions |
US9922642B2 (en) | 2013-03-15 | 2018-03-20 | Apple Inc. | Training an at least partial voice command system |
US9697822B1 (en) | 2013-03-15 | 2017-07-04 | Apple Inc. | System and method for updating an adaptive speech recognition model |
US9582608B2 (en) | 2013-06-07 | 2017-02-28 | Apple Inc. | Unified ranking with entropy-weighted information for phrase-based semantic auto-completion |
US9633674B2 (en) | 2013-06-07 | 2017-04-25 | Apple Inc. | System and method for detecting errors in interactions with a voice-based digital assistant |
US9966060B2 (en) | 2013-06-07 | 2018-05-08 | Apple Inc. | System and method for user-specified pronunciation of words for speech synthesis and recognition |
US9620104B2 (en) | 2013-06-07 | 2017-04-11 | Apple Inc. | System and method for user-specified pronunciation of words for speech synthesis and recognition |
US10657961B2 (en) | 2013-06-08 | 2020-05-19 | Apple Inc. | Interpreting and acting upon commands that involve sharing information with remote devices |
US9966068B2 (en) | 2013-06-08 | 2018-05-08 | Apple Inc. | Interpreting and acting upon commands that involve sharing information with remote devices |
US10176167B2 (en) | 2013-06-09 | 2019-01-08 | Apple Inc. | System and method for inferring user intent from speech inputs |
US10185542B2 (en) | 2013-06-09 | 2019-01-22 | Apple Inc. | Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant |
US9300784B2 (en) | 2013-06-13 | 2016-03-29 | Apple Inc. | System and method for emergency calls initiated by voice command |
US10791216B2 (en) | 2013-08-06 | 2020-09-29 | Apple Inc. | Auto-activating smart responses based on activities from remote devices |
US11894865B2 (en) * | 2013-11-07 | 2024-02-06 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods and devices for vector segmentation for coding |
US9620105B2 (en) | 2014-05-15 | 2017-04-11 | Apple Inc. | Analyzing audio input for efficient speech and music recognition |
US10592095B2 (en) | 2014-05-23 | 2020-03-17 | Apple Inc. | Instantaneous speaking of content on touch devices |
US9502031B2 (en) | 2014-05-27 | 2016-11-22 | Apple Inc. | Method for supporting dynamic grammars in WFST-based ASR |
US9633004B2 (en) | 2014-05-30 | 2017-04-25 | Apple Inc. | Better resolution when referencing to concepts |
US9715875B2 (en) | 2014-05-30 | 2017-07-25 | Apple Inc. | Reducing the need for manual start/end-pointing and trigger phrases |
US10289433B2 (en) | 2014-05-30 | 2019-05-14 | Apple Inc. | Domain specific language for encoding assistant dialog |
US11257504B2 (en) | 2014-05-30 | 2022-02-22 | Apple Inc. | Intelligent assistant for home automation |
US10083690B2 (en) | 2014-05-30 | 2018-09-25 | Apple Inc. | Better resolution when referencing to concepts |
US11133008B2 (en) | 2014-05-30 | 2021-09-28 | Apple Inc. | Reducing the need for manual start/end-pointing and trigger phrases |
US10078631B2 (en) | 2014-05-30 | 2018-09-18 | Apple Inc. | Entropy-guided text prediction using combined word and character n-gram language models |
US10170123B2 (en) | 2014-05-30 | 2019-01-01 | Apple Inc. | Intelligent assistant for home automation |
US10169329B2 (en) | 2014-05-30 | 2019-01-01 | Apple Inc. | Exemplar-based natural language processing |
US9430463B2 (en) | 2014-05-30 | 2016-08-30 | Apple Inc. | Exemplar-based natural language processing |
US9842101B2 (en) | 2014-05-30 | 2017-12-12 | Apple Inc. | Predictive conversion of language input |
US9734193B2 (en) | 2014-05-30 | 2017-08-15 | Apple Inc. | Determining domain salience ranking from ambiguous words in natural speech |
US9966065B2 (en) | 2014-05-30 | 2018-05-08 | Apple Inc. | Multi-command single utterance input method |
US9760559B2 (en) | 2014-05-30 | 2017-09-12 | Apple Inc. | Predictive text input |
US10497365B2 (en) | 2014-05-30 | 2019-12-03 | Apple Inc. | Multi-command single utterance input method |
US9785630B2 (en) | 2014-05-30 | 2017-10-10 | Apple Inc. | Text prediction using combined word N-gram and unigram language models |
US9338493B2 (en) | 2014-06-30 | 2016-05-10 | Apple Inc. | Intelligent automated assistant for TV user interactions |
US9668024B2 (en) | 2014-06-30 | 2017-05-30 | Apple Inc. | Intelligent automated assistant for TV user interactions |
US10904611B2 (en) | 2014-06-30 | 2021-01-26 | Apple Inc. | Intelligent automated assistant for TV user interactions |
US10659851B2 (en) | 2014-06-30 | 2020-05-19 | Apple Inc. | Real-time digital assistant knowledge updates |
US10446141B2 (en) | 2014-08-28 | 2019-10-15 | Apple Inc. | Automatic speech recognition based on user feedback |
US9818400B2 (en) | 2014-09-11 | 2017-11-14 | Apple Inc. | Method and apparatus for discovering trending terms in speech requests |
US10431204B2 (en) | 2014-09-11 | 2019-10-01 | Apple Inc. | Method and apparatus for discovering trending terms in speech requests |
US10789041B2 (en) | 2014-09-12 | 2020-09-29 | Apple Inc. | Dynamic thresholds for always listening speech trigger |
US10074360B2 (en) | 2014-09-30 | 2018-09-11 | Apple Inc. | Providing an indication of the suitability of speech recognition |
US9668121B2 (en) | 2014-09-30 | 2017-05-30 | Apple Inc. | Social reminders |
US10127911B2 (en) | 2014-09-30 | 2018-11-13 | Apple Inc. | Speaker identification and unsupervised speaker adaptation techniques |
US9886432B2 (en) | 2014-09-30 | 2018-02-06 | Apple Inc. | Parsimonious handling of word inflection via categorical stem + suffix N-gram language models |
US9986419B2 (en) | 2014-09-30 | 2018-05-29 | Apple Inc. | Social reminders |
US9646609B2 (en) | 2014-09-30 | 2017-05-09 | Apple Inc. | Caching apparatus for serving phonetic pronunciations |
US11556230B2 (en) | 2014-12-02 | 2023-01-17 | Apple Inc. | Data detection |
US10552013B2 (en) | 2014-12-02 | 2020-02-04 | Apple Inc. | Data detection |
US9711141B2 (en) | 2014-12-09 | 2017-07-18 | Apple Inc. | Disambiguating heteronyms in speech synthesis |
US9865280B2 (en) | 2015-03-06 | 2018-01-09 | Apple Inc. | Structured dictation using intelligent automated assistants |
US9721566B2 (en) | 2015-03-08 | 2017-08-01 | Apple Inc. | Competing devices responding to voice triggers |
US10311871B2 (en) | 2015-03-08 | 2019-06-04 | Apple Inc. | Competing devices responding to voice triggers |
US11087759B2 (en) | 2015-03-08 | 2021-08-10 | Apple Inc. | Virtual assistant activation |
US9886953B2 (en) | 2015-03-08 | 2018-02-06 | Apple Inc. | Virtual assistant activation |
US10567477B2 (en) | 2015-03-08 | 2020-02-18 | Apple Inc. | Virtual assistant continuity |
US9899019B2 (en) | 2015-03-18 | 2018-02-20 | Apple Inc. | Systems and methods for structured stem and suffix language models |
US9842105B2 (en) | 2015-04-16 | 2017-12-12 | Apple Inc. | Parsimonious continuous-space phrase representations for natural language processing |
US10083688B2 (en) | 2015-05-27 | 2018-09-25 | Apple Inc. | Device voice control for selecting a displayed affordance |
US10127220B2 (en) | 2015-06-04 | 2018-11-13 | Apple Inc. | Language identification from short strings |
US10101822B2 (en) | 2015-06-05 | 2018-10-16 | Apple Inc. | Language input correction |
US10356243B2 (en) | 2015-06-05 | 2019-07-16 | Apple Inc. | Virtual assistant aided communication with 3rd party service in a communication session |
US10186254B2 (en) | 2015-06-07 | 2019-01-22 | Apple Inc. | Context-based endpoint detection |
US11025565B2 (en) | 2015-06-07 | 2021-06-01 | Apple Inc. | Personalized prediction of responses for instant messaging |
US10255907B2 (en) | 2015-06-07 | 2019-04-09 | Apple Inc. | Automatic accent detection using acoustic models |
US11500672B2 (en) | 2015-09-08 | 2022-11-15 | Apple Inc. | Distributed personal assistant |
US10671428B2 (en) | 2015-09-08 | 2020-06-02 | Apple Inc. | Distributed personal assistant |
US10747498B2 (en) | 2015-09-08 | 2020-08-18 | Apple Inc. | Zero latency digital assistant |
US9697820B2 (en) | 2015-09-24 | 2017-07-04 | Apple Inc. | Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks |
US11010550B2 (en) | 2015-09-29 | 2021-05-18 | Apple Inc. | Unified language modeling framework for word prediction, auto-completion and auto-correction |
US10366158B2 (en) | 2015-09-29 | 2019-07-30 | Apple Inc. | Efficient word encoding for recurrent neural network language models |
US11587559B2 (en) | 2015-09-30 | 2023-02-21 | Apple Inc. | Intelligent device identification |
US10691473B2 (en) | 2015-11-06 | 2020-06-23 | Apple Inc. | Intelligent automated assistant in a messaging environment |
US11526368B2 (en) | 2015-11-06 | 2022-12-13 | Apple Inc. | Intelligent automated assistant in a messaging environment |
US10049668B2 (en) | 2015-12-02 | 2018-08-14 | Apple Inc. | Applying neural network language models to weighted finite state transducers for automatic speech recognition |
US10223066B2 (en) | 2015-12-23 | 2019-03-05 | Apple Inc. | Proactive assistance based on dialog communication between devices |
US10446143B2 (en) | 2016-03-14 | 2019-10-15 | Apple Inc. | Identification of voice inputs providing credentials |
US9934775B2 (en) | 2016-05-26 | 2018-04-03 | Apple Inc. | Unit-selection text-to-speech synthesis based on predicted concatenation parameters |
US9972304B2 (en) | 2016-06-03 | 2018-05-15 | Apple Inc. | Privacy preserving distributed evaluation framework for embedded personalized systems |
US10249300B2 (en) | 2016-06-06 | 2019-04-02 | Apple Inc. | Intelligent list reading |
US11069347B2 (en) | 2016-06-08 | 2021-07-20 | Apple Inc. | Intelligent automated assistant for media exploration |
US10049663B2 (en) | 2016-06-08 | 2018-08-14 | Apple, Inc. | Intelligent automated assistant for media exploration |
US10354011B2 (en) | 2016-06-09 | 2019-07-16 | Apple Inc. | Intelligent automated assistant in a home environment |
US11037565B2 (en) | 2016-06-10 | 2021-06-15 | Apple Inc. | Intelligent digital assistant in a multi-tasking environment |
US10733993B2 (en) | 2016-06-10 | 2020-08-04 | Apple Inc. | Intelligent digital assistant in a multi-tasking environment |
US10509862B2 (en) | 2016-06-10 | 2019-12-17 | Apple Inc. | Dynamic phrase expansion of language input |
US10067938B2 (en) | 2016-06-10 | 2018-09-04 | Apple Inc. | Multilingual word prediction |
US10490187B2 (en) | 2016-06-10 | 2019-11-26 | Apple Inc. | Digital assistant providing automated status report |
US10192552B2 (en) | 2016-06-10 | 2019-01-29 | Apple Inc. | Digital assistant providing whispered speech |
US10297253B2 (en) | 2016-06-11 | 2019-05-21 | Apple Inc. | Application integration with a digital assistant |
US10269345B2 (en) | 2016-06-11 | 2019-04-23 | Apple Inc. | Intelligent task discovery |
US10089072B2 (en) | 2016-06-11 | 2018-10-02 | Apple Inc. | Intelligent device arbitration and control |
US10521466B2 (en) | 2016-06-11 | 2019-12-31 | Apple Inc. | Data driven natural language event detection and classification |
US11152002B2 (en) | 2016-06-11 | 2021-10-19 | Apple Inc. | Application integration with a digital assistant |
US10553215B2 (en) | 2016-09-23 | 2020-02-04 | Apple Inc. | Intelligent automated assistant |
US10043516B2 (en) | 2016-09-23 | 2018-08-07 | Apple Inc. | Intelligent automated assistant |
US10593346B2 (en) | 2016-12-22 | 2020-03-17 | Apple Inc. | Rank-reduced token representation for automatic speech recognition |
US10755703B2 (en) | 2017-05-11 | 2020-08-25 | Apple Inc. | Offline personal assistant |
US11405466B2 (en) | 2017-05-12 | 2022-08-02 | Apple Inc. | Synchronization and task delegation of a digital assistant |
US10410637B2 (en) | 2017-05-12 | 2019-09-10 | Apple Inc. | User-specific acoustic models |
US10791176B2 (en) | 2017-05-12 | 2020-09-29 | Apple Inc. | Synchronization and task delegation of a digital assistant |
US10482874B2 (en) | 2017-05-15 | 2019-11-19 | Apple Inc. | Hierarchical belief states for digital assistants |
US10810274B2 (en) | 2017-05-15 | 2020-10-20 | Apple Inc. | Optimizing dialogue policy decisions for digital assistants using implicit feedback |
US11217255B2 (en) | 2017-05-16 | 2022-01-04 | Apple Inc. | Far-field extension for digital assistant services |
Also Published As
Publication number | Publication date |
---|---|
KR20030070179A (en) | 2003-08-29 |
US7346516B2 (en) | 2008-03-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7346516B2 (en) | Method of segmenting an audio stream | |
Chou et al. | Robust singing detection in speech/music discriminator design | |
Lu et al. | Content analysis for audio classification and segmentation | |
Lu et al. | A robust audio classification and segmentation method | |
US6785645B2 (en) | Real-time speech and music classifier | |
Lu et al. | Content-based audio classification and segmentation by using support vector machines | |
US9135929B2 (en) | Efficient content classification and loudness estimation | |
US8793127B2 (en) | Method and apparatus for automatically determining speaker characteristics for speech-directed advertising or other enhancement of speech-controlled devices or services | |
Kos et al. | Acoustic classification and segmentation using modified spectral roll-off and variance-based features | |
US6901362B1 (en) | Audio segmentation and classification | |
US20050228649A1 (en) | Method and apparatus for classifying sound signals | |
US7521622B1 (en) | Noise-resistant detection of harmonic segments of audio signals | |
Butko et al. | Audio segmentation of broadcast news in the Albayzin-2010 evaluation: overview, results, and discussion | |
US20050192795A1 (en) | Identification of the presence of speech in digital audio data | |
Zelenák et al. | Speaker diarization of broadcast news in Albayzin 2010 evaluation campaign | |
Zewoudie et al. | The use of long-term features for GMM-and i-vector-based speaker diarization systems | |
Kwon et al. | Speaker change detection using a new weighted distance measure. | |
Bugatti et al. | Audio classification in speech and music: a comparison between a statistical and a neural approach | |
Delacourt et al. | Speaker-based segmentation for audio data indexing | |
US7680654B2 (en) | Apparatus and method for segmentation of audio data into meta patterns | |
Pinquier et al. | Audio indexing: primary components retrieval: robust classification in audio documents | |
Huijbregts et al. | Filtering the unknown: Speech activity detection in heterogeneous video collections | |
Velayatipour et al. | A review on speech-music discrimination methods | |
Zhan et al. | Dragon Systems’ 1998 broadcast news transcription system for Mandarin | |
Kacprzak et al. | Speech/music discrimination for analysis of radio stations |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: LG ELECTRONICS INC., KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SALL, MIKHAEL A.;GRAMNITSKIY, SERGEI N.;MAIBORODA, ALEXANDR L.;AND OTHERS;REEL/FRAME:014058/0840 Effective date: 20030221 |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Free format text: PAYER NUMBER DE-ASSIGNED (ORIGINAL EVENT CODE: RMPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20200318 |