WO2011071928A2 - Détecteur de dialogue et correction - Google Patents
Détecteur de dialogue et correction Download PDFInfo
- Publication number
- WO2011071928A2 WO2011071928A2 PCT/US2010/059296 US2010059296W WO2011071928A2 WO 2011071928 A2 WO2011071928 A2 WO 2011071928A2 US 2010059296 W US2010059296 W US 2010059296W WO 2011071928 A2 WO2011071928 A2 WO 2011071928A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- audio
- dialogue
- tracking
- dialogue tracking
- phonemes
- Prior art date
Links
- 238000012937 correction Methods 0.000 title abstract description 6
- 230000005236 sound signal Effects 0.000 claims abstract description 59
- 238000000034 method Methods 0.000 claims abstract description 41
- 230000000694 effects Effects 0.000 claims description 49
- 238000001914 filtration Methods 0.000 claims description 5
- 238000001228 spectrum Methods 0.000 claims 2
- 238000007689 inspection Methods 0.000 abstract description 6
- 108091006146 Channels Proteins 0.000 description 167
- 238000001514 detection method Methods 0.000 description 18
- 230000004044 response Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 8
- 238000012544 monitoring process Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 1
- 239000006227 byproduct Substances 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000035807 sensation Effects 0.000 description 1
- 238000011895 specific detection Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/233—Processing of audio elementary streams
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/24—Speech recognition using non-acoustical features
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/005—Correction of errors induced by the transmission channel, if related to the coding algorithm
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
- G10L25/57—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for processing of video signals
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
- G10L2015/025—Phonemes, fenemes or fenones being the recognition units
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/78—Detection of presence or absence of voice signals
Definitions
- a typical system could include a video channel and left and right sound channels such as in a stereo television system.
- left and right sound channels such as in a stereo television system.
- the well known intent of using left and right sound channels is to provide a spatially located sound to the viewer whereby sounds created by images at a given location on the television screen are perceived by the viewer as coming from that location.
- the corresponding images and sounds are known as mutual events or MUEVs.
- MUEVs mutual events
- the audio and image MUEVs as perceived by the viewer do not properly correspond they are annoying as the sound is perceived to come from a different location than the image making the sound.
- dialogue e.g. speech of a person in a one way or two way conversation with another
- the speaker is seen in a different location than the sound comes from.
- MUEVs are those events in the video and sound which have a high probability of occurring together, for example the instant change of direction of a thrown baseball and the crack of the bat hitting the ball.
- Other MUEVs include the shape and/or movement of a person's lips and the sound being created.
- the video lip shapes are referred to as visemes or the visual MUEV and the sounds as phonemes or the sound MUEV.
- MUEVs however are not just visemes and phonemes but encompass simultaneously occurring events which have a probability of being related, such as the above baseball direction and bat crack example.
- both audio only for example such as radio
- audio video for example such as television
- dialogue in a particular channel or channels Because sound signals in modern audio only and audio video acquisition and production systems are frequently recorded and carried by multiple sound channels, there is a possibility of the dialogue being misplaced, that is of the dialogue being carried by the wrong audio channel. It is also possible for dialogue to be lost entirely, for example when sound is acquired via a sound effects channel which is subsequently discarded.
- a system sound channel conveys the proper sound signal (e.g. dialogue in the proper channel(s) and/or leading the viewer to perceive sounds as properly corresponding to the image location)
- the sound channel or signal is said to (properly) track and if it does not convey the proper sound signal the channel or signal is said to mistrack.
- the left and right sound channel signals are reversed, that is the left channel carries the right sound signal and vice versa (sometimes called swapping), the sound signals mistrack.
- the dialogue sound signal is missing from the dialogue sound channel(s), the sound signal mistracks.
- the sound of the performers in the television program is conveyed via left and right sound dialogue channels whereas sound effects such as music and other non speech sounds are conveyed by left and right sound effects channels.
- Another example is 5.1 channel sound, sometimes referred to as 3-2 stereo, with a center dialogue channel, front left and right dialogue channels, rear left and right effects channels, and a low frequency effects channel.
- a multiple channel sound system is the Japan Broadcasting Corporation (NHK) experimental Super Hi- Vision television having 22.2 sound channels. These channels are grouped relative to the viewer as 9 above the ear, 10 ear level, 3 below the ear and 2 low frequency effects channels. The various sound channels surround the viewer to provide a highly realistic audio sensation where the sound can be perceived as coming from anywhere within about 300 degrees vertically and 360 degrees horizontally, depending on the location of the viewer relative to the sound transducers (e.g. speakers).
- NHK Japan Broadcasting Corporation
- the location of the sound does not match the location of the image, such as when a person on the left of the image frame is talking but the sound comes from the right sound transducer (speaker).
- the sound which the viewer perceives is intended to come from various directions around the viewer including from ear level, higher and lower directions to correspond to the images which are displayed to the viewer (or previously or about to be displayed to the viewer).
- a sound signal is placed in the wrong channel various annoying effects can occur, such as a speaking person located to the viewer's lower right being heard behind, above, to the left or in some other direction different from where the viewer sees the image of the person speaking.
- Basse does mention that system operators desire to monitor their systems to ensure quality audio and video reaches the viewers and relates prior systems such as cable TV systems where employees monitored the quality. Basse also points out that the cost of hiring employees to monitor every channel in a system can be expensive and notes several problems with utilizing employees to monitor modern systems consisting of as many as 800 TV channels.
- Figure 1 is a diagram of an embodiment of the invention utilized for detecting the loss of proper tracking in a single or multiple channel audio system.
- Figure 2 is a diagram of an embodiment of Channel Phoneme Logic 4 of Figure 1.
- Figure 3 is a diagram of a circuit to correct mistracking in a multiple channel audio system.
- Figure 4 is a partial diagram of an embodiment of the invention utilized for detecting the loss of proper tracking in a system having multiple channel audio and corresponding video.
- Figure 5 is a partial diagram of an embodiment of the invention utilized for detecting the loss of proper tracking in a system having multiple channel audio and corresponding video.
- Figure 6 is a diagram of a circuit to correct mistracking in a system having multiple channel audio.
- Figure 7 is a flow chart demonstrating one embodiment of the present invention.
- the inventive concepts disclosed herein provide an automated system and method for monitoring, reporting and correcting the above described single and multiple channel sound errors caused by one or more instances of a wrong or no particular type of sound signal (as distinguished from no sound at all) being conveyed in a sound channel.
- the embodiment of the present invention includes detection of tracking or mistracking of one or more particular type of sound signals in a single or multiple channel sound only or sound and image system.
- the description of various embodiment of the present invention also includes detection of tracking or mistracking of one or more sound signals in a system including multiple channel sound signals and a corresponding image signal.
- Corresponding sounds and images for example MUEVs, may be utilized to determine if a particular sound signal is carried by the proper sound channel(s) such that the sound MUEVs match the image MUEVs in viewer perceived spatial location. It may also be determined if particular types of sound MUEVs are being carried by the intended channel(s) in a system or otherwise correctly carried and presented to the viewer.
- phonemes corresponding to dialogue are detected in one or more sound signals conveyed by one or more channels and the appropriateness of the detected phonemes being present in the sound channel is determined.
- an action such as one or more of a report, an alarm or a correction is performed.
- the exemplary embodiment includes making the determination of the
- the exemplary embodiment of the invention includes the ability to automatically correct errors by switching one or more sound signal from one sound channel to another in response to the determination of the appropriateness of a sound signal being conveyed in a sound channel.
- Figure 1 shows a diagram of one exemplary embodiment of the present invention as utilized for detection of the loss of proper tracking in a single or multiple channel audio system.
- the audio system may be audio only, audio and video or otherwise as will be known from the present teachings.
- One or more audio signals of the M audio channel(s) 1 is coupled to a phoneme / activity detector 2 which operates to identify activity and the occurrence of phonemes in that audio signal.
- the detector 2 outputs phoneme signal(s) and channel activity signal(s) (or alternatively information corresponding to the detected phonemes and activity) for each of the M audio channels to channel phoneme logic 4 via N signals 3.
- Channel phoneme logic 4 operates to determine if the channel activity and/or phonemes are proper for the particular channel(s) and thus whether the corresponding input audio signal properly tracks. It will be appreciated that the absence of a loss of proper tracking is a confirmation of proper tracking.
- the current invention may detect audio characteristics other than the above mention phonemes and activity.
- the audio MUEV discussed below may be used as audio characteristics for the current invention, where the audio MUEV may include non- phoneme sounds such as applause which is commonly found in audio dialogue.
- the audio characteristics in this specification include phonemes, activity, and non-phoneme sound.
- the phoneme detection which is utilized in 2 may be that disclosed in U.S. Patent Application 20070153089.
- the mouth sound or audio MUEV detection disclosed in U.S. Patent 7,499,104 may be utilized to detect particular mouth sounds or MUEVs which are appropriate for a given audio channel.
- the output 3 indicates MUEVs which are appropriate for the expected type audio signal corresponding to the audio channel being monitored. For example, if the audio channel being monitored is a center dialogue channel, then detection of audio MUEVs such as phonemes corresponding to vowels A, E, and O is performed since these should be frequently found in the center dialogue channel. Other phonemes may be detected as well, for example the sounds M, P, B, N and S.
- Activity may be detected by the presence of a known number of one or more types of phonemes, or may be otherwise determined as is known in the prior art. For example an inspection of the audio signal may be performed to determine if there is significant energy in frequency bands corresponding to those normally carrying voice sounds may be utilized. Such a system is described for example in U.S. Patent 6,836,295 and in particular elements 21-23 of Figure 2 of the '295 disclosure. The output of elements 23 corresponding to the desired frequency bands will provide an indication of activity for that band.
- the phonemes As the phonemes (MUEVs) are detected in 2 they are output via 3 to the channel phoneme logic 4. Alternatively information of the detection of the phonemes may be coupled to 4, for example the number of phonemes detected may be reported every second along with a reporting of the presence of audio in the speech frequency range since no MUEVs are to be expected when there is silence or only audio in frequencies where speech is not commonly found.
- Channel phoneme logic 4 operates to analyze the presence of activity and phonemes detected by 2 and report the results via 5 to other circuitry or operations. If desired 4 can operate to set an alarm when an insufficient number of phonemes is detected within a given period of time. As an example, if no phonemes are detected by 2 within a 3 minute period when audio is present, an alarm can be set. In the exemplary embodiment it is desired that the operator can set parameters independently for reporting and setting alarms. The exemplary parameters are the number of phonemes and the time duration during which that number are determined. For example reporting parameters can be set to report if fewer than one hundred phonemes are detected within a three minute period and alarm parameters can be set to create an alarm if less than ten phonemes are detected within a five minute period.
- parameters may be set for individual phonemes, such as particular ones of vowels, or for groups of phonemes such as vowels and consonants.
- MUEVs other than phonemes may also be detected, for example applause which is commonly found in audio dialogue channels.
- a exemplary embodiment of 4 which operates in a somewhat different manner is described with respect to Figure 2.
- Figure 2 shows one exemplary embodiment of channel phoneme logic 4 when operated with a four channel sound system having left front 7 and right front 8 and left surround
- the circuit of Figure 2 operates to check that dialogue is present in the left and right front channels and not in the left and right surround channels.
- the output 12 is coupled to a filter 14 which operates to reduce or prevent false reporting and alarms which might otherwise happen, for example if there is silence or noise in the front channels or in a momentary presence of dialogue in the surround channels. It is preferred that filter 14 operate as a recursive filter requiring the presence of mistracking signals from 12 for a period of three minutes before reporting or for five minutes before setting an alarm at 13. This recursive filtering in effect provides a running average of the conditions.
- filter 14 operate as a recursive filter requiring the presence of mistracking signals from 12 for a period of three minutes before reporting or for five minutes before setting an alarm at 13. This recursive filtering in effect provides a running average of the conditions.
- One skilled in the art will know from the teachings herein to utilize different time periods as well as different operations for 14 to suit particular applications and desired performance tradeoffs.
- Filter 14 also operates to inspect the number of phonemes in 10 and 11 and to infer activity from those numbers. If the phoneme numbers are very low in 10 or approximately the same in both 10 or 1 1 it is likely that 12 may not accurately indicate tracking and reporting and alarms are to be inhibited until the number of phonemes present on 10 or 11 rises above a known amount and remains so for a known period of time. In one exemplary embodiment it is shown that the known amount be ten phonemes within a period of one minute (without recursive filtering).
- One skilled in the art will know from the teachings herein to utilize different time periods as well as different operations for 14 to suit particular applications and desired performance tradeoffs.
- the channel phoneme logic 4 of Figure 2 operates to detect a swapping of dialogue and surround channels and to report and set an alarm in the event of mistracking. In that event it is desirable to correct the mistracking by redirecting the audio signals into the proper audio channel.
- Figure 3 shows a circuit which performs correction of mistracking when dialogue and surround channels are swapped.
- the left surround 15, left front 16, right front 17 and right surround 18 signals are coupled to a four pole double throw switch 24 which is responsive to the alarm 23 from 4.
- the input signals 15-18 are output as the same signals left surround 19, left front 20, right front 21 and right surround 22.
- an alarm 23 is set and the switch 24 is caused to move to the other position thus returning the mistracked audio signals to their proper channels.
- FIG. 4 and 5 show an exemplary embodiment of the invention where image information is used in conjunction with sound information to ensure proper spatial relationships.
- the system illustrate a scenario of operating with a multiple channel audio input 1.
- a phoneme/activity detection 2, such as that of Figure 1 is utilized to provide phoneme and activity information 3 for each input channel.
- the left and right channels will be considered, although the previously described dialogue and surround channel operation example of Figure 2 and 3 will be understood to be incorporated as well. The present description however will be limited to the left and right channel operation.
- Figure 4 also shows a video input 25 which is coupled to a viseme detection 26 and a frame spatial location element 30.
- Viseme detection 26 may operate for example as described in U.S. Patent Application 20070153089.
- the mouth shape or image MUEV detection disclosed in U.S. Patent 7,499,104 may be utilized to detect particular mouth shapes or MUEVs which are appropriate for a given video channel.
- Viseme detection 26 outputs visemes via 27 which are coupled to a viseme location operation 28.
- Frame spatial location 30 operates to output frame left right information via 31.
- the frame left right signal indicates whether each viseme on 27 is located in the left or right side of the image frame.
- Viseme location operation 28 receives both the viseme and location information and in response thereto outputs left and right visemes via 29.
- the frame left right signal on 31 is a byproduct of the viseme detection operation 26 since when a viseme is found as disclosed in U.S. Patent Application 20070153089 it is known where in the image frame it is located.
- the mouth shapes are found by first locating the face in the frame, then locating the lips in the face and then determining the shape of the lips. It is merely required to convert the precise location of the lips within the frame to a left/right signal by comparing the horizontal address of the viseme to the center point address of the frame.
- the horizontal address of the viseme (e.g. the lips) will range from 0-1919. By comparing that viseme address to 960 to determine which is the larger, the output of that comparison will indicate left (less than 960) or right (960 or greater).
- the horizontal address of the viseme will be a range of addresses corresponding to the size of the viseme. It is desired to utilize the middle or average address for the comparison. For example if the viseme is 12 pixels wide the address might range from 30 to 42 and the middle address 36 would be compared to 960.
- Figure 5 receives the left and right channel phoneme and activity information via 3 from 2 and the left and right viseme information via 29 from 28.
- a left and right phoneme / activity / viseme logic operation 32 operates to inspect the location of the visemes on the image frame in comparison to the corresponding phoneme. For example, a left side visemes would be expected to have consistently corresponding phonemes present in the left front audio signal. If however the corresponding phonemes are present in one of the other three channels the audio mistracks.
- logic 32 illustrates to operate in response to each viseme to identify and spatially locate its corresponding phoneme. It inspects the corresponding audio signal (e.g. if the viseme is in the left of the frame the corresponding audio signal is the left front audio) and if the corresponding phoneme is found it then inspects the remaining channels to see if a corresponding phoneme is found in one or more of them. The outcome of the inspections is noted and the conditions noted. For example a corresponding phoneme is in the right channel and none of the others, or in the right channel and one or more of the others, or not in the right channel but in one or more of the others, or is not found in any of the channels. Inspections are performed for other visemes and the conditions are noted.
- the results of the inspections to find corresponding phonemes is also filtered to ensure that there is sufficient activity in the video and audio channels and also recursively filtered to provide a running average of the results. Reporting is provided and alarms may be set in response to the activities and finding of corresponding phonemes. In particular it is desired to set a left right alarm via 33 indicating that left and right channels have been swapped.
- Figure 6 shows a double pole double throw switch 34, similar to 24 of Figure 3.
- Switch 34 responds to the left right alarm 33 from 32 and operates to pass the input audio signals 15-18 to the output channels 19-22 if the alarm is not present or to swap left and right channels if the alarm is present.
- the invention may be utilized with a range of audio and video channels, from only one to many audio channels and with one or multiple video channels.
- the invention may be utilized with multiple video channel systems such as 3D and surround video.
- the invention may be used with only one or with multiple audio channels such as the NHK Super Hi-Vision system with 22.2 audio channels.
- the invention may be utilized with surround sound systems where sound may be perceived by the viewer as coming from multiple directions. Visemes are detected and their location relative to the viewer or some other reference(s) determined. The location to the viewer may include both locations of visible image and locations which are not visible such as the previously described scenario where an airplane is visually located behind the viewer and not visible.
- Phonemes which correspond to the visemes are then located by searching one or more audio signals.
- the located phonemes are then identified by their spatial location relative to the viewer (or other reference(s)) in response to the audio channel(s) they are found in.
- the spatial location of the viseme is compared to the spatial location of the phoneme to determine if they match.
- Reporting is performed and/or alarms set in response to the matching or mismatching of the spatial locations of corresponding visemes and phonemes.
- the audio signals may be coupled to different audio channels in order to provide proper tracking. It is also possible to improve matching of audio and video by use of audio signal processing. For example dialogue may be electronically removed from an audio signal, leaving other sounds in that signal, and that removed dialogue added to one or more other audio signals.
- Figure 7 is a flow chart demonstrating one embodiment of the invention.
- One or more video signals is inspected to determine activity indicating the likely presence of video MUEVs as shown in block 710, in this example visemes, and also to spatially locate visemes relative to a reference point, as shown in block 712. As described above, activity may be determined in response to the visemes.
- a plurality of audio signals is inspected to determine activity indication the likely presence of audio MUEVs as shown in block 716, in this example phonemes, and to spatially locate phonemes relative to a reference point, as shown in block 714. Again, activity may be determined in response to phonemes.
- the phonemes and visemes are searched to locate corresponding pairs or MUEVs, as shown in block 720.
- the spatial locations of the corresponding MUEVS are identified, as shown in block 730 and those spatial locations are compared to determine their degree of spatial coincidence, as shown in block 740.
- Parameters are set, by operator, as shown in block 760 or otherwise, for filtering the results of the determination of the degree of spatial coincidence of the corresponding MUEVs.
- Parameters may be set during manufacture, for example by programming them into software, storing them in memory, or hard wiring in circuitry. Parameters may also be set and adjusted automatically in response to the audio and/or video signals, for example in response to the average audio level or average audio frequency content.
- results of the determination of the degree of spatial coincidence are filtered in response to the parameters and audio and video activity to determine the average spatial coincidence, as shown in block 750 and a report of the average spatial coincidence is made.
- Parameters are set, as shown in block 760 by operator or otherwise, for comparison to the average spatial coincidence to determine excessive values indicating a loss of spatial coincidence. As above the parameters may be set during manufacture or set and adjusted automatically. [0071] The average spatial coincidence is compared to the parameters, as shown in block 7802 to determine if the average spatial coincidence exceeds the parameters and if so an alarm is set, as shown in block 790.
- audio As generally used in the art, the word audio often, but not always, pertains to sounds likely include dialogue and the signals that carry them. This is a holdover from original radio and TV systems which only had one channel which carried all sounds. Sound is often used more generally to mean audible sounds and the signals that carry them. It is noted however that when used in multiple channel sound systems, audio is now often used to denote any of the sound channels or signals, including those which are intended to carry only sound effects without dialogue. Audio is also used to mean all of the sound channels and signals in a particular program. The embodiment of the invention is described herein in respect to audio, and audio in television systems. The use of terminology including audio and sound in the description of the exemplary embodiment is that commonly used in the art.
- each sound channel conveys an electronic signal representation of the sound.
- These electronic signals may be analog or digital, and may be conveyed by wire, fiber optic, optical, wireless or any other known method.
- Several other television and audio systems which utilize multiple sound channels are known in the art and it is expected that other multiple sound channel systems and equipment will become known in the future.
- the present invention will find application to many of these multiple sound channel systems as well as the associated equipment, transmission systems and methods as will be known to one of ordinary skill in the art from the teachings herein.
- That channel is used herein in a communications theory sense. That channel is the path, whereby signals, information or data are stored, conveyed or transmitted utilizing any of various technologies known in the art.
- the invention may be practiced with audio effects, low frequency audio, electronically generated audio, laugh tracks, applause tracks and any other type of audio signal which is desired to be present and/or carried on one or more particular channel(s).
- the invention may be practiced by inspection of the signal for expected
- MUEVs or other types of information in a channel When speaking of the absence or presence of phonemes, MUEVs or other types of information in a channel, it will be understood that it is the absence or presence of the particular information in the signal (of whatever type) which conveys that information via the channel. Detecting the information in a channel may be performed directly by inspecting the signal carried in the channel. It is also possible to detect the presence of information in that channel indirectly by inspecting the sound from the transducer (e.g. speaker) which converts the signal to sound, such as by using another transducer such as a microphone.
- the transducer e.g. speaker
- dialogue pertains to spoken words such as by humans, cartoon characters and the like. While in the normal sense dialogue pertains to a two way conversation, as used herein it will encompass a one way conversation such as a radio or television announcer broadcasting to a listener.
- frame and image frame refer to the entire frame of images viewed by a viewer whereas image refers to the particular image or images of interest within (or not within but relative to) the frame. Most commonly image refers to the image which corresponds to the sound being discussed, for example the face of a speaker which is analyzed for visemes and for which the sound is being analyzed for phonemes. That face may be in the frame or not, such as when a person in the frame is looking at the speaker which is talking but cannot be seen by the viewer.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Acoustics & Sound (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Stereophonic System (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
La présente invention a trait à un appareil et à un procédé permettant de suivre le dialogue et autres signaux sonores dans un film, à la télévision ou autres systèmes dotés d'un son à voies multiples. Une ou plusieurs voies audio qui sont destinées à transporter la voix des personnes apparaissant dans un programme ou autres types particuliers de sons sont contrôlées afin de déterminer si ce son de cette voie inclut des sons particuliers tels que des MUEV, y compris des phonèmes correspondant aux modèles de voix humaine. Si un nombre incorrect de sons particuliers tels que des phonèmes sont trouvés dans la ou les voies, une mesure telle qu'un compte rendu, une alarme, une correction ou autre mesure est prise. Le contrôle de la ou des voies audio peut être effectué en association avec l'apparition d'images correspondantes associées au son, tel que des visèmes dans le signal vidéo, afin d'améliorer la détermination des types de sons tels que les phonèmes.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US26739309P | 2009-12-07 | 2009-12-07 | |
US61/267,393 | 2009-12-07 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2011071928A2 true WO2011071928A2 (fr) | 2011-06-16 |
WO2011071928A3 WO2011071928A3 (fr) | 2012-03-29 |
Family
ID=44146144
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2010/059296 WO2011071928A2 (fr) | 2009-12-07 | 2010-12-07 | Détecteur de dialogue et correction |
Country Status (2)
Country | Link |
---|---|
US (2) | US9305550B2 (fr) |
WO (1) | WO2011071928A2 (fr) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110381336A (zh) * | 2019-07-24 | 2019-10-25 | 广州飞达音响股份有限公司 | 基于5.1声道的视频片段情感判定方法、装置和计算机设备 |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110311144A1 (en) * | 2010-06-17 | 2011-12-22 | Microsoft Corporation | Rgb/depth camera for improving speech recognition |
IT201900024526A1 (it) * | 2019-12-18 | 2021-06-18 | Technogym Spa | Metodo per migliorare l’esperienza di allenamento di un utente su una macchina ginnica e macchina ginnica implementante tale metodo |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060290810A1 (en) * | 2005-06-22 | 2006-12-28 | Sony Computer Entertainment Inc. | Delay matching in audio/video systems |
US20070153089A1 (en) * | 2003-05-16 | 2007-07-05 | Pixel Instruments, Corp. | Method, system, and program product for measuring audio video synchronization using lip and teeth characteristics |
US20080079851A1 (en) * | 2006-09-29 | 2008-04-03 | Stanger Leon J | Audio video timing measurement and synchronization |
US7424160B1 (en) * | 2003-04-28 | 2008-09-09 | The Directv Group, Inc. | Methods and apparatus for monitoring compressed bitstreams |
Family Cites Families (38)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5202761A (en) * | 1984-11-26 | 1993-04-13 | Cooper J Carl | Audio synchronization apparatus |
US4792974A (en) * | 1987-08-26 | 1988-12-20 | Chace Frederic I | Automated stereo synthesizer for audiovisual programs |
US5644364A (en) * | 1993-04-16 | 1997-07-01 | Avid Technology, Inc. | Media pipeline with multichannel video processing and playback |
US5576843A (en) * | 1993-10-29 | 1996-11-19 | Time Warner Entertainment Co., L.P. | System and method for controlling play of multiple dialog audio tracks of a software carrier |
US5920842A (en) * | 1994-10-12 | 1999-07-06 | Pixel Instruments | Signal synchronization |
US6836295B1 (en) * | 1995-12-07 | 2004-12-28 | J. Carl Cooper | Audio to video timing measurement for MPEG type television systems |
MY123651A (en) * | 1999-04-07 | 2006-05-31 | Dolby Laboratories Licensing Corp | Matrix improvements to lossless encoding and decoding |
US7188193B1 (en) * | 2000-01-20 | 2007-03-06 | Sonic Solutions, A California Corporation | System, method and article of manufacture for a synchronizer component in a multimedia synchronization framework |
US6662026B1 (en) * | 2000-09-28 | 2003-12-09 | International Business Machines Corporation | Apparatus and method for detecting and handling accidental dialing on a mobile communications device |
US7030930B2 (en) * | 2001-03-06 | 2006-04-18 | Ati Technologies, Inc. | System for digitized audio stream synchronization and method thereof |
US6927802B2 (en) * | 2001-11-01 | 2005-08-09 | Warner Bros. Entertainment Inc. | TV receiver with individually programmable SAPchannel |
EP1341386A3 (fr) * | 2002-01-31 | 2003-10-01 | Thomson Licensing S.A. | Système audiovisuel avec retard variable |
US7177816B2 (en) * | 2002-07-05 | 2007-02-13 | At&T Corp. | System and method of handling problematic input during context-sensitive help for multi-modal dialog systems |
US7212248B2 (en) * | 2002-09-09 | 2007-05-01 | The Directv Group, Inc. | Method and apparatus for lipsync measurement and correction |
US6633245B1 (en) * | 2002-12-12 | 2003-10-14 | Lsi Logic Corporation | Method and system for correcting quantization loss during analog to digital to analog signal conversion |
US20070153125A1 (en) * | 2003-05-16 | 2007-07-05 | Pixel Instruments, Corp. | Method, system, and program product for measuring audio video synchronization |
BRPI0410740A (pt) * | 2003-05-28 | 2006-06-27 | Dolby Lab Licensing Corp | método, aparelho e programa de computador para calcular e ajustar o volume percebido de um sinal de áudio |
DE10344638A1 (de) * | 2003-08-04 | 2005-03-10 | Fraunhofer Ges Forschung | Vorrichtung und Verfahren zum Erzeugen, Speichern oder Bearbeiten einer Audiodarstellung einer Audioszene |
US20050144324A1 (en) * | 2003-12-08 | 2005-06-30 | International Business Machines Corporation | Service for analyzing web client dialogs |
US7394903B2 (en) * | 2004-01-20 | 2008-07-01 | Fraunhofer-Gesellschaft Zur Forderung Der Angewandten Forschung E.V. | Apparatus and method for constructing a multi-channel output signal or for generating a downmix signal |
US20050219366A1 (en) * | 2004-03-31 | 2005-10-06 | Hollowbush Richard R | Digital audio-video differential delay and channel analyzer |
US8330859B2 (en) * | 2004-05-14 | 2012-12-11 | Pixel Instruments Corporation | Method, system, and program product for eliminating error contribution from production switchers with internal DVEs |
US7333150B2 (en) * | 2004-05-14 | 2008-02-19 | Pixel Instruments Corporation | Method, system, and program product for eliminating error contribution from production switchers with internal DVEs |
US20050286727A1 (en) * | 2004-06-25 | 2005-12-29 | Victor Company Of Japan, Ltd. | Apparatus for expanding sound image upward |
KR100694060B1 (ko) * | 2004-10-12 | 2007-03-12 | 삼성전자주식회사 | 오디오 비디오 동기화 장치 및 그 방법 |
US9088855B2 (en) * | 2006-05-17 | 2015-07-21 | Creative Technology Ltd | Vector-space methods for primary-ambient decomposition of stereo audio signals |
EP2112652B1 (fr) * | 2006-07-07 | 2012-11-07 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Concept pour combiner plusieurs sources audio codées selon des paramètres |
MX2008012250A (es) * | 2006-09-29 | 2008-10-07 | Lg Electronics Inc | Metodos y aparatos para codificar y descodificar señales de audio basadas en objeto. |
MY145497A (en) * | 2006-10-16 | 2012-02-29 | Dolby Sweden Ab | Enhanced coding and parameter representation of multichannel downmixed object coding |
US20080111887A1 (en) * | 2006-11-13 | 2008-05-15 | Pixel Instruments, Corp. | Method, system, and program product for measuring audio video synchronization independent of speaker characteristics |
US20080253577A1 (en) * | 2007-04-13 | 2008-10-16 | Apple Inc. | Multi-channel sound panner |
US8849432B2 (en) * | 2007-05-31 | 2014-09-30 | Adobe Systems Incorporated | Acoustic pattern identification using spectral characteristics to synchronize audio and/or video |
ES2593822T3 (es) * | 2007-06-08 | 2016-12-13 | Lg Electronics Inc. | Método y aparato para procesar una señal de audio |
US20090030683A1 (en) * | 2007-07-26 | 2009-01-29 | At&T Labs, Inc | System and method for tracking dialogue states using particle filters |
US8155959B2 (en) * | 2007-11-07 | 2012-04-10 | Robert Bosch Gmbh | Dialog system for human agent to correct abnormal output |
EP2272169B1 (fr) * | 2008-03-31 | 2017-09-06 | Creative Technology Ltd. | Décomposition adaptative de signaux audio en composantes primaires et ambiantes |
EP2154911A1 (fr) * | 2008-08-13 | 2010-02-17 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Appareil pour déterminer un signal audio multi-canal de sortie spatiale |
US9324337B2 (en) * | 2009-11-17 | 2016-04-26 | Dolby Laboratories Licensing Corporation | Method and system for dialog enhancement |
-
2010
- 2010-12-07 US US12/962,551 patent/US9305550B2/en not_active Expired - Fee Related
- 2010-12-07 WO PCT/US2010/059296 patent/WO2011071928A2/fr active Application Filing
-
2016
- 2016-02-25 US US15/053,941 patent/US20160241887A1/en not_active Abandoned
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7424160B1 (en) * | 2003-04-28 | 2008-09-09 | The Directv Group, Inc. | Methods and apparatus for monitoring compressed bitstreams |
US20070153089A1 (en) * | 2003-05-16 | 2007-07-05 | Pixel Instruments, Corp. | Method, system, and program product for measuring audio video synchronization using lip and teeth characteristics |
US20060290810A1 (en) * | 2005-06-22 | 2006-12-28 | Sony Computer Entertainment Inc. | Delay matching in audio/video systems |
US20080079851A1 (en) * | 2006-09-29 | 2008-04-03 | Stanger Leon J | Audio video timing measurement and synchronization |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110381336A (zh) * | 2019-07-24 | 2019-10-25 | 广州飞达音响股份有限公司 | 基于5.1声道的视频片段情感判定方法、装置和计算机设备 |
Also Published As
Publication number | Publication date |
---|---|
WO2011071928A3 (fr) | 2012-03-29 |
US20120041762A1 (en) | 2012-02-16 |
US20160241887A1 (en) | 2016-08-18 |
US9305550B2 (en) | 2016-04-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10720153B2 (en) | Name-sensitive listening device | |
US10154360B2 (en) | Method and system of improving detection of environmental sounds in an immersive environment | |
CN107820037B (zh) | 音频信号、图像处理的方法、装置和系统 | |
CN107316651B (zh) | 基于麦克风的音频处理方法和装置 | |
US20160241887A1 (en) | Dialogue Detector and Correction | |
JPH0546910B2 (fr) | ||
CN103581608A (zh) | 发言人检测系统、发言人检测方法和音频/视频会议系统 | |
CN103618986A (zh) | 一种3d空间中音源声像体的提取方法及装置 | |
CN107124647A (zh) | 一种全景视频录制时自动生成字幕文件的方法及装置 | |
CN112468924A (zh) | 一种耳机降噪方法及装置 | |
CN105872205A (zh) | 一种信息处理方法及装置 | |
CN113014844A (zh) | 一种音频处理方法、装置、存储介质及电子设备 | |
CN110381336B (zh) | 基于5.1声道的视频片段情感判定方法、装置和计算机设备 | |
Siddig et al. | Perception Deception: Audio-Visual Mismatch in Virtual Reality Using The McGurk Effect. | |
JP3643372B1 (ja) | 放送信号監視装置及び放送信号監視方法 | |
CN106559729A (zh) | Mic语音自动识别测试系统及方法 | |
Abel et al. | The labial viseme reconsidered: Evidence from production and perception | |
Libal et al. | Multimodal classification of activities of daily living inside smart homes | |
CN104469250B (zh) | 一种信息处理方法及电子设备 | |
Stenzel et al. | Speech reaction time measurements for the evaluation of audio-visual spatial coherence | |
CN107220021A (zh) | 语音输入识别方法、装置及头戴式设备 | |
Hladek et al. | Behavior and Speech Intelligibility in a Changing Multi-talker Environment | |
CN108093296B (zh) | 一种影片自适应播放的方法及系统 | |
US12052550B2 (en) | Method for performing acoustic measurements | |
US20230267942A1 (en) | Audio-visual hearing aid |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 10836554 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 10836554 Country of ref document: EP Kind code of ref document: A2 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 14/12/2012) |