ES2337012B2 - SYSTEM AND METHOD OF ADJUSTMENT OF AUDIO / VIDEO DELAY. - Google Patents

Abstract

System and delay adjustment method audio / video comprising the following stages: extract parameters characteristic of an audio source (3); enter the parameters extracted from the audio source (3) in the video from a video source (1); extract from the audio signal, after video processing, the same parameters extracted in the first stage; get the video signal, after processing the video, the audio parameters entered in the second stage; compare the audio parameters extracted in the third stage and obtained in the fourth; calculate the delay between audio parameters compared; store the calculated delay; Y finally delay the audio signal a delay value configurable depending on the calculated delay and the delay stored from a previous execution of the method.

Description

System and delay adjustment method audio Video.

Field of the Invention

The present invention belongs to the field of digital communications for the production and dissemination of signals from television.

State of the art

The use of digital technologies in teams in charge of audio and video processing of the production processes for television has originated a series of audio and video offset problems due to different delays introduced by the use of these technologies. One of the reasons for this difference in production delays is because, while in the analog world the equipment barely introduced delays in the signal, digital is necessary store the information of each frame to be able to process the image and transmit it later upon departure. Therefore the storing and processing an image that has 25 frames per second without interlacing, means that a delay 1/25 seconds (40 ms), regardless of the time of indicted. If the number of processes involved in the operation were constant, a simple delay in the audio of Nx40 ms would solve the problem, N being the number of processes by What happens the signal. However, the number of processes differs depending on the application you intend to make and the way of operating the technicians who carry it out.

In addition to the above, the delay between the audio and the video may change the control of realization since video mixers are used. The Video mixers are responsible for mixing the different signals that enter the control and are characterized by having several Image processing stages, in each of which you can perform mixing or image insertion operations. This form, when a signal goes through three stages of processing, case typical in professional mixers, the output signal is delayed 120 ms with respect to audio. Thus, the output of a study would have a margin of audio / video desynchronization between 40 and 120 ms.

The problem is aggravated when the signal has to go through various controls or distribution environments, such like IP networks where the audio path is different from the video that make the delay ungovernable.

The audio / video synchronization methods known primarily treat delays derived from the Low quality video transfer on IP networks. One of the most used methods today is based on the processing of images, in order to detect the interlocutor's lips, and in audio processing, in order to adjust as much as possible The sound to the movement of the lips. However, this method presents an important problem that is about CPU consumption required for signal processing.

On the other hand, the digital processing of Audio allows to detect and classify sounds as loud or deaf. In sound, there is a fundamental frequency with an amplitude that stands out over other lower frequency components amplitude, however, in the deaf there are many frequencies that they influence the hue with similar amplitudes and there is no frequency that is imposed on the rest. An example of a vocal vowel is the "a" whose dominant frequency is a frequency around 780 hertz and as a second component a frequency around 1600 hertz. Voice recognition in this case it is based both on the spectral envelope analysis (training frequencies) as in its evolution over time. In this context, the human voice can be considered as "stationary in pieces" with stationary periods of 20 or 40 ms and generating 5 to 20 phonemes per second. This recognition voice, along with the study of the image focusing interest on the location of the lips, it is used to correct the desynchronizations however, this processing involves a load high computational which is a limitation considerable.

It was therefore desirable a method of adjusting audio / video delay that will solve existing problems in the state of the art in this sector.

Description of the invention

The present invention proposes a method of audio / video delay adjustment in order to alleviate problems exposed in the state of the art.

To do this, the method performs an insertion of audio data in the video in order to synchronize the audio and the video Specifically, the method is based on the introduction of significant elements of the audio within the video signal When the audio is voice type. This method is not invasive since that the audio data is not inserted or mixed with the data of image but are entered into free data fields of the own video stream The method also does not require a great capacity to perform the processing since it only studies the sound and data streams of the video stream, that is, you don't need to try the image and therefore reduces the computational load associated with said processing In this way, with this method it is possible to analyze the Audio and video stream, detect image delay with regarding the audio and, once this delay has been detected, adjust staggered an audio delay line to avoid possible cuts or jumps in the sound. This stepped delay setting is can perform by time delay breaks Preset As an example if you have a video with a delay of 200 ms with respect to audio, this adjustment can be made by adding 50 audio delays in 500 ms sections, so that 1.5 seconds the audio and video will be synchronized, or by adding 100 ms jumps in 500 ms sections, in which case in 0.5 seconds I would be in sync. The jump could be from full delay, from this way the synchronized audio and video would not present delays

The method starts from a video source whose output is added the result of processing and extracting parameters from Audio from the audio source. Preferably, the envelope in frequencies along with the amplitudes of the sound that at that time is occurring, these data would accompany the video throughout his career.

From the moment the video has the reference of your audio associated with those sync marks, both the video and the audio can go through the different program production elements, mixers, synchronizers frame, etc.

Once the production processes are finished, the video will be delayed with respect to the audio in a number variable of tables that will vary over time and depend on the different processes and forms of operation they have gone through the signs.

To correct this delay, the audio signal resulting from the production process is treated by a processor which extracts the defined parameter (frequency envelope and amplitudes) Said parameter is compared with the parameters that has the video data flow, result of the insertion carried out before the production processes, obtaining after the comparison a value of the actual delay of the video with respect to the audio, so it will be necessary to delay the audio for synchronization, this is done by the same comparator that controls a variable delay line, the comparator controls remotely delay time added by delay line Variable to the audio signal. The comparator will send the delays necessary to adjust the delay line staggered in the audio to avoid possible cuts or jumps in the sound. Finally, two signals are fully obtained at the output Synchronous: the audio signal and the video signal

In the creation of audiovisual material there are Different types of sources and video production processes. TO the implementation of this method is particularized to each one of them:

- The result of capturing images with a single camera : in this case, the method detects and introduces the envelope of frequencies and amplitudes of sound sounds in the video data stream.

- The result of image editing : in this case, when audio and video of the same sequence is inserted, the data of the frequency envelope of the original image is also copied, in the image resulting from the edition, since Being an audio and video copy, the audio signal must be totally synchronous with the video. The data of the frequency envelope will have been generated by the camera when recording the video and audio and when audio belonging to a different image ( voiceover ) is inserted, the data of the frequency envelope that is in the video signal since it is a reference to the audio that would have associated the image previously.

- The result of the capture in a study : in this case, several cameras and several microphones are involved. First, an association of microphones is made for each camera, where each microphone can be associated with several cameras, that is: one or more audio inputs are associated with a video input and each audio input can be associated with several inputs Of video. In each audio input the sound is processed to extract the value of the envelope in frequencies, this data is introduced into the data flow of the video inputs that are associated with that audio input, specifically before the mixer. You can also associate one or several video signals for each video input in the mixer in order to add several sources of audio data embedded in the video signal for cases where the camera has monitors with people talking in them.

- The result of production in the control of an embodiment : in this case, there are several sources of information in the realization control, these sources go to the mixer where the different signals that make up the program output or study output are selected, each one of the video sources will have the envelope data at frequencies of the sounds associated with the signal so that each time a video signal is selected the spectral envelope data of the audios associated with that input line are copied to the video stream of the mixer or program output.

The delay order from the comparator to the Delay line is performed by default value breaks, Each jump will be called equalization jump value. This equalization jump value can be negative or positive and it will send in sections that will be called equalization tables, this It means that in each equalization box a value of equalization jump. The equalization jump value is a divisor of the picture frame time, for the example of a video of 25 frames per second with a frame time of 40 ms, the values of equalization jump can be; 2020, pm10, \8, pm5, ± 2, ± 1, etc.). The comparator stores the delay value in every moment in which the comparison of the envelope is made spectral of the audio and the envelope associated with the signal of video, the audio envelope data is stored in the comparator and when an envelope value arrives within the video, It is known how long the audio has to be delayed. In this way, yes the delay between video and audio is 200 ms, the delay line It has a stored value of 160 ms and the jump value is ± 20, the value 180 will be sent in the first equalization box to be presented, and if in the following match table it remains the same delay (200 ms), will be updated with the value of 200.

To prevent possible problems of identification of the spectral envelope of the audio inserted in the video with the spectral envelope extracted from the audio processing, Several parameters are set:

- Temporary correction margin : defined as the maximum time to search for the spectral envelope. If those values are not found for the spectral envelope of the video, no temporary correction is made. For example, if a time correction interval of two seconds is set in the delay line comparator, this comparator will store the values of the spectral envelope reported by the audio processor, so that if in a moment t an audio spectral envelope arrives within the video signal, the comparator will look at the different spectral envelopes it has up to t-2 seconds. This parameter will be variable in each installation, depending on the maximum number of processes that the image may suffer. For example, if the image can go through a maximum of 24 storage and processing elements, the maximum correction timeframe will be 24 * 20 ms (in the case of 50 images per second). The value is set in the comparator.

- Margin of identification : defined as the difference, expressed as a percentage, below which two spectral envelopes are considered as identical, that is, that the value embedded in the video corresponds to that of the audio being analyzed and They are considered equal. If this margin is not exceeded, there is no delay. This comparison is made in the comparator and the value is set in the functional testing phase.

Therefore, the delay adjustment method Audio / video, includes the following stages:

to)

extract characteristic parameters of an audio source;

b)

enter the parameters extracted from the audio source in the video from a source video;

C)

extract from the audio signal, after video processing, the same parameters extracted in a);

d)

get the video signal, after the Video processing, audio parameters entered in b);

and)

compare audio parameters extracted in c) and obtained in d);

F)

calculate the delay between audio parameters compared in e);

g)

store delay calculated;

h)

delay the audio signal a value of configurable delay depending on the calculated delay and the delay stored from a previous execution of the method.

Preferably, the parameter extraction characteristic includes the extraction of the spectral envelope and of the sound sound amplitudes of the audio source in the moment of extraction. However, other parameters may be used. that characterize the audio such as sound peaks or Different levels in frequency.

Preferably, in processes in which the video It comes from different video sources:

-

if he inserted audio and video come from the same sequence, copied the characteristic parameters of the source image audio in the destination image;

-

if he audio is inserted with a different image, the copies are not copied characteristic parameters of the source image audio in the target image, not containing the target image envelope data Spectral of no sound.

Optionally, if the video and audio proceed from different video and audio sources, the method of adjusting audio / video delay further comprises the following stages:

-

associate, in each video input, the minus an audio signal;

-

process the audio inputs of each video input extracting the characteristic parameters of each audio input;

-

enter, in each of the signals video, the characteristic parameters as a reference value in the data flow of the video inputs in which you are associated;

-

associate, in each video input, the minus a video signal to enter the parameters characteristic of the audio signals of the devices Viewers that enter the image field.

       \ newpage
    

The introduction of parameters extracted from The audio source in the video can be performed within the field of data specified in each of the standards. The different video standards contain fields where you can host data additional, for example, in MPEG2 hosts PES packages (Packetised Elementary Stream) of data and in JPEG2000 boxes of XML.

Preferably, the delay of the audio signal It comprises the following stages:

-

check if the delay value calculated in f) differs from the stored delay of the execution previous of the method;

-

select a delay jump value if the delay value calculated in f) differs from the delay stored;

-

update delay value stored increasing or decreasing said delay a value equal to delay jump value selected;

-

delay audio delay value stored.

Likewise, preferably, the selection of the delay jump is performed between the time dividers of picture.

The audio / video delay adjustment system understands:

?

a means of processing configured to perform the following stages:

i)

extract characteristic parameters of an audio source;

j)

enter the parameters extracted from the audio source in the video from a source video;

k)

extract from the audio signal, after video processing, the same parameters extracted in a);

l)

get the video signal, after the Video processing, audio parameters entered in b);

?

a comparator configured for perform the following stages:

m)

compare audio parameters extracted in k) and obtained in l);

n)

calculate the delay between audio parameters compared in m);

or)

store delay calculated;

?

a configured delay line to perform the following stage:

p)

delay the audio signal a value of configurable delay depending on the calculated delay and the delay stored from a previous execution of the method.

Preferably, the processing means comprise an audio processing block configured for the extraction of characteristic parameters by extracting the spectral envelope and the amplitudes of the sound sound of the Audio source at the time of extraction.

Preferably, in systems where the video It comes from different video sources:

-

if he inserted audio and video come from the same sequence, the media processing are configured to copy the parameters audio characteristics of the source image in the image destination;

-

if he audio is inserted with a different image, the means of processing are set to clear parameters characteristics of the audio that the image had.

Optionally, if the video and audio proceed from different video and audio sources, the means of processing are configured to additionally perform the following stages:

-

associate, in each video input, the minus an audio signal;

-

process the audio inputs of each video input extracting the characteristic parameters of each audio input;

-

enter, in each of the signals video, the characteristic parameters as a reference value in the data flow of the video inputs in which you are associated;

-

associate, in each video input, the minus a video signal to enter the parameters characteristic of the audio signals of the devices Viewers that enter the image field.

Preferably, the means of processing are configured to enter the parameters extracted from the audio source in the video within the field of data specified in each of the standards.

Additionally, the delay of the audio signal You can understand the following stages:

-

check if the delay value calculated in f) differs from the stored delay of the execution previous of the method;

-

select a delay jump value if the delay value calculated in f) differs from the delay stored;

-

Send the delay value to the audio delay line;

-

update delay value stored increasing or decreasing said delay a value equal to delay jump value selected.

Additionally, the audio comparator can be configured to select the delay jump between the frame time dividers.

And also, preferably the comparator of Audio will be set to:

-

establish a margin of identification corresponding to the maximum difference admitted to consider that the characteristic parameters are identical;

-

search within a timeframe of configurable correction, the characteristic parameters received in the audio signal, ordering the delay line a delay of the audio signal if the audio comparator obtains the parameters features inserted in the corresponding video, under a identification margin, to the characteristic parameters received In the audio signal.

Brief description of the drawings

Then, to facilitate the understanding of the invention, by way of illustration but not limitation, will be described an embodiment of the invention that refers to a series of figures.

Figure 1 shows the block diagram of the processes comprising the present invention.

Figure 2 shows the block diagram of the processes comprising an embodiment of the invention. In specifically, the case in which a presenter talks to an editor remote and showing the editor on screen in the same studio in the that the presenter is.

Detailed description of one embodiment

An embodiment of the invention would be the case in which, from the realization control, is shown on a screen a presenter dialoguing next to a really located editor in a remote site but it appears as if the editor were in the same study as the presenter.

In this case, the image shows notably the difference in sound, since the presentation of the image of the editor is associated with a delay, not present in the sound, since it goes directly to the audio mixing console. Normally, in these circumstances, it is usually kept a short time the image of both on screen and is usually passed to the signal that arrives in order that the viewer does not notice the low quality of issue.

The way to synchronize these sounds is shown in figure 2. The signal from remote editor 6 is transmitted to the headquarters through links 7. At reception 8, the audio and video to take out the characteristics of audio 12, which subsequently they are added to video 9 (this process can be done at the source so that the video can arrive with the data of the Audio). The image of the editor will pass through at least one preselector or by a mixer 10 so there will be a variable delay that will add to the delay presented by the study screen 11, while that the audio goes directly to the studio audio mixer fifteen.

The presenter's camera 18 will be capturing the image of the presenter while the studio screen 11 You will have the remote editor in picture. The studio microphone 13 will be associated with the same video mixer input 20, of way its signal 14 is processed and inserted in image 19 from the presenter's camera 18. The video mixer 20 it has associated the input of the presenter's camera 18 and the signal of screen video and audio included in it. The mixer main 20 with has the possibility to delay a time preset to adjust the delay from the display of the study 11. Thus, the video output of the video mixer 20 will have embedded the audios features of the remote editor and the of the presenter in the studio when the entrance of the presenter's camera 18. The outputs of the audio mixers 15 and video 20 are processed using the method proposed in the present invention with audio processing 16, comparator 21 which detects the delays and the audio delay line 17 governed by comparator 21. Thus, when you are speaking the presenter in the study, comparator 21 will adjust the delay to the signal to arrive while the remote editor speaks will adjust the delay to the delay of the entire supply chain generated by remote editor 6.

Once the invention has been clearly described, notes that the particular realizations above described are subject to modifications of detail provided that do not alter the fundamental principle and the essence of the invention.

Claims (15)

1. Audio / video delay adjustment method, characterized in that it comprises the following stages:

to)

extract characteristic parameters of an audio source (3);

b)

enter the parameters extracted from the audio source (3) in the video from a video source (one);

C)

extract from the audio signal, after video processing, the same parameters extracted in a);

d)

get the video signal, after the Video processing, audio parameters entered in b);

and)

compare audio parameters extracted in c) and obtained in d);

F)

calculate the delay between audio parameters compared in e);

g)

store delay calculated;

h)

delay the audio signal a value of configurable delay depending on the calculated delay and the delay stored from a previous execution of the method.

2. An audio / video delay adjustment method according to claim 1, characterized in that the extraction of characteristic parameters comprises the extraction of the spectral envelope and the amplitudes of the sound from the audio source (3) at the time of extraction . 3. Audio / video delay adjustment method according to any of the preceding claims, characterized in that in processes in which the video comes from different video sources (1):

-

if he inserted audio and video come from the same sequence, copied the characteristic parameters of the source image audio in the destination image;

-

if he audio is inserted with a different image, the copies are not copied characteristic parameters of the source image audio in the target image, not containing the target image envelope data Spectral of no sound.

4. Audio / video delay adjustment method, according to any of the preceding claims, characterized in that the video and audio come from different video (1) and audio (3) sources, comprising the following steps:

-

associate, in each video input, the minus an audio signal;

-

process the audio inputs of each video input extracting the characteristic parameters of each audio input;

-

enter, in each of the signals video, the characteristic parameters as a reference value in the data flow of the video inputs in which you are associated.

5. Audio / video delay adjustment method according to any of the preceding claims, characterized in that the introduction of the parameters extracted from the audio source (3) in the video (1) is carried out within the data field specified in each One of the standards. 6. Audio / video delay adjustment method according to any of the preceding claims, characterized in that delaying the audio signal comprises the following steps:

-

check if the delay value calculated in f) differs from the stored delay of the execution previous of the method;

-

select a delay jump value if the delay value calculated in f) differs from the delay stored;

-

update delay value stored increasing or decreasing said delay a value equal to delay jump value selected;

-

delay audio delay value stored.

7. Audio / video delay adjustment method according to claim 6, characterized in that the selection of the delay jump is made between the frame time dividers. 8. Audio / video delay adjustment system, characterized in that it comprises:

?

a means of processing configured to perform the following stages:

i)

extract characteristic parameters of an audio source (3);

j)

enter the parameters extracted from the audio source (3) in the video from a video source (one);

k)

extract from the audio signal, after video processing, the same parameters extracted in a);

l)

get the video signal, after the Video processing, audio parameters entered in b);

?

a comparator (2) configured to perform the following stages:

m)

compare audio parameters extracted in k) and obtained in l);

n)

calculate the delay between audio parameters compared in m);

or)

store delay calculated;

?

a delay line (5) configured to perform the following stage:

p)

delay the audio signal a value of configurable delay depending on the calculated delay and the delay stored from a previous execution of the method.

9. Audio / video delay adjustment system according to claim 8, characterized in that the processing means comprise an audio processing block (4) configured for extracting characteristic parameters by extracting the spectral envelope and amplitudes of the sound sound of the audio source (3) at the time of extraction. 10. Audio / video delay adjustment system according to any of claims 8-9, characterized in that the video comes from different video sources (1), wherein:

-

if he inserted audio and video come from the same sequence, the media processing are configured to copy the parameters audio characteristics of the source image in the image destination;

-

if he audio is inserted with a different image, the means of processing are set to clear parameters characteristics of the audio that the image had.

11. Audio / video delay adjustment system according to any of claims 8-10, characterized in that the video and audio come from different video (1) and audio (3) sources, the processing means being configured for Perform, in each video input, the following stages:

-

associate, in each video input, the minus an audio signal;

-

process the audio inputs of each video input extracting the characteristic parameters of each audio input;

-

enter, in each of the signals video, the characteristic parameters as a reference value in the data flow of the video inputs in which you are associated.

12. Audio / video delay adjustment system according to any of claims 8-11, characterized in that the processing means are configured to enter the parameters extracted from the audio source (3) into the video (1) within the field of data specified in each of the standards. 13. Audio / video delay adjustment system according to any of claims 8-12, characterized in that the audio comparator (2) is configured to perform the following steps:

-

check if the delay value calculated in f) differs from the stored delay of the execution previous of the method;

-

select a delay jump value if the delay value calculated in f) differs from the delay stored;

-

Send the delay value to the audio delay line (5);

-

update delay value stored increasing or decreasing said delay a value equal to delay jump value selected.

14. Audio / video delay adjustment system according to claim 13, characterized in that the audio comparator (2) is configured to select the delay jump between the frame time dividers. 15. Audio / video delay adjustment system according to any of claims 8-14, characterized in that the audio comparator (2) is configured to:

-

establish a margin of identification corresponding to the maximum difference admitted to consider that the characteristic parameters are identical;

-

search within a timeframe of configurable correction, the characteristic parameters received in the audio signal, ordering the delay line (5) a delay of the audio signal if the audio comparator (2) obtains the corresponding characteristic parameters inserted in the video, under a margin of identification, to the characteristic parameters received in the audio signal.