JP2012244413A - Signal monitoring device and program, and signal correction device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a relative delay between a video signal and an audio signal when a delay occurs between the video signal and the audio signal.SOLUTION: A transmit side device 10 comprises a signal extraction unit 11 which extracts an audio signal from a video signal in which the audio signal is multiplexed, a feature quantity calculation unit 12 which calculates a sound feature quantity a(i) from the audio signal extracted by the signal extraction unit 11, and an auxiliary data multiplexing unit 13 which multiplexes the sound feature quantity a(i) calculated by the feature quantity calculation unit 12 in the video signal which has had the audio signal multiplexed therein and outputs a video signal having the audio signal and the sound feature quantity a(i) multiplexed therein. A receive side device 20 comprises a signal and feature quantity extraction unit 21 which extracts the audio signal and the sound feature quantity a(i) from the video signal in which the audio signal and the sound feature quantity a(i) have been multiplexed, a feature quantity calculation unit 22 which calculates a sound feature quantity b(i) from the audio signal extracted by the signal and feature quantity extraction unit 21, and a relative delay calculation unit 24 which calculates a video and audio relative delay quantity 25 from the sound feature quantity a(i) extracted by the signal and feature quantity extraction unit 21 and the sound feature quantity b(i) calculated by the feature quantity calculation unit 22.

Description

  The present invention relates to a signal monitoring device and program for detecting a relative delay or error between a video signal and an audio signal generated in the transmission process when transmitting a video signal and an audio signal, and a signal correction for correcting the relative delay. The present invention relates to a device and a program.

  Audio signals are often multiplexed and transmitted in an auxiliary data area of SDI (Serial Digital Interface) for TV video signals. A method for multiplexing an audio signal to the auxiliary data area of SDI is defined in, for example, ARIB standard BTA S-006. In the case of HDTV (1080 / 59.94 / I) used in Japan, approximately 1600 audio signals (48 kHz sampling) are multiplexed for each frame period of the video signal. In the transmission process, an error may occur in a video signal or an audio signal due to a malfunction of a device or noise, or an unintended signal may be added or inserted due to an operational error. Also, video and audio may be separated and processed, and audio signals may be multiplexed and transmitted again. In this case, a delay difference occurs between the video signal and the audio signal depending on the signal processing. When a delay occurs between video and audio, the sample of the audio signal multiplexed in the video frame period is different from the original sample.

  The ARIB technical document ARIB TR-B29 (Non-Patent Document 1) describes a method for using video and audio feature quantities to perform signal fault monitoring. The video and audio feature quantities on the sending side are calculated for each frame and transmitted as metadata along with the video and audio signals. At the monitoring point on the receiving side, the video and audio feature quantities are received from the received video and audio signals for each frame. And is compared with the feature amount calculated and transmitted on the transmission side. If there is an error in the signal during the transmission process, the feature value calculated on the receiving side will be different from the feature value calculated on the transmitting side. Can detect signal errors.

  FIG. 1 shows the relationship between the video signal and audio signal and the video feature value and audio feature value for each frame on the transmission side and the reception side. This example shows a case where the video signal is delayed by 0.5 frames from the audio, and the number of audio samples per frame is described as 1600.

  As a prior art document of a technique for detecting a relative delay between a video signal and an audio signal generated in a transmission process, for example, there is Patent Document 1. In Patent Document 1, time information sampled with a reference signal of a digital video signal and a digital audio signal is added to video compression data and audio compression data on the transmission side and transmitted, and on the reception side, the received video compression data and Extracts the time information added to the audio compression data and detects the difference between the extracted time information and the time information sampled with the reference signal of the digital video signal and digital audio signal obtained by expanding the received video compression data and audio compression data. Based on the detected difference information, a delay time that is a difference between an input time on the transmission side of the video signal and the audio signal and an output time on the reception side is detected, and the video signal is determined from the delay time of the video signal and the delay time of the audio signal. And obtaining a value indicating the difference between the delay times of the audio signal (summary, paragraphs 0035- See 036).

JP 2004-104730 A

ARIB Technical Document ARIB TR-B29 “Metadata for Video / Audio Signal Fault Monitoring in Broadcast Chains” [online], [April 6, 2011 Search], Internet <http: //www.arib. or.jp/english/html/overview/doc/4-TR-B29v1_0.pdf>

  In order to detect the relative delay between the video signal and the audio signal, the temporal correspondence between the video signal and the audio signal needs to be known. Furthermore, a means for detecting temporal correspondence from the received video signal and audio signal is required.

Also, if there is a delay between the video signal and the audio signal in the transmission process, the audio signal multiplexed in the received video signal frame period will be different from the audio signal multiplexed on the transmission side. Even if there is no error, the speech feature amount calculated on the receiving side is different from the speech feature amount calculated on the transmitting side, and the transmitted and received speech feature amounts do not match. That is, in FIG. 1, the audio feature AI _T (1) is generated from the audio sample 1-1600 for the video frame 1 on the transmission side, while the audio sample for the video frame 1 is generated on the reception side. A voice feature value AI _R (1) is generated for 801-2400, and AI _T (1) and AI _R (1) do not match even if there is no error in the transmission process.

  An object of the present invention is to provide an apparatus for detecting a relative delay between a video signal and an audio signal when a delay occurs between the video signal and the audio signal due to a processing delay generated as a result of signal processing in a transmission process. It is.

  Also, when a signal error caused by equipment failure or transmission error in the transmission process is monitored, if a delay occurs between the video signal and the audio signal, the relative delay between the video signal and the audio signal is detected. An object of the present invention is to provide an apparatus for detecting an error in an audio signal by correcting a delay.

  Another object of the present invention is to provide a device that delays one of the two signals on the basis of the relative delay between the detected video signal and the audio signal and corrects the delay between the video signal and the audio signal. .

  Of the inventions disclosed in this specification, the outline of typical ones will be briefly described as follows.

  (1) A signal monitoring apparatus that receives an audio signal and a video signal obtained by multiplexing a first audio feature quantity that is an audio feature quantity of the audio signal, the audio signal and the first audio feature quantity being obtained from the video signal. A signal / feature amount extraction unit for extracting a signal, a feature amount calculation unit for calculating a second audio feature amount from the audio signal extracted by the signal / feature amount extraction unit, and a first one extracted by the signal / feature amount extraction unit And a relative delay calculating unit that calculates a video / audio relative delay amount from the first audio feature amount and the second audio feature amount calculated by the second feature amount calculating unit.

  (2) In the above (1), the first audio feature amount and the second feature amount extracted by the signal / feature amount extraction unit based on the video / audio relative delay amount calculated by the relative delay calculation unit. A first sound feature value obtained by correcting a delay from the second sound feature value calculated by the calculation unit; a phase adjustment unit that obtains a second sound feature value; and a delay obtained by correcting the delay obtained by the phase adjustment unit. 1 is a signal monitoring device including a speech feature amount comparison unit that performs error detection by comparing one speech feature amount with a second speech feature amount.

  (3) Either the audio signal or the video signal in the video signal in which the audio signal is multiplexed based on the signal monitoring device of (1) above and the video / audio relative delay amount output from the signal monitoring device. And a video / audio signal correction unit that corrects such a delay.

  (4) A program for causing a computer to function as a signal monitoring device that receives an audio signal and a video signal in which a first audio feature quantity that is an audio feature quantity of the audio signal is multiplexed. And a signal / feature amount extraction means for extracting the first sound feature amount, a feature amount calculation means for calculating a second sound feature amount from the sound signal extracted by the signal / feature amount extraction means, and the signal / feature A program for functioning as a relative delay calculating unit that calculates a video / audio relative delay amount from the first audio feature amount extracted by the amount extracting unit and the second audio feature amount calculated by the second feature amount calculating unit It is.

  (5) A program that causes a computer to function as a signal monitoring device that receives an audio signal and a video signal in which a first audio feature quantity that is an audio feature quantity of the audio signal is multiplexed. And a signal / feature amount extraction means for extracting the first sound feature amount, a feature amount calculation means for calculating a second sound feature amount from the sound signal extracted by the signal / feature amount extraction means, and the signal / feature A relative delay calculating unit that calculates a video / audio relative delay amount from the first audio feature amount extracted by the amount extracting unit and the second audio feature amount calculated by the second feature amount calculating unit; and the relative delay calculating unit. Based on the video / audio relative delay amount calculated by the means, a delay is generated from the first audio feature amount extracted by the signal / feature amount extraction unit and the second audio feature amount calculated by the second feature amount calculation unit. The first sound corrected A phase adjustment unit that obtains the collected amount and the second audio feature amount, and an error detection is performed by comparing the first audio feature amount and the second audio feature amount obtained by correcting the delay obtained by the phase adjustment unit. This is a program for functioning as a voice feature amount comparison unit.

  (6) A program that causes a computer to function as a signal correction device that receives an audio signal and a video signal in which a first audio feature quantity that is an audio feature quantity of the audio signal is multiplexed. And a signal / feature amount extraction means for extracting the first sound feature amount, a feature amount calculation means for calculating a second sound feature amount from the sound signal extracted by the signal / feature amount extraction means, and the signal / feature A relative delay calculating unit that calculates a video / audio relative delay amount from the first audio feature amount extracted by the amount extracting unit and the second audio feature amount calculated by the second feature amount calculating unit; and the relative delay calculating unit. Based on the video / audio relative delay amount output from the means, it functions as a video / audio signal correction means for correcting the delay of either the audio signal or the video signal in the video signal multiplexed with the audio signal. It is a program for.

  According to the present invention, when transmitting a video signal and an audio signal, a relative delay between the video signal and the audio signal when a delay occurs between the video signal and the audio signal due to a processing delay resulting from the signal processing in the transmission process. An apparatus for detecting the delay can be obtained.

  In addition, according to the present invention, when a signal error caused by a device failure or a transmission error in the transmission process is monitored, when a delay occurs between the video signal and the audio signal, the relative delay between the video signal and the audio signal is determined. Thus, a device for detecting an error in a speech signal can be obtained by correcting and comparing the relative delay between the speech feature amount transmitted from the transmission side and the speech feature amount calculated on the reception side.

  Further, according to the present invention, there is provided an apparatus that delays whichever signal is advanced based on the relative delay between the detected video signal and the audio signal, and eliminates the delay between the video signal and the audio signal. be able to.

It is a figure which shows the relationship between the video feature-value and audio | voice feature-value for every image | video signal and audio | voice signal and each flame | frame in a transmission side and a receiving side. It is a figure which shows the signal monitoring apparatus of Example 1 of this invention. It is a figure which shows the example of the audio | voice feature-value calculated by the transmission side apparatus. It is a figure which shows the example of the audio | voice feature-value calculated by the receiving side apparatus. It is a figure which shows the example of window function w (i). It is a figure which shows the example (when video / audio relative delay amount = 0) of the detection result r (i) according to the audio | video / video relative delay amount. It is a figure which shows the example (in the case of video / audio relative delay amount = 0.25 frame (video delay and audio advance)) of the detection result r (i) according to the audio / video relative delay amount. It is a figure which shows the example (in the case of video / audio relative delay amount = 1.5 frames (video delay and audio advance)) of the detection result r (i) according to the audio / video relative delay amount. It is a figure which shows the example (in the case of video / audio relative delay amount = 30 frames (video delay and audio advance)) of the detection result r (i) according to the audio / video relative delay amount. It is a figure which shows the signal monitoring apparatus of Example 2 of this invention. It is a figure which shows the signal correction apparatus of Example 3 of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  The configuration of the signal monitoring apparatus according to the first embodiment of the present invention is shown in FIG. The present embodiment relates to a signal monitoring device (video / audio relative delay measuring device) for detecting a relative delay between a video signal and an audio signal generated in a transmission process when transmitting a video signal and an audio signal.

  As shown in FIG. 2, the signal monitoring device includes a reception-side device 20 provided on the reception side that receives a signal from the transmission-side device 10.

  The transmission-side apparatus 10 extracts an audio signal from the video signal multiplexed with the audio signal, and calculates an audio feature amount a (i) for each video frame period from the audio signal extracted by the signal extraction unit 11. And the audio feature quantity a (i) calculated by the feature quantity calculation unit 12 is multiplexed with the video signal multiplexed with the audio signal, and the audio signal and the audio feature quantity a (i) are multiplexed. And an auxiliary data multiplexing unit 13 for outputting the video signal.

  The receiving-side device 20 includes a signal / feature amount extraction unit 21 that extracts an audio signal and an audio feature amount a (i) from a video signal in which the audio signal and the audio feature amount a (i) are multiplexed, and a signal / feature amount extraction. A feature amount calculating unit 22 that calculates an audio feature amount b (i) for each video frame period from the audio signal extracted by the unit 21, and an audio feature amount a (i) and feature amount extracted by the signal / feature amount extracting unit 21. The relative delay calculation unit 24 that calculates the video / audio relative delay amount 25 from the audio feature amount b (i) calculated by the calculation unit 22, and the audio feature amount b (i) calculated by the feature amount calculation unit 22 as the audio signal and An auxiliary data multiplexing unit 23 that multiplexes the audio feature a (i) with the multiplexed video signal and outputs the audio signal and the video signal with the audio feature a (i) and b (i) multiplexed. Prepare. Note that the auxiliary data multiplexing unit 23 is not required when signal monitoring is not performed in the receiving side device 20 and later.

  The video signal in which the audio signal is multiplexed is a signal obtained by multiplexing the audio signal with the video signal every predetermined time (for example, video frame period), and the feature amount calculation unit 12 performs the predetermined time (for example, video frame) from the audio signal. The audio feature quantity a (i) for each period) is calculated, and the feature quantity calculation unit 22 calculates the audio feature quantity b (i) for each predetermined time (for example, video frame period) from the audio signal. When a 48 kHz sampling audio signal is transmitted together with a 1080 / 59.94 / I HDTV signal, the number of audio samples per frame is about 1600 samples.

  Also, the relative delay calculation unit 24 calculates the phase difference between the audio feature quantity a (i) and the audio feature quantity b (i) by, for example, the phase-only correlation method, and the video / audio relative delay quantity 25 is calculated. Get.

  In this example, on the transmitting side and the receiving side, video signals are input / output by SDI (serial digital interface), and audio signals are multiplexed and transmitted on SDI as auxiliary data. The audio feature amounts a (i) and b (i) calculated for each video frame period are multiplexed in the auxiliary data area of the next video frame. Assume that there is no relative delay difference between the video signal and the audio signal on the transmission side on the basis of the relative delay between the video signal and the audio signal on the transmission side. However, the video signal is not limited to SDI.

  An HD-SDI signal (video + audio) 14 in which an audio signal is multiplexed into an auxiliary data packet is input to the transmission side apparatus 10, and an audio feature quantity a (i) is multiplexed into the auxiliary data packet by an auxiliary data multiplexing unit 13. Then, an HD-SDI signal (video + audio + audio feature quantity a (i)) 15 in which the audio signal and the audio feature quantity a (i) are multiplexed in the auxiliary data packet is transmitted toward the receiving side. In FIG. 2, signal transmission and signal processing from the transmission-side device 10 to the reception-side device 20 are collectively described as transmission / processing.

  Received is an HD-SDI signal (video + audio + audio feature a (i)) 26 in which the audio signal and audio feature a (i) received from the transmission side through transmission and signal processing are multiplexed in the auxiliary data packet. Input to the side device 20, the receiving side device 20 outputs an HD-SDI signal (video + audio + audio feature amount a (i) + audio feature amount b (i)) 27 and video / audio relative delay amount 25.

  For example, ARIB technical document ARIB TR-B29 “Metadata for video / audio signal failure monitoring in the broadcast chain” (Non-patent Document 1) specifies the audio feature quantity and the method of multiplexing these auxiliary data packets. Has been. In the present embodiment, amplitude information or phase information defined in ARIB TR-B29 is used as the audio feature amount. However, the voice feature amount and the multiplexing method to these auxiliary data packets are not limited to these.

  The signal extraction unit 11 extracts an audio signal multiplexed in the auxiliary data area of the HD-SDI signal (video + audio) 14.

  The feature amount calculation unit 12 calculates an audio feature amount a (i) for each video frame period based on ARIB TR-B29 from the audio signal extracted by the signal extraction unit 11.

  The auxiliary data multiplexing unit 13 multiplexes the audio feature quantity a (i) calculated by the feature quantity calculating unit 12 on the HD-SDI signal (video + audio) 14 as auxiliary data based on ARIB TR-B29.

  The signal / feature quantity extraction unit 21 extracts the audio signal and the audio feature quantity a (i) multiplexed in the auxiliary data area of the HD-SDI signal (video + audio + audio feature quantity a (i)) 26.

  The feature quantity calculation unit 22 calculates the audio feature quantity b (i) for each video frame period based on ARIB TR-B29 from the audio signal extracted by the signal / feature quantity extraction unit 21.

  The auxiliary data multiplexing unit 23 uses the audio feature quantity b (i) calculated by the feature quantity calculation unit 22 as HD-SDI signal (video + audio + audio feature quantity a (i)) 26 as auxiliary data based on ARIB TR-B29. To multiplex.

  The relative delay calculation unit 24 calculates the video / audio relative delay amount 25 from the audio feature amount a (i) extracted by the signal / feature amount extraction unit 21 and the audio feature amount b (i) calculated by the feature amount calculation unit 22. .

  If there is no relative delay between the video signal and the audio signal during the transmission process, the audio feature b (i) calculated on the receiving side and the audio sample to be calculated for the audio feature a (i) calculated on the transmitting side are Match. However, if a relative delay occurs between the video signal and the audio signal, the audio signal period corresponding to the video frame period will be shifted in time, and the audio feature calculation target audio samples are different on the transmission side and the reception side. As a result, the voice feature amounts do not match. Based on the transmitted and received audio feature values a (i) and b (i), the relative delay calculation unit 24 calculates the video / audio relative delay amount 25 using the phase-only correlation method as follows.

The audio feature quantity samples for the past N (= 2 ⁿ ) frames including the current video frame at the transmitting side and the receiving side are respectively a (i), b (i) (i = −2 ⁿ⁻¹ ,..., 2 ^{n -1-1} ). Here, i represents a video frame. The number of frames N used for calculating the video / audio relative delay amount may be set in accordance with the video / audio relative delay amount and accuracy to be detected and the calculation method of the discrete Fourier transform described later. The power of 2 is for ease of fast Fourier transform.

a (i) and b (i) are multiplied by the window function w (i) = (1 + cos (πi / 2 ^n-1 )) / 2, and then subjected to discrete Fourier transform to A (j), B (j ) The discrete Fourier transform and the discrete inverse Fourier transform are represented by symbols F {} and F ⁻¹ {}, respectively.
A (j) = F {a (i) w (i)}
B (j) = F {b (i) w (i)}
The reason for applying the window function is to perform a discrete Fourier transform on a finite number of samples.

Calculate the normalized mutual power spectrum R (j) = A (j) B ^* (j) / | A (j) B ^* (j) | from A (j) and B (j) And || represents an absolute value), and a phase-only correlation function r (i) is obtained by a discrete inverse Fourier transform of R (j).
R (j) = A (j) B ^* (j) / | A (j) B ^* (j) | (* indicates phase conjugation, || indicates absolute value)
r (i) = F ^-1 {R (j)}
If a (i) and b (i) are the same, r _p (0) = 1.0 is obtained.

From r (i), the maximum value r _p of r (i) and its position i _p are detected.
r _p = r (i _p ) (0 <r _p ≦ 1)
When r _p = 1.0 and i _p = 0, it is determined that there is no delay, and the following processing is unnecessary.

second maximum value r _s of r (i) and detects the position i _s.
r _s = r (i _s ) (0 <r _s <r _p )
Sign = -1 if the second maximum value is below the maximum value, sign = -1 if the second maximum value is above the maximum value r _p , sign = 1, and if the second maximum value is not adjacent to the maximum value With sign = 0, the phase difference is detected by the following calculation.
Phase difference = i _p + sign * {(1-r _p ) + r _s } / 2
The sign of the phase difference indicates whether a (i) or b (i) is advanced or delayed.

  In this embodiment, a simple method for obtaining an approximate value of the phase difference from the maximum value of r (i) and the second maximum value is used, but a more advanced method may be used.

  Thus, by calculating the phase difference between a (i) and b (i) by the phase-only correlation method, the video / audio relative delay amount generated in the transmission process can be obtained. However, the video / audio relative delay amount may be obtained using a method other than the phase-only correlation method.

  Next, an example of the audio feature amount, an example of the window function, and an example of the detection result r (i) corresponding to the video / audio relative delay amount will be described with reference to FIGS. 3 to 9, the horizontal axis is a frame.

  FIG. 3 shows an example of the voice feature amount a (i) calculated by the transmission side device 10. Since there is no relative delay difference between the video signal and the audio signal on the transmission side, the video / audio relative delay amount = 0. Here, the audio in-phase information (AII) defined in ARIB TR-B29 is used as the audio feature amount.

  FIG. 4 shows an example of the audio feature b (i) calculated by the receiving-side device 20 when a video / audio relative delay of 1.5 frames (about 2400 audio samples @ 48 kHz) occurs in the transmission / processing process. Show.

  FIG. 5 shows the window function w (i) when N = 256.

FIG. 6 shows r (i) when the video / audio relative delay amount = 0. It can be seen that r _p = r (0), i _p = 0, and the delay is zero.

FIG. 7 shows r (i) in the case of video / audio relative delay amount = 0.25 frame (400 audio samples @ 48 kHz, video delay / audio advance). r _p = r (0) = 0.752, i _p = 0, r _s = r (1) = 0.2429, i _s = 1 and the second maximum value is the upper neighbor of the maximum value, sign = 1 And
Phase difference _{= i p + sign * {(} 1 - r p) + r s} / 2 = 0 + 1 * {(1 - 0.752) + 0.242} is calculated as / 2 = 0.245 frames. Since the phase difference is 0.245 frames, the video / audio relative delay amount is about 0.25 frames.

FIG. 8 shows r (i) when the video / audio relative delay amount = 1.5 frames (video delay / audio advance). r _p = r (1) = 0.533, i _p = 1, r _s = r (2) = 0.482, i _s = 2, and the second maximum value is adjacent to the maximum value, sign = 1 And
Phase difference = i _p + sign * {(1-r _p ) + r _s } / 2 = 1 + 1 * {(1-0.533) + 0.482} / 2 = 1.47 frames. Since the phase difference is 1.47 frames, the video / audio relative delay is about 1.5 frames.

FIG. 9 shows r (i) when the video / audio relative delay amount = 30 frames (video delay / audio advance). r _p = r (30) = 0.761, i _p = 30, r _s = r (-26) = 0.095, i _s = -26, and the maximum value and the second maximum value are not adjacent, sign = 0 And
Phase difference = i _p = 30 frames is calculated. Since the phase difference is 30 frames, the video / audio relative delay amount is 30 frames.

As described above, FIGS. 7 to 9 show examples of video delay and audio advance. Conversely, in the case of video advance and audio delay, i _{p which} is the maximum value of r (i) is of course on the negative side.

  According to the present embodiment, the audio signal is multiplexed with the video signal and transmitted, so that it can be handled on the transmission side as having no time difference between them. Further, the audio feature amount calculated on the transmission side is transmitted as small size data together with the video signal and the audio signal. This audio feature amount (same as the video feature amount) is transmitted in synchronization with the video frame, and even when the video and audio are separated in the transmission process and the audio signal is multiplexed again, the video signal and the auxiliary data area The positional relationship of the audio feature amount does not change. On the reception side, a phase difference can be obtained by calculating a voice feature value from the received voice signal and applying a phase-only correlation method to the voice feature value calculated and transmitted on the transmission side.

  In the case where signal monitoring is performed even after the receiving side device 20, any section of the transmitting side device, the receiving device, the second, third,... Receiving device is provided by providing the second, third,. Can be obtained.

  The configuration of the signal monitoring apparatus according to the second embodiment of the present invention is shown in FIG. The present embodiment relates to a signal monitoring device (video / audio signal monitoring device) for detecting an error between a video signal and an audio signal generated in the transmission process, and a relative delay occurs between the video signal and the audio signal in the transmission process. This is a device that can detect an error in an audio signal even in the case of an error.

  As shown in FIG. 10, the signal monitoring device includes a reception-side device 40 provided on the reception side that receives a signal from the transmission-side device 30.

The transmitting-side device 30 extracts a sound signal and a video signal from a video signal multiplexed with a sound signal, and a sound feature amount AI _T (i) from the sound signal and the video signal extracted by the signal extraction unit 31. and a feature amount calculation unit 32 that calculates the image feature VI _T (i), audio feature feature amount calculation unit 32 has calculated AI _T (i) and the image feature VI _T a (i) the audio signals are multiplexed And an auxiliary data multiplexing unit 33 that outputs the video signal multiplexed with the audio signal, the audio feature amount AI _T (i), and the video feature amount VI _T (i).

The receiving-side device 40 receives the audio signal, the video signal, the audio feature quantity AI _T (i), and the video feature from the video signal in which the audio signal, the audio feature quantity AI _T (i), and the video feature quantity VI _T (i) are multiplexed. A signal / feature amount extraction unit 41 that extracts the amount VI _T (i), and a feature amount calculation unit that calculates the audio feature amount AI _R (i) and the video feature amount VI _R (i) from the extracted audio signal and video signal. 42, and a relative audio / video relative delay amount 45 calculated from the audio feature amount AI _T (i) extracted by the signal / feature amount extraction unit 41 and the audio feature amount AI _R (i) calculated by the feature amount calculation unit 42. Based on the delay calculation unit 44 and the video / audio relative delay amount 45 calculated by the relative delay calculation unit 44, the phase adjustment unit 46 that adjusts the phase of AI _T (i) and the phase of AI _R (i) are corrected. a phase adjusting unit 47 compares the voice characteristic amount is corrected by the phase adjusting unit 46, 47 AI _T (i) and speech features AI _R (i) Ri and audio feature amount comparison section 48 for detecting, by comparing the image feature amount signal, the feature extraction unit 41 has extracted VI _T (i) the image feature feature quantity calculating unit 42 is calculated and VI _R (i) A video feature quantity comparison unit 49 that performs error detection and an audio feature quantity AI _R (i) and a video feature quantity VI _R (i) as an audio signal, a voice feature quantity AI _T (i), and a video feature quantity VI _T (i ) Is multiplexed into the multiplexed video signal, and the audio signal and audio feature quantities AI _T (i) and AI _R (i), and the video feature quantities VI _T (i) and VI _R (i) are multiplexed. And an auxiliary data multiplexing unit 43 for outputting. Note that the auxiliary data multiplexing unit 43 is not required when signal monitoring is not performed after the receiving side device 40.

The video signal in which the audio signal is multiplexed is a signal obtained by multiplexing the audio signal with the video signal every predetermined time (for example, video frame period), and the feature amount calculation unit 32 performs a predetermined time (from the audio signal and the video signal). For example, an audio feature value AI _T (i) and a video feature value VI _T (i) are calculated for each video frame period), and the feature value calculation unit 42 calculates a predetermined time (for example, a video frame period) from the audio signal and the video signal. The audio feature amount AI _R (i) and the video feature amount VI _R (i) are calculated for each. Also, the relative delay calculation unit 44 calculates the phase difference between the audio feature value AI _T (i) and the audio feature value AI _R (i) by, for example, the phase-only correlation method to calculate the video / audio relative A delay amount (phase difference) 45 is obtained.

An HD-SDI signal (video + audio) 34 in which an audio signal is multiplexed in an auxiliary data packet is input to the transmission side device 30, and the audio feature amount AI _T (i) and the video feature amount VI _T ( i) is multiplexed into the auxiliary data packet, and the audio signal and audio feature value AI _T (i) and video feature value VI _T (i) are multiplexed into the auxiliary data packet (video + audio + video feature value). VI _T (i) + voice feature quantity AI _T (i)) 35 is transmitted toward the receiving side. In FIG. 10, signal transmission and signal processing from the transmission-side device 30 to the reception-side device 40 are collectively described as transmission / processing.

An HD-SDI signal (video + audio + video feature VI _T (i) in which the audio signal, audio feature AI _T (i) and video feature VI _T (i) received from the transmission side are multiplexed in the auxiliary data packet + Audio feature amount AI _T (i)) 50 is input to the receiving side device 40, and the receiving side device 40 receives the HD-SDI signal (video + audio + video feature amount VI _T (i) + video feature amount VI _R (i ) + Voice feature quantity AI _T (i) + voice feature quantity AI _R (i)) 51. In addition, the receiving-side device 40 displays the sound delay state on a display device (not shown) using the video / audio relative delay amount (phase difference) 45 that is the output of the relative delay calculation unit 44, and the video feature amount comparison unit 49, Using the output of the audio feature quantity comparison unit 48, video state display and audio state display are performed on a display device (not shown).

  In the present embodiment, video space information, video time information, audio amplitude information, and audio phase information defined in ARIB TR-B29 (Non-Patent Document 1) are used as video feature amounts and audio feature amounts. However, the video feature value, the audio feature value, and the multiplexing method of these to the auxiliary data packet are not limited to these.

  The signal extraction unit 31 extracts the video signal and the audio signal multiplexed in the auxiliary data area from the HD-SDI signal (video + audio) 34.

The feature amount calculation unit 32 calculates the video feature amount VI _T (i) and the audio feature amount AI _T (i) based on ARIB TR-B29 from the video signal and the audio signal.

The auxiliary data multiplexing unit 33 multiplexes the video feature amount VI _T (i) and the audio feature amount AI _T (i) as the auxiliary data on the HD-SDI signal (video + audio) 34 based on ARIB TR-B29.

The signal / feature quantity extraction unit 41 uses the HD-SDI signal (video + audio + video feature quantity VI _T (i) + audio feature quantity AI _T (i)) 50 to multiplex the video signal and the auxiliary data area. A signal and video feature amount VI _T (i) and an audio feature amount AI _T (i) are extracted.

The relative delay calculation unit 44 calculates a video / audio relative delay amount 45 from the audio feature amount AI _T (i) extracted by the signal / feature amount extraction unit 41 and the audio feature amount AI _R (i) calculated by the feature amount calculation unit 42. calculate.

The phase adjustment unit 46 and the phase adjustment unit 47 are calculated by the audio feature amount AI _T (i) extracted by the signal / feature amount extraction unit 41 and the feature amount calculation unit 42 based on the calculated video / audio relative delay amount, respectively. The phase of the audio feature value AI _R (i) is corrected. In fact, since the correction is made by delaying the one that is advanced with respect to the one that is behind, the correction is made (delayed) by either one of the phase adjustment units 46 and 47.

The audio feature amount comparison unit 48 compares the audio feature amount AI _T (i) corrected by the phase adjustment unit 46 and the audio feature amount AI _R (i) corrected by the phase adjustment unit 47 for each frame.

The video feature amount comparison unit 49 compares the video feature amount VI _T (i) extracted by the signal / feature amount extraction unit 41 with the video feature amount VI _R (i) calculated by the feature amount calculation unit 42 for each frame.

The auxiliary data multiplexing unit 43 uses the video feature value VI _R (i) and the audio feature value AI _R (i) as auxiliary data based on ARIB TR-B29 as HD-SDI signals (video + audio + video feature value VI _T (i ) + Voice feature amount AI _T (i)) 50. These VI _T (i) and AI _T (i) are transmitted in synchronization with the video frame.

On the transmission side, the transmission side device 30 calculates a predetermined video feature amount VI _T (i) and audio feature amount AI _T (i) for each video frame period, and multiplexes them in the auxiliary data area of the next frame. When a 48 kHz sampling audio signal is transmitted together with a 1080 / 59.94 / I HDTV signal, the number of audio samples per frame is about 1600 samples.

On the receiving side, the video feature amount VI _T (i) and the audio feature amount AI _T (i) calculated and multiplexed on the transmitting side are received together with the video signal and the audio signal. On the receiving side, the receiving side device 40 calculates the video feature amount VI _R (i) and the audio feature amount AI _R (i) based on the same definitions as those calculated and transmitted from the received video signal and audio signal on the transmitting side. calculate.

If there is no relative delay between the video signal and the audio signal during the transmission process, the target audio of the audio feature value AI _R (i) calculated on the receiving side and the audio feature value AI _T (i) calculated on the transmitting side The samples match. However, if there is a relative delay between the video signal and the audio signal, the audio sample corresponding to the video frame period will be shifted in time, and the audio sample for which the audio feature value is calculated will be different between the transmission side and the reception side. Thus, the voice feature amounts do not match.

  Based on the transmitted and received audio feature quantities, the relative delay calculation unit 44 calculates the video / audio relative delay quantity 45 using the phase only correlation method.

  That is, by applying the phase-only correlation method between the speech feature value sequence calculated from the received speech signal and the speech feature value sequence calculated and transmitted on the transmission side, the phase difference of the speech feature value between transmission and reception is reduced. Ask. The phase difference of the audio feature amount between transmission and reception corresponds to the relative delay between video and audio.

The calculation method of the relative delay calculation unit 44 of the present embodiment is the same as that of the relative delay calculation unit 24 of the first embodiment. If a (i) and b (i) in the first embodiment are replaced with AI _T (i) and AI _R (i), the operation is exactly the same. Therefore, although not repeated here, as in the first embodiment, AI _T A phase-only correlation function r (i) is obtained from (i) and AI _R (i), and a phase difference (video / audio relative delay amount) is obtained from the maximum value, the second maximum value, and their positions. The sign of the phase difference indicates whether AI _T (i) or AI _R (i) is advanced (delayed). If AI _T (i) and AI _R (i) are the same, then r _p (0) = 1.0 is obtained.

  In this way, the video / audio relative delay amount generated in the transmission process is obtained.

  In the first embodiment, the example of the audio feature amount, the example of the window function, and the example of the detection result r (i) corresponding to the video / audio relative delay amount have been described with reference to FIGS. The same applies to the examples.

Based on this video / audio relative delay amount 45, the phase adjustment unit 46 extracts the audio feature amount AI _T (i) extracted by the signal / feature amount extraction unit 41 and the audio feature amount AI _R calculated by the feature amount calculation unit 42. Correct the phase of (i). The process for giving the phase difference can be expressed as follows, for example.

Assuming that the Fourier transform of f (t) is F (ω), the Fourier transform of f (t-t ₀ ) obtained by shifting f (t) by t ₀ is expressed by F (ω) exp (-jωt ₀ ) (J is an imaginary unit), and the Fourier transform A (k) is multiplied by exp (-jωt ₀ ) (where phase difference t ₀ , ω = 2πk) is applied to the speech feature sequence a (i), and the inverse Fourier transform is performed. Get.

  Since the audio feature amount is calculated in units of frames, in the case of an integer frame delay, the phase adjustment unit 46 may shift the feature amount sample by a phase difference. In the case of a delay of a fractional frame, it is necessary to perform the above interpolation processing according to the delay amount.

  An audio signal abnormality is detected by obtaining a difference in audio feature amount between transmission and reception using the phase-corrected audio feature amount. If the difference is within a predetermined threshold, it is determined that there is no abnormality.

  As for the video feature amount, a video signal abnormality is detected by obtaining a difference between the transmission and reception feature amounts. If the difference is within a predetermined threshold, it is determined that there is no abnormality.

  In the present embodiment, the delay-corrected speech feature value is obtained by shifting the phase of the speech feature value on the transmission side or the reception side in accordance with the phase difference of the speech feature value between transmission and reception. By using the delay-corrected voice feature quantity, a difference in voice feature quantity between transmission and reception is obtained. This makes it possible to detect an audio signal error due to a transmission process failure or noise even when a relative sound delay occurs.

  When the signal monitoring is performed even after the receiving side device 40, the second, third,... Receiving devices are provided, so that any section of the transmitting side device, the receiving device, the second, third,. Relative delay detection and error detection are possible.

  In the second embodiment, both the audio feature amount and the video feature amount are multiplexed. However, the phase adjustment units 46 and 47 and the audio feature amount comparison are performed in the first embodiment that multiplexes only the audio feature amount as in the second embodiment. The unit 48 may be provided to detect an audio signal abnormality.

  Further, in the first embodiment, only the audio feature amount is multiplexed. However, as in the second embodiment, both the audio feature amount and the video feature amount are multiplexed, and a video feature amount comparison unit 49 is provided so that a video signal is provided. An abnormality may be detected.

  FIG. 11 shows the configuration of the signal correction apparatus according to the third embodiment of the present invention. In this embodiment, even when a relative delay occurs between the video signal and the audio signal in the transmission process, the relative delay between the two signals is detected, and correction is performed to delay the signal that is advanced by the detected delay. This is a device that eliminates the relative delay.

  As shown in FIG. 11, the signal correction device includes a reception-side device 60 that receives a video signal and a signal obtained by multiplexing an audio feature and an audio signal synchronized with a video frame. Further, this is a configuration in which a video / audio signal correction unit 66 is added to the signal monitoring device 20 of the first embodiment except for the auxiliary data multiplexing unit 23.

  The reception-side device 60 includes a signal / feature amount extraction unit 61 that extracts an audio signal and an audio feature amount a (i) from a video signal in which the audio signal and the audio feature amount a (i) are multiplexed, and a signal / feature amount extraction unit. A feature amount calculation unit 62 that calculates an audio feature amount b (i) for each video frame period from the audio signal extracted by the unit 61; an audio feature amount a (i) and a feature amount extracted by the signal / feature amount extraction unit 61 A relative delay calculation unit 64 that calculates a video / audio relative delay amount 65 from the audio feature amount b (i) calculated by the calculation unit 22, and a video / audio signal correction that delays the video signal or the audio signal by the calculated relative delay. Part 66.

  The operations until the video / audio relative delay amount 65 is calculated are the same as those in the first embodiment.

  The video / audio signal correcting unit 66 is provided with a buffer corresponding to the assumed delay amount so that the [video + audio (auxiliary data area)] signal can stay, and the audio signal is accessed by accessing the auxiliary data area. In other words, delay or advance delay adjustment is performed in accordance with the video / audio relative delay amount. If the video / audio relative delay amount is video delay / audio advance, the audio signal (sampling data) multiplexed in the auxiliary data area is delayed by the calculated delay amount. If the video advance / audio delay occurs, the audio signal in the auxiliary data area is moved to a video signal delayed by the calculated delay amount.

  In this way, the video / audio signal correction unit 66 can output a video signal and an audio signal in which the relative delay of the video / audio during the transmission process is corrected. Further, if necessary, a device corresponding to the transmission-side device 10 of the first embodiment may be provided in the subsequent stage, and the audio feature amount for the audio signal may be calculated again and added to the next video frame.

  The signal monitoring apparatus according to the first and second embodiments and the signal correction apparatus according to the third embodiment can be configured by a computer and a program, except for the HD-SDI interface unit. Moreover, you may comprise a part or all of the program with a hardware.

  As mentioned above, the invention made by the present inventor has been specifically described based on the above embodiments. However, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Of course.

DESCRIPTION OF SYMBOLS 10 ... Transmission side apparatus, 11 ... Signal extraction part, 12 ... Feature-value calculation part, 13 ... Auxiliary data multiplexing part, 14 ... HD-SDI (video + audio | voice) input into the transmission side apparatus 10, 15 ... Transmission side apparatus HD-SDI (video + audio + audio feature quantity a (i)) output from 10, 20... Receiving side device, 21... Signal / feature quantity extraction unit, 22 .. feature quantity calculation unit, 23 .. auxiliary data multiplexing unit , 24 ... relative delay calculation unit, 25 ... video / audio relative delay amount, 26 ... HD-SDI (video + audio + audio feature amount a (i)) input to the reception side device 20, 27 ... reception side device 20 HD-SDI (video + audio + audio feature value a (i) + audio feature value b (i)), 30... Transmission side device, 31... Signal extraction unit, 32. Auxiliary data multiplexing unit 34... HD-SDI (video + audio) input to the transmission-side device 30 35. HD-SDI (video +) output from the transmission-side device 30 Audio + video feature amount VI _T (i) + audio feature amount AI _T (i)), 40... Receiving side device, 41... Signal / feature amount extraction unit, 42... Feature amount calculation unit, 43. 44 ... Relative delay calculation unit, 45 ... Video / audio relative delay amount, 46, 47 ... Phase adjustment unit, 48 ... Audio feature amount comparison unit, 49 ... Video feature amount comparison unit, 50 ... Input to receiving apparatus 40 HD-SDI (video + audio + video feature VI _T (i) + audio feature VI _T (i)), 51 ... HD-SDI (video + audio + video feature VI _T output from the receiving device 40 (i) + image feature value VI _R (i) + audio feature value VI _T (i) + audio feature value VI _R (i)), 60... reception side device, 61... signal / feature value extraction unit, 62. Amount calculation unit, 64 ... relative delay calculation unit, 65 ... video / audio relative delay amount, 66 ... video / audio signal correction unit

Claims (6)

A signal monitoring device that receives an audio signal and a video signal in which a first audio feature amount that is an audio feature amount of the audio signal is multiplexed,
A signal / feature amount extraction unit for extracting an audio signal and a first audio feature amount from the video signal;
A feature amount calculator that calculates a second speech feature amount from the speech signal extracted by the signal / feature amount extractor;
A relative delay calculation unit that calculates a video / audio relative delay amount from the first audio feature amount extracted by the signal / feature amount extraction unit and the second audio feature amount calculated by the second feature amount calculation unit;
A signal monitoring device comprising: Based on the video / audio relative delay amount calculated by the relative delay calculation unit, the first audio feature amount extracted by the signal / feature amount extraction unit and the second audio calculated by the second feature amount calculation unit A phase adjustment unit for obtaining a first audio feature quantity and a second audio feature quantity, in which a delay is corrected from the feature quantity;
A speech feature amount comparison unit that performs error detection by comparing the first speech feature amount with the delay corrected by the phase adjustment unit and the second speech feature amount;
The signal monitoring apparatus according to claim 1, further comprising: A signal monitoring device according to claim 1;
A video / audio signal correction unit that corrects a delay of either the audio signal or the video signal in the video signal multiplexed with the audio signal based on the video / audio relative delay amount output from the signal monitoring device; ,
A signal correction apparatus comprising: A program that causes a computer to function as a signal monitoring device that receives an audio signal and a video signal in which a first audio feature quantity that is an audio feature quantity of the audio signal is multiplexed,
A signal / feature amount extraction means for extracting an audio signal and a first audio feature amount from the video signal;
Feature quantity calculation means for calculating a second voice feature quantity from the voice signal extracted by the signal / feature quantity extraction means;
Functions as a relative delay calculation unit that calculates a video / audio relative delay amount from the first audio feature amount extracted by the signal / feature amount extraction unit and the second audio feature amount calculated by the second feature amount calculation unit. Program to let you. A program that causes a computer to function as a signal monitoring device that receives an audio signal and a video signal in which a first audio feature quantity that is an audio feature quantity of the audio signal is multiplexed,
A signal / feature amount extraction means for extracting an audio signal and a first audio feature amount from the video signal;
Feature quantity calculation means for calculating a second voice feature quantity from the voice signal extracted by the signal / feature quantity extraction means;
A relative delay calculating unit that calculates a video / audio relative delay amount from the first audio feature amount extracted by the signal / feature amount extracting unit and the second audio feature amount calculated by the second feature amount calculating unit;
Based on the video / audio relative delay amount calculated by the relative delay calculating means, the first audio feature amount extracted by the signal / feature amount extraction unit and the second audio calculated by the second feature amount calculation unit. Phase adjusting means for obtaining a first voice feature quantity and a second voice feature quantity, in which a delay is corrected from the feature quantity;
A program for functioning as a speech feature amount comparison unit that performs error detection by comparing the first speech feature amount with the delay corrected by the phase adjustment unit and the second speech feature amount. A program that causes a computer to function as a signal correction device that receives an audio signal and a video signal in which a first audio feature amount that is an audio feature amount of the audio signal is multiplexed,
A signal / feature amount extraction means for extracting an audio signal and a first audio feature amount from the video signal;
Feature quantity calculation means for calculating a second voice feature quantity from the voice signal extracted by the signal / feature quantity extraction means;
A relative delay calculating unit that calculates a video / audio relative delay amount from the first audio feature amount extracted by the signal / feature amount extracting unit and the second audio feature amount calculated by the second feature amount calculating unit;
Video / audio signal correction means for correcting the delay of either the audio signal or the video signal in the video signal multiplexed with the audio signal based on the video / audio relative delay amount output from the relative delay calculation means Program to function as.

Images