JP6343171B2 - Receiver - Google Patents

Description

  The present invention relates to a receiving apparatus that receives signals transmitted from a plurality of different transmission paths.

  In recent years, with the progress of digitalization of broadcasting and broadbandization of communication, research and development have been conducted on a system that realizes a broadcasting / communication cooperation service. The system includes a first imaging device, a first transmission device that transmits a broadcast signal generated based on data (audio signal and video signal) transmitted from the first imaging device, as a broadcast wave, Two image pickup devices, a second sending device for sending communication signals generated based on data (audio signals and video signals) sent from the second image pickup device by IP transmission, and receiving broadcast signals and communication signals It is composed of a receiving device (receiver). Hereinafter, the first sending device and the second sending device are collectively referred to as a sending device (sending machine).

In addition, the receiving apparatus receives a case where multimedia content composed of components of a plurality of pieces of information (video, audio, subtitles, etc.) is distributed using a plurality of different transmission paths (for example, broadcasting and communication). Needs to have a function of presenting the reproduction timing of each component in synchronization.
This is because the time required for each packet constituting the component to reach the receiver differs depending on the transmission path.

For example, the transmitter adds a time stamp to each component.
The time stamp is added to each distribution unit (distribution unit of video / audio frames, etc.) of each component, and indicates the time when the data of the distribution unit should be presented by the receiver. The receiver uses this time stamp to synchronize the playback timing of each component.

  Specifically, the receiver buffers the components that arrive early based on the value of the time stamp, and synchronizes with the components that arrive late, thereby matching the presentation timing.

  Further, as described above, the time stamp is added on the transmitter side. The transmitter has a system clock common to the entire system, and a time stamp is added to the distribution unit of each component based on the system clock.

JP 2012-10009 A JP 2012-205075 A

In such a system, there is an independent transmitter for each path, and the time stamp depends on the system clock for each transmitter. Various methods for performing presentation synchronization by synchronizing the system clock of each transmitter have been proposed (for example, Patent Documents 1 and 2).
However, in such a system, it is necessary for the service provider side to provide a mechanism for synchronizing the system clock of each transmitter.

  An object of the present invention is to provide a receiving apparatus capable of performing synchronization control of signals transmitted from a plurality of different transmission paths without adjusting a time stamp on the transmitter side.

  A receiving apparatus according to the present invention includes a first communication unit that receives a first signal transmitted from a first transmission path, and a first separation unit that separates the first signal into a first video signal and a first audio signal. A first buffer for storing the first video signal and the first audio signal, a first video decoding unit for decoding the first video signal, and a first audio decoding unit for decoding the first audio signal A first feature extraction unit that extracts feature information from the first speech signal decoded by the first speech decoding unit to generate a first feature signal, and a second transmission path that is different from the first transmission path A second communication unit that receives the second signal transmitted from the second communication unit; a second separation unit that separates the second signal into a second video signal and a second audio signal; and the second video signal and the second audio. A second buffer unit for storing signals; a second video decoding unit for decoding the second video signal; A second speech decoding unit that decodes the second speech signal; a second feature extraction unit that extracts feature information from the second speech signal decoded by the second speech decoding unit and generates a second feature signal; A delay processing unit that calculates a delay amount of the second feature signal based on the first feature signal, and a recovered clock that determines a timing for outputting the recovered clock based on the delay amount calculated by the delay processing unit The first buffer unit outputs the first video signal to the first video decoding unit when the reproduction clock generated by the reproduction clock generation unit is input, and the first buffer unit outputs the first video signal to the first video decoding unit. An audio signal is output to the first audio decoding unit, the first feature extraction unit converts the first audio signal into information of a frequency domain waveform by Fourier transform, and the second feature extraction unit Second voice It is configured to convert the information in the frequency domain waveform by Fourier transform No..

  According to this configuration, the receiving device calculates the delay amount of the second feature signal based on the first feature signal, delays it by a time corresponding to the delay amount, and stores the delay amount in the first buffer unit. Since one video signal is output to the first video decoding unit, the first audio signal is output to the first audio decoding unit, and the first signal and the second signal are synchronized, the time stamp is not adjusted on the transmitter side. Thus, synchronization control of signals transmitted from a plurality of different transmission paths can be performed.

  In the receiving device, the first feature extraction unit reduces the information amount by sampling the first audio signal at a predetermined sampling frequency, and performs a Fourier transform on the first audio signal with the reduced information amount. The second feature extraction unit reduces the amount of information by sampling the second audio signal at a predetermined sampling frequency, and the second audio with the information amount reduced. A configuration may be employed in which information is converted into frequency domain waveform information by performing Fourier transform on the signal.

  According to such a configuration, the receiving apparatus reduces the amount of information while leaving a characteristic part of the first signal received by the first communication unit, and a characteristic of the second signal received by the second communication unit. Since the amount of information is reduced while leaving a portion, the delay amount can be calculated efficiently in the delay processing unit.

  In the receiving device, the first feature extraction unit generates the first feature signal by removing a high frequency component from the first audio signal obtained by the Fourier transform, and the second feature extraction unit The second feature signal may be generated by removing a high frequency component from the second sound signal obtained by Fourier transform.

  According to such a configuration, the receiving device generates the first feature signal by reducing the amount of information while leaving the characteristic portion of the first signal received by the first communication unit, and also receives the signal by the second communication unit. Since the second feature signal is generated by reducing the information amount while leaving the characteristic part of the second signal, the delay processing unit can efficiently calculate the delay amount.

  According to the present invention, synchronization control of signals transmitted from a plurality of different transmission paths can be performed.

It is a figure which shows the structure of a broadcast communication cooperation system. It is a figure which shows the specific structure of a broadcast communication cooperation system. It is a flowchart with which it uses for description about the process sequence of a receiver. It is a flowchart with which it uses for description about the process sequence of a 1st feature extraction part. It is a flowchart with which it uses for description about the process sequence of a delay process part.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
As shown in FIG. 1, the broadcasting / communication cooperation system 100 includes a broadcasting station 1, a broadcasting antenna 2, a receiver 3, and an IP transmission server 4, and realizes a broadcasting / communication cooperation service. It is. Specifically, in the broadcasting / communication cooperation system 100, in the receiver 3, a broadcasting service based on an ISDB (Integrated Services Digital Broadcasting) system and a communication service using an IP network such as the Internet N are linked. The user can use the broadcasting / communication cooperation service by the receiver 3. In this embodiment, the broadcasting / communication cooperation system 100 is described as being compatible with the ISDB method, but is not limited to the ISDB method, and may be another method.

  A broadcast signal broadcast from the broadcast station 1 (broadcast antenna 2) is the same as a broadcast signal of a conventional digital broadcast broadcast by a broadcast facility, and will be described in detail later, but will be described in ARIB (Association of Radio Industries and Broadcast). : Radio Industry Association of Japan) Specified in the standard.

  Although not shown, the broadcasting station 1 has a general digital broadcasting broadcasting facility including a program organization facility, a program transmission facility, a transmission facility, and the like.

  The broadcast station 1 broadcasts a broadcast signal including content. The content includes a program that is a normal content broadcast according to a broadcast schedule and an event that is an emergency content that occurs asynchronously with the program.

  The receiver 3 of the broadcasting / communication cooperation system 100 configured as described above has a function of simultaneously processing broadcast waves and video and audio distributed from the IP network by the ISDB method which is a standard for handling video and audio. Yes. Hereinafter, a configuration example for realizing the function will be described.

  As illustrated in FIG. 2, the receiver 3 includes a first communication unit 11, a first separation unit 12, a first buffer unit 13, a first video decoding unit 14, a first audio decoding unit 15, One feature extraction unit 16, second communication unit 17, second separation unit 18, second buffer unit 19, second video decoding unit 20, second audio decoding unit 21, and second feature extraction unit 22 And a delay processing unit 23 and a reproduction clock generation unit 24. In the following description, the first communication unit 11 receives a broadcast signal, and the second communication unit 17 receives a communication signal transmitted via an IP network such as the Internet N. However, the present invention is not limited to this, and the first communication unit 11 may receive a communication signal and the second communication unit 17 may receive a broadcast signal.

For example, when a baseball broadcast is performed, the broadcasting station 1 captures a video of a pitcher by the first camera 101 and an image of a batter by the second camera 102. The microphone 103 collects the voice of the stadium support or the announcer.
The broadcast station 1 encodes video captured by the first camera 101 using the first video encoding unit 104, encodes audio collected by the microphone 103 using the first audio encoding unit 105, and encodes the encoded video and audio. Are multiplexed by the first multiplexing unit 106, and the multiplexed signal is broadcasted as a broadcast signal to each home via the broadcast antenna 2.
In addition, the broadcast station 1 transmits the video captured by the second camera 102 and the sound collected by the microphone 103 to the IP transmission server 4.
The IP transmission server 4 encodes the video captured by the second camera 102 with the second video encoding unit 107, encodes the audio collected by the microphone 103 with the second audio encoding unit 108, and encodes the encoded video The voice is multiplexed by the second multiplexing unit 109, and the multiplexed signal is transmitted as a communication signal via an IP network such as the Internet N.
That is, the broadcast signal and the communication signal include different images, but the sound is the same.

  The first communication unit 11 receives the first signal transmitted from the first transmission path. Specifically, the first communication unit 11 receives a broadcast signal (first signal) broadcast from the broadcast station 1 via the broadcast antenna 2.

The first separation unit 12 separates the first signal into a first video signal and a first audio signal.
The first buffer unit 13 stores the first video signal and the first audio signal.
The first video decoding unit 14 decodes the first video signal.
The first audio decoding unit 15 decodes the first audio signal.
The first feature extraction unit 16 extracts feature information from the first speech signal decoded by the first speech decoding unit 15 to generate a first feature signal.

  The second communication unit 17 receives a second signal transmitted from a second transmission path different from the first transmission path. Specifically, the second communication unit 17 receives a communication signal (second signal) transmitted from the IP transmission server 4 via an IP network such as the Internet N.

  When the receiver 3 receives the broadcast signal (first signal) by the first communication unit 11, the receiver 3 extracts a unique content ID included in the broadcast signal and sends the content ID to the IP transmission server 4. By transmitting, a communication signal related to the broadcast signal received by the first communication unit 11 is requested. The IP transmission server 4 transmits content corresponding to the content ID transmitted from the receiver 3 to the receiver 3.

The second separator 18 separates the second signal into a second video signal and a second audio signal.
The second buffer unit 19 stores the second video signal and the second audio signal.
The second video decoding unit 20 decodes the second video signal.
The second audio decoding unit 21 decodes the second audio signal.
The second feature extraction unit 22 extracts feature information from the second audio signal decoded by the second audio decoding unit 21 to generate a second feature signal.

The delay processing unit 23 calculates the delay amount of the second feature signal based on the first feature signal.
The reproduction clock generation unit 24 determines the timing for outputting the reproduction clock based on the delay amount calculated by the delay processing unit 23.
The first buffer unit 13 outputs the first video signal to the first video decoding unit 14 when the reproduction clock generated by the reproduction clock generation unit 24 is input, and the first audio signal is output to the first audio decoding unit. 15 is output.

  Also, the first video signal decoded by the first video decoding unit 14, the first audio signal decoded by the first audio decoding unit 15, the second video signal decoded by the second video decoding unit 20, The second audio signal decoded by the second audio decoding unit 21 is supplied to the output unit 25. The output unit 25 includes a display and a speaker (not shown).

  Here, the processing procedure of the receiver 3 will be described with reference to the flowchart shown in FIG. In the following description, the first communication unit 11 receives a broadcast signal (first signal) broadcast from the broadcast station 1 via the broadcast antenna 2, and the second communication unit 17 receives an IP such as the Internet N. It is assumed that a communication signal (second signal) transmitted from the IP transmission server 4 via the network is received.

In step S1, the first separation unit 12 separates the first signal into a first video signal and a first audio signal.
In step S2, the first audio decoding unit 15 decodes the first audio signal.
In step S3, the first feature extraction unit 16 extracts feature information from the first audio signal and generates a first feature signal.

In step S4, the second separator 18 separates the second signal into a second video signal and a second audio signal.
In step S5, the second audio decoding unit 21 decodes the second audio signal.
In step S6, the second feature extraction unit 22 extracts feature information from the second audio signal to generate a second feature signal.
In step S7, the delay processing unit 23 calculates the delay amount of the second feature signal based on the first feature signal.

  In step S <b> 8, the recovered clock generation unit 24 determines the timing for outputting the recovered clock based on the delay amount calculated by the delay processing unit 23. The reproduction clock generation unit 24 supplies the reproduction clock to the first buffer unit 13 at the determined timing.

  In this way, the receiver 3 calculates the delay amount of the second feature signal based on the first feature signal, delays it by a time corresponding to the delay amount, and stores the delay amount in the first buffer unit 13. Since the first video signal is output to the first video decoding unit and the first audio signal is output to the first audio decoding unit, the first signal and the second signal are synchronized. Without adjusting the time stamp on the IP transmission server 4) side, the receiver 3 itself can perform synchronization control of signals transmitted from a plurality of different transmission paths.

  The first feature extraction unit 16 reduces the amount of information by sampling the first audio signal at a predetermined sampling frequency, Fourier-transforms the first audio signal with the reduced information amount, and obtains the first obtained by the Fourier transform. The configuration may be such that the first feature signal is generated by removing the high frequency component from one audio signal.

  The second feature extraction unit 22 reduces the amount of information by sampling the second audio signal at a predetermined sampling frequency, performs a Fourier transform on the second audio signal with the reduced information amount, and obtains a second obtained by the Fourier transform. The second feature signal may be generated by removing the high frequency component from the two audio signals.

Here, the processing procedure of the first feature extraction unit 16 will be described with reference to the flowchart shown in FIG. The processing procedure of the second feature extraction unit 22 is basically the same as the following procedure.
In step S11, the first feature extraction unit 16 performs sampling at a frequency fs (for example, 22 kHz) lower than the sampling frequency (for example, 44.1 kHz) used for the first audio signal in the broadcast station 1. . This step has the effect of thinning out the information of the first audio signal and reducing the entire data amount. Note that the sampling frequency fs is variable.
In step S12, the first feature extraction unit 16 performs a Fourier transform on the sampled first audio signal. By this step, the first audio signal is converted into frequency domain waveform information.
In step S13, the first feature extraction unit 16 removes the high frequency component from the first audio signal based on the cutoff frequency fc, and generates a first feature signal. Note that the cutoff frequency fc is variable.

  According to such a configuration, the receiver 3 generates the first feature signal by reducing the amount of information while leaving the characteristic portion of the first signal received by the first communication unit 11, and the second communication unit. Since the second feature signal is generated by reducing the amount of information while leaving the characteristic portion of the second signal received in step 17, the delay amount can be calculated efficiently in a short processing time in the delay processing unit 23 at the subsequent stage. Can do.

  The delay processing unit 23 calculates a correlation function based on the first feature signal and the second feature signal, performs an inverse Fourier transform on the calculated correlation function, and calculates a delay amount from the correlation function obtained by performing the inverse Fourier transform. The structure to do may be sufficient.

  Here, the processing procedure of the delay processing unit 23 will be described with reference to the flowchart shown in FIG. Hereinafter, the first feature signal is referred to as frequency domain waveform information X (f), and the second feature signal is referred to as frequency domain waveform information Y (f).

In step S21, the delay processing unit 23 calculates a correlation function Z (f) in the frequency domain.
Z (f) = X ^* (f) × Y (f) (1)
Note that * indicates conjugation.

In step S22, the delay processing unit 23 performs inverse fast Fourier transform (inverse FFT) to calculate a correlation function Z (t) in the time domain.
Z (t) = F ⁻¹ (X ^* (f) × Y (f)) (2)

  In step S23, the delay processing unit 23 calculates a delay amount τ. For example, the delay processing unit 23 calculates t that maximizes the correlation function Z (t), and sets the maximum value t as the delay amount τ.

  According to this configuration, the receiver 3 calculates a correlation function based on the first feature signal and the second feature signal, performs inverse FFT on the calculated correlation function, and delays from the correlation function that has performed inverse FFT. The first video signal stored in the first buffer unit 13 is output to the first video decoding unit 14 and the first audio signal is decoded by the first audio decoding. Since the first signal and the second signal are synchronized with each other, the receiver 3 itself transmits a plurality of different transmissions without adjusting the time stamp on the signal transmission (broadcast station 1 and IP transmission server 4) side. Synchronous control of signals transmitted from the path can be performed.

  In the present embodiment, the configuration and operation of the receiver that mainly controls the synchronization of signals transmitted from a plurality of different transmission lines without adjusting the time stamp on the transmitter side have been described. However, the present invention may be configured as a method and a program for performing synchronization control of signals transmitted from a plurality of different transmission paths without adjusting the time stamp on the transmitter side without including each component. .

  Further, the program for realizing the function of the receiver may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed.

  The “computer system” here includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a hard disk built in the computer system.

  Furthermore, “computer-readable recording medium” means that a program is dynamically held for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It is also possible to include one that holds a program for a certain time, such as a volatile memory inside a computer system that becomes a server or client in that case. Further, the program may be for realizing a part of the above-described functions, and may be capable of realizing the above-described functions in combination with a program already recorded in the computer system. .

DESCRIPTION OF SYMBOLS 1 Broadcasting station 2 Broadcasting antenna 3 Receiver 4 IP transmission server 11 1st communication part 12 1st isolation | separation part 13 1st buffer part 14 1st video decoding part 15 1st audio | voice decoding part 16 1st characteristic extraction part 17 2nd Communication unit 18 Second separation unit 19 Second buffer unit 20 Second video decoding unit 21 Second audio decoding unit 22 Second feature extraction unit 23 Delay processing unit 24 Reproduced clock generation unit 25 Output unit 100 Broadcast communication cooperation system

Claims (3)

A first communication unit that receives the first signal transmitted from the first transmission path;
A first separation unit for separating the first signal into a first video signal and a first audio signal;
A first buffer for storing the first video signal and the first audio signal;
A first video decoding unit for decoding the first video signal;
A first voice decoding unit for decoding the first voice signal;
A first feature extraction unit that extracts feature information from the first speech signal decoded by the first speech decoding unit to generate a first feature signal;
A second communication unit that receives a second signal transmitted from a second transmission path different from the first transmission path;
A second separation unit for separating the second signal into a second video signal and a second audio signal;
A second buffer for storing the second video signal and the second audio signal;
A second video decoding unit for decoding the second video signal;
A second voice decoding unit for decoding the second voice signal;
A second feature extraction unit that extracts feature information from the second speech signal decoded by the second speech decoding unit to generate a second feature signal;
A delay processing unit that calculates a delay amount of the second feature signal based on the first feature signal;
A reproduction clock generation unit that determines a timing for outputting a reproduction clock based on the delay amount calculated by the delay processing unit;
The first buffer unit outputs the first video signal to the first video decoding unit when the reproduction clock generated by the reproduction clock generation unit is input, and outputs the first audio signal to the first video signal. Output to the voice decoder,
The first feature extraction unit converts the first audio signal into information of a frequency domain waveform by Fourier transform,
The second feature extraction unit is a receiving device that converts the second audio signal into information of a frequency domain waveform by Fourier transform. The first feature extraction unit reduces the amount of information by sampling the first audio signal at a predetermined sampling frequency, and performs a Fourier transform on the first audio signal with the reduced information amount to perform a frequency domain. Converted into waveform information,
The second feature extraction unit reduces the amount of information by sampling the second audio signal at a predetermined sampling frequency, and performs a Fourier transform on the second audio signal with the reduced amount of information, thereby generating a frequency domain. The receiving apparatus according to claim 1, wherein the receiving apparatus converts into waveform information. The first feature extraction unit generates the first feature signal by removing a high frequency component from the first audio signal obtained by the Fourier transform,
The receiving apparatus according to claim 1 or 2, wherein the second feature extraction unit generates the second feature signal by removing a high frequency component from the second audio signal obtained by the Fourier transform.

Images