JP6935843B2 - Transmitter and transmission method, and receiver and reception method - Google Patents

Description

The techniques disclosed in the present specification include a transmission device and a transmission method for transmitting information to be overlaid and displayed on a video of a broadcast program, and a receiving device and a receiving device for receiving and displaying information to be overlaid and displayed on the video of a broadcast program. Regarding the receiving method.

Digital broadcasting services that use MPEG2-TS as the media transport method are being operated all over the world. Further, in the next-generation digital broadcasting standard, a change from MPEG2-TS to the MMT (MPEG Media Transport) system is being considered (see, for example, Patent Document 1). In the MMT method, it is easy to use a combination of different transmission lines, and it can be commonly used for a plurality of transmission lines for broadcasting and communication.

In digital broadcasting, a data broadcasting application can be transmitted by a broadcast wave, and various information linked to a broadcast program on the air can be provided through the data broadcasting application. For example, a broadcasting system for broadcasting commentary data to be displayed in a video frame together with video data of a program has been proposed (see, for example, Patent Document 2).

In addition, a proposal has been made for a broadcasting system that provides navigation information that reflects the contents of a moving image for each arbitrary section of a broadcasting program. Navigation information represents hyperlinks, buttons, textual information, etc. to other content. For example, in a broadcast program of travel information around the world, a menu consisting of a plurality of buttons according to the country being introduced and a plurality of navigation information representing supplementary explanations are provided (see, for example, Patent Document 3).

Japanese Unexamined Patent Publication No. 2014-2000 54 Special Table 2004-500770 Japanese Unexamined Patent Publication No. 2001-359090

An object of the technique disclosed herein is to provide an excellent transmission device and transmission method capable of transmitting information to be superimposed and displayed on a video of a broadcast program.

A further object of the technique disclosed herein is to provide an excellent receiving device and receiving method capable of receiving and displaying information to be superimposed and displayed on a video of a broadcast program.

The technology disclosed in the present specification has been made in consideration of the above-mentioned problems, and the first aspect thereof is.
A transmitter that transmits broadcast signals including video,
A data transmitter that transmits data for displaying an object that moves in synchronization with the movement of the video,
It is a transmission device provided with.

According to the second aspect of the technique disclosed herein, the data transmitter of the transmitter according to the first aspect provides information about the position and size of the object on the screen and the display data of the object. It is configured to transmit said data including.

According to the third aspect of the technique disclosed in the present specification, the data transmission unit of the transmission device according to the first aspect intermittently transmits the data together with a time stamp indicating the display time of the data. It is configured as follows.

According to a fourth aspect of the technique disclosed herein, the data transmitter of the transmitter according to the first aspect includes interpolation information for determining the interpolation position of the object until the next timing. The data is configured to be transmitted intermittently.

According to a fifth aspect of the technique disclosed herein, the data transmitter of the transmitter according to the fourth aspect contains the interpolated information including at least one of the velocity vector information or the angular velocity information of the object. Is configured to send.

According to a sixth aspect of the technique disclosed herein, the data transmitting unit of the transmitting device according to the first aspect is configured to transmit the data including the link destination information of the object. There is.

According to the seventh aspect of the technique disclosed herein, the data transmission unit of the transmitter according to the first aspect sets the position and size of the object on the screen and the reference destination of the display data in a predetermined format. It is configured to send the document file described in.

According to an eighth aspect of the technique disclosed herein, the data transmitter of the transmitter according to the seventh aspect packages the document file and a file of display data referenced from the document file. , It is configured to send in package units.

According to the ninth aspect of the technique disclosed in the present specification, the data transmission unit of the transmission device according to the eighth aspect specifies whether or not to retain each display data included in the package and the retention period. It is configured to send.

According to the tenth aspect of the technique disclosed herein, the data transmission unit of the transmitter according to the ninth aspect refers to a file of display data packaged and transmitted together with other document files. It is configured to send more document files.

In addition, the technology according to the eleventh aspect of the technology disclosed in the present specification is
A transmission step that transmits a broadcast signal including video,
A data transmission step of transmitting data for displaying an object that moves in synchronization with the movement of the video, and
It is a transmission method having.

In addition, the technology according to the twelfth aspect of the technology disclosed in the present specification is
A receiver that receives broadcast signals including video,
A data receiving unit that receives data for displaying an object that moves in synchronization with the movement of the image, and a data receiving unit.
A control unit that executes a process for displaying the object on the screen of the broadcast video based on the received data, and a control unit.
It is a receiving device provided with.

According to the thirteenth aspect of the technique disclosed herein, the data receiving unit of the receiving device according to the twelfth aspect provides information on the position and size of the object on the screen and the display data of the object. Upon receiving the data including the data, the control unit is configured to display the display data of the object at the position and size on the screen specified by the data.

According to the fourteenth aspect of the technique disclosed herein, the data receiving unit of the receiving device according to the twelfth aspect intermittently receives the data together with a time stamp indicating the display time of the data. The control unit is configured to control the display timing of the object based on the display time specified by the time stamp.

According to the fifteenth aspect of the technique disclosed herein, the data receiving unit of the receiving device according to the fourteenth aspect includes interpolation information for obtaining the interpolation position of the object until the next timing. The data is intermittently received, and the control unit is configured to display the object at an interpolation position obtained based on the interpolation information.

According to the sixteenth aspect of the technique disclosed herein, the data receiving unit of the receiving device according to the twelfth aspect sets the position and size of the object on the screen and the reference destination of the display data in a predetermined format. It is configured to receive the document file described in.

According to the 17th aspect of the technique disclosed in the present specification, the data receiving unit of the receiving device according to the 16th aspect is a package unit including the document file and a file of display data referenced from the document file. It is configured to receive at.

According to the eighteenth aspect of the technique disclosed herein, the data receiving unit of the receiving device according to the seventeenth aspect receives the package in which the necessity and retention period of each display data are specified. However, the control unit is configured to hold the display data until the retention period expires.

According to the nineteenth aspect of the technique disclosed herein, the data receiving unit of the receiving device according to the eighteenth aspect refers to a file of display data packaged and transmitted together with other document files. Upon further receiving the document file, the control unit is configured to display the object using the display data that is packaged and transmitted together with other document files referenced from the document file.

In addition, the technology according to the twentieth aspect of the technology disclosed in the present specification is
A reception step for receiving a broadcast signal including video,
A data receiving step that receives data for displaying an object that moves in synchronization with the movement of the image, and
A control step that executes a process for displaying the object on the screen of the broadcast video based on the received data, and
It is a receiving method having.

According to the technique disclosed in the present specification, there is provided an excellent transmission device and transmission method capable of transmitting display information of an object to be overlaid in accordance with the movement of a video of a broadcast program in time synchronization with the video. can do.

Further, according to the technique disclosed in the present specification, an excellent receiving device and receiving device capable of receiving display information of an object in time synchronization with a broadcast signal and overlaying the object according to the movement of an image. A method can be provided.

The effects described in the present specification are merely examples, and the effects of the present invention are not limited thereto. In addition, the present invention may exert additional effects in addition to the above effects.

Still other objectives, features and advantages of the techniques disclosed herein will become apparent by more detailed description based on embodiments and accompanying drawings described below.

FIG. 1 is a diagram schematically showing a configuration example of the digital broadcasting system 10. FIG. 2 is a diagram showing a protocol stack 200 in a digital broadcasting system 10 using the MMT method. FIG. 3 is a diagram showing a configuration example of the broadcast transmission system 11. FIG. 4 is a diagram showing a configuration example of a receiver 12 that receives a broadcast signal from the broadcast transmission system 11. FIG. 5 is a diagram showing an image of a broadcast signal 500 transmitted by the MMT method. FIG. 6 is a diagram showing a mechanism for designating each asset related to the broadcasting service from the MP table in the PA message. FIG. 7 is a diagram showing a syntax example 700 of the MP table. FIG. 8 is a diagram showing a syntax example 800 of the MPU time stamp descriptor. FIG. 9 is a diagram for explaining a mechanism for acquiring a file related to data broadcasting transmitted by MMT. FIG. 10 is a diagram showing the configuration of MPU and MFU of general-purpose data. FIG. 11 is a diagram showing a syntax example 1100 of MFU of general-purpose data. FIG. 12 is a diagram showing a display example of a broadcast screen displaying a display object synchronized with a video of a broadcast program. FIG. 13 is a diagram showing a configuration example of an MPU that transmits a general-purpose data stream in which a moving object list description document and display data are packaged. FIG. 14 is a diagram showing a display example of a display object described by the moving object list description document shown in FIG. FIG. 15 is a diagram showing a configuration example of an MPU consisting only of the MFU of the moving object list description document, which does not include display data. FIG. 16 is a diagram showing a syntax example 1600 of additional information (mold_info) in an MFU that transmits a moving object list description document. FIG. 17 is a diagram showing a syntax example 1700 of additional information (material_info) in an MFU that transmits a material (display data). FIG. 18 is a diagram showing a syntax example 1800 of a moving object list description document. FIG. 19 is a diagram for explaining a mechanism for interpolating the display position of the display object based on the interpolation information. FIG. 20 is a diagram illustrating a scenario in which a moving object list description document is transmitted by a general-purpose data stream. FIG. 21 is a flowchart showing a processing procedure when the receiver receives a general-purpose data stream. FIG. 22 is a flowchart showing a processing procedure for overlaying a display object that the receiver synchronizes with the video based on the general-purpose data stream.

Hereinafter, embodiments of the techniques disclosed in the present specification will be described in detail with reference to the drawings.

A. System Configuration FIG. 1 schematically shows a configuration example of the digital broadcasting system 10. The illustrated digital broadcasting system 10 includes a broadcasting transmission system 11 and a receiver 12.

The broadcast transmission system 11 applies the MMT method to the transmission of broadcast signals, and transmits each component constituting the broadcast service as an IP (Internet Protocol) packet. Specifically, the broadcast transmission system 11 puts the code of the video signal and the audio signal of the broadcast program, and the signal of the content (data broadcasting application, etc.) and the subtitle related to the broadcast program on the MMTP (MMT Protocol) payload. Then, it is converted into an MMTP packet, further converted into an IP packet, and transmitted in the form of a TLV (Type Length Value) packet on a broadcasting transmission line. Here, the components related to the main body of the broadcast program such as video, audio, and subtitles are synchronous media. Further, the content, that is, the data broadcasting application encoded in the HTML (HyperText Transfer Protocol) 5 format or the like is an asynchronous (Non-Timed) medium.

The broadcast transmission system 11 also performs MMT-SI signaling (notification), which is control information related to a broadcast service that transmits synchronous and asynchronous media. The MMT-SI has a message, a table, and a descriptor. The table carries messages as containers. Some of the parameters shown in messages and tables are described in the form of descriptors.

On the other hand, the receiver 12 receives the TLV packet sent from the broadcast transmission system 11 on the broadcast transmission line. The receiver 12 then decodes the synchronous transmission media constituting the broadcast program such as video, audio, and subtitles from the received packet, and presents the image, audio, and subtitles. Further, when the receiver 12 acquires each data file for data broadcasting from the received packet, the receiver 12 activates an application engine such as an HTML browser to display the data broadcasting linked to the broadcasting program and the video of the TV program. Display in a part or all area of the screen.

The receiver 12 also receives MMT-SI, which is control information related to the broadcasting service. The receiver 12 controls reception of transmission media such as video, audio, subtitles, and data broadcasting, and output (display, audio output) control on the receiver 12 based on the MMT-SI.

FIG. 2 shows a protocol stack 200 in a digital broadcasting system 10 using the MMT method.

One broadcasting service is composed of video 201, audio 202, subtitle 203, application 204, and content download 205 components. The video 201 is encoded 211 in the HEVC (High Efficiency Video Coding) format, the audio 202 is encoded 212 in the AAC (Advanced Audio Coding) format, and the subtitle 203 is subtitle-encoded 213. Application 204 also includes an EPG (Electronic Program Guide), but is encoded 214 in, for example, HTML5 format.

On the MMT layer 220, the video signal and audio signal coding components 211 and 212 of the broadcast program are MFU (Media Fragment Unit) and MPU (Media Processing Unit), and are put on the MMTP payload and converted into MMTP packets. Further, the subtitle signal of the broadcast program and the application (data / content) coding components 213 and 214 related to the broadcast program are also referred to as MFU and MPU, and are put on the MMTP payload and converted into MMTP packets. Further, MMT-SI221, which is control information (indicating the configuration of a broadcast program) related to MMT, which is a media transport method, is also put on the MMTP payload and converted into an MMTP packet. Details of the control information MMT-SI related to MMT will be described later. The MPU is a unit of processing, and the MFU is a smaller unit than the MPU. The data transmission method 215 of the content download 205 includes four types: a subtitle / character super transmission method, an application transmission method, an event / message transmission method, and a general-purpose data transmission method. Of these, the application transmission method is used for data transmission services asynchronous with broadcast programs. In addition, the general-purpose data transmission method is a method of transmitting various types of data synchronously or asynchronously, and is used for data used by a player who presents data other than video, audio, and subtitles, and data used in a multimedia service. It can be applied to streaming.

In UDP (User Datagram Protocol) / IP layer 530, MMTP packets are converted into IP packets. In addition, the NTP (Network Time Protocol) packet 206 containing the current time information for the synchronous media is also converted into an IP packet. Further, these IP packets are converted into TLV packets by the TLV layer 240 and transmitted on the broadcast transmission line 250 which is the lowest physical layer. Further, the TLV-SI241 related to the TLV multiplexing format for multiplexing IP packets is also converted into a TLV packet and transmitted on the broadcast transmission line 250. The transmission slot in which the TLV packet is multiplexed is identified by using the TLV stream identification information (TLV_stream_id) from the TMCC (Transmission and Multiplexing Control) signal 251 of the transmission line.

FIG. 3 shows a configuration example of a broadcast transmission system 11 that transmits a broadcast signal having a protocol stack structure as shown in FIG. The illustrated broadcast transmission system 11 includes a clock unit 301, a signal transmission unit 302, a video encoder 303, an audio encoder 304, a subtitle / character super encoder 305, a signaling encoder 306, and a file encoder 307. An electronic data processing system (Electronic Data Processing System: EDPS) 308, a TLV signaling encoder 309, an IP service multiplexer (MUX) 310, a TLV multiplexer (MUX) 311 and a modulator / transmitter 312. I have.

The clock unit 301 generates time information synchronized with the time information acquired from the NTP server (not shown), and sends an IP packet including this time information to the IP service multiplexer 310.

The signal transmission unit 302 is, for example, a recording / playback device such as a studio of a TV broadcasting station or a VTR, and transmits stream data such as video, audio, subtitles, and character supermarkets, which are synchronous media, to a video encoder 303 and an audio system, respectively. Send to encoder 304, subtitle / character super encoder 305. Further, the signal transmission unit 302 sends a data broadcasting application which is an asynchronous medium and general-purpose synchronous and asynchronous data to the file encoder 307.

The electronic data processing system 308 is a scheduler of a TV broadcasting station and a source of files. The electronic data processing system 308 sends data broadcasting applications, which are asynchronous media, and synchronous and asynchronous general-purpose data to the file encoder 307. Further, the electronic data processing system 308 sends control information indicating the configuration of the broadcasting service to the signaling encoder 306. The electronic data processing system 308 also sends control information regarding IP packet multiplexing to the TLV signaling encoder 309.

The video encoder 303 HEVC-encodes the video signal transmitted from the signal transmission unit 302, further packetizes it, and sends an IP packet including an MMT packet of the video signal to the IP service multiplexer 310. Further, the audio encoder 304 AAC-encodes the audio signal transmitted from the signal transmission unit 302, further packetizes it, and sends an IP packet including the MMT packet of the audio signal to the IP service multiplexer 310. Further, the subtitle / character super encoder 305 encodes the subtitle signal and the character super signal transmitted from the signal transmission unit 302, further generates an MPU in the unit of presentation processing, and generates an IP packet including the MMT packet of the subtitle. Send to IP service multiplexer 310.

When the broadcast transmission system 11 transmits a plurality of broadcast services, HEVC coding of the video signal, AAC coding of the audio signal, and coding processing of the subtitle / character super are performed for each broadcast service, and each broadcast service # It is sent to IP service multiplexers 310-1, ..., 310-N corresponding to 1 to # N.

The signaling encoder 306 generates control information (MMT-SI) indicating the configuration of a broadcast signal based on the information transmitted from the electronic data processing system 308, and the MMT packet in which the MMT-SI is arranged in the payload section. The IP packet containing the above is sent to the IP service multiplexer 310. The MMT-SI has messages, tables, and descriptors (described above). The signaling encoder 306 generates MMT-SI for each broadcast service and also generates MMT-SI across broadcast services.

The file encoder 307 encodes general-purpose data used in an HTML5 format data broadcasting application or data broadcasting application transmitted from the signal transmission unit 3102 or the electronic data processing system 308, further packetizes the MMT packet, and converts the MMT packet into a packet. The containing IP packet is sent to the IP service multiplexer 310.

The broadcast transmission system 11 is equipped with a plurality of IP service multiplexers 310-1, ..., 310-N for each broadcast service (broadcast channel) # 1 to #N to be transmitted. The IP service multiplexer 310 of the channel for each broadcasting service is an IP including each of video, audio, subtitles, (broadcasting service unit) MMT-SI, data broadcasting application, and general-purpose data sent from each encoder 303 to 307. The packet is multiplexed to generate a TLV packet including a broadcast signal for each broadcast service and MMT-SI across the broadcast service.

The TLV signaling encoder 309 generates a TLV packet that arranges the control information (TLV-SI) regarding the multiplexing of the IP packet in the payload unit based on the information transmitted from the electronic data processing system 308.

The TLV multiplexer 311 multiplexes the TLV packets generated by each IP service multiplexer 310-1 to 10-N and the TLV signaling encoder 309, respectively, to generate a TLV stream identified by the TLV stream identification information. do.

The modulation / transmission unit 312 performs RF modulation processing on the TLV stream generated by the TLV multiplexer 311 and sends it to the broadcast transmission line.

The operation of the broadcast transmission system 11 shown in FIG. 3 will be described.

The video signal transmitted from the signal transmission unit 302 is supplied to the video encoder 303. In the video encoder 303, the video signal is HEVC-encoded and further packetized to generate an IP packet including an MMT packet of the HEVC-encoded video signal. This IP packet is sent to the IP service multiplexer 310.

Further, the same processing is performed on the audio signal, the subtitle, and the character super signal transmitted from the signal transmission unit 302. That is, an IP packet including an MMT packet of the AAC-encoded audio signal generated by the audio encoder 304 is sent to the IP service multiplexer 310, and the subtitle-encoded signal generated by the subtitle / character super encoder 305. An IP packet, including an MMT packet, is sent to the IP service encoder 310.

Further, in the signaling encoder 306, control information (MMT-SI) indicating the configuration of a broadcast signal or the like is generated based on the information transmitted from the electronic data processing system 308, and this MMT-SI is arranged in the payload section. An IP packet including an MMT packet is generated. These IP packets are sent to the IP service multiplexer 310.

Further, general-purpose data used in a data broadcasting application, a data broadcasting application, or the like transmitted from the signal transmitting unit 302 or the electronic data processing system 308 is supplied to the file encoder 307. In the file encoder 307, HTML5 format data broadcasting application and general-purpose data are encoded and further packetized, and an IP packet including this MMT packet is generated. This IP packet is sent to the IP service multiplexer 310.

In each IP service multiplexer 310-1, ..., 310-N, video, audio, subtitles, MMT-SI (broadcast service unit and cross-service), data broadcasting application (cross-broadcast service unit and service) sent from each encoder 303-307 (broadcast service unit and cross-service). (HTML5 document), IP packets containing each of general-purpose data are multiplexed to generate TLV packets constituting one channel.

The TLV signaling encoder 309 generates a TLV packet in which the control information (TLV-SI) relating to the multiplexing of the IP packet is arranged in the payload unit based on the information transmitted from the electronic data processing system 308.

In the TLV multiplexer 311, the TLV packets generated by each IP service multiplexer 310-1 to 10-N and the TLV signaling encoder 309 are multiplexed to generate a TLV stream. The modulation / transmission unit 312 performs RF modulation processing on the TLV stream generated by the TLV multiplexer 311 and sends the RF modulation signal to the broadcast transmission line.

FIG. 4 shows a configuration example of the receiver 12 that receives the broadcast signal from the broadcast transmission system 11. The receiver 12 shown is a tuner / demodulation unit 401, an MMT demultiplexer (DEMUX) 402, a clock recovery unit 403, a video decoder 404, an audio decoder 405, a character super decoder 406, and subtitles. Decoder 407, multimedia (MM) cache 408, SI cache 409, broadcasting system control unit 410, application (App) engine 411, communication interface (IF) 412, scaler 414, and synthesizer 415- It is equipped with 418.

The tuner / demodulation unit 401 selects and receives a broadcast signal, performs demodulation processing, and obtains a TLV stream. The MMT demultiplexer 402 performs demultiplexing processing and depackaging processing on the TLV stream. The demultiplexer 402 includes a packet filter 402-1 and an SI filter 402-2.

The packet filter 402-1 filters IP packets based on the TLV stream identification information and the IP address, and further filters the MMTP packets from the IP packets based on the information in the MMTP header to obtain video, audio, and characters. Synchronous transmission media such as super and subtitles are distributed to a video decoder 404, an audio decoder 405, a character super decoder 406, and a subtitle decoder 407, respectively. Further, the packet filter 402-1 distributes each coding component of multimedia (data broadcasting application) and general-purpose data to the multimedia (MM) cache 408. Further, the packet filter 402-1 distributes the MMTP packet carrying the signaling information to the SI filter 402-2. Then, the SI filter 402-2 filters the signaling information SI and caches it in the SI cache 410.

The clock recovery unit 403 generates time information synchronized with this time information based on the current time information contained in the NTP packet filtered by the packet filter 402-1 in the MMT demultiplexer 402, and each of them generates time information synchronized with this time information. To decode the synchronous transmission media, the output is output to the video decoder 404, the audio decoder 405, the character super decoder 406, and the subtitle decoder 407, respectively.

The video decoder 404 decodes the coded video signal obtained by the MMT demultiplexer 402 to obtain a baseband video signal. Further, the audio decoder 405 decodes the coded audio signal obtained by the MMT demultiplexer 402 to obtain a baseband audio signal. Further, the character super decoder 406 and the subtitle decoder 407 decode the character super and subtitle coded signals obtained by the MMT demultiplexer 402, respectively, to obtain the character super and subtitle display signals, respectively.

The broadcast system control unit 410 controls the broadcast service reception operation of the entire receiver 12 based on the MMT-SI cached in the SI cache 409. For example, when the broadcasting system control unit 410 analyzes the execution control information (MH-AIT) of the application included in the MMT-SI and finds the data broadcasting application instructed to start, the broadcasting system control unit 410 determines the application engine 411. Instruct the data broadcasting presentation process.

The application engine 411 is, for example, an HTML browser or the like, and processes a data broadcasting application (HTML5 document or the like) cached in the multimedia cache 408 to generate a data broadcasting display signal and an audio signal. In addition, the application engine 411 acquires data files (media data used for displaying data broadcasting, linked applications, etc.) necessary for displaying data broadcasting from the IP network via the communication interface 412. You can also.

Further, in the present embodiment, it is assumed that general-purpose data used in the data broadcasting application is streamed. The application engine 411 acquires general-purpose data to be streamed and reflects it in the display of the application.

The scaler 414 scales the video signal (broadcast video) decoded by the video decoder 404 according to the screen size of the receiver 12.

The synthesis unit 415 synthesizes the audio signal decoded by the audio decoder 405 and the audio signal for data broadcasting reproduced by the application engine 411 to generate an audio signal for output.

The compositing unit 416 synthesizes the character super display decoded by the character super decoder 4206 and the subtitle display decoded by the subtitle decoder 407. Further, the synthesis unit 417 synthesizes the broadcast video scaled by the scaler 414 and the display signal of the data broadcast generated by the application engine 411. Further, the synthesis unit 418 in the latter stage synthesizes the broadcast video on which the display of the data broadcast is superimposed and the character super display and the subtitle display output from the synthesis unit 416 to generate a video signal for output.

The operation of the receiver 12 shown in FIG. 4 will be described.

The tuner / demodulation unit 401 receives the broadcast signal, performs demodulation processing, and obtains a TLV stream. In the MMT demultiplexer 402, the TLV stream is subjected to demultiplexing processing and packetization processing, and encoding of NTP time information, video, audio, character superimposition and subtitles, data broadcasting application, and general-purpose data. The signal and signaling information are extracted and distributed to a video decoder 404, an audio decoder 405, a character super decoder 406, a subtitle decoder 407, a multimedia (MM) cache 408, and an SI filter 402-1.

Further, the NTP packet extracted by the demultiplexer 402 is distributed to the clock recovery unit 403. The clock recovery unit 403 generates time information synchronized with this time information based on the time information carried in the NTP packet. That is, the clock recovery unit 403 generates time information that matches the time information generated by the clock unit 401 on the broadcast transmission system 11 side. The generated time information is output to the video decoder 404, the audio decoder 405, the character super decoder 406, and the subtitle decoder 407 to decode each synchronous transmission medium.

The coded video signal extracted by the MMT demultiplexer 402 is sent to the video decoder 404 for decoding, and a baseband video signal is obtained. The character super-encoded signal extracted by the demultiplexer 402 is sent to the character super decoder 406 for decoding, and a character super display signal is obtained. Further, the subtitle coded signal extracted by the demultiplexer 402 is sent to the subtitle decoder 407 for decoding, and a subtitle display signal is obtained.

The broadcasting system control unit 410 controls the entire reception processing operation by the receiver 12 including the processing of the data broadcasting application based on the MMT-SI received via the SI filter 402-2 and the SI cache 409.

The application engine 411 processes a data broadcasting application (HTML5 document, etc.) cached in the multimedia cache 408, and generates a data broadcasting display signal and an audio signal. Further, in the present embodiment, it is assumed that general-purpose data used in the data broadcasting application is streamed. In the application engine 411, the acquired general-purpose data is reflected in the display of the application.

In the scaler 414, scaling processing of the video signal (broadcast video) decoded by the video decoder 404 is performed. Further, the compositing unit 416 synthesizes the character super display decoded by the character super decoder 406 and the subtitle display decoded by the subtitle decoder 407. Subsequently, the compositing unit 417 synthesizes the broadcast video scaled by the scaler 414 and the display signal of the data broadcast generated by the application engine 411. Further, in the subsequent synthesis unit 418, the broadcast video on which the data broadcast display is superimposed and the character super display and the subtitle display output from the synthesis unit 416 are combined to generate a video signal for output.

Further, the synthesis unit 415 synthesizes the audio signal decoded by the audio decoder 405 and the audio signal for data broadcasting reproduced by the application engine 411 to generate an audio signal for output.

B. Configuration of Broadcast Signals FIG. 5 shows an image of a broadcast signal 500 transmitted from a broadcast transmission system 11 to a receiver 12 on a broadcast transmission line according to the MMT method.

Broadcast signals are composed of synchronous media such as video, audio, and subtitles related to the main part of the broadcast program, and asynchronous media such as data broadcasting applications. In the MMT method, it is easy to use these synchronous and asynchronous media in a combination of different transmission lines. In the example shown in FIG. 5, as the broadcast signal 500, MMT transmission lines 501 to 506 for each type of data such as video, audio, subtitles, data broadcasting applications, general-purpose data, and control information (MMT-SI) related to broadcasting services are used. Has been done.

In each MMT transmission line, the encoded media data is converted into an MPU format, converted into an MMTP packet, and transmitted as an IP packet. Each MMT transmission line corresponds to one IP data flow. The IP data flow referred to here means that the values of the five fields of the source IP address, the destination IP address, the protocol type of the IP header, the source port number, and the destination port number of the IP header and the UDP header are all the same. Is a set of IP packets.

One broadcast service is a package composed of a plurality of components such as video, audio, subtitles, and data broadcasting application (however, one broadcast service corresponds to one channel, that is, a broadcast program). In the MMT transmission scheme, components are defined as "assets". Each asset is identified by unique asset identification information (asset_id). The asset identification information uniquely corresponds to the component tag that identifies the component (video assets correspond to video components, audio assets correspond to audio components, subtitle assets correspond to subtitle components, and data broadcasting. The data file assets for the application correspond to the data file components).

Each asset is composed of a set (logical group) of one or more MPUs that share the same asset identification information. The MPU is a format that serves as a transmission unit in the MMT system, and is composed of one or more MFUs. The MPU is transmitted to each asset via a dedicated ES (Elementary Stream), that is, an MMT transmission line. Each MPU is identified by the asset identification information and the sequence number of the MPU on the corresponding transmission line.

In the example shown in FIG. 5, the transmission line 501 transmits an MMTP packet of a coded video signal composed of an MPU logical group having common asset identification information for the video signal. Similarly, the transmission line 502 transmits an MMT packet of a coded voice signal consisting of an MPU logic group having common asset identification information for voice signals, and the transmission line 503 has common asset identification information for subtitle signals. The MMTP packet of the coded subtitle signal consisting of the MPU group is transmitted, and the MMTP packet of the coding application consisting of the MPU logical group having common asset identification information for the data broadcasting application is transmitted on the transmission lines 504-1 and 504-2. Will be done. Further, in the transmission line 505, it is transmitted as an MMTP packet of a coded general-purpose data signal composed of an MPU group having common asset identification information.

Multiple assets of the same type may be transmitted within one package (broadcasting service). In the example shown in FIG. 5, the assets of the two data broadcasting applications are assigned different asset identification numbers and are transmitted on dedicated ESs, that is, MMT transmission lines 504-1 and 504-2, respectively. For example, on one MMT transmission line 504-1, the assets of a program-linked data broadcasting application linked to a broadcast program are transmitted, and on the other MMT transmission line 504-2, a program non-linked data broadcasting application not linked to a broadcast program ( Assets such as weather forecasts and news) are transmitted. It is also assumed that the assets of the data broadcasting application will be provided independently from two or more delivery segments.

Further, in the MMT method, a plurality of transmission lines for broadcasting and communication can be commonly used. For example, asynchronous media such as data broadcasting (HTML5 document, etc.) is transmitted together with synchronous media of the same broadcasting service using transmission lines 504-1 and 504-2 of the broadcasting signal 500 as shown in FIG. In addition, transmission can also be performed via a communication transmission line (not shown) such as an IP network.

The transmission line 505 is used to transmit various data by a general-purpose data transmission method in a broadcasting system using MMT. General-purpose data transmission can be used for streaming data to HTML5 format data broadcasting applications.

The transmission line 506 is used for signaling in the MMT system, that is, transmission of MMT-SI, which is control information indicating information related to the configuration of the MMT package and the broadcasting service. On the transmission line 506, the MMT message including the MMT-SI is converted into an MMTP packet and repeatedly transmitted by the carousel method. In FIG. 5, the transmission line for TLV-SI is not shown.

MMT-SI has messages, tables, and descriptors. Transmission line 506 Examples of the message to be transmitted include PA message 510, M2 section message 520, and data transmission message 530.

The PA message 510 is control information indicating the configuration of a broadcast program, and is a container for storing an MP (MMT Package) table 511 that describes information constituting the package such as a list of assets and their positions.

The MP (MMT Package) table 511 is a table showing basic reception control information for each broadcasting service, and specifically, gives information constituting a package such as a list of assets and location information of assets. The MP table 511 also includes MPT descriptors such as application service descriptors. The application service descriptor indicates entry information of a data broadcasting application (for example, entry information of a data transmission message or each M2 section message that transmits MH-AIT and EMT, respectively). Further, an MPU time stamp descriptor that provides an MPU presentation time is arranged in the MPU table 511.

The PA message 510 is an entry point of a broadcasting service, and fixed packet identification information (for example, 0x0000) is assigned to the MMTP packet that transmits the PA message 510. Therefore, on the receiver side, the PA message 510 can be acquired by directly designating the fixed packet identification information on the MMT transmission line 506. Then, with reference to the MP table 511 transmitted in the PA message 510, each asset (video, audio, subtitle, data broadcasting application, general-purpose data) and other signaling messages constituting the package (broadcast program) are indirectly specified. can do.

Although omitted in FIG. 5, examples of the table stored in the PA message 510 include a PL (Package List) table (PLT) and an LC (Layout Configuration) table (LCT) in addition to the MP table. The PLT shows a list of IP data flows and packet IDs that transmit PA messages of the MMT package provided as a broadcast service and IP data flows that transmit the IP service. Further, the LCT is used to associate the layout information for presentation with the layout number. The PA message is a container that can store up to three tables, an MP table, a PL table, and an LC table.

The M2 section message 520 is a message that transmits the section extension format of the MPEG-2 Systems, and is a container that stores the section format signaling tables one by one. In the M2 section message 520, tables such as MH-AIT (Application Information Table) 521 and EMT (Even Message Table) 522 are stored one by one.

The MH-AIT521 is a table that specifies dynamic control information about the application and additional information necessary for execution. Specifically, in the application information loop, the MH-AIT521 indicates the processing method of each application to be controlled by the table by application_control_code, and arranges various application information descriptors that describe the information of the application. Here, application_control_code describes application processing methods such as application (automatic startup of application), present (state in which application can be executed), kill (termination of application), and presentation (acquisition and retention of application (pre-cache)). show. In addition, the transmission protocol descriptor, which is one of the application information descriptors, specifies the location information according to the transmission method of the application and the transmission method (common when the application is HTTP / HTTP and MMT asynchronous transmission). Then, specify the location information in the URL (Uniform Location Protocol) format).

The EMT522 is a signaling table used in the event message transmission method, and stores information (event message descriptor) regarding an event message (a synchronous / asynchronous message from a broadcasting station to an application on a receiver). The event message transmission method provides a means for transmitting message information from a broadcasting station to a data broadcasting application running on a receiver immediately or at a specified time.

The data transmission message 530 is a message for transmitting control information regarding transmission of a data broadcasting application. The data transmission message 530 includes a data directory management table (Data Directory Management Table: DDMT) 531 and a data asset management table (Data Asset Management Table: DAMT) 532, and a data content management table (Data ContactControl) 532. It is a container that can store up to three tables at the same time. The data directory management table and the data asset management table are essential for the data broadcasting service, but the data content management table is optional.

Here, the data directory management table 531 is a table for managing the data broadcasting application in the directory unit (in other words, the production unit of the data broadcasting application). Since the table describes the directory structure for items (directories and subdirectories and files contained in directories) contained in one package, it is possible to separate the file structure of the application from the file transmission structure. can. The table shows the correspondence between the path name of each node such as a directory or file and the node tag that identifies each node in the data transmission message.

Further, the data asset management table 532 is a table for managing the data broadcasting application for each asset, and describes the version information for each MPU with the configuration of the MPU in the asset. Specifically, the table identifies the download identification information of each component and the node tag (file or directory) included in each MPU that transmits the component in the data transmission message and the node on the MMT transmission path. Indicates the correspondence with the item identification information to be identified.

Further, the data / content management table 533 is a table for managing the data broadcasting application for each presentation unit (PU). The table shows presentation unit identification information that identifies a data broadcasting application for each presentation unit, and a node tag that identifies a node (file or directory) included in each presentation unit in a data transmission message. In addition, the table can include link destination presentation unit identification information that identifies the presentation unit linked to the presentation unit and information related to cache control of each presentation unit, and file data for a data broadcasting application on the receiver side. It can be used for flexible and effective cache control.

In the transmission method of a data broadcasting application by MMT, by utilizing the above three types of signaling tables 531 to 533 transmitted by a data transmission message, a transmission data structure for each file and a directory structure for content (data broadcasting application) production can be obtained. Can independently represent the data structure of the usage unit such as application unit and presentation unit. The configuration of each signaling table 531 to 533 is specified in ARIB STD B-60.

The packet identification information of the message or table transmitted as MMT-SI may be fixed or indirectly specified from another table. Of these, the PA message is the entry point of the broadcasting service and is assigned fixed packet identification information (for example, 0x0000). The MP table transmitted by the PA message shows the asset identification information and the location information of each asset (video, audio, subtitle, data broadcasting application, etc.) constituting the package (broadcast program). Therefore, on the receiver side, the PA message can be acquired by directly designating the fixed packet identification information on the MMT transmission path. Then, as shown in FIG. 6, each asset (video, audio, subtitle, data broadcasting application, etc.) related to the broadcasting service and other signaling messages are indirectly transmitted by referring to the MP table transmitted by the PA message. Can be specified in. The MP table arranges an MH-data coding method descriptor for each asset other than video and audio in order to identify the data coding method.

FIG. 7 shows a syntax example 700 of the MP table. The meaning of each parameter of the MP table will be described below.

table_id is an 8-bit fixed value (0x20) that identifies the table as an MP table. version is an 8-bit integer value indicating the version number of the MP table. For example, if some parameters (descriptors) that make up the MP table are also updated, the version is incremented by +1. Length is a 32-bit length parameter that indicates the size of the MP table in bytes, which is counted immediately after this field. In addition, MPT_mode shows the operation when this MP table is divided into subsets.

MMT_package_id_length indicates the size of the text information of the package identification information (MMT_package_id) in bytes. In the subsequent package identification information loop, the package identification information is indicated in byte units (MMT_package_id_byte) for MMT_package_id. The package identification information is identification information as an entire package having assets such as all signals (video, audio, subtitles) transmitted by a broadcast signal (IP data flow) and file data as components. This identification information is text information, and the upper 16 bits have the same value as the service identification information for identifying the service.

MPT_descriptor_length indicates the size of the MP table descriptor area in bytes. In the MP table descriptor loop, the contents of the MP table descriptor are described in byte units (MPT_descriptors_byte). The MPT descriptor describes basic information related to reception control of the entire corresponding broadcasting service (package).

number_of_assets is an 8-bit parameter indicating the number of assets (signals, files) as elements constituting a broadcasting service (package). Loops of asset information are arranged for the number of number_of_asset. In one asset information loop, each parameter of asset identification information (asset_id) as information of individual assets, general location information (MMT_general_location_info), and asset descriptor (asset_descriptor) is arranged. The information arranged in the asset information loop will be described below.

identifier_type indicates the ID system of the MMTP packet flow. If it is an ID system indicating asset identification information (asset_id), it is set to 0x00. assert_id_scheme indicates the format of the asset identification information. assert__id_length indicates the size of the text information of the asset identification information in bytes. In the subsequent asset identification information loop, the asset identification information is indicated in byte units (asset_id_byte).

assert_type indicates the type of asset (video, audio, superimpose / subtitle, application, etc.) as a 32-bit long character string. assert_clock_relation_flag is a flag indicating the presence or absence of the clock information field of the asset. When the flag is 1, the clock information identification field (asset_clock_relation_id) and the timescale flag field (asset_timescale_flag) are present, and when the flag is 0, these fields are not present. The location_count indicates the number of location information of the asset, and in the location information loop repeated by the number of subsequent location_counts, the location information of the corresponding asset, MMT_general_location_info, is indicated.

assert_type indicates the type of asset (video, audio, superimpose / subtitle, application, etc.) as a 32-bit long character string. The types of assets represented by asset_type (characters) are shown in Table 1 below.

assert_clock_relation_flag is a flag indicating the presence or absence of the clock information field of the asset. When the flag is 1, the clock information identification field (asset_clock_relation_id) and the timescale flag field (asset_timescale_flag) are present, and when the flag is 0, these fields are not present. The location_count indicates the number of location information of the asset, and in the location information loop repeated by the number of subsequent location_counts, the location information of the corresponding asset, MMT_general_location_info, is indicated. The location information of the asset is described in the form of packet identification information (packet_id: PID) on the MMT transmission path to which the asset is acquired. Therefore, the asset identification information can be pulled on the MP table to retrieve the corresponding packet identification information on the MMT transmission line (IP data flow) (see FIG. 6).

assert_descriptor_length indicates the size of the text information of the asset descriptor (asset_descriptor) in bytes. In the asset descriptor loop indicated by the reference number 701, the contents of the descriptor for each asset are indicated in byte units (asset_descriptors_byte). As the asset descriptor placed in the asset information loop (second loop of the MP table), the "MH-data encoding method descriptor" used to identify the data coding method and the presentation time of the MPU are displayed. "MPU Timestamp Descriptor" to provide, "MPU Timestamp Descriptor" to provide decryption time of access units in MPU, "Asset Group" to provide group relations of assets and priority within group Examples thereof include a "descriptor", a "MPU presentation area specification descriptor" that provides the presentation position of the MPU, and a "dependency descriptor" that provides the asset ID of the dependent asset.

FIG. 8 shows a syntax example 800 of the MPU timestamp descriptor arranged in the asset information loop of the MP table. The MPU timestamp descriptor indicates the presentation time of the first access unit in the presentation order in the MPU. The meaning of each parameter of the MPU time stamp descriptor will be described below.

descriptor_tag indicates an 8-bit fixed value that identifies the descriptor 800. descriptor_length is an 8-bit area for writing the byte length of the data of the descriptor following this field.

mpu_sequence_number indicates the sequence number of the MPU that describes the time stamp in the descriptor. mpu_presentation_time indicates the presentation time of the MPU indicated by the above sequence number in a 64-bit NTP time stamp format.

In MH-AIT, the location information of each application is described in URL format, in other words, a path name consisting of a combination of a directory node name and a file name. On the other hand, the path name of the asset can be converted into the identification information (packet identification information) on the MMT transmission path by using the data directory management table and the data asset management table transmitted by the data transmission message. With reference to FIG. 9, a mechanism for acquiring a file related to data broadcasting transmitted by MMT will be briefly described.

The receiver refers to the application information loop in the MH-AIT transmitted by the M2 section message to detect the application to be started (for example, the application for which automatic start (autostart) is specified by the application control code). be able to. In addition, the receiver can acquire the location information of the application, that is, the path name from the transmission protocol descriptor arranged in the MH-AIT. In the case of a broadcast-transmitted application, the node tag of the file corresponding to the specified pathname can be obtained from the data directory management table in the data transmission message, as indicated by reference number 901.

Then, as shown by reference number 902, from the data asset management table also in the data transmission message, the component tag of the data asset to which the item with the node tag obtained in the data directory management table is transmitted, Download identification information, MPU sequence number, and item identification information can be obtained.

Further, as shown by the reference number 903, when the location information of the asset having the component tag obtained in the data asset management table is acquired from the MP table, the corresponding file is actually found as shown by the reference number 904. You can identify the data assets to be transmitted.

Then, within the specified data asset, the download identification information obtained from the data asset management table and the download identification information described in the header area of the MMTP packet that transmits the item are used to repeat the file corresponding to the carousel. The unit of transmission can be uniquely identified. As shown by the reference number 905, among the items repeatedly transmitted, the item having the MPU sequence number and the item identification information obtained from the data asset management table can be specified as a desired file.

C. Transmission of general-purpose data streams The general-purpose data transmission method is a method of transmitting various types of data synchronously or asynchronously, and is used for data and multimedia services used by players who present data other than video, audio, and subtitles. It can be applied to the streaming of data to be used. In general-purpose data transmission, various data used in an HTML5 format data broadcasting application can be used for broadcasting as a stream. There are synchronous general-purpose data transmitted by a synchronous MPU and asynchronous general-purpose data transmitted by an asynchronous MPU. The presentation time is indicated by the MPU time stamp descriptor (described above) in the MPU of the synchronous general-purpose data.

In the processing of general-purpose data transmission, the MPU is the unit of processing. The MPU is a unit that includes data to be used at a certain timing and can perform processing such as presentation in MPU units. FIG. 10 shows the configuration of the MPU and MFU of general-purpose data.

The general-purpose data MPU is composed of one or more MFUs without arranging MPU metadata and movie fragment metadata. Each MFU is composed of a general-purpose data header part and a data part. Data of any size can be stored in the data unit of each MFU. For example, one MFU can send one file.

FIG. 11 shows an example of MFU syntax 1100 for general-purpose data. Hereinafter, the meaning of each parameter of MFU of general-purpose data will be described.

The component_tag (component tag) is a label for identifying the component stream, and is a component tag that uniquely corresponds to the asset identification information of the general-purpose data stream given in the MH-stream identification descriptor of the MP table. A 16-bit value is written.

data_type_length indicates the length of the subsequent data type description (data_type_byte) in bytes. The data_type_byte indicated by the reference number 1101 is an area for storing the MIME (MULTIPURPOSE Internet Mail Extension) type of the data to be transmitted by the MFU.

additional_info_length indicates the length of the subsequent additional information description in bytes. The additional_info_byte indicated by the reference number 1102 is an area for storing additional information regarding the data transmitted by the MFU.

data_length indicates the length of the data byte of the subsequent general-purpose data in bytes. The data_byte indicated by the reference number 1103 is an area for storing general-purpose data.

The component tag, data type, and additional information correspond to the general-purpose data header part. data_byte corresponds to the data unit.

As a use case of general-purpose data, the present specification proposes a technique for transmitting data for changing a display object from moment to moment as a general-purpose data stream to a data broadcasting application in HTML5 format. In the following, as an embodiment of the technique disclosed in the present specification, a technique for transmitting control information of a display object as a general-purpose data stream to a data broadcasting application that overlays a display object on a video of a broadcast program. Will be described in detail.

For example, when it is desired to display a display object that is synchronized with the video of a broadcast program, the control information of the display object that changes from moment to moment can be transmitted by a synchronous general-purpose data stream that is time-synchronized with the video. Then, the data broadcasting application can overlay the display object on the video according to the movement of the video based on the control information received as the general-purpose data stream.

FIG. 12 shows a display example of a broadcast screen displaying a display object synchronized with the video of the broadcast program. The figure is a broadcast screen of a soccer game, and a "balloon" -shaped display object that presents information on participating players (for example, player name, position, number of goals, etc.) follows the movement of each player. It is displayed as an overlay.

FIG. 12 is only one example of overlaying a display object on the screen of a broadcast program. Similar display object overlays can be applied to broadcast programs that relay various sports other than soccer, such as baseball, basketball, golf, athletics, judo, and car racing. Of course, the overlay display of the same display object can be applied to broadcast programs of various genres such as dramas and music programs other than sports.

As a method of expressing information about the display object synchronized with the video, for example, a moving object list description (MOLD) document described below can be used. The moving object list description document has a document structure that describes a list of information about display objects for the number of objects displayed on the screen at the same time. In the moving object list description document, the position and size of the display object on the screen and the display data itself or the acquisition destination information of the display data (reference data reference number or URL, etc.) are described as information for each display object. do it.

From the transmitting side (broadcasting station), the moving object list description document and the display data (image file) of the display object to be referenced are packaged as a general-purpose data stream together with a time stamp indicating the display time of each package. Broadcast and transmit intermittently. Then, on the receiver side, when the information for drawing each display object is taken out from the moving object list description document, the display object synchronized with the video is displayed on the screen of the broadcast program based on the display time specified by the time stamp. It is possible to display an overlay on.

The time stamp information indicating the display time of each package can be indicated by the MPU time stamp descriptor that provides the presentation time of the MPU. The MPU timestamp descriptor is one of the asset descriptors 701 placed in the second loop of the MP table. An example of the syntax of the MPU time stamp descriptor is shown in FIG. 8, but the sequence number of the MPU for transmitting the moving object list description document may be specified by mpu_sequence_number, and the display time may be specified by mpu_presentation_time.

In addition, as information about the display object, the interpolation information for obtaining the interpolation position of each display object until the next timing is also described in the moving object list description document. The interpolation information includes information for obtaining the interpolation position for each time, which is a subdivision of the time until the next timing. It is not realistic to transmit a general-purpose data stream for each video frame because of the large amount of information. The general-purpose data stream is transmitted intermittently, and the receiver side displays the display object at the interpolation position obtained based on the interpolation information described in the moving object list description document until the next timing. The display object can be moved smoothly between. In the display example of the broadcast program shown in FIG. 12, the display object can be overlaid by interpolation so as to smoothly follow the soccer player who always moves around on the pitch.

For example, by describing the velocity vector information of the display object as the interpolation information, it is possible to linearly interpolate the interpolation position of the display object. Furthermore, by including the angular velocity information, it becomes possible to interpolate the interpolation position of the display object with an arc curve. The velocity vector and angular velocity can be obtained by analyzing the movement of the corresponding player.

Further, as the information about the display object, the URL information of the hyperlink may be further described in the moving object list description document. On the receiver side, when the viewer of the video operates the display object, the information can be acquired from the link destination specified based on the URL specified for the display object and displayed.

In addition, the display data of the display object that is commonly referred to by multiple moving object list description documents is not packaged with the moving object list description document and transmitted every time, but is represented by the first moving object list. It may be packaged with the description document and transmitted, and may be referred to in common from the moving object list description document transmitted thereafter. By transmitting general-purpose data consisting only of moving object list description documents that refer to external display data, the amount of transmission information can be reduced (that is, the transmission band can be saved). When the display data is updated, if it is packaged with the moving object list description document and transmitted, the effect of bandwidth saving will be enhanced. However, in order to enable the display object to be displayed even by the receiver tuned in the middle of the program, it is preferable to transmit general-purpose data in which the display data is packaged at regular intervals.

Further, the general-purpose data stream may be intermittently transmitted at regular intervals, or the transmission cycle may be variably controlled. For example, the transmission cycle of the general-purpose data stream may be controlled according to the movement of the display object. For example, when the movement of the display object is fast, transmission can be performed at 1-second intervals, making it easier to obtain a more accurate interpolation position of the display object. On the contrary, when the movement of the display object is slow, the transmission information amount may be reduced (that is, the transmission band is saved) by setting the interval at 5 seconds.

FIG. 13 schematically shows the structure of a moving object list description document that describes information about display objects for the number of objects and a general-purpose data stream that packages display data referenced from the document.

This general purpose data stream is transmitted as a synchronous MPU. As already described with reference to FIG. 10, in the processing of general-purpose data transmission, the MPU is a processing unit, and the MPU is composed of a plurality of MFUs. When information about a display object is transmitted as general-purpose data, a moving object list description document and a package of display data referenced from the document are transmitted in one MPU as a processing unit.

In FIG. 13, reference number 1300 is an MPU that transmits one package, and the moving object list description document is stored in the first MFU shown by reference number 1310 and is referred to from this moving object list description document. The display data (image file) of each display object is stored in the subsequent MFU 1320-1, ..., 1320-N.

The moving object list description document stored in the data unit of the head MFU 1310 consists of a list of display object information for N objects to be displayed on the screen. As shown in FIG. 14, when four display objects # 1 to # 4 are displayed on the screen at the same time, N = 4, and the moving object list description document lists information about four display objects. ..

In the moving object list description document, the tag information (Tag) for identifying the display object and the position (Position) for displaying the display object on the screen are used as information about the display object # 1 in the information block indicated by the reference number 1311. ) And size (Size), velocity vector information (Vector) and angular velocity information (θ) as interpolating information, and information on display data of the display object (MediaData). As described above, the URL information of the hyperlink may be further included as the information of the display object (omitted in FIG. 13). Although omitted for the sake of simplification of the drawings, it should be understood that the other information blocks also describe similar information about the corresponding display objects.

The information (MediaData) of the display data of the display object can be represented by the acquisition destination information (reference data reference number, URL, etc.) of the display data. In the example shown in FIG. 13, it is assumed that the display data (image file) of the display object is also packaged with the moving object list description document, and the acquisition destination information such as the URL is transmitted in the same MPU. , The location information of the MFU that stores the corresponding image file shall be specified. Alternatively, the display data itself, that is, the binary data, can be described in the information block of the moving object list description document.

Further, reference number 1330 is an MP table, and specifies asset identification information and location information of each asset (video, audio, subtitle, data broadcasting application, etc.) constituting the broadcasting signal. The display time of the MPU 1300 is specified by the MPU time stamp descriptor 1331 arranged for the asset of the general-purpose data stream in the second loop of the MP table. That is, in the MPU time stamp descriptor 1331, the sequence number of the MPU 1300 for transmitting the moving object list description document is specified by mpu_sequence_number, and the display time is specified by mpu_presentation_time.

FIG. 13 shows a configuration example of an MPU that packages and transmits a moving object list description document and display data referenced from the document, and the display data (image file) of each display object is in the same MPU. It is transmitted by the subsequent MFU. As a modification of this, as shown in FIG. 15, an MPU that does not include display data and consists only of the MFU of the moving object list description document is also assumed. Such a moving object list description document refers to the display data transmitted by the MFU included in the preceding MPU. The amount of transmission information is reduced (that is, the transmission band is saved) by packaging the display data commonly referred to in multiple moving object list description documents into only one moving object list description document and transmitting the data. can do. By introducing a mechanism to specify the retention period of the display data (described later), when the receiver receives the MPU in which the display data is packaged, the display data is cached until the specified retention period. It is also possible to refer to the retention period from the moving object list description document transmitted from. When updating the display data, the updated display data and the packaged moving object list description document may be transmitted again. However, it is preferable to transmit the MPU in which the display data is packaged at regular intervals so that the display object can be displayed even by the receiver tuned in the middle of the program.

In addition, when the same function is realized by the broadcasting system of the MPEG TS system instead of the MMT system, the same information is transmitted by the PES (Packetized Elementary Stream) packet and the display time is indicated by the PTS (Presentation TimeStamp). You can do it like this.

FIG. 11 shows an example of MFU syntax 1100 for general-purpose data. The usage of each parameter of the MFU when applied to the transmission of the moving object list description document and the display data referenced from the document will be further described below.

The data_type_byte indicated by the reference number 1101 specifies the type of data to be transmitted by the MFU with a character string having a data length specified by data_type_length. The MFU in the MPU shown in FIG. 13 transmits either a moving object list description document (MOLD) or an image file of display data. The types of data specified by the character string data_type_byte are shown in Table 2 below. In the case of a moving object list description document, "mld" may be specified, in the case of JPEG format display data, "jpg", and in the case of PNG format display data, "png" may be specified by data_type_byte.

Further, in the additional information area (additional_info_byte) shown by the reference number 1102, additional information corresponding to the data to be stored in the data unit is described. That is, if the MFU transmits a moving object list description document, additional information for the moving object list description document is described. In the case of other MFUs that transmit materials of display objects such as display data, additional information about the materials is described in the additional information area.

The general-purpose data itself transmitted by the MFU is stored in the data unit (data_byte) indicated by the reference number 1103. When "mld", that is, the moving object list description document is specified in the preceding parameter data_type_byte, the file of the moving object list description document is stored in this data unit. In addition, when "jpg" or "png" is specified in data_type_byte, the display data in JPEG format or PNG format is displayed.

FIG. 16 shows a syntax example 1600 of additional information (mold_info) in the MFU that transmits the moving object list description document. Hereinafter, the meaning of each parameter of the additional information (mold_info) will be described.

The material_location is a field indicating in 2 bits the acquisition destination information of the display data referred to from the moving object list description document (that is, stored in the data unit). The meanings of the material_location values are shown in Table 3 below. When transmitting the display data with the same MPU as the moving object list description document (see FIG. 13), 00 is specified for material_location. Further, when the display data is transmitted by the previous MPU, 01 is specified in material_location. Further, when the display data is referred to on the Internet, 10 is specified in material_location. When 01 and 10 are specified for material_location, the transmission bandwidth is saved because the display data is not packaged in the same MPU as the moving object list description document (see FIG. 15). As will be described later, the MFU that transmits the display data can specify the retention period. Therefore, if the already transmitted display data is within the retention period, 01 can be specified for material_location.

estimated_next_MPU_time indicates the estimated transmission time of the next MPU (which transmits the moving object list description document). On the receiver side, the display position of the display object is interpolated in consideration of the predicted transmission time of the next MPU (see FIG. 13).

Further, FIG. 17 shows a syntax example 1700 of additional information (material_info) in the MFU that transmits the material (display data) referred to from the moving object list description document. Hereinafter, the meaning of each parameter of the additional information (material_info) will be described.

material_name_length indicates the length of the subsequent material name description (material_name_byte) in bytes. The material_name_byte is an area for storing the material reference name used when the material transmitted in the data unit of the MFU is referred from the moving object list description document.

Persistent_flag is a data retention flag that specifies whether or not the display data (JPEG or PNG image file) stored in the data unit (data_byte) of the MFU should be retained even after the receiver finishes the processing of the MPU. be. When 1 is specified for persistent_flag, the retention period of the display data stored in the data unit of the MFU is described in the explore_date field.

The data retention flag and the data retention period can be specified for each MFU, that is, for each display data in the general-purpose data header section.

On the receiver side, the display data in which 1 is specified in the persistent_flag of the general-purpose data header portion is continuously retained (cached) for the retention period specified by expire_date even after the processing of the MPU is completed. The retained display data can also be referred to (that is, reused) from the moving object list description document transmitted in the subsequent MPU. In short, the subsequent MPU does not need to package the display data referenced from the moving object list description document and transmit it in the same MPU, so that the transmission bandwidth can be saved.

FIG. 18 shows a syntax example 1800 of a moving object list description document. When "mld" is specified in the parameter data_type_byte in the general-purpose data header section of the MFU, the moving object list description document as shown in the figure is stored in the data section. In the figure, parameters with @ at the beginning indicate attribute information, and parameters without @ indicate elements. The meaning of each parameter will be described below.

mold indicates that the data is a moving object list description document. At the same time, the information of the moving object mo is arranged in the document as many as the number of objects displayed on the screen.

Each moving object mo is identified by an object tag. An object tag as an attribute value is described in the moTag field.

As information of each moving object mo, object position information (position), object size information (size), object movement vector information (mv), object display data information (media), and link destination information at the time of object operation. (Link) is included.

The position information of the object includes the horizontal coordinate x and the vertical coordinate y of the display position (upper left corner in the case of a rectangular area) of the object on the screen as attribute values, and each pixel position is indicated by an integer value.

The object size information (size) includes the width x and the height y of the rectangular area indicating the size of the object to be displayed on the screen as attribute values, and each is indicated by a pixel value (integer value).

The motion vector information (mv) of the object is an attribute value of a velocity vector consisting of the horizontal velocity x and the vertical velocity y of the object and the angular velocity information θ (provided that the movement of the object is a curved motion represented by an arc). And the integer values indicating each are described. On the receiver side, the motion vector information (mv) can be used to determine the interpolation position of the object until the timing of the next MPU (that is, the next moving object list description document) (described above).

The information (media) regarding the display data indicates the type (JPEG or PNG) of the display data of the object and the acquisition destination information of the display data of the object. The acquisition destination of the display data is roughly classified into either the same MPU, the previous MPU, or the Internet, and is specified by the parameter material_location in the additional information (mold_info) of the general-purpose data header section (described above). For the display data acquisition destination information, the acquisition destination at the location specified by material_location is specified in the url format, and the image file in the format specified by the display data type is placed in the acquisition destination specified by the url. Is the premise. If 0 is specified for the display data acquisition destination information, the display data binary data may be stored in the display data type area.

The link destination information (link) at the time of object operation specifies the link destination information of the object in the url format. When the viewer of the video operates the display object, the information can be acquired from the link destination specified by the link destination information (link) and displayed.

FIG. 19 illustrates a mechanism for interpolating the display position of a display object based on the interpolation information specified in the moving object list description document. Reference number 1901 indicates the display position (X1, Y1) of the display object specified in the moving object list description document transmitted at a certain timing T1. Further, the reference number 1902 indicates the display position (X2, Y2) of the same display object specified in the moving object list description document transmitted at the next timing T2. Reference numbers 1911 to 1914 are interpolation positions for each time when the time interval of the timing is subdivided. In the moving object list description document transmitted at the timing T1, it is assumed that the velocity vector (x, y) and the angular velocity θ are specified as the interpolation information of this display object. Based on the velocity vector (x, y), it is possible to linearly interpolate the subdivided interpolation positions 1911, ... For each time, and further, based on the angular velocity θ, interpolate the interpolation positions 1911, ... It is possible to do.

The application engine 411, which executes an application that overlays display objects on the video, sequentially acquires the moving object list description document transmitted by the general-purpose data stream and interpolates based on the interpolation information to perform the video. It is possible to display a display object that moves smoothly according to the movement of.

FIG. 20 illustrates a scenario in which a moving object list description document is transmitted as a general purpose data stream. In the figure, a broadcasting signal transmitted from a certain broadcasting company (broadcasting station) is illustrated, and the horizontal axis is the time axis. The broadcast signal includes a video asset 2010, an audio asset 2020, a data (application) asset 2030, and a general-purpose data asset 2040 as components constituting a broadcast program. The broadcast signal may further include other assets, but will be omitted for the sake of simplification of the drawings.

Further, the broadcast signal includes a control signal 2050 that transmits control information (MMT-SI), such as a signaling table such as an MP table and a signaling message that serves as a container for the signaling table.

In the figure, the data asset 2030 transmits a data broadcasting application that overlays the display object on the video of the broadcast program, and the general-purpose data asset 2040 transmits the control information of the display object, that is, the moving object list description document. Is assumed to be transmitted intermittently as a general-purpose data stream. Further, the display time of each MPU transmitted by the general-purpose data asset 2040 is specified by the MPU time stamp descriptor of the MP table (described above).

Reference number 2041 indicates an MPU that packages the moving object list description document and the display data (image file) of the display object to be referenced from now on. The moving object list description document is stored in the data part of the first MFU, and the display data referenced from this moving object list description document is stored one by one in the second and subsequent MFUs (FIG. 13). See).

Reference number 2042 also indicates an MPU that does not package the display data. This MPU transmits only one MFU that stores the moving object list description document. The moving object list description document refers to the display data already transmitted by the preceding MPU2041 (see FIG. 15). However, it is premised that the displayed data to be referred to is the general-purpose data header portion of the MFU, the data retention is specified by persistent_flag, and the data retention period specified by explore_date has not passed. Similarly, the MPU indicated by the reference number 2043 also transmits only the MFU of the moving object list description document. The amount of transmission information is reduced (that is, the transmission band is saved) by packaging the display data commonly referred to in multiple moving object list description documents into only one moving object list description document and transmitting the data. can do.

Reference number 2044 indicates an MPU that packages the moving object list description document and the display data of the display object to be referenced. Similar to the MPU shown by reference number 2041, the moving object list description document is stored in the data part of the first MFU, and the display data referenced from this moving object list description document is stored in the second and subsequent MFUs. They are stored one by one (same as above).

When the display data is updated, the MPU 2044 including the updated display data and the packaged moving object list description document may be transmitted again. Alternatively, in order to enable the display object to be displayed even by the receiver tuned in the middle of the program, the MPU2044 in which the display data is packaged is transmitted after a certain period of time has passed since the MPU2041 was transmitted.

The MPU time stamp descriptor in the MP table indicated by reference number 2051 specifies the display time T10 of the display object by the moving object list description document transmitted by the MPU indicated by reference number 2041. Similarly, the MPU time stamp descriptor in the MP table indicated by reference number 2052 specifies the display time T20 of the MPU indicated by reference number 2042, and the MPU time stamp descriptor in the MP table indicated by reference number 2053 is a reference number. The MPU display time T30 indicated by 2043 is specified, and the MPU time stamp descriptor in the MP table indicated by reference number 2054 specifies the MPU display time indicated by reference number 2044.

On the receiver side, when MPU2041 transmitted by broadcast wave is received, the moving object list description document transmitted by the first MFU is analyzed, and the display data transmitted by the second and subsequent MFUs is displayed in the header part. Holds only the specified retention period.

Then, when the display time T10 specified by the MPU time stamp descriptor of the MP table 2051 arrives, the receiver displays each display according to the description content of the moving object list description document received by the MPU 2041 as shown by the reference number 2061. Overlay the object on the video of the broadcast program.

Further, the receiver specifies each display object in the moving object list description document at the time T11 in which the time interval is subdivided until the display time T20 specified for the next MPU 2042 arrives. The interposition position is obtained based on the interposition information (velocity vector, angular velocity), and the display of each display object is updated as shown by the reference number 2062.

After that, when the display time T20 specified by the MPU time stamp descriptor of the MP table 2052 arrives, the receiver precedes as indicated by the reference number 2063 according to the description contents of the moving object list description document received by the MPU 2042. With reference to the display data transmitted by the MPU 2041, each display object is overlaid on the image of the broadcast program.

FIG. 21 shows the processing procedure when the receiver receives the general-purpose data stream in the form of a flowchart. The illustrated processing procedure corresponds to the process of acquiring the received and decoded general-purpose data in the HTML browser (application engine 411).

When the receiver receives the MP table in the PA message which is the entry point of the broadcasting service, it checks whether or not the MPU time stamp descriptor of the general-purpose data asset exists (step S2101).

If the MPU time stamp descriptor of the general-purpose data asset exists (Yes in step S2101), the receiver stores the information of the MPU time stamp descriptor, that is, the MPU sequence number and its presentation time (Yes). Step S2102).

When the receiver receives the MPU of the general-purpose data asset (Yes in step S2103), the receiver stores the general-purpose data MPU together with the information of the MPU time stamp descriptor specifying the MPU sequence number (step S2104).

Here, reference is made to material_location in the general-purpose data header portion of the MFU at the beginning of the MPU (that is, the MFU that transmits the moving object list description document). Then, when material_location = 10, that is, when the acquisition destination of the display data is specified on the Internet (Yes in step S2105), the receiver is a moving object list description document stored in the data part of the MFU. The display data (image file) of each display object is acquired based on the designated URL of the display data acquisition destination (step S2106).

Further, FIG. 22 shows a processing procedure for overlaying a display object that the receiver synchronizes with the video based on the general-purpose data stream in the form of a flowchart. The illustrated processing procedure corresponds to a process of updating the display of a display object based on the latest general-purpose data in an HTML browser (application engine 411).

Each time the display timing of the video frame arrives (Yes in step S2201), the receiver checks whether or not it matches the MPU presentation time specified in the stored MPU time stamp descriptor (step S2202).

Then, when the display timing this time is the MPU presentation time specified in the stored MPU time stamp descriptor (Yes in step S2202), the receiver transmits the MFU at the head of the corresponding MPU. Based on the description of the moving object list description document, the display object is overlaid on the video frame with reference to the material data (step S2203).

Further, the receiver deletes the display data packaged in the MPU received before the MPU at the presentation time, except that the data retention flag (persistent_flag) is set to 1 (step). S2204).

Further, the receiver checks whether all the displayed data held is within the retention period (step S2205). Then, if there is display data whose retention period has expired, it is deleted (step S2206).

If the display timing this time is not the MPU presentation time specified in the stored MPU time stamp descriptor (No in step S2202), the receiver is a moving object of the MPU to be processed. -For each display object listed in the list description document, the interposition position at the current time is interpolated and calculated based on the element information of the motion vector (mv) (that is, the velocity vector information and the angular velocity information), and the object's The display is updated (step S2207).

As described above, the techniques disclosed in the present specification have been described in detail with reference to the specific embodiments. However, it is self-evident that a person skilled in the art can modify or substitute the embodiment without departing from the gist of the technique disclosed herein.

Although the present specification has mainly described embodiments in which the technology disclosed in the present specification is applied to a broadcasting system adopting the MMT method for media transmission, the gist of the technology disclosed in the present specification is limited to this. It's not something. Similarly, the techniques disclosed herein can be applied to various types of broadcasting systems that have a mechanism for streaming data used in data broadcasting applications. For example, when the same function is realized in the MPEG TS system broadcasting system, the same information may be transmitted by the PES packet and the display time may be indicated by the PTS.

In short, the techniques disclosed in this specification have been described in the form of examples, and the contents of the present specification should not be interpreted in a limited manner. The scope of claims should be taken into consideration in order to determine the gist of the technology disclosed herein.

The technology disclosed in the present specification can also have the following configuration.
(1) A transmitter that transmits a broadcast signal including video, and
A data transmitter that transmits data for displaying an object that moves in synchronization with the movement of the video,
A transmitter equipped with.
(2) The data transmission unit transmits the data including the position and size of the object on the screen and information on the display data of the object.
The transmitter according to (1) above.
(3) The data transmission unit intermittently transmits the data together with a time stamp indicating the display time of the data.
The transmitter according to any one of (1) and (2) above.
(4) The data transmission unit intermittently transmits the data including interpolation information for obtaining the interpolation position of the object until the next timing.
The transmitter according to any one of (1) to (3) above.
(5) The data transmission unit transmits the interpolation information including at least one of the velocity vector information or the angular velocity information of the object.
The transmitter according to (4) above.
(6) The data transmission unit transmits the data including the link destination information of the object.
The transmitter according to any one of (1) to (5) above.
(7) The data transmission unit transmits a document file in which the position and size of the object on the screen and the reference destination of the display data are described in a predetermined format.
The transmitter according to (1) above.
(8) The data transmission unit packages the document file and the display data file referenced from the document file, and transmits the package unit.
The transmitter according to (7) above.
(9) The data transmission unit specifies the necessity and retention period of each display data to be included in the package and transmits the data.
The transmitter according to (8) above.
(10) The data transmission unit further transmits the document file that refers to the display data file packaged and transmitted together with other document files.
The transmitter according to (9) above.
(11) A transmission step of transmitting a broadcast signal including video, and
A data transmission step of transmitting data for displaying an object that moves in synchronization with the movement of the video, and
Transmission method with.
(12) A receiver that receives broadcast signals including video, and
A data receiving unit that receives data for displaying an object that moves in synchronization with the movement of the image, and a data receiving unit.
A control unit that executes a process for displaying the object on the screen of the broadcast video based on the received data, and a control unit.
A receiving device comprising.
(13) The data receiving unit receives the data including the position and size of the object on the screen and the information about the display data of the object.
The control unit displays the display data of the object at the position and size on the screen specified by the data.
The receiving device according to (12) above.
(14) The data receiving unit intermittently receives the data together with a time stamp indicating the display time of the data.
The control unit controls the display timing of the object based on the display time specified by the time stamp.
The receiving device according to any one of (12) and (13) above.
(15) The data receiving unit intermittently receives the data including interpolation information for obtaining the interpolation position of the object until the next timing.
The control unit displays the object at an interpolation position obtained based on the interpolation information.
The receiving device according to (14) above.
(16) The data receiving unit receives a document file in which the position and size of the object on the screen and the reference destination of the display data are described in a predetermined format.
The receiving device according to (12) above.
(17) The data receiving unit receives the document file and a file of display data referenced from the document file in a package unit.
The receiving device according to (16) above.
(18) The data receiving unit receives the package in which the necessity and retention period of each display data are specified.
The control unit holds the display data until the retention period expires.
The receiving device according to (17) above.
(19) The data receiving unit further receives the document file that refers to the display data file packaged and transmitted together with the other document file.
The control unit displays an object using display data that is packaged and transmitted together with other document files referenced from the document file.
The receiving device according to (18) above.
(20) A reception step for receiving a broadcast signal including video, and
A data receiving step that receives data for displaying an object that moves in synchronization with the movement of the image, and
A control step that executes a process for displaying the object on the screen of the broadcast video based on the received data, and
Receiving method having.

10 ... Digital broadcasting system, 11 ... Broadcast transmission system, 12 ... Receiver 301 ... Clock unit, 302 ... Signal transmission unit 303 ... Video encoder, 304 ... Audio encoder 305 ... Subtitle / character super encoder 306 ... Signaling encoder , 307 ... File Encoder 308 ... Electronic Data Processing System,
309 ... TLV signaling encoder 310 ... IP service multiplexer 311 ... TLV multiplexer 312 ... Modulation / transmitter 401 ... Tuner demodulation unit, 402 ... MMT demultiplexer 403 ... Clock recovery unit, 404 ... Video decoder 405 ... Audio decoder 406 ... Character super decoder, 407 ... Subtitle decoder 408 ... Multimedia cache, 409 ... SI cache 410 ... Broadcast system control unit 411 ... Application engine 412 ... Communication interface 414 ... Scaler 415-418 ... Synthesis Department

Claims (5)

A transmitter that transmits broadcast signals including video,
A data transmitter that sends a document file described in a predetermined format for displaying moving objects, and a data transmitter.
Equipped with
The data transmission unit transmits the document file including the position information of the object and the interpolation information for obtaining the interpolation position of the object until the next timing.
Transmitter. The document file is an HTML5 format data broadcasting application.
The transmitting device according to claim 1. A transmission step that transmits a broadcast signal including video,
A data transmission step that sends a document file described in a given format for displaying moving objects, and
Have,
In the data transmission step, the document file is transmitted including the position information of the object and the interpolation information for obtaining the interpolation position of the object until the next timing.
Sending method. A receiver that receives broadcast signals including video,
A data receiver that receives a document file described in a predetermined format for displaying moving objects, and a data receiver.
A control unit that executes a process for displaying the object on the screen of the video based on the received document file.
Equipped with
The document file contains the position information of the object and the interpolation information for obtaining the interpolation position of the object until the next timing.
The control unit displays the object at an interpolation position obtained based on the position information and the interpolation information.
Receiver. A reception step for receiving a broadcast signal including video,
A data receiving step that receives a document file described in a given format for displaying moving objects, and a data receiving step.
A control step for executing a process for displaying the object on the screen of the video based on the received document file, and a control step.
Have,
The document file contains the position information of the object and the interpolation information for obtaining the interpolation position of the object until the next timing.
In the control step, the object is displayed at an interpolation position obtained based on the position information and the interpolation information.
Receiving method.

Images